Tool to manage openinverter via CAN

[johu]

That looks sane.
I'd go for delete & recreate because then you can map the same item multiple times.

[johu]

I have now settled for this:
index 0x3000 is map TX
- subindex 0 sets COB Id
- subindex 1 sets item ID (2 bytes) bit position and bit length (1 byte each)
- subindex 2 sets gain (3 bytes, gain * 1000) and offset (1 byte signed) and commits the addition

index 0x3001 is map RX

index 0x3100-0x317F lets you read all TX maps
- subindex 0 gets COB Id
- subindex 2i+1 gets ID, position, length as above where i is the item index on that COB id
- subindex 2i+2 gets gain and offset as above

index 0x3180-0x31FF lets you read all RX maps

writing to 0x31zz (subindex - 1)*i deletes the item, addressing as above

[davefiddes]

Seems workable. If you let me know when you've got some code I can work against I'll update the tool.

[johu]

Cool
You can use this one: https://github.com/jsphuebner/stm32-sin ... safer_can2

[johu]

I've recently found that the json stream contains random 0-bytes. It occurs both with the candownload.py script and the ESP32 so the culprit is probably the STM firmware.

I think I know why: PutChar() runs in the lowest priority main loop and the CAN RX IRQ can interrupt it before it fills up the buffer. That leaves some zeros in there.

[davefiddes]

It's been well behaved for me. I mostly have been testing in opmode off which might account for that.

Slow progress on my side as summer has arrived here at long last. Hope to make more progress next week.

[johu]

I've been thinking this over and it is easy to reproduce but hard to fix. Put a bit of interrupt load on the system and the download fails or at least contains 0 bytes. That's because the main loop does not manage to put enough bytes into the buffer before the next segment request fires.

Waiting for all bytes in the ISR is no solution either as that would result in classic priority inversion.

Also I don't want to disable the RX IRQ while printing as that would distract normal CAN operation and I totally don't want comms to get in the way of more important things.

So for now it means json download is only possible with low load. Could also filter the 0 bytes in reception...

[davefiddes]

Managed to get my "inverter" (an STM32 Bluepill on a breadboard) into Run mode and can now replicate this reliably. Should definitely filter it client side even if you are able to fix the problem. I need to be a bit cleverer in processing the incoming data stream as binary and remove the NUL bytes. Should be straight forward.

[johu]

I have now increased the buffer for printing to 64 bytes. Once the queue runs dry the transfer is terminated. I've tested with the inverter software in run mode and the spurious chars are gone.

UPDATE: code committed

[davefiddes]

I've pushed a fix to the tool which filters out any erroneous bytes anyway. I found it harder to replicate than I first thought so had to rely on unit tests which is not a bad outcome.

[johu]

I've pushed a fix to the tool which filters out any erroneous bytes anyway. I found it harder to replicate than I first thought so had to rely on unit tests which is not a bad outcome.

Ah good. Did you test with the code mentioned above or previous version?

[davefiddes]

It was the 5.27.R-sine release version. I don't have a reliable way to update the firmware on my board at the moment.

[celeron55]

I'm trying to use this to communicate
- with my focci board (https://github.com/uhi22/foccci)
- running the ccs32clara firmware (https://github.com/uhi22/ccs32clara)
- using a Lawicel CANUSB adapter (https://www.canusb.com/products/canusb/)
- on Fedora Linux 38

This is not going exactly swimmingly.

I first tried this type of configuration when running oic:

Code: Select all

interface = slcan
channel = /dev/ttyUSB0
bitrate = 500000

One problem is that in this mode, the CANUSB adapter does not ACK any messages on the bus unless someone has the device open. As a result, the target board ends up flooding the bus with re-send attempts right until the point where oic opens the device, after which the CANUSB adapter starts to respond to ACKs, and right when oic closes the device, the adapter appears to stop responding to ACKs again. To be exact, I don't have a CAN analyzer that would prove me it's the ack bit, but the behavior exactly matches that which in my experience results from the missing ACK bit.

Anyway, given this gotcha, I still could attempt using oic. Here is the result:

Code: Select all

$ oic scan
Scanning for devices. Please wait...

No nodes found

$ oic -n 22 serialno
Transfer aborted by client with code 0x05040000
SDO communication error: No SDO response received

$ oic -n 22 listparams
Transfer aborted by client with code 0x05040000
Traceback (most recent call last):
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/lib/python3.11/site-packages/canopen/sdo/client.py", line 65, in read_response
    response = self.responses.get(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/bin/oic", line 8, in <module>
    sys.exit(cli())
...
... (long traceback cut out)
...
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/lib/python3.11/site-packages/canopen/sdo/client.py", line 68, in read_response
    raise SdoCommunicationError("No SDO response received")
canopen.sdo.exceptions.SdoCommunicationError: No SDO response received

Ok. Well, it's clearly not working. We have another method to try, though: We can use slcand to turn this into a socketcan interface. Let's try:

Code: Select all

$ sudo slcand -F -o -c -f -s6 /dev/ttyUSB0 can0  # Speed 6 = 500kbpsps

We'll leave that running on the foreground.

We can immediately see on the scope that CANUSB starts to respond to ACKs as the target board stops flooding the bus.

Now, the interface still needs to be brought up first.

Code: Select all

$ sudo ip link set can0 up

And then we'll configure python-can to use socketcan:

Code: Select all

interface = socketcan
channel = can0
bitrate = 500000

Now it'll surely work, right? Right? There's nothing we can do but try to call oic again:

Code: Select all

]$ oic scan
Scanning for devices. Please wait...

CAN error: Failed to transmit: No buffer space available [Error Code 105]

Oh no! What does this even mean?

Then the rest of it:

Code: Select all

$ oic -n 22 serialno
Transfer aborted by client with code 0x05040000
SDO communication error: No SDO response received

$ oic -n 22 listparams
Transfer aborted by client with code 0x05040000
Traceback (most recent call last):
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/lib/python3.11/site-packages/canopen/sdo/client.py", line 65, in read_response
    response = self.responses.get(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/bin/oic", line 8, in <module>
    sys.exit(cli())
             ^^^^^
...
... (long traceback cut out)
...
  File "/home/celeron55/programs/openinverter_can_tool/openinverter-can-tool/venv/lib/python3.11/site-packages/canopen/sdo/client.py", line 68, in read_response
    raise SdoCommunicationError("No SDO response received")
canopen.sdo.exceptions.SdoCommunicationError: No SDO response received

No luck at all.

Looking on the scope, the target board appears to be sending stuff with a bit length of about 2us which matches the 500kbaud we are setting.

Disconnecting the target board from the bus lets us notice the CANUSB adapter is also sending frames with a 2us bit length, which matches. We can also see the bus is being flooded by retransmitted frames from the PC side, as nobody is ACKing them now.

Now in this socketcan mode, I can also use

Code: Select all

candump -tz can0

to look at the data sent by the target board. There doesn't seem to be any obvious problem with the data.

So, it looks like the basic CAN communication is working (I haven't for example swapped H and L, and the bitrate is correct), yet oic fails to get any useful results.

One problem is, I don't know what this protocol is supposed to look like when it's working, so I can't determine any problems at the frame level. The board is very talkative, and I don't know whether this protocol is supposed to be hidden within the same frame ids, or whether new frame ids should pop up. I have attached the "candump -tz can0" output of running the previously listed three oic commands. The vast majority of the data is not related to oic, though.

Anyone got any ideas?

[johu]

Those frames are from uhis first test implementation
You need to run the recent OI style firmware to be able to use oic

EDIT: apart from that I also found slcand more reliable

[uhi22]

Easy Story You are using the old Clara, before Johannes integrated all the Openinverter features. It sends on hard-coded CAN IDs starting with 0x567. No CANopen support at all. So the tool is perfectly right in saying "no response".
Just pull and compile the latest Clara.

[celeron55]

Wait what? I thought I pulled, compiled and flashed the latest clara!

This is supposed to be 0cb9745a3bea0e8171c03f190f38181dfb1a5f1c, and I built it on 9th February, just a week ago. The hex file I used has that date on it.

I'll go continue in the clara thread as this is off-topic here. viewtopic.php?p=67386#p67386

[celeron55]

It all worked out, now to the point that I need to access the CAN mappings on the board, by using OIC. What's the status on that?

EDIT: I started working on it now

[davefiddes]

Glad you got things working. The setup of CAN sessions on Linux can be a bit of a pain though it's good when you get it running.

Happy to accept PRs. I built the tool in python so it would be approachable to more people than typical OI projects. I've been away from OI and car related projects for a bit.

In addition to adding commands to allow basic editing of individual of CAN mappings I had intended to add loading and saving of all board mappings for backup purposes. My original idea was to read and write CAN mappings in DBC files using one of the common python DBC libraries. When I dug into it this got quite complicated. Perhaps simple custom JSON files would be useful as an intermediate step. DBC files would be very useful to allow other tools like SavvyCAN to interpret the CAN stream in terms that OI users might understand.

[celeron55]

It sure is fairly approachable to me at least. Just another Python tool!

CAN map listing and manipulation works now:

https://github.com/celeron55/openinvert ... ree/canmap

I'll submit a PR when I finish tweaking this.

Code: Select all

$ oic -n 22 can list
CAN mappings:
(empty)
$ oic -n 22 can add tx 0x101 5 0 1 0 0
CAN mapping added succesfully
$ oic -n 22 can list
CAN mappings:
* 0x3100 tx param_id=5 pos=0 len=0 gain=0 offset=0
$ oic -n 22 can add tx 0x102 5 0 1 0 2
CAN mapping added succesfully
$ oic -n 22 can list
CAN mappings:
* 0x3100 tx param_id=5 pos=0 len=0 gain=0 offset=0
* 0x3101 tx param_id=5 pos=0 len=0 gain=0 offset=0
$ oic -n 22 can add rx 0x201 5 0 1 0 0
CAN mapping added succesfully
$ oic -n 22 can list
CAN mappings:
* 0x3100 tx param_id=5 pos=0 len=0 gain=0 offset=0
* 0x3101 tx param_id=5 pos=0 len=0 gain=0 offset=0
* 0x3181 rx param_id=5 pos=0 len=0 gain=0 offset=0
$ oic -n 22 can remove 0x3101
0x3101
CAN mapping removed succesfully
$ oic -n 22 can list
CAN mappings:
* 0x3100 tx param_id=5 pos=0 len=0 gain=0 offset=0
* 0x3181 rx param_id=5 pos=0 len=0 gain=0 offset=0

[davefiddes]

Neat!

[johu]

Good progress! Can you look up param_id in the json to be able to use the friendly name both when mapping and listing? Was trying to conceal those IDs is much as possible (viewtopic.php?p=52550#p52550)

BTW here is the intermediate json format I use on the ESP32

Code: Select all

[{"isrx":false,"id":264,"paramid":2009,"position":8,"length":16,"gain":1,"offset":0,"index":12544,"subindex":2},
{"isrx":false,"id":264,"paramid":2008,"position":24,"length":8,"gain":1,"offset":0,"index":12544,"subindex":4},
{"isrx":false,"id":265,"paramid":1,"position":0,"length":8,"gain":0,"offset":10,"index":12545,"subindex":2},
{"isrx":false,"id":265,"paramid":2006,"position":8,"length":16,"gain":1,"offset":0,"index":12545,"subindex":4},
{"isrx":false,"id":265,"paramid":2010,"position":24,"length":8,"gain":1,"offset":0,"index":12545,"subindex":6},
{"isrx":false,"id":265,"paramid":2000,"position":40,"length":8,"gain":0.289000005,"offset":0,"index":12545,"subindex":8},
{"isrx":true,"id":256,"paramid":6,"position":0,"length":16,"gain":1,"offset":0,"index":12672,"subindex":2},
{"isrx":true,"id":258,"paramid":3,"position":8,"length":16,"gain":1,"offset":0,"index":12673,"subindex":2},
{"isrx":true,"id":258,"paramid":4,"position":24,"length":8,"gain":1,"offset":0,"index":12673,"subindex":4},
{"isrx":true,"id":258,"paramid":5,"position":48,"length":8,"gain":0.5,"offset":0,"index":12673,"subindex":6},
{"isrx":true,"id":258,"paramid":23,"position":40,"length":1,"gain":1,"offset":0,"index":12673,"subindex":8},
{"isrx":true,"id":258,"paramid":2016,"position":0,"length":1,"gain":0,"offset":0,"index":12673,"subindex":10}]

[uhi22]

What is index and subindex?

[johu]

Thats helper info in case you want to delete a mapping.
See here: viewtopic.php?p=60996#p60996

[celeron55]

I don't quite get the protocol used to list the mappings. Somehow multiple mappings appear or disappear on the listing when I add or delete one. The index/subindex thing where sometimes they're used as a pair that can be translated to something, sometimes index is a command and sometimes subindex means something on its own is very confusing without a protocol spec.

Also, getting an abort response means various things at various points in the listing protocol. It doesn't seem to be a very good design from my standpoint.

EDIT:

Good progress! Can you look up param_id in the json to be able to use the friendly name both when mapping and listing? Was trying to conceal those IDs is much as possible (viewtopic.php?p=52550#p52550)

Already did that. It was the easiest part in all of this. The difficult part is in the listing protocol of the mappings.

[celeron55]

Maybe this is correct?

Code: Select all

Listing TX mappings
SDO 0x3100.0: can_id=0x101
SDO 0x3100.1: dataposlen=0x80007d3
SDO 0x3100.2: gainofs=0x3e8
-> Decoded mapping: tx 0 2 can_id=0x101 param='temp1'(id=2003) pos=0 len=8 gain=1.0 offset=0
SDO 0x3100.3: dataposlen=0x82007d5
SDO 0x3100.4: gainofs=0x3e8
-> Decoded mapping: tx 0 4 can_id=0x101 param='temp3'(id=2005) pos=32 len=8 gain=1.0 offset=0
SDO 0x3100.5: not found
SDO 0x3101.0: can_id=0x102
SDO 0x3101.1: dataposlen=0x81007d5
SDO 0x3101.2: gainofs=0x3e8
-> Decoded mapping: tx 1 2 can_id=0x102 param='temp3'(id=2005) pos=16 len=8 gain=1.0 offset=0
SDO 0x3101.3: not found
SDO 0x3102.0: not found
Listing RX mappings
SDO 0x3180.0: can_id=0x202
SDO 0x3180.1: dataposlen=0x8100005
SDO 0x3180.2: gainofs=0x3e8
-> Decoded mapping: rx 0 2 can_id=0x202 param='soc'(id=5) pos=16 len=8 gain=1.0 offset=0
SDO 0x3180.3: dataposlen=0x8100005
SDO 0x3180.4: gainofs=0x3e8
-> Decoded mapping: rx 0 4 can_id=0x202 param='soc'(id=5) pos=16 len=8 gain=1.0 offset=0
SDO 0x3180.5: not found
SDO 0x3181.0: can_id=0x203
SDO 0x3181.1: dataposlen=0x10100013
SDO 0x3181.2: gainofs=0x3e8
-> Decoded mapping: rx 1 2 can_id=0x203 param='maxcur'(id=19) pos=16 len=16 gain=1.0 offset=0
SDO 0x3181.3: dataposlen=0x10200011
SDO 0x3181.4: gainofs=0x3e8
-> Decoded mapping: rx 1 4 can_id=0x203 param='maxpower'(id=17) pos=32 len=16 gain=1.0 offset=0
SDO 0x3181.5: dataposlen=0x830000e
SDO 0x3181.6: gainofs=0x3e8
-> Decoded mapping: rx 1 6 can_id=0x203 param='lockpwm'(id=14) pos=48 len=8 gain=1.0 offset=0
SDO 0x3181.7: not found
SDO 0x3182.0: can_id=0x201
SDO 0x3182.1: dataposlen=0x8300006
SDO 0x3182.2: gainofs=0x3e8
-> Decoded mapping: rx 2 2 can_id=0x201 param='batvtg'(id=6) pos=48 len=8 gain=1.0 offset=0
SDO 0x3182.3: not found
SDO 0x3183.0: not found
CAN mappings:
* SDO 0x3100.2: tx 0 2 can_id=0x101 param='temp1'(id=2003) pos=0 len=8 gain=1.0 offset=0
* SDO 0x3100.4: tx 0 4 can_id=0x101 param='temp3'(id=2005) pos=32 len=8 gain=1.0 offset=0
* SDO 0x3101.2: tx 1 2 can_id=0x102 param='temp3'(id=2005) pos=16 len=8 gain=1.0 offset=0
* SDO 0x3180.2: rx 0 2 can_id=0x202 param='soc'(id=5) pos=16 len=8 gain=1.0 offset=0
* SDO 0x3180.4: rx 0 4 can_id=0x202 param='soc'(id=5) pos=16 len=8 gain=1.0 offset=0
* SDO 0x3181.2: rx 1 2 can_id=0x203 param='maxcur'(id=19) pos=16 len=16 gain=1.0 offset=0
* SDO 0x3181.4: rx 1 4 can_id=0x203 param='maxpower'(id=17) pos=32 len=16 gain=1.0 offset=0
* SDO 0x3181.6: rx 1 6 can_id=0x203 param='lockpwm'(id=14) pos=48 len=8 gain=1.0 offset=0
* SDO 0x3182.2: rx 2 2 can_id=0x201 param='batvtg'(id=6) pos=48 len=8 gain=1.0 offset=0

EDIT: Made it even longer with more mappings