Tool to manage openinverter via CAN

[davefiddes]

I have just released v0.0.9:

What's Changed

• Custom CAN message mappings can be managed by the tool. Many thanks to @celeron55 for this functionality.
• CAN message maps can be exported to a local JSON file and reloaded onto the same or a different device
• CAN message maps can be exported to a DBC file for use with common tools like SavvyCAN
• oic log now takes special wildcard parameters ALL, PARAMS or VALUES which saves having to specify each parameter
• It is possible to specify the CAN node with the OIC_NODE environment variable easing use of the tool with non-default node ids
• Be more forgiving of badly formed parameter databases (fixes issue #4)
• Scan all possible nodes up to nodeid 127 (fixes issue #3)
• Major internal refactoring allowing more complete unit tests
• Remove python 3.7 support and add python 3.12 support

The CAN mapping support is as full featured as I could make it and basically the same as what is possible with the WiFi interface. There are a number of rough edges in stm32-sine and libopeninv derived projects which are waiting for @johu reviewing a couple of fixes.

The CAN mapping to DBC export functionality is noteworthy. It really makes it a lot easier to figure out what is going into and out of a given openinverter device using SavvyCAN. You don't need to spend hours carefully defining your own custom messages or stare blankly at screens of hex digits hopefully. It just takes seconds to export a complex map and load it into SavvyCAN:

[uhi22]

Wow, at the beginning I was sceptical ("Who the hell wants a command line tool?"), now I'm really impressed. Great work, many thanks.

[davefiddes]

Thanks. FWIW I've been building GUIs for control systems for 30 years. Doing it properly takes more effort than I'm prepared to commit. It would be possible to add one in the future as the guts of the network protocol has all been abstracted. For now I just need something minimal that can manage an inverter entirely via CAN.

My next stops are CAN upgrade and VCU (throttle, start/stop, etc) support.

[js1tr3]

Hi Dave,
I am very excited to try your tool.
Can you post an example of how to use it for mapping can parameters?

[davefiddes]

Sure.
Say you wanted to map the heatsink and motor temperatures from an openinverter board to an outgoing CAN message (for a custom instrument display to read for example). You might use a 16-bit length field for each temperature with a scale factor of 10 so you can get sub-degree precision. To put the two parameters in a single CAN message with hex ID 0x100 you would do:

Code: Select all

$ oic can add tx 0x100 tmphs 0 16 10
Adding CAN tx mapping with can_id=0x100 param='tmphs' position=0 length=16 gain=10.0 offset=0
CAN mapping added successfully.
$ oic can add tx 0x100 tmpm 16 16 10
Adding CAN tx mapping with can_id=0x100 param='tmpm' position=16 length=16 gain=10.0 offset=0
CAN mapping added successfully.

This will then immediately start sending a CAN message with ID 0x100 (decimal 256) with the two temperatures occupying the first 32-bits of the message every 100ms (or 10ms depending on the canperiod setting). You can list what the device is doing with:

Code: Select all

$ oic can list
0x100:
 tx.0.0 param='tmphs' pos=0 length=16 gain=10.0 offset=0
 tx.0.1 param='tmpm' pos=16 length=16 gain=10.0 offset=0

Don't forget to save the CAN mapping to Flash on the inverter:

Code: Select all

$ oic cmd save
Command sent successfully

To look at the messages in SavvyCAN you need to export the CAN map:

Code: Select all

$ oic can export --format dbc inverter-temperatures.dbc
Parameter CAN message map exported

If you load inverter-temperatures.dbc into SavvyCAN it will look like this in the Signal Editor:

Which would look something like this when running and interpreting the CAN messages:

Ignore the crazy temperatures, I just have a resistor rather than a temperature sensor.

[davefiddes]

Just discovered that the python click library I use in openinverter_can_tool provides a way to configure shell command completion for systems that use the bash, zsh or fish shells (Linux and Mac basically). To set it up run the following command:

Code: Select all

$ _OIC_COMPLETE=bash_source oic > ~/.oic-complete.bash

Then add the following line to your ~/.bashrc file (the leading period and space are important):

Code: Select all

. ~/.oic-complete.bash

Once you login again you should then be able to press Tab to complete command and option names. Pressing Tab twice will display the list of possible options. For example:

Code: Select all

$ oic <tab><tab>
cache       cmd         listparams  log         save        serialno
can         dumpall     load        read        scan        write
$ oic log -<tab><tab>
-s              --timestamp     --help          
--step          --no-timestamp  

Looks like it might be possible to extend the completion to include parameter names in a future release.

[a768phoenix]

Hey Dave,

oic is a great command line tool to manage my new FOCCCI , but i have some issues with the creation of the can mapping for my boostech bms v3.

i used the following command for rx mapping

Code: Select all

/opt/venv-python/bin/oic -n 22 can add rx 0xb12 ChargeCurrent 31 -16 0.100000001 0

and get this error output

Code: Select all

can_id: 0xb12 out of range. Expected 0x000-0x7ff or 0-2047

It seems that OIC works with the CAN2.0a standard, but boostech has opted for the CAN2.0b standard .
The FOCCCI firmware has already been adapted by Johannes aka johu (version 0.40.5.B) and i flashed it onto the FOCCCI.

Here is a short pcan viewer dump to show the boostech can ids and some messages.

Code: Select all

   111)     14871.4  Rx     18E54024  8  FC 85 0E 22 0C 06 FF FF 
   112)     14872.0  Rx     00000310  8  0D AA FF F4 02 BC 07 00 
   113)     14872.5  Rx     00000311  8  00 27 59 00 27 57 00 5A 
   114)     14873.1  Rx     00000312  8  11 14 82 00 00 00 59 4A 
   115)     14873.6  Rx     00000313  6  0D AC 03 E8 00 82 
   116)     14874.0  Rx     00000314  4  00 01 88 44 
   117)     14874.5  Rx     00000314  4  01 01 88 44 
   118)     14874.9  Rx     00000314  4  02 01 88 45 
   119)     14875.3  Rx     00000314  4  03 01 88 B4 
   120)     14875.7  Rx     00000314  4  04 01 88 B4 
   121)     14876.1  Rx     00000314  4  05 01 88 B4 

Is there a way to implement the can2.0b standard in oic to manage the "boostech" mappings in FOCCCI.

best regards

[johu]

I should add that I added an experimental feature that lets you register an id < 2047 as an extended 29-bit id by adding 0x800 to it. This will probably see a semantic change to support actual extended IDs but for now we wanted to run with this. It seems oic has its own range check for id<0x7ff

EDIT: Maybe I should clarify here again: this latest bodge was added to force the CAN hardware to filter on an extended ID even though the ID would bit into the standard 11-bit range

[johu]

For testing you could also just use can-tools cansend command:

Code: Select all

cansend can0 616#23.01.30.00.12.0b.00.00
cansend can0 616#23.01.30.01.04.00.1F.F0
cansend can0 616#23.01.30.02.64.00.00.00

Not 100% sure. You can also intercept oic can messages when adding with ID 0x313 and then change byte 4 and 5 of the first message to 0xb12 or 0x12 0x0b, respectively

[davefiddes]

Is there a way to implement the can2.0b standard in oic to manage the "boostech" mappings in FOCCCI.

Hmm. I see that could be a problem. I've raisedan issue to cover this. Unfortunately it's a bit more than loosening up a range check and has consequences within the tool. I'll need a week or so to get to it.

@johu: With this latest bodge the CAN SDO mapping API is really starting to creak at the seams. I think that we need to have a revised V2 API. It seems that splitting each parameter into its own sub-index would make life a lot simpler for both devices and clients. Given that very little code has been released with working mapping support maintaining parallel implementations seems unnecessary. I'd propose bumping the main index constants to new values. Obviously I'd prefer to implement both ends of the link to ensure good end to end testing as I know you're busy.

[johu]

Not quite ready to go ahead here, still interested to understand which exact protocol you're proposing.

It's really just that, a bodge to cover up another bodge. No one knows why boostech decided to use standard ids but then tunnel them into extended ids.

So far it was good enough to imply everything below 2047 to be standard and everything above to be extended. The current implementation even supports extended ids. To make this less ambiguous the "force extended id" flag would look better in bit 30 instead of but 12.

[davefiddes]

I would propose separating out all of the packed parameters so something like this:

Code: Select all

Index: 0x3100
Sub-Index 0: can_id : UNSIGNED32 # Interpreted as per libopeninv canId
Sub-Index 1: param_id1: UNSIGNED32
Sub-Index 2: position1: UNSIGNED32
Sub-Index 3: length1: SIGNED32
Sub-Index 4: gain1: FLOAT
Sub-Index 5: offset1: SIGNED32
Sub-Index 6: param_id2: UNSIGNED32
Sub-Index 7: position2: UNSIGNED32
...

This plays nicer with CANopen SDO standard data types and leaves room for expansion. Obviously we implement range checks to the sizes we expect. Yes, this would use more frames but it's a one-time configuration task and pretty small compared to fetching all the parameters every couple of seconds.

It seems from my testing that the offset is pretty small as an int8_t especially for larger bit-length mappings.

I don't have a concrete use case but it seems that a pre-offset might be useful. So you do something like: (value + pre_offset)* gain + offset. This would let you map linear values quite flexibly to almost anything. Just an idle thought. Edit: Having slept on it this is largely pointless if offset has sufficient dynamic range for the mapping in question.

[celeron55]

I would love to have the ability to map big endian values, as I generally use that. Foccci/clara is currently forcing me to bodge extra little-endian or 8-bit values to messages in my cars to make things work.

I don't expect this to ever be added, but looks like now is the time to mention this in any case.

[johu]

That is already possible: viewtopic.php?f=41&t=4468 and Dave also added it for sending messages

[davefiddes]

I've just discovered a nasty wrinkle with the openinverter_can_tool. The esp32-web-interface doesn't like sharing a CAN bus with can mapping commands sent via oic. It seems that the web interface picks up replies not initiated by it which then causes it to be very confused. The ESP32 needs to be rebooted before it will start behaving correctly again. I'm not sure if this is fixable and it may be a byproduct of the way the CANopen protocol works.

Edit: Raised https://github.com/jsphuebner/esp32-web ... e/issues/8 to cover it. It only seems to happen when there is a CAN map in place otherwise the tools co-exist reasonably.

[johu]

Yes if I remember correctly CANOpen is designed to have only one bus master or whatever you call it. But still neat to have it fixed

[johu]

So while working on the canmap I found the change that allows communicating signed numbers. Unfortunately that is a breaking change if you have previous CAN mappings that rely on using the full number of bits, or am I missing something?

If we opt to change the protocol we should add a flag field for things like endianess and signedness.

Speaking of backward compatibility Maybe we should drag along both implementations for some time until everyones upgraded their oic and esp and whatnot and then delete the legacy protocol. Still got plenty of indexes left

[davefiddes]

What a glorious muddle.

TX mappings are and have always been signed.

That said a TX mapping continues to overflow in a "useful" way to behave as if it was unsigned. So this new unit test I wrote passes just fine:

Code: Select all

static void send_map_le_full_scale_throttle_position_in_first_word()
{
    canMap->AddSend(Param::pot, CanId, 0, 12, 1.0, 0);
    Param::SetFloat(Param::pot, 4095);

    canMap->SendAll();

    ASSERT(FrameMatches({ 0xff, 0x0f, 0, 0, 0, 0, 0, 0 }));
}

RX mappings were sometimes signed and other times unsigned when they worked. I made them mostly signed except for single bit values which I special cased to be 1 or 0 (and I should have twigged something was wrong here). Two OI nodes with complementary TX and RX mappings should be able to interchange params in a way they could not previously.

This new unit test used to pass just fine but now fails (i.e. confirming that the previous behaviour was unsigned):

Code: Select all

static void receive_map_le_full_scale_throttle_position_in_first_word()
{
    canMap->AddRecv(Param::pot, CanId, 0, 12, 1.0, 0);

    SendFrame({ 0xff, 0x0f, 0, 0, 0, 0, 0, 0 });

    ASSERT(Param::GetInt(Param::pot) == 4095);
}

I'm thinking about starting a new PR with:

• a safe subset of PR #8 avoiding signed mappings
• Big-endian TX mapping
• properly clamping TX mappings at 0..2^length. From inspection it looks like horrendous things can happen on overflow right now.

Then a further PR extending CAN mapping:

• Signed mappings
• Add a new SDO interface (as above)
• Extend the size of offset to be useful
• Proper extended CAN ID support

I will not be supporting multiple SDO interfaces in openinverter_can_tool at this point. Also, I will not be extending it to support the partial extended CAN frames. It's an experimental tool. Sorry.

I'm happy to add the code into libopeninv to support multiple interfaces if you wish it.

I would suggest that a "type" field would be a better way of encapsulating signed and unsigned. It would allow the addition of 32-bit float mappings as well. Looking at SavvyCAN that's certainly something that is found in the wild and would be simple to add at this point.

I prototyped a separate parameter for big-endian vs. little-endian. I found it confusing. The negative numbers for length work well when thinking about how the bits end up in the resulting CAN frame.

Does this seem a reasonable starting point? This would likely take the rest of the summer assuming I can work at this near full time as I have been recently.

[johu]

Continued a bit further on the force_extended_id branch: https://github.com/jsphuebner/libopenin ... f974f84fa8
Now the "force extended" bit is always bit 29. So to force-register an 11-bit ID as 29-bit add 0x20000000 to it. So 0x20000312 in the example above.

Code: Select all

cansend can0 616#23.01.30.00.12.03.00.20
cansend can0 616#23.01.30.01.04.00.1F.F0
cansend can0 616#23.01.30.02.64.00.00.00

I think for now phoenix can probably live with sending 3 CAN raw messages to set up his mapping. On the other hand it seems like it's just the range check (that you could consider redundant as the firmware does the same check) that keeps it from working with oic.

In the same commit I made signed RX mappings a compile time constant for simplicity. I know the inverter can do without signed quantities so I'd simply leave it off for that. All other projects can then choose whether it's safe to migrate to signed RX. So I guess the safe #8 is done.

Clamping sounds very reasonable.

Concerning the other changes please check how stuff is saved to flash. You'll find one migration function that I added when I did a rework of the data structures. So whatever you change to the data structures needs to be legacy-loaded from flash so people don't loose their mapping when upgrading firmware. It would probably be worthwhile to add some magic version byte in the beginning so detecting the format version doesn't just rely on the failed CRC check. "No wonder god built the world in 7 days, he didn't have a legacy base to build upon"

And finally, the current extended CAN ID support is proper as soon as "CAN_EXT" is defined. Again, the above bodge was only added to force-register a standard id as an extended ID in the STM32 hardware filter. Because filtering for 0x312 doesn't catch 0x00000312 (extended)

[davefiddes]

I really don't understand this extended ID encoding you have introduced.

As I understand it 29-bit extended CAN frames are different from 11-bit standard frames. This is orthogonal from the actual ID number and we have to configure our CAN HW to distinguish between when sending or receiving.

Internally in libopeninv we use a uint32_t for the canId which can represent either ID length comfortably. Ditto the current CAN SDO protocol uses an unsigned32 for the can_id field (I had forgotten this - from the way you were talking I thought it was one of the over-packed frames like gain/offset).

I don't understand:

• What the purpose behind "forcing" a can ID to be extended
• Why the bit position for "forcing" moves about between bit-11 and bit-29
• Why we can't just use a static bit 31 (i.e. the "sign" bit) as a standard vs. extended flag

I'm looking at adding a simple flag to indicate that an extended CAN ID is to be specified. For example:

Code: Select all

oic can add rx 0x312 ChargeCurrent 31 -16 0.100000001 0 --extended

I would really like to just OR the ID with 0x80000000 when sending the add SDO request to Sub-Index 0.

If a device doesn't support extended IDs it can surely just reject the add request. Either because the ID is 0x80000312 and thus > 0x7ff or because it explicitly recognises it as an extended frame ID which it does not support.

Internally I need to know whether a CAN ID is extended or not to correctly generate DBC files. The cantools library I use has a simple flag parameter to specify that a message uses an extended ID. This all seems straight forward to add.

You should be able to do all this in esp32-web-interface with a checkbox and/or new table column.

[johu]

As I understand it 29-bit extended CAN frames are different from 11-bit standard frames. This is orthogonal from the actual ID number and we have to configure our CAN HW to distinguish between when sending or receiving.

Internally in libopeninv we use a uint32_t for the canId which can represent either ID length comfortably. Ditto the current CAN SDO protocol uses an unsigned32 for the can_id field (I had forgotten this - from the way you were talking I thought it was one of the over-packed frames like gain/offset).

Ack.

What the purpose behind "forcing" a can ID to be extended

In hardware 0x312 and 0x00000312 are two different ids. So far we just implied everything below 0x7ff to be a standard id and all above to be an extended id. So if we want to register 0x312 as extended we need a way to explicitly specify this.

Why the bit position for "forcing" moves about between bit-11 and bit-29

This is no longer relevant on the outside interface. Internally with the symbol "CAN_EXT" we can select whether 16 or 32 bit of RAM (and flash) are used for the ids. This was done to squeeze as many frames as possible into a 1k flash page. So internally when CAN_EXT is not defined we store the EXT flag in bit 11, otherwise bit 29.

Why we can't just use a static bit 31 (i.e. the "sign" bit) as a standard vs. extended flag

That's what we do now. I just chose bit 29 (0-based) because it's the first unused bit.

oic can add rx 0x312 ChargeCurrent 31 -16 0.100000001 0 --extended

Perfect!

[davefiddes]

Ah. I hadn't appreciated you were _implying_ something was an extended id based on the value.

Bit 29 for the flag it is.

[davefiddes]

I've just released v0.1.0:

What's Changed

• CAN message mappings now support both standard CAN 2.0a and extended CAN 2.0b frames.
• Fix DBC export of CAN RX message maps to address issue #6 . Signals in RX maps are now unsigned. Signals in TX maps remain signed. All should match the expected behaviour of libopeninv projects.
• It is now possible to quickly clear all maps by issuing "oic can clear" or "oic can clear all" rather than having to clear tx and then rx.
• Adopt semantic version numbering from this release forwards

To create a map with an extended CAN ID just add --extended at the end of the can add command:

Code: Select all

$ oic can add rx 0x312 ChargeCurrent 31 -16 0.100000001 0 --extended
Adding CAN rx mapping with can_id=0x00000312 param='ChargeCurrent' position=31 length=-16 gain=0.100000001 offset=0
CAN mapping added successfully.

When you list the can maps the extended IDs are 8 hex digits in length rather than 3 for a standard ID irrespective of what the ID is. This follows other CAN command line tools like can-tools:

Code: Select all

$ oic can list
0x100:
 tx.0.0 param='tmphs' pos=0 length=16 gain=10.0 offset=0
 tx.0.1 param='tmpm' pos=16 length=16 gain=10.0 offset=0
0x00000312:
 rx.1.0 param='ChargeCurrent' pos=31 length=-16 gain=0.1 offset=0

It's worth noting that the tool supports much more capability for creating CAN maps with extended CAN IDs than libopeninv currently supports. It will let you know, politely, that the functionality is not supported where appropriate. For now stick to CAN RX maps with IDs of 0x7fff or less.

[johu]

Foccci is the first firmware to support at least 0x00000312 kind of mappings. No real extended IDs are yet possible as I foolishly started out with only space for 11-bit IDs in the map. Don't want to change it now so people don't loose their maps.
When the first use case comes up I guess we'll have to bite the bullet and restore the mappings. At least oic makes this easy.

[a768phoenix]

hey guys, i wanted to give you a big shout out.
the new FOCCCI firmware and the new OIC version work perfectly together.

image.png (8.48 KiB) Viewed 3584 times