Tool to manage openinverter via CAN

davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Great. Looks prettier in Wireshark :)
Screenshot from 2023-02-08 14-54-40.png
johu wrote: Wed Feb 08, 2023 1:46 pm By that you mean SDO upload or including the output of genparamdb with every build?
Automatically uploading from the running firmware to oic as needed and not cached on disk.

I'm going to keep genparamdb in my c2000 project for the foreseeable. Too many other vital bits missing from that code base at the moment to be distracted.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Works on the template :)

Code: Select all

debian@beaglebone:~$ python3 candownload.py 
{ 
   "canspeed": {"unit":"0=125k, 1=250k, 2=500k, 3=800k, 4=1M","value":2.00,"isparam":true,"minimum":0.00,"maximum":4.00,"default":2.00,"category":"Communication","i":0},
   "canperiod": {"unit":"0=100ms, 1=10ms","value":0.00,"isparam":true,"minimum":0.00,"maximum":1.00,"default":0.00,"category":"Communication","i":1},
   "testparam": {"unit":"Hz","value":0.00,"isparam":true,"minimum":-100.00,"maximum":1000.00,"default":0.00,"category":"Testing","i":2},
   "opmode": {"unit":"0=Off, 1=Run","value":0.00,"isparam":false},
   "version": {"unit":"4=1.00.R-name","value":4.00,"isparam":false},
   "lasterr": {"unit":"0=NONE, 1=TESTERROR,","value":0.00,"isparam":false},
   "testain": {"unit":"dig","value":121.00,"isparam":false},
   "cpuload": {"unit":"%","value":0.00,"isparam":false},
   "serial": {"unit":"","value":"87215032","isparam":false}
}
On the first reply I always specify a size of 64k as I don't know in advance how long the JSON string will be. Seems to work anyway.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Cool. It looks like it's entirely allowed to not specify the data size if you don't know it. Looking at the code, the python canopen libraryseems to be happy with that response.

Are you sticking with the official CANopen dictionary Index of 0x1021? Spec says that should be in EDS format (a fancy INI file). Probably best to give it a custom Index number.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Mapped it to 0x5001, 0.
Timed the JSON download, it takes about 10s on the sine firmware

Checkout the port here: https://github.com/jsphuebner/stm32-sin ... can_module
Attachments
stm32_sine.bin
(50.09 KiB) Downloaded 54 times
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Just thinking how to do automatic versioning of the data items. Maybe a simple checksum over all uids would suffice? Could just be calculated with the preprocessor. Then the CAN tool can decide whether or not to download the json data
Also the "serial" object that I just insert on the fly upsets the parser as it has no id. Maybe objects without id can just be ignored.

Also found a potential issue with test parameters. These all have id=0 and are not saved to flash. Seems we need to think of something new. Maybe id>x are not stored to flash

EDIT: just timed the CAN download on my laptop, there it's just 3.5s
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

I like that idea. I could cache the parameter database by checksum and node ID. It doesn't have to be perfect (e.g. crypto hash) as we have the uid as backup to resolve any confusion.

I'll take a look and see how to integrate support for what you've done. At 3.5 sec per command it's not painful to do the simple thing and download every time. On a busy CAN bus it'll no doubt be a different story.
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Not having much luck getting the stm32-sine new_can_module branch going with your binary or if I build it myself. It just seems to crash the CAN bus with protocol errors whenever any CAN transmission is attempted. Tried wiping the CAN map and saving to flash before upgrading but that just stops the problems until I send an SDO request. stm32-template works fine. This is with a bluepill.

I see what you mean about "serial" and "testparam". Everything needs an ID of some sort to be accessible over CAN. How about assigning IDs starting at 3000 for generated values?

Bit worried about the reset command. Doesn't look like the SDO response will get a chance to get out of the device before it goes away. Also, there's no mutex with terminal initiated parameter saves and loads which could lead to fun times.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Silly... The canspeed enum has changed. Set it to 800k to run at 500k. Just did a fresh clone and that did the trick.
Yes with ids > 3000 or rather 16384 that could be done. That said, "serial" isn't a true data item that you can retrieve with an id. I bodged it in and it was quite messy if you want it to display something meaningful to the user (i.e. a hex number, not a fixed point number).
EDIT: I could hard code serial in the SDO handler. Right now you can retrieve all 3 words on SDO index 0x5000, subindex 0-2. I could instead map it to 0x21FF, subindex 0-2 (would correspond to id 65280-65282).

Yes noticed with the lacking SDO response. But heyho, you get the boot message instead. Otherwise it would mean send SDO response, find out whether it has really been sent (how do you even do that?) and then reset. A bit too much perfectionism for me. Same with simultaneous save over CAN and terminal. Congratulations to anyone who manages to do that :P
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Changing the CAN speed to 800k fixed it.

I see what you mean about "serial". I think that special casing it and using the 0x5000 index to fetch using a dedicated command is the simplest approach for now. It would sit OK alongside CAN bootloader support in the tool. I'll filter out things from the JSON parameter database I don't understand.

You can get confirmation that a CAN frame has been acknowledged by someone if you grub deep enough in the registers. Seems a lot of work. The approach I've used with IoT systems with much success is to respond and to wait a tactical period (e.g. 100ms) before doing the deed. Works for most systems for most of the time.

Will look at getting support for all this stuff implemented now.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Alright, that could be feasible (waiting)

Currently doing some more work on the boot loader to spit out the UID. Will eliminate the "magic" for CAN updates and just wait for the 4 correct id bytes.

EDIT: committed and pushed
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

davefiddes wrote: Fri Feb 10, 2023 2:00 pm Will look at getting support for all this stuff implemented now.
Could you add a continous polling mode that just outputs comma separated values of the chosen items? Could be used for logging.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

That would be useful for slow changing signals. I'll add that to the list.

Somewhat relatedly I have an idea for the medium term to manage can mapping with the tool. My thought was that it might be possible to configure can mappings from a DBC file. That way you could design the can map in SavvyCAN then load it onto the device and then the data would just show up ready for high-speed logging and graphing. The API would need a lot of boring work so I've put that on the back burner for now. It's easier for me to just hard code CAN mapping in my firmware then I don't have to worry about writing a mapping storage backend and a full featured CAN API to manage.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

There is already an SDO index range for CAN mapping but I'd describe it as somewhat dodgy (maybe you've seen it). I think I will revisit that also and maybe split it into a segmented download or adjacent subindex or something. So you'd be sending the first half of the mapping info to subindex 0, the second half to subindex 1 which then also does the final mapping.
I know there is something with PDO mapping in canopen, maybe that could be used. That said PDO are are also tied to the node id which is not useful for us.

On the latest libopeninv I've added the parameter id sum to index 0x5000, subindex 3
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Not only seen it I've written unit tests to understand how it works and its limitations. Would like to approach this from an end-to-end point of view to avoid clunky APIs.

Would be good to suggest DBC export to someone working on the web interface. It would be entirely possible to output a can map to DBC ready to use. Strikes me as very useful.
johu wrote: Sun Feb 12, 2023 5:28 pm On the latest libopeninv I've added the parameter id sum to index 0x5000, subindex 3
Any chance you could move this to 0x5001 subindex 1? It seems to be more in common with the database string itself and not much to do with the serial number(s).

I've pushed the code to auto-load the DB if you want to give it a go. It'll take me a couple of days to finish up the remaining features before I make a release.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

davefiddes wrote: Sun Feb 12, 2023 6:13 pm Not only seen it I've written unit tests to understand how it works and its limitations. Would like to approach this from an end-to-end point of view to avoid clunky APIs.

Would be good to suggest DBC export to someone working on the web interface. It would be entirely possible to output a can map to DBC ready to use. Strikes me as very useful.
Ah yes, for that we need read/upload support also. The information is contained in the json but "proper" SDO semantics shouldn't be hard to add.
Open to suggestions on the leanest format over SDO.
davefiddes wrote: Sun Feb 12, 2023 6:13 pm Any chance you could move this to 0x5001 subindex 1? It seems to be more in common with the database string itself and not much to do with the serial number(s).
Yeah, a bit philosophical, this. Put it there for ease of implementation. If you check the code for 0x5001 it flags into the userspace and expects a longer blob of data rather than a simple integer.
I've pushed the code to auto-load the DB if you want to give it a go. It'll take me a couple of days to finish up the remaining features before I make a release.
Great! will check it out tomorrow.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

davefiddes wrote: Sun Feb 12, 2023 6:13 pm I've pushed the code to auto-load the DB if you want to give it a go. It'll take me a couple of days to finish up the remaining features before I make a release.
How do I obtain the latest version? pip?
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

I've not released a new version. Wanted to get more functionality in before the hassle of releasing it. You need a development clone:

Code: Select all

git clone https://github.com/davefiddes/openinverter_can_tool.git
cd openinverter_can_tool
pip install -e .[dev,test]
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Hmm :(

Code: Select all

Traceback (most recent call last):
  File "/home/debian/.local/bin/oic", line 33, in <module>
    sys.exit(load_entry_point('openinverter-can-tool', 'console_scripts', 'oic')())
  File "/home/debian/.local/bin/oic", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/debian/.local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 169, in load
    return functools.reduce(getattr, attrs, module)
AttributeError: module 'openinverter_can_tool' has no attribute 'cli'
Python 3.7.3
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

Bleh. Instructions clearly need some work.

It's probably best to run in a python virtualenv when running development versions. First time set up would look like:

Code: Select all

git clone https://github.com/davefiddes/openinverter_can_tool.git
cd openinverter_can_tool
virtualenv venv
. venv/bin/activate
pip install -e .[dev,test]
Then when you run "oic" it'll pick up the development code. Your bash shell will get an annotation showing the virtualenv is active. Yopu can go anywhere and run "oic" and it'll pick it up from the modified path.

Code: Select all

(venv) [dave@snowman openinverter-can-tool]$
To return your shell python environment to normal run:

Code: Select all

deactivate
Next time you want to run the dev version of the tool run:

Code: Select all

cd openinverter_can_tool
. venv/bin/activate
You might need to install the python3-virtualenv package with apt-get/dnf.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Thanks, tried it and all looked well but then same end result:

Code: Select all

(venv) debian@beaglebone:~/openinverter-can-tool$ oic
Traceback (most recent call last):
  File "/home/debian/.local/bin/oic", line 33, in <module>
    sys.exit(load_entry_point('openinverter-can-tool', 'console_scripts', 'oic')())
  File "/home/debian/.local/bin/oic", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/debian/.local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 169, in load
    return functools.reduce(getattr, attrs, module)
AttributeError: module 'openinverter_can_tool' has no attribute 'cli'
EDIT: I had to change clone to git clone https://github.com/davefiddes/openinverter-can-tool.git so dashes instead of underscores. Am I cloning the wrong repo?

EDIT2: problem is only on beaglebone, on my laptop it runs fine
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Yes, runs fine on my laptop, even w/o venv. Pulling a dumpall from the BMS takes 7s
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

johu wrote: Mon Feb 13, 2023 7:03 pm EDIT2: problem is only on beaglebone, on my laptop it runs fine
Hmm. What version of Debian is it running? I've got an old Sheevaplug running Debian Bullseye that seemed to run it ok.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

davefiddes wrote: Mon Feb 13, 2023 7:17 pm Hmm. What version of Debian is it running? I've got an old Sheevaplug running Debian Bullseye that seemed to run it ok.
BeagleBoard.org Debian Buster IoT Image 2020-04-06 (I think based on debian 10.3). It's the newest image I found.

Meanwhile I have messed it up so badly that not even the pip install version runs anymore

EDIT: got it running again by tracking down all oic "binaries" and packages, reinstalling python and installing again. The release version, that is
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
davefiddes
Posts: 211
Joined: Mon Jan 18, 2021 12:39 pm
Location: Edinburgh, Scotland, UK
Has thanked: 14 times
Been thanked: 35 times

Re: Tool to manage openinverter via CAN

Post by davefiddes »

I've made a new release of openinverter_can_tool. It has the following new features:
  • Automatic parameter database download from the device being configured *
  • Retrieve device serial number use with the CAN bootloader
  • Command the device to:
    • reset
    • load parameters from flash
    • save parameters to flash
    • reset parameters to defaults **
    • stop **
    • start **
  • Log a list of parameters to a CSV file or the console at periodic intervals
  • Slightly clearer and less verbose error handling
* - Requires a device built from libopeninv commit c83352d or later. For example stm32-sine on the new_can_module branch.
** - Requires firmware support which hasn't been implemented yet

Things that it doesn't do but are on the roadmap:
  • Cache parameter databases locally to speed up commands and reduce the load on the CAN bus
  • Automatic firmware upgrade over CAN
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Tool to manage openinverter via CAN

Post by johu »

Works a treat on my beaglebone!
Weird how pip doesn't seem to allow upgrades? Had to uninstall and reinstall.

Where does it cache the json files?
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Post Reply