Great work, is the plan to transition the web interface over to using the binary data for plotting?
Got the repos forked and cloned, hope to get the changes in and pushed up later today.
I've added a 'binlogging' command to the inverter firmware which takes an argument of 1 or 0 (on or off). It switches to 2250 and then starts streaming 24byte blocks (max possible at 2250kbps) of data (one on each run of the control loop). Just dummy data for testing at the moment.
The plan for today is to add new file creation at the start of logging (at the moment it just creates a single file on boot). All the time stamping and sequential naming stuff is in the already so should be fairly quick. At the moment if it finds a RTC it uses the time and date for filenames, if not it reverts to numbered sequential filenames. On boot it also checks SD card space and deletes the oldest files until 2GB is left (oldest determined from date if there is a RTC, otherwise filename number).
Logging start automatically when there are no connections to the ESP on WiFi and stops as soon as a WiFi connection is made.
The data received from the inverter is streamed straight to the card so not too easy to use at the moment. The plan to address this is to have the inverter send an ascii json header when logging starts which will define the packing of the binary blocks. This will get copied to the start of the card by the ESP firmware. The idea is that whatever uses the file can just parse the header to figure out how to process the rest of the file. This has the big advantage in that the header is created by the inverter and so it changes as the inverter build does and - so there should be no problems with compatibility between firmware builds and data analysis. This is the next bit to add in to the inverter code.
Once that's done, hopefully today, we should have a usable logger. It will be necessary to remove the SD card to get at the data but it's workable. At that point I'll pass it over to Bigpie to look at the web side. One thing I haven't done is add the web hooks for web access to the SD card file system. I think the best way is to map it as a virtual subdirectory on the root of the existing SPIFFS handler, that way the existing hooks can handle it. If I have time I will do it today, otherwise small test bin file/s can be placed in the SPIFFS root folder to allow web development in the meantime.
Not sure how to process the data yet. Options are either:
- Have the web app translate it to something else (say SavvyCan format and create a .dbc for it) while it is downloaded.
- Download as is and have a separate PC app which translates it to another format (SavvyCan, CSV, others?).
- Download as is and have a dedicated app for processing (may well do a very rough version of this for testing).
- Any other ideas?
Edit - Plan for the binary block is:
- Incrementing message counter - 8bits = 1B
- 4xADc vals at 12bits each (encoder cos, encoder sin, I1, I2) - 48bits = 6B
- 3xpwm vals at 14bits each - 42bits (+2 flag bits on each for desat, opmode, etc)= 6B
- 5xFP vals scaled to 16 bit each (iqref, idref, Ifw, Uq, Ud) = 10B
- checksum at 8bits = 1B
Total of 24bytes.
Plan for synchronisation on binary data when decoding is to scan through till get a valid csum. If this and the next csum are both valid treat it as synced but keep checking. Compare message counter on each pass to check for missing blocks. It's not 100% robust but hopefully good enough? Would a CRC rather than a checksum be worth the increased complexity? Should we sacrifice a data value to allow a more robust scheme?
Not too worried about corruption on the card, more corruption on the serial link (shouldn't be an issue if the ESP32 is on the inverter board but some concern if it is on wires).
Edit - Changed bit mappings above