ESP32 Based Web Interface & Data Logger

Discussion about various user interfaces such as web interface, displays and apps
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

The esp codes not fussy about what comes back, whatever it receives gets logged to the sd card. As long as you have a way of decoding the files from the sd card the stm can send just about anything you like when in logging mode.
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Sure, you'd just need to send "binstream -1 il1,il2" once
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Following from this thread viewtopic.php?t=3520 had a thought on how to make the logging a bit more flexible.

Essentially the idea is to add a second logging mode that uses the standard 'stream' command instead of binary logging. This would work in the same way as the binary logging in that when there were no WiFi connections the ESP would send a stream command (the actual command would be read from a text file in the SPIFFS file system (so that it can be updated/changed via the existing web interface upload functionality). Any data received back would then be written to the SD card. The stream command results in a comma delimited ascii data stream which means the data on the card could be directly imported into any standard graphing or analysis software.

Would probably be worth adding a time stamp to the front of each line. Would also be worth adding the stream command used at the top of each SD card file to aid analysis.

This should work with completely standard OI firmware. The only possible change needed would be to increase the number of parameters that the stream command can handle from the current value of 10 to say 20 to increase the logging capability.

Not sure when I'll get to actually doing this though.
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

I've started coding the CAN backend of ESP32 web interface but ran into some roadblocks: the CAN driver behaves weirdly when SDOs are requested in a for-loop. Instead of sending out 0x601#40 00 50 01 00 00 00 00 for example, it sends out and all zero message 0x601#0 0 0 0 0 0 0 0 0

If I do one request per call of the loop() it works as intended, I can even download the full json then. I use miwagner/ESP32CAN@^0.0.1
Now I'm not sure which way to go now, bug hunt the CAN issue or try to convert the data paths to WebSocket.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Might be worth a look at the CAN library gvret uses on the ESP32 https://github.com/collin80/ESP32RET , that's the one I hope to use.

Sounds like yours isn't interrupt based?
arber333
Posts: 3241
Joined: Mon Dec 24, 2018 1:37 pm
Location: Slovenia
Has thanked: 74 times
Been thanked: 223 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by arber333 »

Pete9008 wrote: Tue May 23, 2023 5:42 pm Might be worth a look at the CAN library gvret uses on the ESP32 https://github.com/collin80/ESP32RET , that's the one I hope to use.

Sounds like yours isn't interrupt based?
I am already using Collins code to run Outlander charger, DCDC, AC compressor, Heater and Tesla DCDC with ESP32 board.
I recommend using interrupts as this allows to use CAN telegrams out of sync.
I use timer interrupt to time telegrams to 100ms and cyclic central telegram 0x285 at 10ms.
I also use watchdog at 2s interval to verify chip behaves...
Code is here... https://github.com/arber333/ESP32-VCU/b ... _code2.ino
User avatar
uhi22
Posts: 554
Joined: Mon Mar 14, 2022 3:20 pm
Location: Ingolstadt/Germany
Has thanked: 83 times
Been thanked: 392 times

Re: ESP32 Based Web Interface & Data Logger

Post by uhi22 »

There is a longer discussion of ESP32 CAN controller issues here: https://github.com/espressif/esp-idf/issues/4276
Not sure whether this is related to the topic here, and also have no clue whether there is a stable solution in the end. My understanding at the moment is: The CAN works fine, as long as there is nearly no bus load, and with increasing bus load, the troubles are starting, because the hardware buffer handling is not correctly designed.
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

That could be what I'm seeing. I'll switch to Collins library now https://github.com/collin80/esp32_can and will handle messages in an interrupt, after handling send out the next request.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

This can interface is terrible! I've tried Collins driver but didn't manage to send any message with it.
Now I'm using the espressif twai api but even that is not working stable. I have no idea what it is that makes it work stable for one run and then at the next change it fails with another mode again.

Current source code attached.

Here's the traffic it creates:

Code: Select all

ID	Extended	Bus	LEN	D1	D2	D3	D4	D5	D6	D7	D8
601	false		1	8	40	0	50	0	0	0	0	0
581	false		1	8	43	0	50	0	56	FF	6D	6
601	false		1	8	40	0	50	1	0	0	0	0
581	false		1	8	43	0	50	1	51	83	57	49
601	false		1	8	43	0	50	1	51	83	57	49
So the last 601 message instead of requesting (0x40 in byte 0) is just a copy of the last received message. wtf...

UPDATE: I think I found it! The twai_message_t tx_frame mustn't reside on the stack, it must remain in memory until the frame is sent or enqueued or something. Now it works.
Attachments
oi_can.cpp
(5.39 KiB) Downloaded 58 times
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Finally managed to implement the "json", "set" and "stream" command via CAN. Now trying to decide whether I should mix it in with the existing code or start with a fresh code base?
As it stands the UART comms is turned off. A unified web interface would let you choose perhaps between UART0, 1, 2 or CAN (nodeid 1-63)

UPDATE: https://github.com/jsphuebner/esp32-web-interface
I left the uart code in there but removed the binary logging as that won't work over CAN or would cover a different use case.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Added update via CAN also. It runs quite quickly.
https://github.com/jsphuebner/esp32-web ... 73fc33220b

EDIT: I should add, the update now runs autonomously once it has been started. So even if the wifi connection collapses the update will finish.

Now Save/Restore/Stop etc. commands to be added and also CAN mapping via CAN :) Needs a better implementation in canmap.cpp as well
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Not sure whether there is still any interest in the data logging side of this, if so then there is a bit of progress described over in my build thread (viewtopic.php?p=57268#p57268).

If there is any interest let me know and I'll look at porting the code back to the F103.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I am for sure. Picked up a Yaris inverter to try your auto tune, binary logging and resistor mod outside of the car :D
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Great stuff :) are you going to use it with the Leaf or the Outlander front motor?

I'll look at porting it over to the auto-setup branch (or another branch off that branch).

First I'm going to have a go at exporting the data from the decoder app directly to PulseView's native file format (mentioned in the other thread it's not displaying the data right and my suspicion is that it's the PulseView csv importer that's the problem).
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Eventually leaf motor in a car, but while it's in garage I'll run auto tune on leaf and outlander front motor.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

The decoder app for the binary logs has now been tidied up and is on github: https://github.com/Pete9008/LoggingDecode

It's a Qt console application and now accepts command line arguments for input and output file names and options to select what processing is done. It can generate a JSON file containing the inverter parameters, two CSV files (one for high rate core control loop data and a lower rate one containing all spot parameters) and a PulseView native format file containing the core control loop data.

The default is to generate all output files and takes around 30sec to process 1min of input data. Skipping the motor data csv output file halves this (might be sensible to change the default to do this!).

It should be backward compatible with old logs but I haven't tried it. The spot data is a new feature and currently only works with my F405 build but will look at porting it back to the F103 code.

Note:
The old CSV PulseView import method has been found to corrupt the data so the new native format should always be used with PulseView.

The native PulseView format needs the data files to be zipped into a single archive. The program uses a command line call to do this so is only likely to work on Linux systems. To work on other platforms the command and arguments for the process call in the code will need to be changed to suit. Alternatively the .PulseViewZip folder could be manually zipped.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Sun Jun 04, 2023 12:43 pm I am for sure. Picked up a Yaris inverter to try your auto tune, binary logging and resistor mod outside of the car :D
Any idea when you might be doing this? Might be interested in popping over when you do as it looks like the auto setup code may need a little work!

Edit - Also found an old Bluepill board in the cupboard so going to try and get some F103 logging hardware setup here.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Working on assembly of the boards at the moment. Just about there with the inverter board so hopefully by the weekend should have check the various power rails and verified PWM etc.

Need to get the esp32 BOM parts ordered next, but could just run web interface on a esp32 devboard mess from last time.

WFH so pretty much always around
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Thanks, no rush at all so whenever you're ready to try it.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Got my test setup now. Need to debug my lack of resolver readings, next.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Is the resolver issue using the auto setup code or 5.27? If the former then it's sounding increasingly likely that I've messed something up in the code (Romale had a similar issue).

Got a logging setup here now using an old bluepill board but haven't got anything done on porting the code over yet.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I'll try a known working version later. I broke out the picoscope and it looked like the exciter was running. I didn't hear the pwm start either.
You're welcome to come mess around with my setup if you want.

EDIT. Put an old version on and still not working. So will have to go back over the Yaris board thread and debug.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Thu Jun 15, 2023 8:12 am You're welcome to come mess around with my setup if you want.
Thanks, that would be good. :)

Probably makes sense to get the logging code ported across first though (and I could do with getting my laptop sorted out too - needs the OS updating and all the dev tools putting on it).
User avatar
Proton
Posts: 137
Joined: Sat May 06, 2023 2:23 am
Location: Georgia/US
Has thanked: 104 times
Been thanked: 18 times

Re: ESP32 Based Web Interface & Data Logger

Post by Proton »

johu wrote: Wed May 31, 2023 2:22 pm
This is good news because I really do not like the Wifi option. I do not like to leave a digital trail everywhere I go. I do not use wifi in my house on any devices.
I was thinking that after i am done setting up the motor, to open the motor and damage the wifi card so it will never transmit. for sure I will not leave it in there.
Hopefully I will be able to use the CAN solution.
Post Reply