ESP32 Based Web Interface & Data Logger

Discussion about various user interfaces such as web interface, displays and apps
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Did a quick and dirty decoder in python. Not quite as advanced as yours as in it doesn't calculate id and iq.
It uses the descriptor in [] array format.

EDIT: my attempt at the firmware is here: https://github.com/jsphuebner/stm32-sin ... arylogging

Code: Select all

[
   {"name":"counter","size":8,"scale":1,"signed":false},
   {"name":"angle","size":14,"scale":1,"signed":true},
   {"name":"idc","size":14,"scale":1,"signed":true},
   {"name":"il1","size":14,"scale":1,"signed":true},
   {"name":"il2","size":14,"scale":1,"signed":true},
   {"name":"pwm1","size":14,"scale":1,"signed":false},
   {"name":"opmode","size":2,"scale":1,"signed":false},
   {"name":"pwm2","size":14,"scale":1,"signed":false},
   {"name":"desat","size":1,"scale":1,"signed":false},
   {"name":"pwm3","size":14,"scale":1,"signed":false},
   {"name":"iqref","size":14,"scale":1,"signed":true},
   {"name":"idref","size":14,"scale":1,"signed":true},
   {"name":"ifw","size":14,"scale":1,"signed":true},
   {"name":"uq","size":16,"scale":1,"signed":true},
   {"name":"ud","size":16,"scale":1,"signed":true},
   {"name":"csum","size":8,"scale":1,"signed":false}
]
I think it would make sense to use the scaling to enable the converter to convert to SI without prior knowledge. Like 360/65536 for angle.

Code: Select all

import json, math

dataFile = open('dump.bin', "rb")
paramFile = open('params.json', 'wb')
outFile = open('data.csv', 'wb')

c = dataFile.read(1)

openBraces = (c == b'{') + 0

while openBraces > 0:
	paramFile.write(c)
	c = dataFile.read(1)
	
	if c == b'{':
		openBraces = openBraces + 1
	elif c == b'}':
		openBraces = openBraces - 1

paramFile.write(b'}')
paramFile.close()

while c != b'[':
	c = dataFile.read(1)
	
openBrackets = 1

descriptorJson = c
while openBrackets > 0:
	c = dataFile.read(1)
	descriptorJson = descriptorJson + c
	
	if c == b'[':
		openBrackets = openBrackets + 1
	elif c == b']':
		openBrackets = openBrackets - 1

items = json.loads(descriptorJson)
dataFile.read(2) #skip \r\n

bitPos = 0
comma = ''
for item in items:
	item['start'] = bitPos
	item['mask'] = (1 << item['size']) - 1
	bitPos = bitPos + item['size']
	
	outFile.write((comma + item['name']).encode())
	comma = ','
	
outFile.write(b'\n')

totalBits = sum([i['size'] for i in items])
recordSize = math.ceil(totalBits / 8)

record = dataFile.read(recordSize)
recordsRead = 1

def getBits(record, start, mask):
	byteidx = start // 8
	word = int(record[byteidx])
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 8)
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 16)
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 24)
	word = word >> (start % 8)
	return word & mask

while len(record) == recordSize:
	comma = ''
	for item in items:
		value = getBits(record, item['start'], item['mask'])
		outFile.write((comma + str(value)).encode())
		comma = ','
	outFile.write(b'\n')
	record = dataFile.read(recordSize)
	recordsRead = recordsRead + 1
	
print(recordsRead)
print(len(record))
One question: why are we logging idc? It is just calculated from uq, uq, id and iq.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Good idea, the scaling is already used that way for current so makes sense to extend it to the others (as long as no precision is lost when data is saved to file).

Idc is just in there because there was space and at the time I was interested in achievable power. It would probably make more sense to change it to something else now, for example frq would make analysis of the data much easier.

I was also thinking of doing another version that uses all the space to save all the current samples to get a better idea what effect different adc sampling schemes might have (particularly relevant for low L motors). Could do with something for sin builds too.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Added more logs from today if of any interest
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Thanks, I'll have a look, might be a day or two though.

Any improvements in the sd card reliability?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Killed the RTC somehow. The conversion app seems to not like some of the logs though, they're short 10 min drives but it's processing 100s on minutes of data. 0000000005.bin for example so not sure if it's related to the SD card or the conversion program at the moment
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Seem to remember thinking I'd killed mine at one point when doing the code. Turned out that it had just got into some kind of non resposive state. Pulling the battery and power supply for 10min sorted it out - might be worth a try if you haven't already?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I'll give that a try, but no major loss. Just need to get a clear log at min to try workout fluxlinkage and the crit param.

Hopefully I get some time this weekend to take a look
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Yahha777
Posts: 52
Joined: Wed Feb 27, 2019 7:03 am
Location: Belorussia. Borisov
Has thanked: 15 times
Been thanked: 1 time
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by Yahha777 »

Hello everybody! I have a couple of questions: I have GPIOs 16 and 17 connected to MG 1 in my ESP 32. It works! And MG 2 is connected to GPIO 9 and 10.
I make changes to the code:

Code: Select all

#define DBG_OUTPUT_PORT Serial
#define INVERTER_PORT UART_NUM_2
#define INVERTER_RX 9
#define INVERTER_TX 10
The following messages come to the debug port:
rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1344
load:0x40078000,len:13836
load:0x40080400,len:3608
entry 0x400805f0
ets Jun 8 2016 00:22:57

Moreover, even if the ESP is flashed with UART GPIO 16 17 and connect MG 2 to the outputs as in the diagram, then the ESP also constantly reboots.
is it possible to use UART1 in this code? If so, how and what can be changed?
Attachments
Communication.png
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

You have to be a bit careful which pins you use, some conflict with the internal flash memory in the ESP32 module.

Think you need to avoid pins 6-11 in particular.

Edit - also seem to remember that UART1 can conflict with some things too but afraid I can't remember what :(
Yahha777
Posts: 52
Joined: Wed Feb 27, 2019 7:03 am
Location: Belorussia. Borisov
Has thanked: 15 times
Been thanked: 1 time
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by Yahha777 »

Pete9008 wrote: Sat Nov 12, 2022 5:22 pm You have to be a bit careful which pins you use, some conflict with the internal flash memory in the ESP32 module.


Edit - also seem to remember that UART1 can conflict with some things too but afraid I can't remember what :(
Tried to use GPIO 1, 3. Works. 6-11 seems to be limited when using memory.
Thank you!
arber333
Posts: 3241
Joined: Mon Dec 24, 2018 1:37 pm
Location: Slovenia
Has thanked: 74 times
Been thanked: 223 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by arber333 »

I have a question regarding Collins CAN library for ESP32.
I use it quite succesfully. It works directly and you can send CAN telegrams par function. However when i tried to condition the change of msg within that function i couldnt get it to work.
With DUE i could program conditional loop within CAN telegram function and i could change particular byte here. But now with ESP32 i cant seem to get it to work. When i apply correct pin to start charger nothing changes within function. It seems like it would require to completely change function not just change one byte...

Anyone has any idea how to implement dynamic changes within function?

Code like this...

Code: Select all

void sendCANframeD() { // Heater
 CAN_FRAME txFrame;
    txFrame.rtr = 0;
    txFrame.id = 0x188; // // 0x188 03 50 A2 40 00 00 00 00 
    txFrame.extended = false;
    txFrame.length = 8; // Data payload 8 bytes
        if(digitalRead(Heater_pin) == LOW) { // if heater is ON        
    txFrame.data.uint8[0] = 0x03;
        }   
        else {
    txFrame.data.uint8[0] = 0x00;             
        }
    txFrame.data.uint8[1] = 0x50;
    if(Heatertemp > 55) { // if temp is higher than 55deg  
    txFrame.data.uint8[2] = 0x00; // when 55deg power goes to 0A
        } 
        else if(Heatertemp > 50) { // if temp is higher than 50deg
    txFrame.data.uint8[2] = 0x32; 
        }         
        else {
    txFrame.data.uint8[2] = 0xA2; // Full power to start heating
        }
    txFrame.data.uint8[3] = 0x40;
    txFrame.data.uint8[4] = 0x00;
    txFrame.data.uint8[5] = 0x00;         
    txFrame.data.uint8[6] = 0x00;
    txFrame.data.uint8[7] = 0x0;
    CAN0.sendFrame(txFrame);
}
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Can't see of anything that would stop the above from working.

I'd be a bit suspicious about whether the Heater_pin is working/setup right or whether it is on a pin that conflicts with something else. Have you tried putting some debug terminal prints in to confirm whether its state is being correctly read?
arber333
Posts: 3241
Joined: Mon Dec 24, 2018 1:37 pm
Location: Slovenia
Has thanked: 74 times
Been thanked: 223 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by arber333 »

Pete9008 wrote: Thu Dec 08, 2022 5:43 pm Can't see of anything that would stop the above from working.

I'd be a bit suspicious about whether the Heater_pin is working/setup right or whether it is on a pin that conflicts with something else. Have you tried putting some debug terminal prints in to confirm whether its state is being correctly read?
I tested the pin by a digital test. Turning on a LED on command. And it works. Also i added a buzzer to the function and it works as long as it is part of the command within conditional. If i put it inside a separate function however i cant get that function to change on command.
I would like to know if that is esp32 related or maybe my fault.
Did anyone sucessfuly tested CAN function in such a way?
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

The Arduino implementation of some peripherals did seem a bit flaky for the ESP32. I ended up having to use direct calls to the espressif api to get the UART and SD card to work properly (see https://docs.espressif.com/projects/esp ... index.html for more details on the api).

One other thought, have you tried dividing the problem? Move the IO read into the main loop updating a global variable and then use that variable in the CAN send function (same as you do with Heatertemp).
4GSHARK
Posts: 5
Joined: Thu Jan 12, 2023 6:37 am
Been thanked: 2 times

Re: ESP32 Based Web Interface & Data Logger

Post by 4GSHARK »

Does any one have the Sine firmware and bootloader, that works with esp32 web interface hex. file to share. SMD Update dont work so i think i need hex file only, I saw there is a modified firmware that was posted but it wasnt compiled. im not very CMD knowledgeable to make one.

I have 5.20 Sine installed and I can not see parameters at all in web parameters section. There are no values.

I have v6 LDU board with ESP32-Whoom Module.
Attachments
image_2023-02-11_145331464.png
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

I never tested the ESP32 code on my github for firmware updates and I seem to remember that when Johannes tried it he had to make a few changes (detailed above somewhere), they never got pulled due to github forking issues. @johu - if you can post the changes somewhere I'll update the repository?

If you can't see the parameters you also have a comms problem too.

If you want a simpler option the ESP8266 is currently a more stable/mature solution (the ESP32 option is still being developed).
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Just realized I never forked your repo, will clean that all up later (or you give me write access, will push to another branch).
For my scenario I changed the inverter UART to UART0.
Most of all I bugfixed the update process:

Code: Select all

-  int pages = (file.size() + PAGE_SIZE_BYTES - 1) / PAGE_SIZE_BYTES;
+  uint8_t pages = (file.size() + PAGE_SIZE_BYTES - 1) / PAGE_SIZE_BYTES;

-    snprintf(uartMessBuff, UART_MESSBUF_SIZE, "%d", pages);
-    uart_write_bytes(INVERTER_PORT, uartMessBuff, strnlen(uartMessBuff, UART_MESSBUF_SIZE));
+    uart_write_bytes(INVERTER_PORT, &pages, 1);
That was it, it all works then. I'm already shipping that on the Tesla SDU and Nissan Leaf kit.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

johu wrote: Sun Feb 12, 2023 9:43 am Just realized I never forked your repo, will clean that all up later (or you give me write access, will push to another branch).
Whatever is easiest (although you may have to explain how yo do it - I'm still a gitub novice). Might be neater to just add it as a new repository or branch on your github so everything is in one place?
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

In the meantime I've added Johannes changes (hopefully all of them but untested) to a branch - https://github.com/Pete9008/esp32-web-i ... /tree/johu

Hopefully this will work with ESP32 modules that are integrated onto boards.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Board arrived today Pete, thanks. Need to get my head around what's changes are needed in the inverter firmware to get this working again with the latest release code.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
mario
Posts: 23
Joined: Fri Dec 14, 2018 10:20 pm
Has thanked: 67 times
Been thanked: 10 times

Re: ESP32 Based Web Interface & Data Logger

Post by mario »

My branch is https://github.com/mmadzar/GN02475inv and inverter firmware update code wasn't changed, just added some custom code that no one else needs. :)
I can confirm that firmware update code works. I uploaded v5.26.Y firmware last night without any problems. Just uploaded bin file and that was it.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Tue Apr 04, 2023 12:20 pm Board arrived today Pete, thanks. Need to get my head around what's changes are needed in the inverter firmware to get this working again with the latest release code.
You and me both!

Thinking about it my board was built up largely of bits and pieces I already had so the parts list might not be quite as good as it could be. Let me know if anything doesn't make sense.
mario wrote: Tue Apr 04, 2023 12:29 pm My branch is https://github.com/mmadzar/GN02475inv and inverter firmware update code wasn't changed, just added some custom code that no one else needs. :)
I can confirm that firmware update code works. I uploaded v5.26.Y firmware last night without any problems. Just uploaded bin file and that was it.
Thanks for posting, I'd noticed your fork and wondered whether it was someone on here :)

Would you be able to give a bit more detail on what your additions are as they could well be useful to someone (I'm curious about the MQTT stuff in particular)?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I've been having a bit of a read, the code on Johannes GitHub doesn't contain your binary logging changes, but does contain binstream that works differently. As I understand, the binstream that's in released code doesn't send the values each loop, so not ideal.
I've not braved a look at the BOM yet, but I guess adding your flavour of binary logging would be where I go next.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Not sure, the ideal would be to get Johanne's version working but need to have another look at the code to see what would be needed.

In terms of seeing what's happening with your current sensors either approach should work but I'd prefer to have loop by loop data if possible.

Been struggling a bit with getting stuff done recently but hopefully back on it soon.
User avatar
johu
Site Admin
Posts: 5683
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Yeah binstream sends out items in 4-byte packets as fast as the serial allows. So depending on the number of items it can be slower. Most of all I don't think it's implemented in Petes ESP code.

Will get back to that and implement Petes more condensed protocol
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Post Reply