Page 10 of 13

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Nov 11, 2022 4:02 pm
by johu
Did a quick and dirty decoder in python. Not quite as advanced as yours as in it doesn't calculate id and iq.
It uses the descriptor in [] array format.

EDIT: my attempt at the firmware is here: https://github.com/jsphuebner/stm32-sin ... arylogging

Code: Select all

[
   {"name":"counter","size":8,"scale":1,"signed":false},
   {"name":"angle","size":14,"scale":1,"signed":true},
   {"name":"idc","size":14,"scale":1,"signed":true},
   {"name":"il1","size":14,"scale":1,"signed":true},
   {"name":"il2","size":14,"scale":1,"signed":true},
   {"name":"pwm1","size":14,"scale":1,"signed":false},
   {"name":"opmode","size":2,"scale":1,"signed":false},
   {"name":"pwm2","size":14,"scale":1,"signed":false},
   {"name":"desat","size":1,"scale":1,"signed":false},
   {"name":"pwm3","size":14,"scale":1,"signed":false},
   {"name":"iqref","size":14,"scale":1,"signed":true},
   {"name":"idref","size":14,"scale":1,"signed":true},
   {"name":"ifw","size":14,"scale":1,"signed":true},
   {"name":"uq","size":16,"scale":1,"signed":true},
   {"name":"ud","size":16,"scale":1,"signed":true},
   {"name":"csum","size":8,"scale":1,"signed":false}
]
I think it would make sense to use the scaling to enable the converter to convert to SI without prior knowledge. Like 360/65536 for angle.

Code: Select all

import json, math

dataFile = open('dump.bin', "rb")
paramFile = open('params.json', 'wb')
outFile = open('data.csv', 'wb')

c = dataFile.read(1)

openBraces = (c == b'{') + 0

while openBraces > 0:
	paramFile.write(c)
	c = dataFile.read(1)
	
	if c == b'{':
		openBraces = openBraces + 1
	elif c == b'}':
		openBraces = openBraces - 1

paramFile.write(b'}')
paramFile.close()

while c != b'[':
	c = dataFile.read(1)
	
openBrackets = 1

descriptorJson = c
while openBrackets > 0:
	c = dataFile.read(1)
	descriptorJson = descriptorJson + c
	
	if c == b'[':
		openBrackets = openBrackets + 1
	elif c == b']':
		openBrackets = openBrackets - 1

items = json.loads(descriptorJson)
dataFile.read(2) #skip \r\n

bitPos = 0
comma = ''
for item in items:
	item['start'] = bitPos
	item['mask'] = (1 << item['size']) - 1
	bitPos = bitPos + item['size']
	
	outFile.write((comma + item['name']).encode())
	comma = ','
	
outFile.write(b'\n')

totalBits = sum([i['size'] for i in items])
recordSize = math.ceil(totalBits / 8)

record = dataFile.read(recordSize)
recordsRead = 1

def getBits(record, start, mask):
	byteidx = start // 8
	word = int(record[byteidx])
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 8)
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 16)
	byteidx = byteidx + 1
	if byteidx < len(record):
		word = word + (int(record[byteidx]) << 24)
	word = word >> (start % 8)
	return word & mask

while len(record) == recordSize:
	comma = ''
	for item in items:
		value = getBits(record, item['start'], item['mask'])
		outFile.write((comma + str(value)).encode())
		comma = ','
	outFile.write(b'\n')
	record = dataFile.read(recordSize)
	recordsRead = recordsRead + 1
	
print(recordsRead)
print(len(record))
One question: why are we logging idc? It is just calculated from uq, uq, id and iq.

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Nov 11, 2022 4:35 pm
by Pete9008
Good idea, the scaling is already used that way for current so makes sense to extend it to the others (as long as no precision is lost when data is saved to file).

Idc is just in there because there was space and at the time I was interested in achievable power. It would probably make more sense to change it to something else now, for example frq would make analysis of the data much easier.

I was also thinking of doing another version that uses all the space to save all the current samples to get a better idea what effect different adc sampling schemes might have (particularly relevant for low L motors). Could do with something for sin builds too.

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Nov 11, 2022 6:36 pm
by Bigpie
Added more logs from today if of any interest

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Nov 11, 2022 6:47 pm
by Pete9008
Thanks, I'll have a look, might be a day or two though.

Any improvements in the sd card reliability?

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Nov 11, 2022 7:46 pm
by Bigpie
Killed the RTC somehow. The conversion app seems to not like some of the logs though, they're short 10 min drives but it's processing 100s on minutes of data. 0000000005.bin for example so not sure if it's related to the SD card or the conversion program at the moment

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Nov 12, 2022 1:15 pm
by Pete9008
Seem to remember thinking I'd killed mine at one point when doing the code. Turned out that it had just got into some kind of non resposive state. Pulling the battery and power supply for 10min sorted it out - might be worth a try if you haven't already?

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Nov 12, 2022 2:59 pm
by Bigpie
I'll give that a try, but no major loss. Just need to get a clear log at min to try workout fluxlinkage and the crit param.

Hopefully I get some time this weekend to take a look

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Nov 12, 2022 4:54 pm
by Yahha777
Hello everybody! I have a couple of questions: I have GPIOs 16 and 17 connected to MG 1 in my ESP 32. It works! And MG 2 is connected to GPIO 9 and 10.
I make changes to the code:

Code: Select all

#define DBG_OUTPUT_PORT Serial
#define INVERTER_PORT UART_NUM_2
#define INVERTER_RX 9
#define INVERTER_TX 10
The following messages come to the debug port:
rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1344
load:0x40078000,len:13836
load:0x40080400,len:3608
entry 0x400805f0
ets Jun 8 2016 00:22:57

Moreover, even if the ESP is flashed with UART GPIO 16 17 and connect MG 2 to the outputs as in the diagram, then the ESP also constantly reboots.
is it possible to use UART1 in this code? If so, how and what can be changed?

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Nov 12, 2022 5:22 pm
by Pete9008
You have to be a bit careful which pins you use, some conflict with the internal flash memory in the ESP32 module.

Think you need to avoid pins 6-11 in particular.

Edit - also seem to remember that UART1 can conflict with some things too but afraid I can't remember what :(

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Nov 12, 2022 6:36 pm
by Yahha777
Pete9008 wrote: Sat Nov 12, 2022 5:22 pm You have to be a bit careful which pins you use, some conflict with the internal flash memory in the ESP32 module.


Edit - also seem to remember that UART1 can conflict with some things too but afraid I can't remember what :(
Tried to use GPIO 1, 3. Works. 6-11 seems to be limited when using memory.
Thank you!

Re: ESP32 Based Web Interface & Data Logger

Posted: Wed Dec 07, 2022 2:44 pm
by arber333
I have a question regarding Collins CAN library for ESP32.
I use it quite succesfully. It works directly and you can send CAN telegrams par function. However when i tried to condition the change of msg within that function i couldnt get it to work.
With DUE i could program conditional loop within CAN telegram function and i could change particular byte here. But now with ESP32 i cant seem to get it to work. When i apply correct pin to start charger nothing changes within function. It seems like it would require to completely change function not just change one byte...

Anyone has any idea how to implement dynamic changes within function?

Code like this...

Code: Select all

void sendCANframeD() { // Heater
 CAN_FRAME txFrame;
    txFrame.rtr = 0;
    txFrame.id = 0x188; // // 0x188 03 50 A2 40 00 00 00 00 
    txFrame.extended = false;
    txFrame.length = 8; // Data payload 8 bytes
        if(digitalRead(Heater_pin) == LOW) { // if heater is ON        
    txFrame.data.uint8[0] = 0x03;
        }   
        else {
    txFrame.data.uint8[0] = 0x00;             
        }
    txFrame.data.uint8[1] = 0x50;
    if(Heatertemp > 55) { // if temp is higher than 55deg  
    txFrame.data.uint8[2] = 0x00; // when 55deg power goes to 0A
        } 
        else if(Heatertemp > 50) { // if temp is higher than 50deg
    txFrame.data.uint8[2] = 0x32; 
        }         
        else {
    txFrame.data.uint8[2] = 0xA2; // Full power to start heating
        }
    txFrame.data.uint8[3] = 0x40;
    txFrame.data.uint8[4] = 0x00;
    txFrame.data.uint8[5] = 0x00;         
    txFrame.data.uint8[6] = 0x00;
    txFrame.data.uint8[7] = 0x0;
    CAN0.sendFrame(txFrame);
}

Re: ESP32 Based Web Interface & Data Logger

Posted: Thu Dec 08, 2022 5:43 pm
by Pete9008
Can't see of anything that would stop the above from working.

I'd be a bit suspicious about whether the Heater_pin is working/setup right or whether it is on a pin that conflicts with something else. Have you tried putting some debug terminal prints in to confirm whether its state is being correctly read?

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Dec 09, 2022 7:36 am
by arber333
Pete9008 wrote: Thu Dec 08, 2022 5:43 pm Can't see of anything that would stop the above from working.

I'd be a bit suspicious about whether the Heater_pin is working/setup right or whether it is on a pin that conflicts with something else. Have you tried putting some debug terminal prints in to confirm whether its state is being correctly read?
I tested the pin by a digital test. Turning on a LED on command. And it works. Also i added a buzzer to the function and it works as long as it is part of the command within conditional. If i put it inside a separate function however i cant get that function to change on command.
I would like to know if that is esp32 related or maybe my fault.
Did anyone sucessfuly tested CAN function in such a way?

Re: ESP32 Based Web Interface & Data Logger

Posted: Fri Dec 09, 2022 11:08 am
by Pete9008
The Arduino implementation of some peripherals did seem a bit flaky for the ESP32. I ended up having to use direct calls to the espressif api to get the UART and SD card to work properly (see https://docs.espressif.com/projects/esp ... index.html for more details on the api).

One other thought, have you tried dividing the problem? Move the IO read into the main loop updating a global variable and then use that variable in the CAN send function (same as you do with Heatertemp).

Re: ESP32 Based Web Interface & Data Logger

Posted: Sat Feb 11, 2023 7:51 pm
by 4GSHARK
Does any one have the Sine firmware and bootloader, that works with esp32 web interface hex. file to share. SMD Update dont work so i think i need hex file only, I saw there is a modified firmware that was posted but it wasnt compiled. im not very CMD knowledgeable to make one.

I have 5.20 Sine installed and I can not see parameters at all in web parameters section. There are no values.

I have v6 LDU board with ESP32-Whoom Module.

Re: ESP32 Based Web Interface & Data Logger

Posted: Sun Feb 12, 2023 9:16 am
by Pete9008
I never tested the ESP32 code on my github for firmware updates and I seem to remember that when Johannes tried it he had to make a few changes (detailed above somewhere), they never got pulled due to github forking issues. @johu - if you can post the changes somewhere I'll update the repository?

If you can't see the parameters you also have a comms problem too.

If you want a simpler option the ESP8266 is currently a more stable/mature solution (the ESP32 option is still being developed).

Re: ESP32 Based Web Interface & Data Logger

Posted: Sun Feb 12, 2023 9:43 am
by johu
Just realized I never forked your repo, will clean that all up later (or you give me write access, will push to another branch).
For my scenario I changed the inverter UART to UART0.
Most of all I bugfixed the update process:

Code: Select all

-  int pages = (file.size() + PAGE_SIZE_BYTES - 1) / PAGE_SIZE_BYTES;
+  uint8_t pages = (file.size() + PAGE_SIZE_BYTES - 1) / PAGE_SIZE_BYTES;

-    snprintf(uartMessBuff, UART_MESSBUF_SIZE, "%d", pages);
-    uart_write_bytes(INVERTER_PORT, uartMessBuff, strnlen(uartMessBuff, UART_MESSBUF_SIZE));
+    uart_write_bytes(INVERTER_PORT, &pages, 1);
That was it, it all works then. I'm already shipping that on the Tesla SDU and Nissan Leaf kit.

Re: ESP32 Based Web Interface & Data Logger

Posted: Sun Feb 12, 2023 9:50 am
by Pete9008
johu wrote: Sun Feb 12, 2023 9:43 am Just realized I never forked your repo, will clean that all up later (or you give me write access, will push to another branch).
Whatever is easiest (although you may have to explain how yo do it - I'm still a gitub novice). Might be neater to just add it as a new repository or branch on your github so everything is in one place?

Re: ESP32 Based Web Interface & Data Logger

Posted: Sun Feb 12, 2023 12:44 pm
by Pete9008
In the meantime I've added Johannes changes (hopefully all of them but untested) to a branch - https://github.com/Pete9008/esp32-web-i ... /tree/johu

Hopefully this will work with ESP32 modules that are integrated onto boards.

Re: ESP32 Based Web Interface & Data Logger

Posted: Tue Apr 04, 2023 12:20 pm
by Bigpie
Board arrived today Pete, thanks. Need to get my head around what's changes are needed in the inverter firmware to get this working again with the latest release code.

Re: ESP32 Based Web Interface & Data Logger

Posted: Tue Apr 04, 2023 12:29 pm
by mario
My branch is https://github.com/mmadzar/GN02475inv and inverter firmware update code wasn't changed, just added some custom code that no one else needs. :)
I can confirm that firmware update code works. I uploaded v5.26.Y firmware last night without any problems. Just uploaded bin file and that was it.

Re: ESP32 Based Web Interface & Data Logger

Posted: Tue Apr 04, 2023 6:18 pm
by Pete9008
Bigpie wrote: Tue Apr 04, 2023 12:20 pm Board arrived today Pete, thanks. Need to get my head around what's changes are needed in the inverter firmware to get this working again with the latest release code.
You and me both!

Thinking about it my board was built up largely of bits and pieces I already had so the parts list might not be quite as good as it could be. Let me know if anything doesn't make sense.
mario wrote: Tue Apr 04, 2023 12:29 pm My branch is https://github.com/mmadzar/GN02475inv and inverter firmware update code wasn't changed, just added some custom code that no one else needs. :)
I can confirm that firmware update code works. I uploaded v5.26.Y firmware last night without any problems. Just uploaded bin file and that was it.
Thanks for posting, I'd noticed your fork and wondered whether it was someone on here :)

Would you be able to give a bit more detail on what your additions are as they could well be useful to someone (I'm curious about the MQTT stuff in particular)?

Re: ESP32 Based Web Interface & Data Logger

Posted: Tue Apr 04, 2023 6:31 pm
by Bigpie
I've been having a bit of a read, the code on Johannes GitHub doesn't contain your binary logging changes, but does contain binstream that works differently. As I understand, the binstream that's in released code doesn't send the values each loop, so not ideal.
I've not braved a look at the BOM yet, but I guess adding your flavour of binary logging would be where I go next.

Re: ESP32 Based Web Interface & Data Logger

Posted: Tue Apr 04, 2023 6:39 pm
by Pete9008
Not sure, the ideal would be to get Johanne's version working but need to have another look at the code to see what would be needed.

In terms of seeing what's happening with your current sensors either approach should work but I'd prefer to have loop by loop data if possible.

Been struggling a bit with getting stuff done recently but hopefully back on it soon.

Re: ESP32 Based Web Interface & Data Logger

Posted: Wed Apr 05, 2023 8:29 am
by johu
Yeah binstream sends out items in 4-byte packets as fast as the serial allows. So depending on the number of items it can be slower. Most of all I don't think it's implemented in Petes ESP code.

Will get back to that and implement Petes more condensed protocol