ESP32 Based Web Interface & Data Logger

[mjc506]

I guess with johu's binstream command, you'll be specifying what you want logged? And yours is fixed so you know what's coming?

Oh, sorry. Yes, a full param dump to go with each log would be handy. Can't see why the json wouldn't be good enough? Doesn't have to be massively fast, only has to be done once per power cycle(?)

[Pete9008]

Agree, think Ill add it next time im revising the code.

I haven't put the tcp streaming in yet as I've struggled to figure out how to get it to work with the web interface. Is it used in place of the web interface, does every response from the inverter (whether due to the tcp connection or the web interface) get routed to it, or does the code need to keep track of who sent the command and send the response there? Or is it just for binary streaming?

Really need to figure out the use case for it first!

[Pete9008]

Still need to get the sdcard module to play with but I'm up and running for the most part. What is it you're wanting adding to the web interface? Listing of the sdcard files and downloading? Anything else?

One other thought on this re decoding the binary data.

Would it be possible to convert the binary file to a csv file in the client browser using some java or similar? No idea if this is even possible but if it was it would allow a quick download of the binary file but then have it decoded to a csv (or Savvycan format?) before it is saved to disk without needing any extra conversion utility.

[Bigpie]

I've ordered the same mircoSD card reader you've got, should be here tomorrow. An option could be to have the webserver return a binary blob and use javascript to generate the CSV.

Using websockets would be an option to have the webserver send data to the client.

[Pete9008]

I've ordered the same mircoSD card reader you've got, should be here tomorrow. An option could be to have the webserver return a binary blob and use javascript to generate the CSV.

Using websockets would be an option to have the webserver send data to the client.

Afraid you've lost me there but it sounds like something might be possible?

Edit - also realised that the way I've streamed the json header in the stm code isn't very clever and could briefly disrupt the control loop. Shouldn't be an issue for testing but I need to sort it before it's used properly.

[Pete9008]

Is there any equivalent of the sleep or delay functions that can be used in the OpenInverter firmware, specifically within the terminal task?

[johu]

Why do you need a delay?

Did some philosophical reasoning on your approach and I do see a point. The programmers would know best what they want dumped and how many bits to reserve for it. I think I'd convert it to a bit field. Using the SOMETHING_LIST trick I'd generate the json header and the bit field declaration from the same data source.

I'd implement this as a regular terminal command which first dumps the bitfield-json (as opposed to the parameters json) and then calls something like PwmGeneration::StartDump() or so.

On the ESP side you would issue

Code: Select all

fastuart 2 //switch to 2.25 mbps
json //get parameters json
binlog run //dump bitfield json and start the binary log
binlog stop //stop binary log
fastuart 1 //if your prefer 1 mbps

[Pete9008]

Glad you agree

Have you got a link for the SOMETHING_LIST trick, it's not something I've heard of?

Not sure about using a bitfield, the trouble is is still enforces the compiler byte alignment and so won't achieve the same packing density (unless there is another trick I'm missing?).

For now I'm going to stick with the binlogging command as it is just one command to send which then takes care of everything. If it needs changing later it can be and for now I'd rather focus on the bits that get to a working system. I'm sure that once it's used in anger for the first time there will be plenty of other bits that need changing too!

Hoping to spend a bit of time on it later today to tidy things up, add the json parameter send and convert it to use your parameter store idea for passing the data.

[Pete9008]

Revised version of both the stm and esp code now on github which include all the bug fixes and changes mentioned in the previous posts.

[Pete9008]

I'm wondering if we need two logging schemes.

The first, the one that Ive been working on, is intended for finding intermittent bugs in the core control code but isn't that usefull for anything else due to the limited flexibility and number of messages.

The second is Johannes configurable approach that can be tailored to whatever the user wants but isn't quick enough to dump the full control loop data.

The web page would then contain an option to allow the user to select which one starts when the web page is closed. This would give the best of both worlds, if there are control problems, or control loop changes have been made, you enable the control loop logging, for anything else you set up your own (or have a pre-configured default?) configurable logging.

[johu]

Have you got a link for the SOMETHING_LIST trick, it's not something I've heard of?

Yes, check for example the Param class. It uses PARAM_LIST to create various arrays and enums.

[johu]

Did an experiment

Code: Select all

struct bla
{
   uint32_t counter:8;
   uint32_t angle:16;
   int32_t idc:11;
   int32_t il1:11;
   int32_t il2:11;
   uint32_t pwm1:10;
   uint32_t pwm2:10;
   uint32_t pwm3:10;
   uint32_t opmode:2;
   uint32_t desat:1;
   int32_t iqref:11;
   int32_t idref:11;
   uint32_t ifw:10;
   uint32_t uq:17;
   uint32_t ud:17;
   uint32_t crc:8;
} __attribute__((packed));

This, as expected, is 21 bytes and would hold all values as integers, currents up to 1024A, PWM divided down to 10 bits.
Removing the packed attribute of course bloats it to 28 bytes.
CRC could just be some byte from the CRC32 that is built in.

Regarding software structure (not important for testing now!) the code doesn't belong in terminal.cpp because how would you now use it in a non-FOC project? Check out "Single responsibility principle". So I think assembling the data in pwmgeneration and just passing the structure as a uint8_t* to a generic BinarySend() function will retain said principle. Don't worry about it for now, I can fix it once everything works, just wanted to give some insight on what I think is important to keep software maintainable

[Pete9008]

Did an experiment

Code: Select all

struct bla
{
   uint32_t counter:8;
   uint32_t angle:16;
   int32_t idc:11;
   int32_t il1:11;
   int32_t il2:11;
   uint32_t pwm1:10;
   uint32_t pwm2:10;
   uint32_t pwm3:10;
   uint32_t opmode:2;
   uint32_t desat:1;
   int32_t iqref:11;
   int32_t idref:11;
   uint32_t ifw:10;
   uint32_t uq:17;
   uint32_t ud:17;
   uint32_t crc:8;
} __attribute__((packed));

This, as expected, is 21 bytes and would hold all values as integers, currents up to 1024A, PWM divided down to 10 bits.
Removing the packed attribute of course bloats it to 28 bytes.
CRC could just be some byte from the CRC32 that is built in.

I stand corrected, that's a much nicer way of doing it

I was sure I'd tried that before and it didn't work. It would have been ages ago with a different compiler though.

Regarding software structure (not important for testing now!) the code doesn't belong in terminal.cpp because how would you now use it in a non-FOC project? Check out "Single responsibility principle". So I think assembling the data in pwmgeneration and just passing the structure as a uint8_t* to a generic BinarySend() function will retain said principle. Don't worry about it for now, I can fix it once everything works, just wanted to give some insight on what I think is important to keep software maintainable

That's more the way I'd originally planned to do it so suits me fine. I'd somehow got the impression that you wanted to keep it out of the control loop and pull the data out of the parameter store so changed it over (or was that just for the configurable parameter streaming approach?).

Edit - might tweak a couple of those bit widths though. We have the space so it would be nice to have a little more resolution and range in places.

[johu]

I found JLCPCB stocks an ESP32 (ESP32-WROOM-32E) module that they can solder directly to the board for about $3 They also have an ESP8266 module but that is out of stock. So I think I will make a variant of the Nissan Leaf adapter board that takes this and as I understood the web interface runs pretty much as is on the ESP32.

If that goes well I will put it on the other boards also. Not sure about adding an SD card slot or just use the streaming variant - with the clear downside that it requires a receiver to be running. But could be just another ESP32 that's listening?

[Yahha777]

I found JLCPCB stocks an ESP32 (ESP32-WROOM-32E) module that they can solder directly to the board for about $3 They also have an ESP8266 module but that is out of stock. So I think I will make a variant of the Nissan Leaf adapter board that takes this and as I understood the web interface runs pretty much as is on the ESP32.

If that goes well I will put it on the other boards also. Not sure about adding an SD card slot or just use the streaming variant - with the clear downside that it requires a receiver to be running. But could be just another ESP32 that's listening?

Yes, that's a good idea! In terms of hardware, I solved it. Now it would be even better to run both UARTs for two controllers

just came from JLC. While there is no time to fill in the components.

[johu]

Very nice! Yes those dual usart would be ideal for this board.
Did you do it in KiCAD? Is it on github?

[Yahha777]

Very nice! Yes those dual usart would be ideal for this board.
Did you do it in KiCAD? Is it on github?

Yes - it's made in KICAD. This is my second attempt to create a board, the first had some errors. Not on GitHub. I can share.

[Bigpie]

Make sure you get an SDIO mode one, not SPI, and preferably one with pull up resistors on it. There is supposed to be an issue with SDIO on some ESP32 modules, if they have 3v3 internal flash an eeprom fuse may need setting although I couldn't find a definitive list of which modules are affected. Best thing to do is try it and if it works leave it, if it doesn't then look at the fuse. Mine was a WROOM-32E module and it was fine without anything doing.

Finally getting round to looking at this, having difficulty working out the pins for SDIO. Don't suppose you have a diagram? I've got the same SDCARD and RTC modules that you did.

[Pete9008]

Afraid I couldn't find a diagram so used this as the reference, slot2 https://docs.espressif.com/projects/esp ... slave.html.

Think I also put the pinouts for the sdio and rtc in the github readme.

[Bigpie]

Doh, should learn to read those

Code: Select all

CLK to pin2, CMD to pin15, D0 to Pin2

Are both CLK and D0 to Pin 2?

[Pete9008]

Doh x2!

Clk is on pin 14

[Bigpie]

Added setting the RTC from the web interface.
Will look at the SD Card downloading next.

[Pete9008]

Looking good

Let me know if there is anything I can do to help?

[Bigpie]

Do you have any log files? I dont have it hooked up to an stm32 at the minute. A couple of demo log files and description of the data formatting in there would be useful for the task of converting the binary data to csv

[Pete9008]

Not handy but can generate one later. It will only be dummy data though as my stm32 isn't hooked up to any sensors.

It might be best leaving any file translation for now, as it seems fairly likely to change when we start testing, and just save as a bin file.

The bin files actually contain a json description of the binary format at the start of the file (along with a json parameter dump from the inverter firmware). The idea is that eventually it will be possible to automatically parse them rather than needing to know what format they were saved in.

Edit - File attached. Only a few seconds as wasn't sure what the file attachment limit is but shows the format. Parameter json dump is first followed by json field description for file followed by the binary data. All the fields are static except for the angle which increments.

2022-09-07-04-01-38_0.bin

(1.77 MiB) Downloaded 39 times