ESP32 Based Web Interface & Data Logger

[Pete9008]

I've been thinking about how to do high speed logging of inverter data for a while now using a STM32 and a SD card. In a conversation over here viewtopic.php?p=45951#p45951 Bigpie suggested using an ESP32. Initially I wasn't keen but having looked at the ESP32 specs, and had a think about it, I think he's spot on with his suggestion

Now the standard web interface does plotting, and it does it very well, but while trying to diagnose intermittent problems it has a couple of limitations. First the user has to be connected and plotting data to capture it - not ideal for intermittent problems. Second it has limited time resolution, for example if grabbing 1000points it manages 0.6sec resolution, at 100points it gives 0.3sec resolution. This doesn't sound too bad until you realise that in this 300ms the motor control loop has run 2650 times. We do get the min, max and average for each sample but even those only capture 3% of the control loop runs. Now for analysing vehicle behaviour this is more than enough but when trying to figure out why a control loop is misbehaving, or why an inverter is tripping out, it leaves a little to be desired.

To try and address this the idea is to produce a version of the web interface that:

• Runs on the ESP32
• Provides as much of the functionality of the current ESP8266 based code as possible
• When there are no connections to the web page it instead enters a fast logging mode and writes the data to an attached SD card
• In the fast logging mode it logs all the essential data on every run of the control loop (estimated to be around 200kB/sec, or 12MB/min)
• Allows files from the SD card to be browsed and downloaded via the web interface (or ftp?, ssh?)

For anyone not familiar the current ESP8266 is a single core device that shares one processor between the wifi/web stack and the user application, the ESP32 is a dual core device meaning that the user application can pretty much hog a core all the time. Each processor on the ESP32 is also around 3x faster than the one on the 8266 The trouble is the latest web interface doesn't run on it.

So the plan is to port the current web interface build to the ESP32, add support for the SD card and add a fast logging mode to the firmware. I'm going to do the C based bits and Bigpie is going to do the web page based bits (which I know nothing about) and then hopefully use the product to figure out why his inverter keeps tripping out.

[Pete9008]

The initial plan is to put something together based on dev boards:

That's an ESP32 board, a SD card breakout and a RTC module. The Real Time Clock is to allow time stamping of recorded files to make it easier to identify which one is which.

If it all works it will hopefully get put on a PCB, something like this with the ESP32 module on the top and the SD card and RTC on the underside so that it can be used in place of the current module if needed:

So, on to the questions:

There are two github repositories that I'm aware of, Johannes' latest and greatest https://github.com/jsphuebner/esp8266-web-interface and 0xBedz's ESP32 port https://github.com/Bedz01/esp32-web-interface-port which only supports PlatformIO builds and doesn't include the SWD based device programming functionality.

I'm inclined to start with Johannes as it has more recent changes, supports both Arduino and PlatforIO builds and also appears to use a more recent version of the graph plotting library and use 0xBedz's as a reference for the changes needed to move to the ESP32. If anyone has a better idea please let me know?

Johannes' build also supports both Arduino and PlatformIO based builds. Arduino is simpler to use but PlatformIO offers additional debug options if needed. Any advise on which to go with?

Finally is there anything else that we should be thinking about on this?

Edit - Another question. The logger will generate large amounts of data, far too much to process with a spreadsheet. There are a few open source data analysis packages which might do it - does anyone have any experience or recommendations here? The other option is a custom app? Final though is make it possible to import the logs into the simulator, could the simulator then be tuned to match the logs to improve the accuracy of future simulations? Are there any other options?

[Bigpie]

Platform.io all the way for me, much been dependency handling.

I had a thought on handling the data. If the log files are packaged up to be like a can log from SavvyCan we can just open them with it and use a DBC for decoding and graphing. Splitting the files in to smaller blocks probably makes sense along with flagging when a desat event is dectected.

I can handle the web server side of things if you want to use 0xBedz as a starting point. I've done a few ESP projects with web sever and sockets.

[Pete9008]

Platformio it is.

I like that idea re SavvyCan! Only slight concern is file size, think the wifi could be quite slow to download big files. If savvycan has a binary format then that would be a good way to go.

Just curious, why do you prefer 0xBedz as a starting point?

[mjc506]

I was going to suggest that it may be possible to at least graph stuff in the browser - doesn't need a masses amount of support from the server (ESP) but relying on javascript for graphing a few million points may be optimistic...

No concerns with wifi speed imho. I've a few ESP32s here for 'playing' with, they seem to manage ~20mbps over TCP (UDP is a little faster). Even if it takes a few minutes to download the logs from a run... so what? That's even quick enough to do basic 'live' logging if needed (this could probably be in-browser)

I'd be tempted to look file formats used by software oscilloscopes like Pulseview. That software seems to manage to display several millions of points quite easily, and can show several channels at once etc etc. Of course, it's aimed at analysing data rather than calculating based on analogue signals, but it can do it... It's GPL, so potentially customisable to suit OI purposes.

[Pete9008]

I was wondering about trying to graph a reduced data set in the browser (say every 1000th point or averaged, min, max), just enough to confirm whether the file was the one you wanted. It would be a nice feature to have but that's down to Bigpie. I wouldn't have a clue where to start with java graphing.

The files could get big, 360MB for a half hour run (if kept in binary, more than twice that if ascii). At 20Mbps or 2MBps that's 3minutes. Hmm. Ok that's not as bad as I thought after all

Pulseview or similar sound worth bearing in mind. It could be very big data sets though. It might be a case of trying a few things on a dummy data set to see what works.

It will normally only be a small fraction of a file that is of real interest so something that provides an overview good enough to allow that section to be identified and pulled out to then be analysed with something else might be another way to go.

Edit - how about live streaming, a TCP or UDP port that an app could connect to to get the full live data stream. Not quite sure what you would use it for though??

Edit2 - is that 20Mbps sustained or burst? I've read a couple of posts mentioning a high initial data rate that then drops with time.

[Pete9008]

I'm stuck with the ESP32 already

Visual Studio Code and the Platformio plugin is installed and has loaded up the project ESP32 version of the web interface fine. It builds and uploads the app and the file system OK. Health LED on the dev board is flashing (now I've changed the pin to match the hardware) but I can't get the web page up.

The access point is created correctly and I can connect to it but browsing to 192.168.4.1 initially gave a could not connect error. Changing the local address in the ESP32 code from 1.1.1.1 to 192.168.4.1 (same as gateway address) changes this to FileNotFound. Now the SPIFFS appears to build, uploads and, according to the platformio monitor is mounted correctly. The root seems to be set to the index.html file correctly but I can't seem to get any further.

Any suggestions?

In the meantime I'm going to connect up an ESP8266 based board and put Johannes' version on to see if that will work.

Edit - Still stuck.
The ESP8266 works fine as long as I use the 'Upload Filesystem Image OTA' rather than 'Upload Filesystem Image' (later refuses to connect to the serial port??)
Trying the same with the ESP32 gives the same result in the browser as before (FileNotFound but now also get '[E][WebServer.cpp:647] _handleRequest(): request handler not found' in the serial monitor.

Edit2 - Giving up for now but when I get back to it going to try porting the ESP8266 code to the ESP32 and see what breaks.

[johu]

Holiday is almost over so I might as well chime in here

So I wouldn't mind taking care of the inverter side of things. Right now all we have is the stream command, a little known fact is it can stream indefinitely until it receives an interrupt. Its shortcoming is the ASCII format which kind of bloats data and also record size depends on the data (30000.00 takes longer to send then 1.00).

An improvement could be a "binstream" command that streams out 32-bit signed ints (wouldn't wanna go 16 bit as things like ud/uq exceed it). Also data rate can be upped further, that is down to experiments. Maybe 10 MBit/s.
I'd also add a "BinSend()" function to the terminal that takes a binary blob as is and sends it without waiting for a terminating char.

Also the new command should trigger, e.g. on the first element. The first element could be "angle" then data is only streamed out when angle changes resulting in data being sent only when the motor is actually moving.

Also a wrapping cycle counter as a regular spot value could be helpful.

[mjc506]

A binary stream would be a fantastic addition! Agree on the 32bit, any point in being able to select which variables to stream, or just always send everything?

Yes, datarate is more likely to be limited by the serial connection (and SD card maybe? not sure how good the ESPs are with that). I've just done a test here, sustained approx 20mbps over wifi in each direction (just dumping the bits at the receiving end, no processing) but that does drop a bit with signal strength. But like you say, even 5-10min transfer time for a 30min 'run' isn't terrible, especially as the majority of logging will be pretty short (of course there's always the long-term stuff for intermittent problems, but even then...) Depending how quick the ESP can close/open files, can always split 'long' logs into shorter sections and timestamp them.

[Pete9008]

Holiday is almost over so I might as well chime in here

Welcome back, hope you had/are having a great holiday

So I wouldn't mind taking care of the inverter side of things. Right now all we have is the stream command, a little known fact is it can stream indefinitely until it receives an interrupt. Its shortcoming is the ASCII format which kind of bloats data and also record size depends on the data (30000.00 takes longer to send then 1.00).

An improvement could be a "binstream" command that streams out 32-bit signed ints (wouldn't wanna go 16 bit as things like ud/uq exceed it). Also data rate can be upped further, that is down to experiments. Maybe 10 MBit/s.
I'd also add a "BinSend()" function to the terminal that takes a binary blob as is and sends it without waiting for a terminating char.

The BinSend approach is exactly what I was thinking, allows a reasonable degree of flexibility (can be used to do any size atomic variable or blocks containing multiple mixed data sizes to maximise packing). For example it would make sense to capture the adc results as 12-bit, desat as 1-bit, and so on. One blob would be sent at the end of each run of the control loop.

10Mbps would be very nice but my experience is that I'd only expect to achieve it reliably on a PCB, once any kind of cabling gets involved 2Mbps is probably more realistic. Question is whether to allow it to be adjusted or go with a fixed safe(ish) value?

Also the new command should trigger, e.g. on the first element. The first element could be "angle" then data is only streamed out when angle changes resulting in data being sent only when the motor is actually moving.

Not sure on that, even if the angle doesn't change it could still be useful to see the data, for example if tuning PI loops with a locked rotor or seeing how integrals or currents decay once the rotor has stopped. Maybe a feature to add once the basic logging is working?

Also a wrapping cycle counter as a regular spot value could be helpful.

That's a good idea, maybe combine it with a crc or checksum?

Final question, any thoughts on your build vs the existing ESP32 version as a starting point?

[Pete9008]

A binary stream would be a fantastic addition! Agree on the 32bit, any point in being able to select which variables to stream, or just always send everything?

Yes, datarate is more likely to be limited by the serial connection (and SD card maybe? not sure how good the ESPs are with that). I've just done a test here, sustained approx 20mbps over wifi in each direction (just dumping the bits at the receiving end, no processing) but that does drop a bit with signal strength. But like you say, even 5-10min transfer time for a 30min 'run' isn't terrible, especially as the majority of logging will be pretty short (of course there's always the long-term stuff for intermittent problems, but even then...) Depending how quick the ESP can close/open files, can always split 'long' logs into shorter sections and timestamp them.

Hoping to use SDIO mode to improve SD card speed. SPI mode data that I've seen is of the order of 200kBps which is a bit marginal.

A reliable 20Mbps would be good!

I'd prefer to always stream everything.

Edit - main problem I've had with SD cards in the past is the spec allows them to pause for up to 100ms (cards vary, some are much better than others in this) while doing background tasks so some degree of buffering is needed. That same buffering would allow for the opening and closing

[Pete9008]

Quick update.

Gave up on the ESP32 port mentioned above, not sure what I was missing but couldn't get it to work.

Instead I've started from Johannes' latest ESP8266 version and ported it across to the ESP32. Seems to build without errors, downloads to the board and now brings up the full web page. Haven't tested it with an OpenInverter board yet but I'm going to quit while I'm ahead and leave that for another day!

@Bigpie - let me know if you want a copy now or whether you are OK to wail till the SD card stuff is in there?

Edit - Connects to the inverter and pulls out the parameters, plots graphs and gauges correctly Not tried the firmware updates yet but really am going to quit while I'm ahead now!

Edit2 - Now it's working I realise why it took me a while to get the inverter moved to UART2. I was assuming that when I got it moved correctly all activity on UART0 would stop, but it didn't. On ESP reset the following debug data is still written out on UART0:

Code: Select all

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1344
load:0x40078000,len:13836
load:0x40080400,len:3608
entry 0x400805f0
[   298][E][WiFiSTA.cpp:317] begin(): connect failed! 0x300a
[   314][E][WiFiClient.cpp:335] setOption(): fail on fd -1, errno: 9, "Bad file number"

and then when opening the gauges window (which appears to work fine) this:

Code: Select all

[226041][E][WebServer.cpp:647] _handleRequest(): request handler not found
[226400][E][WebServer.cpp:647] _handleRequest(): request handler not found
[226488][E][WebServer.cpp:647] _handleRequest(): request handler not found
[226743][E][WebServer.cpp:647] _handleRequest(): request handler not found

No idea whether this is common to the ESP8266 too but thought it worth mentioning.

Edit3 - Got to say I'm well impressed by the web interface, hadn't realised how much it could do!

[Bigpie]

I'm easy, if you want to stick it up on github I'll fork and play around, can wait till you're ready though

[Pete9008]

I'm easy, if you want to stick it up on github I'll fork and play around, can wait till you're ready though

I suppose what I should do is fork Johannes and apply the changes I've made to that, just not too sure how to go about doing it. Do you fork it on the github web site, then clone that down to a local copy or is it all done from the command line?

[Pete9008]

SD card in SDIO mode is now working with the ESP32. Still in a test app rather than the web interface to make performance testing easier.

The results so far are:
100MB binary file write - 40seconds.
100MB binary file write with flush every 1024bytes - 240seconds.
100MB binary file write with flush every 1MB - 40seconds,

So the write speed is around 2.5MBps, more than enough for what we need. Probably worth testing at some point with a cheap no brand card to see what that gets but don't have one handy.

Without the flush the file can be corrupted if the power is removed unexpectedly (which is fairly likely in our application). Adding a flush on every 1024byte write slows it down far too much (and probably doesn't do the card life much good either). A compromise of a flush every 1MB works well. The nice way to do it would be to detect the power going down and close the file but I can't see a way of doing that.

Going to get the RTC working next and then get them integrated into the web app.

Edit - Not tested read speed but it's likely to be a bit faster and since the write speed is already quicker than the WiFi it's not going to make a much difference anyway.

[mjc506]

Not bad

I think there's a way of disabling brownout detection, and then monitoring '5V' with the ADC, but that'd be rather unreliable... Although there might be enough bulk capacitance bulk capacitance on the 12V line to catch something. Will probably depend too much on the individual system and loads. A proper battery backup seems excessive probably not critical anyway!

[Pete9008]

Question for anyone familiar with ESP32 code.

The RTC is working and I'm trying to hook it up so that it is used by the filesystem for the created/modified file time stamps. Now I've done this before using FatFS on Arduino sketches but the ESP32 SDIO mode uses vfs_fat_sdmmc instead and I can't find any documentation on how to do it on this.

Any ideas?

Edit - Please ignore the above. You need to use the external battery backed RTC to initialise the on chip RTC and the filesystem then uses that.

[Pete9008]

Test platform now working. External RTC is used to initialise the on chip clock on boot and that is then used for timestamping (actually a better solution as accessing the external rtc would have slowed down the file writes).

Also noticed while testing that the card write speed seemed to be slowing down as it filled up. It was still achieving 600kBps but should have been better. Bit of digging suggested that a larger block write would improve things so upped write buffer from 1k to 16k. It's now managing 5MBps

Need to tidy up the test app and use it to test a few other bits and pieces like file naming but once that's done it can be pushed into the web interface.

Not too sure what to do about clearing space on the card. Essentially if it fills up there will be continual interruptions to the logging as it stops to delete old files. At the moment thinking the simplest way is to clear say 2GB worth of space on the card at boot up (by deleting the oldest files, should give a few hours worth of logging) which is relatively easy to do and should cover most scenarios.

[mjc506]

Nice

Nothing wrong with relying on the user to keep an eye on free space etc (especially if there'll be a 'start logging' button)

[Pete9008]

Trouble is the plan is that if there are no connections to the web server it will start logging in the background so that you should always have a record of unexpected events.

Experience is that you have never remembered to press the button when something interesting happens

Wonder how long a card will last, probably find it will have worn out by the time you go looking for something!

[mjc506]

Ah, of course. Good point!

Dashcam cards seem to last ok. I wonder how hard it would be to get the ESP talking SATA... hahah

[Pete9008]

Ah, of course. Good point!

Dashcam cards seem to last ok. I wonder how hard it would be to get the ESP talking SATA... hahah

Only if you're offering to do it!

[mjc506]

I suppose if you stream data to a laptop, that's technically there, right? hahaha

[Pete9008]

I've just done a test here, sustained approx 20mbps over wifi in each direction (just dumping the bits at the receiving end, no processing) but that does drop a bit with signal strength.

Would you be willing to share the block of esp32 code you used for this? Sure I can work something out if needed but starting from known working code is a lot quicker, especially without a debugger.

Out of interest have you tried the esp-prog debugger, if so is it any good?

[mjc506]

I don't think I even saved the script (arduino, sorry) but something like this:

Code: Select all

const uint ServerPort = 8888;
WiFiServer Server(ServerPort);
WiFiClient RemoteClient;
 
void setup()
{
  Server.begin();
}

void loop()
{
  uint8_t ReceiveBuffer[30];
  while (RemoteClient.connected() && RemoteClient.available())
  {
    int Received = RemoteClient.read(ReceiveBuffer, sizeof(ReceiveBuffer)); //receive data
    //RemoteClient.write(ReceiveBuffer, Received); //send data
  }
}

Then just

Code: Select all

time dd if=/dev/zero bs=9000 count=1000 > /dev/tcp/$ESP32/$8888

for various amounts of data and just calculating that way.

I could say I've given esp-prog much of a chance, I briefly had a look at it while I was trying to get my head around the stm32 and gave up quite quickly!