ESP32 Based Web Interface & Data Logger

Discussion about various user interfaces such as web interface, displays and apps
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Thank you!

It always surprises me just how simple working code can be and yet how you can go through so many versions of non-working code before you get to it!

No problem with Arduino. VS Code and PlatformIO refuse to create new projects at the moment (something to do with admin rights?) so have been using Arduino for the test code.

Really must get to grips with the linux command line, I've spent far too long relying using windows. Any chance that you have another line of Linux black magic to receive tcp the data?

Had a look at the esp-prog debugger but looks like the JTAG lines it needs on the ESP32 are shared with the SDIO SD card interface so no use for this project.

With your code I now have all the code functions needed so can start integrating it into the web app :)

Edit - figured out the command line syntax but why use > instead of of= on the dd?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Data Logger

Post by Bigpie »

Excellent progress. I'm finishing tiling bathroom floor today hopefully so can take a proper look tomorrow if you get chance to github it.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Bigpie wrote: Sat Oct 01, 2022 11:46 am Excellent progress. I'm finishing tiling bathroom floor today hopefully so can take a proper look tomorrow if you get chance to github it.
Strange you should say the, it's exactly what I 'should' be doing :)

Hopefully have something that is mostly there up on github later today.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Not too sure that this is going to work the way we were hoping. Problems:
  1. There seems to be something funny with the comms and the two seem to lose sync when switching between baud rates.
  2. The STM32F103 can only run at 2250000baud on USART3 (not ideal but not too bad).
  3. And the big one - when combined with the web interface, and with serial data coming from the serial port, the ESP32 doesn't seem to be able to keep up with the data rates. Not sure where the bottleneck is at the moment but the max data rate seems be half to a quarter of what is needed.
Giving up for now. Going to have to go back to the test app and build it up to see when it breaks.

BTW - I have forked the source for the web interface and foc code in github but currently it is all the standard code as I haven't worked out how to push changes back up yet.
User avatar
mjc506
Posts: 343
Joined: Wed Sep 09, 2020 9:36 pm
Location: Wales, United Kingdom
Has thanked: 30 times
Been thanked: 28 times

Re: ESP32 Based Data Logger

Post by mjc506 »

Pete9008 wrote: Sat Oct 01, 2022 11:02 am Really must get to grips with the linux command line, I've spent far too long relying using windows. Any chance that you have another line of Linux black magic to receive tcp the data?
I used netcat to receive, but then did some bitrate calculating on the ESP side. I did initially use the network interface counters on the laptop side, but that includes overhead. I think netcat should be able to spit out numbers too, but gave up with the manpages.
Pete9008 wrote: Sat Oct 01, 2022 11:02 am Had a look at the esp-prog debugger but looks like the JTAG lines it needs on the ESP32 are shared with the SDIO SD card interface so no use for this project.
Ah, that's a shame. I recall that it's possible to build with GDB hooks, but that probably changes enough things in the code to make bugs disappear :) Very easy, however, to do some 'printf style' debugging - set up a separate tcp server on the ESP (different port) and just point netcat at it to receive messages in realish time.
Pete9008 wrote: Sat Oct 01, 2022 11:02 am Edit - figured out the command line syntax but why use > instead of of= on the dd?
Not an expert, but I think it makes dd stream the bytes rather then try and write in blocks (like it would to a real file)
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

mjc506 wrote: Sun Oct 02, 2022 10:17 am I used netcat to receive, but then did some bitrate calculating on the ESP side. I did initially use the network interface counters on the laptop side, but that includes overhead. I think netcat should be able to spit out numbers too, but gave up with the manpages.
Thanks, will give that a try.
mjc506 wrote: Sun Oct 02, 2022 10:17 am Ah, that's a shame. I recall that it's possible to build with GDB hooks, but that probably changes enough things in the code to make bugs disappear :) Very easy, however, to do some 'printf style' debugging - set up a separate tcp server on the ESP (different port) and just point netcat at it to receive messages in realish time.
Learnt my embedded coding in the days when printf was the the accepted way to do things, trouble is it's slow and can effect execution time too. When that mattered what I used to do was just send a single byte using a direct write to the Tx register, only took a single instruction cycle and if carefull with timings it could provide a pretty good idea of program flow without printf's impact on execution time. I've got lazy these days though and I find it really frustrating writing code without a debugger!
mjc506 wrote: Sun Oct 02, 2022 10:17 am Not an expert, but I think it makes dd stream the bytes rather then try and write in blocks (like it would to a real file)
That makes sense :)

On the ESP32 code I've realised I made a couple of fairly stupid mistakes yesterday. Looking at it again now I don't have a good ground connection for the serial connection - might explain some of the funny issues I've been seeing on the comms. Also had forgotten to increase the serial port default buffer size so it is highly likely that it's overflowing by quite a lot. Going to fix those now and see what it does.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Not great news.

Spent a bit more time on it and ported the comms code into my test app which makes debugging a bit easier. Essentially:
  • With just the SDIO I get 5000kBps write speed.
  • If I also have the inverter USART enabled and receiving data, even if I don't actually process the data, this drops to 500kbps
  • If I process the data and also have the bare minimum of the WiFi stack running this drops to 50kBps
Not 100% sure but based on what I've seen so far it looks like the ESP32 doesn't use DMA on the USART. If so this would explain the symptoms. I can find a few posts online looking for/asking if there is a DMA based implementation but so far haven't been able to find one.

Not too sure what to do now. Using the standard interrupt based libraries isn't going to be fast enough and writing a DMA based device driver is a big job. Will have another look for a DMA based USART driver but unless I can find an existing one it might be the end of this project :(

Edit - found the source for the ESP32 UART drivers and they don't support DMA, can't find any 3rd party DMA solutions either. Did a couple of tests without the SD card but just trying to receive serial data at sustained high rates and it tops out at similar levels. The only possibility I can see it to use a lower baud rate but 2250kbps was already marginal so don't think that's worth it.

Edit2 - Espressif do seem to document a number of low level uart access functions, may see if using those help. Can't understand why it seems to be so bad, there's no DMA but the uart does have a FIFO so it should be a lot better.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

It's working :D Well at least the test app is!

It's writing to the card and keeping up with the serial port at 2250kbps, no lost bytes and no corruption :) The trick seems to be to bin the Arduino HardwareSerial class and instead use Espressif IDF API calls. With those it just works. It also looks like it might have fixed a couple of the comms quirks that I was experiencing before (they may also have been there in the 8266 version as there are a few delays and extra linefeeds that may not be needed any more).

The IDF API is still interrupt based but it is using the Rx FIFO and also has a driver that incorporates a ring buffer making buffering very easy.

Just need to put a wrapper round the API calls to make it easier to use, integrate it onto the web interface and then strip out all the bodges that I've put in while trying to get it to work.

I remember why I don't get on with Arduino now!
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Data Logger

Post by johu »

Check out the latest, totally untested commit of https://github.com/jsphuebner/stm32-sine

EDIT: and great work on the ESP32 I should say :D

I have added the terminal command "binstream" that works the same as the normal "stream" command but streams data as 32-bit binary data. Can handle up to 20 items.

I have also implemented "fastuart 2" that switches to 2250 kbps

BTW sending changes to your github is a 2-stage process.

First of all you need to be working on a cloned copy from github (not some local copy and not a downloaded zip file). That means you need to have git installed, either command line or some GUI client like TortoiseGIT on windows.

Now you can make changes and you locally "commit" them.
Then you "push" your local commit to your github repository.

Unfortunately the latter step has been complicated as you need to have certificate authentication set up with github. Far too much to describe it here...
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Great work, is the plan to transition the web interface over to using the binary data for plotting?

Got the repos forked and cloned, hope to get the changes in and pushed up later today.

I've added a 'binlogging' command to the inverter firmware which takes an argument of 1 or 0 (on or off). It switches to 2250 and then starts streaming 24byte blocks (max possible at 2250kbps) of data (one on each run of the control loop). Just dummy data for testing at the moment.

The plan for today is to add new file creation at the start of logging (at the moment it just creates a single file on boot). All the time stamping and sequential naming stuff is in the already so should be fairly quick. At the moment if it finds a RTC it uses the time and date for filenames, if not it reverts to numbered sequential filenames. On boot it also checks SD card space and deletes the oldest files until 2GB is left (oldest determined from date if there is a RTC, otherwise filename number).

Logging start automatically when there are no connections to the ESP on WiFi and stops as soon as a WiFi connection is made.

The data received from the inverter is streamed straight to the card so not too easy to use at the moment. The plan to address this is to have the inverter send an ascii json header when logging starts which will define the packing of the binary blocks. This will get copied to the start of the card by the ESP firmware. The idea is that whatever uses the file can just parse the header to figure out how to process the rest of the file. This has the big advantage in that the header is created by the inverter and so it changes as the inverter build does and - so there should be no problems with compatibility between firmware builds and data analysis. This is the next bit to add in to the inverter code.

Once that's done, hopefully today, we should have a usable logger. It will be necessary to remove the SD card to get at the data but it's workable. At that point I'll pass it over to Bigpie to look at the web side. One thing I haven't done is add the web hooks for web access to the SD card file system. I think the best way is to map it as a virtual subdirectory on the root of the existing SPIFFS handler, that way the existing hooks can handle it. If I have time I will do it today, otherwise small test bin file/s can be placed in the SPIFFS root folder to allow web development in the meantime.

Not sure how to process the data yet. Options are either:
  1. Have the web app translate it to something else (say SavvyCan format and create a .dbc for it) while it is downloaded.
  2. Download as is and have a separate PC app which translates it to another format (SavvyCan, CSV, others?).
  3. Download as is and have a dedicated app for processing (may well do a very rough version of this for testing).
  4. Any other ideas?
Edit - Plan for the binary block is:
  • Incrementing message counter - 8bits = 1B
  • 4xADc vals at 12bits each (encoder cos, encoder sin, I1, I2) - 48bits = 6B
  • 3xpwm vals at 14bits each - 42bits (+2 flag bits on each for desat, opmode, etc)= 6B
  • 5xFP vals scaled to 16 bit each (iqref, idref, Ifw, Uq, Ud) = 10B
  • checksum at 8bits = 1B
Total of 24bytes.

Plan for synchronisation on binary data when decoding is to scan through till get a valid csum. If this and the next csum are both valid treat it as synced but keep checking. Compare message counter on each pass to check for missing blocks. It's not 100% robust but hopefully good enough? Would a CRC rather than a checksum be worth the increased complexity? Should we sacrifice a data value to allow a more robust scheme?

Not too worried about corruption on the card, more corruption on the serial link (shouldn't be an issue if the ESP32 is on the inverter board but some concern if it is on wires).

Edit - Changed bit mappings above
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Question for anyone familiar with git.

I'm just trying to push changes to github and have just realised that one of the changed files (terminal class) is not in the stm32_sine repo but instead in the libopeninv repo so I can't add, commit or push it.

What is the best way to resolve this?


Edit - Still having problems with the above (wondering whether I need to fork and clone the libopeninv into my local project instead of the one that is there, apply the changes and then push that?) but the ESP32 code is on github https://github.com/Pete9008/esp32-web-interface
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Data Logger

Post by Bigpie »

libopeninv is a submodule, pretty sure you need to fork it and add a new remote to your checkout https://git-scm.com/book/en/v2/Git-Tools-Submodules
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Bigpie wrote: Mon Oct 03, 2022 4:29 pm libopeninv is a submodule, pretty sure you need to fork it and add a new remote to your checkout https://git-scm.com/book/en/v2/Git-Tools-Submodules
Working on it :)

Having trouble getting git to forget the old submodules before it will let me add the replacement!
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Wow, that was a fight!

Johannes had made changes to the libopeninv library this morning which meant when I forked it I got a different version to the one I had been working on :(

After a LOT of faffing about I've got the whole libopeninv repo forked which gave me access to the old tag, which I could then branch, checkout, modify, commit and push before doing it all again with stm32-sine!

The firmware that matches the ESP logging code is here https://github.com/Pete9008/stm32-sine. It's based on Johannes 5.20 version with no mods other than those to enable fast logging. It is TOTALLY UNTESTED!
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

To summarise where this is up to the ESP32 code and the stm32-sine inverter code are both in a state where they should be useable for logging. Note that the stm32-sine code has only had minimal changes but HAS NOT BEEN TESTED on any real inverter hardware.

The STM32 code is here: https://github.com/Pete9008/stm32-sine It's based on the 5.20 release.
The ESP32 code is here: https://github.com/Pete9008/esp32-web-interface

The ESP code should be able to do most things that the ESP8266 code can although the firmware update functions are also totally untested.

The connections to the ESP32 module are detailed in the repository README.

The ESP32 will start logging to a card whenever there is no connection to the ESP WiFi and will continue logging till either power is removed or a connection occurs. The data written to the card is in binary format and there is not currently anything that can parse it (there will be soon).

The serial connection to the ESP32 runs at 2.25Mbps when it is logging so you need to be extra careful with connections if the module is off board, keep wires as short as possible and make sure they are well screened, to minimise potential for signal corruption.

If a I2C RTC chip is connected to the ESP32 module it will be used for timestamping files. If not the sequentially numbered filenames will be used.

Yet to be added:
  1. Some way to acess the logged files over the web page/WiFi (at the moment the SD card must be removed to read).
  2. Some way to set the time on the RTC if fitted.
  3. Some way to convert/analyse the binary logged files.
  4. Add tcp data streaming.
  5. Test everything properly!

Edit - @Bigpie - I haven't done any SD file system access function yet, or any RTC access functions. If you let me know what you need I'll have a look at it.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Data Logger

Post by Bigpie »

Nice work. I'll take a look in morning.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Data Logger

Post by johu »

Nice work as always :)
Just a bit confused about your changes to the STM firmware as now there would be two ways to stream out binary data. Yours is better in terms of the value count being streamed out each cycle. My implementation is less intrusive in the control code and more flexible but only manages to stream out 6 values each PWM cycle because each one occupies 32 bits.
Now I'm thinking of a way to combine the advantages of both methods. For example specifying something like

Code: Select all

binstream -1 il1:2,il2:2,ud:3,uq:3
So each item is assigned a size in bytes.
Doing it that way also allows the header generation to be shifted to the ESP code, would allow selecting the items to be logged in the web interface, and, sorry, would keep the control code implementation cleaner.

The checksum implementation is a good idea, I'd just use the hardware CRC generator of the STM32 that is already used when saving parameters or receiving the firmware update.
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

I'd already done most of this before you made your changes and couldn't face starting again so finished it off as was.

Not too bothered how its done as long as it allows the core data data to be sent locked to the control loop execution. My goal with this is to be able to see exactly what the control loop is doing in real time. The streaming approach is more a way of improving the data quantity using the current approach. I see the two approaches as complementary.

I agree about the untidy effect on the control loop though. Sure there is a neater way but just put it together fairly quickly to test the concept.

Hardware CRC would be a good idea too, had forgotten it was there.

Edit - not sure specifying it bytes would be enough, struggled to get everything in even when cutting things right down.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Not sure I've explained what I'm trying to achieve very well so thought it worth trying to clarify it.

The OpenInverter firmware is a fantastic piece of software but as far as I can tell (may well have missed something) there was no real way to fully test the core control loop functions and no way to identify exactly what the problem was when something unexpected happened in the car.

The simulator was done to provide a way of exercising the control loop while watching its behaviour to allow it to be put through various test cases, i.e. a way of exhaustively testing it. However that is not much use if you don't know the exact test case that needs creating to reproduce a real world problem.

The data logging is intended to allow the exact control loop behaviour, including the exact motor data that it is processing on each run, to be collected. This data can then be used to generate a test case for further debugging. For me the logging must therefore be integrated into the control loop so that it collects the exact data being used by it. For example the motor current 50us early or 50us late is of little use, the data logged needs to be the same data the control loop used and also match the logged resolver data to be of most value.

Does that make sense?

Edit - also to explain the reasoning, both Bigpie and Ev8 are still experiencing unexplained behaviour. Ev8's WiFi isn't good enough to capture data, Bigpie can capture data but the resolution isn't good enough to work out what's going on. I'd like whatever logging solution we come up with to be able to handle both these cases. For my own install I'd like something that can be fitted and forgotten about but that will log in the background just in case something happens.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

One other point worth noting, this only works with the foc code build. The build probably will fail for the sine version but even if it builds it won't do any logging. This will need sorting at some point.
User avatar
johu
Site Admin
Posts: 5682
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Data Logger

Post by johu »

Oh yes, the trigger point is, well, a good point ;)

That said the data in the Param-module also reflects the exact sample from the control loop as the control loop is responsible for writing these values. With "my" implementation I just have to make sure that the data is collected when the control loop exits rather than at some random time. Plus fitting it all into 24 bytes...

I'm also still having unwanted behaviour as I learned on our road trip. Field weakening is still brought in in a lump and it kind of feels like early day turbo cars that suddenly start boosting. Also full throttle yields less power than low throttle in the FW region. At least now the misbehaviour is not as catastrophic anymore (for me)
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

I like the idea of using the param module to collate all the data, that would neaten things up a lot :) It might need a few extra params adding (I would like the resolver Injected conversions to be included as I think that is a critical measurement that doesn't have much test coverage at the moment). At that point all it would need is a trigger from the control loop to the terminal class to cause a block send.

My thoughts are that we should have both schemes. Yours which provides improved high data rate logging to be used in the web plots and mine which provides a fixed format block that can be sent once every control loop run (obviously with different data for the sine and foc builds) for ofline logging. For the logging I prefer the idea of a fixed block - my use for it is to sit logging in the background until it's needed, and when it is it should have all the core data in the log, not just what happened to be left set up from the last time it was used. It also makes it more useful as a log for non-technical user to submit when odd things happen as it should always be configured right.

On the unexpected behaviour I had another though last week (see viewtopic.php?p=45671#p45671). It handles Ifw a little differently and in simulation at least is easily the best set of results that I've seen to date. Code mods are:

In pwmgeneration:

Code: Select all

if (opmode == MOD_RUN && initwait == 0)
      {
         s32fp ifw = fwController.RunFW(Param::GetInt(Param::amp), idMtpa);
         //dController.SetRef(idMtpa + ifw);
         dController.SetRef(ifw);
         Param::SetFixed(Param::ifw, ifw);
      }
and in picontroller:

Code: Select all

int32_t PiController::RunFW(s32fp curVal, s32fp idMtpa)
{
   //the idea here is to force the ouput current to follow idMtpa in -increases
   //but only bring it down using the controller
   s32fp err = refVal - curVal;
   esum += err;

   //anti windup
   esum = MIN(esum, maxSum);
   esum = MAX(esum, minSum);

   int32_t y = FP_TOINT(err * kp + (esum / frequency) * ki);
   int32_t ylim = MAX(y, minY);
   ylim = MIN(ylim, maxY);

   if(idMtpa < ylim)
   {
      //set I term to reflect new output value
      if (ki != 0)
         esum = FP_FROMINT((idMtpa * frequency) / ABS(ki));
      return idMtpa;
   }
   else
      return ylim;
}
It completely eliminates the problem when Mtpa derived -Id is removed when coming off the throttle (although it's there for Mtpa it's pretty effective for FW too and so when it disappears it causes a big blip in the control loops which they may or may not recover in time). It also seems to work well with low ki and kp values which may help it come in a little more slowly.

Edit - forgot to say the simulation results referred to above looking at the revised Mtpa/FW integration also used sync compensation which appeared to help too.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Data Logger

Post by Bigpie »

Still need to get the sdcard module to play with but I'm up and running for the most part. What is it you're wanting adding to the web interface? Listing of the sdcard files and downloading? Anything else?
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Make sure you get an SDIO mode one, not SPI, and preferably one with pull up resistors on it. There is supposed to be an issue with SDIO on some ESP32 modules, if they have 3v3 internal flash an eeprom fuse may need setting although I couldn't find a definitive list of which modules are affected. Best thing to do is try it and if it works leave it, if it doesn't then look at the fuse. Mine was a WROOM-32E module and it was fine without anything doing.

On the code it's pretty much up to you, whatever you consider useful. My thoughts are some way to list and download files and a way to check the time and set the time on the RTC. You could have a look at plotting but probably best waiting till the file format is fixed?

If you need any more handlers adding to the C code let me know. Also I was just thinking of adding the SD card file system as a virtual subdirectory under the SPIFFS root, let me know if that would work for you?

Edit1 - There is also the option of whether to move the existing web plotting over to use Johannes new fast binary mode too? It would need to be backwards compatible though.

Edit2 - It would also be great if you could see if the inverter firmware update functionality still works as I haven't been able to test it (my test board doesn't have the bootloader on it).

Edit3 - mentioning backward compatibility has made me realise that if the new esp32 code is used with old inverter code it will constantly be sending the start logging command. Need to change it so that it tries say 3 times and then gives up. Will change next time I'm in the code. Also just to remind myself I think there is a fclose missing in the case where logging fails to start, need to fix that at the same time.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Data Logger

Post by Pete9008 »

Post in another thread has made me realise that we should add a list of all parameters to the front of the logged file too. Would a json command at the start of the log do it or is there a better way?
Post Reply