ESP32 Based Web Interface & Data Logger

[Bigpie]

Nice, thanks

*EDIT* I must have wired up the sdcard reader wrong.
"Failed to initialize the card (0x107). Make sure SD card lines have pull-up resistors in place.
Couldn't start SD_MMC"

I assume you've not used external pullups?

[Pete9008]

No, just the ones on the card. The ones on the card are a little higher in value than recommended but worked fine on mine.

Have you checked your connections against the espressif link just in case I got anything else wrong in the readme? How long are your connections? (mine were about 20cm which I was half expecting to cause problems but were absolutely fine).

There is supposedly a boot config issue (see the espressif link for details) but if it is booting ok then that shouldn't be a problem.

[Bigpie]

Connections look good, jumper wires are only about 20cm too.
I don't know if this is my issue: https://docs.espressif.com/projects/esp ... licts-dat2 I don't really understand it

[Pete9008]

Its a line that gets read on boot to decide whether to power the flash at 1.8v or 3v3 that can conflict with the card. If it was an issue then the program wouldn't load or run with the card connected.

Silly question I know but are the 3v3 and gnd to the card ok?

[Bigpie]

The font on my board is awful G16 looked like G14. Now trying to get the sdcard mounted It's formatted FAT32

Code: Select all

  if (SD_MMC.begin("/sdcard", false, true, 40000, 5U)) 

doesn't seem to do the trick

Code: Select all

E (328) sdmmc_sd: sdmmc_check_scr: send_scr returned 0xffffffff
E (328) vfs_fat_sdmmc: sdmmc_card_init failed (0xffffffff).
[   331][E][SD_MMC.cpp:132] begin(): Failed to mount filesystem. If you want the card to be formatted, set format_if_mount_failed = true.
Couldn't start SD_MMC
[   631][E][WiFiClient.cpp:335] setOption(): fail on fd -1, errno: 9, "Bad file number"

[Pete9008]

Ok, I didn't try that, the card I had was already formatted fat32.

Can you format it on the PC then try?

Ill have a look at getting it to auto format the card next time im working on it.

Edit - think I misunderstood, your card is already formatted. Still sounds like it could be connection problems, possibly when it tries to change from 1-bit to 4-bit mode? Could do with more verbose error messages!

[Bigpie]

It's already FAT32 formatted, I'll fiddle some more. Should the DET pin be connected to anything?

[Pete9008]

No, it's the optional card present detect switch so not needed.

I'll try disconnecting a few wires on mine to see if I can get the same error.

[Pete9008]

The font on my board is awful G16 looked like G14. Now trying to get the sdcard mounted It's formatted FAT32

Code: Select all

  if (SD_MMC.begin("/sdcard", false, true, 40000, 5U)) 

doesn't seem to do the trick

Code: Select all

E (328) sdmmc_sd: sdmmc_check_scr: send_scr returned 0xffffffff
E (328) vfs_fat_sdmmc: sdmmc_card_init failed (0xffffffff).
[   331][E][SD_MMC.cpp:132] begin(): Failed to mount filesystem. If you want the card to be formatted, set format_if_mount_failed = true.
Couldn't start SD_MMC
[   631][E][WiFiClient.cpp:335] setOption(): fail on fd -1, errno: 9, "Bad file number"

Sorry, just read this properly and had misunderstood before (you've already tried the command to autoformat using the above command).

If I disconnect any pin I get the same error as you.

If I disconnect one of the extra 4-bit mode pins (D1-D3) but then use the following line to start the interface:

Code: Select all

if (SD_MMC.begin("/sdcard", true, false, 40000, 5U)) 

It then opens the file system OK (as this command forces it into 1-bit mode). Might be worth trying this on yours? Not sure it will log fast enough in 1-bit mode though (will try that next) but it would still be good to know whether tours can open the file system this way?

Edit - Looks like it's just about fast enough in 1-bit mode (I only had 1 lost message in a 25MB file). I am using a pretty quick card though so it could well be too slow with other cards.

Do you have any manufacturer details for the ESP32 module that you are using? Wondering whether it does anything different to mine on any of the pins?

[Bigpie]

I do not, it's just a generic board.

It looks like letting it auto format worked, I've changed back to

Code: Select all

 if (SD_MMC.begin()) 

and its now logging blank files to the cd card, so I can continue

[Pete9008]

Great stuff! Looking forward to seeing some real data

Need to figure out how to analyse it now!

[Bigpie]

Code: Select all

WiFi.softAPgetStationNum() == 0

Might want to re-think this a little. I have it connect to a WiFi network instead of AP and unable to list contents while it's auto logging away.

[Pete9008]

Good catch, that will need changing. Ill have a look for the equivalent test for network mode.

Can you just comment it out for the time being to force it to stop logging?

[Bigpie]

The length filenames are a bit of an issue. I've got a page that's showing the contents of the sdcard, but

Code: Select all

 if (SD_MMC.exists(path)) 

is false when the filenames are used. Any idea how to configure the library for longer filenames?

[Pete9008]

Fairly sure it is already using long filenames. It is definitely able to delete the files when clearing card space.

I did have a bit of trouble with it needing the leading slash on the name though. I know the documentation says it doesn't but it lies! Assume it must be different for the vfs file system the esp libs use?

Think I used that command in the delete oldest files function, might be worth seeing what I did there?

Edit - sorry above is not very clear, the file system seems to need the leading slash for files in the root directory. This seemed to work fine.

Code: Select all

         snprintf(filename, 50, "/%010d.bin", nextFileIndex++);
  }
  while(SD_MMC.exists(filename));

[Bigpie]

Code: Select all

void handleSdCardList() {
  
  if (!haveSDCard) {
    server.send(200, "text/json", "{\"error\": \"No SD Card\"}");
    return;
  }
  File root = SD_MMC.open("/");
  if(!root){
    server.send(200, "text/json", "{\"error\": \"Failed to open directory\"}");
    return;
  }
  if(!root.isDirectory()){
    server.send(200, "text/json", "{\"error\": \"Root is not a directory\"}");
    return;
  }
  File sdFile = root.openNextFile();
  String output = "[";
  int count = 0;
  while(sdFile && count < 200){
    if (output != "[") output += ',';
    output += "\"";
    output += String(sdFile.name());
    output += "\"";
    sdFile = root.openNextFile();

    count++;
  }
  output += "]";
  server.send(200, "text/json", output);
  return;
}

Think limiting to 200 files is ok? Not sure of a way of getting pagination working other than just opening each file in turn but only adding to the result after x amount which seems to cause a slow response.

Anyway, I've done a PR https://github.com/Pete9008/esp32-web-interface/pull/1
It should work enough for now, so I need to make up a protoboard now and update the firmware on the inverter for it.

[Pete9008]

Great stuff, looks good. Not had a chance to try it yet but will aim to over the weekend.

200 files should be plenty Thinking about it it would make more sense for the ESP code to limit the number of files on the card to 200 rather than just do it on spare space. Will change it when I do the pull.

Also planning of updating the stm code to match Johannes latest at the same time, assume you are happy with that?

[Bigpie]

Happy with that.

[Pete9008]

Heads up for anyone following this - DO NOT use the stm32 repos on my github. I tried to merge Johannes changes and have no idea what I've done but the result is a right mess

Think I'm going to have to delete the repos and start again using Johannes latest as the starting point and apply my changes to it - but can't face starting it this evening and don't want to delete the repos now in case I figure out another option in the meantime.

[mjc506]

You should be able to 'roll back' to pre-merge at the very least? My git is rusty, but I remember doing it far too many times...

[Bigpie]

What's the matter with it? Its my changes merged in and it looks to match what I'm using at the min.

[Pete9008]

Probably but I'm not sure what to do differently the second time round and even if it looked like it worked I'm not sure I'd completely trust it.

The trouble is that there were a lot of changes to the main branch (both the main project and the libopeninv subproject, both of which I'd made changes to) between me forking it and finishing the changes. I ended up having to fork an earlier version of libopeninv to get it to work last time.

I'm thinking it's safer to do a fresh fork of Johannes latest, check it out and then add in the much smaller number of changes that I've made before pushing it back. That limits the amount of git command line work and maximises my chances of getting it to work!

Git is obviously very powerful, I just don't seem to be able to get the hang of it

What's the matter with it? Its my changes merged in and it looks to match what I'm using at the min.

Your bit merged fine (at least I think it did, haven't tried compiling it yet). The above shouldn't affect the esp32 repository at all.

Johannes and my changes wouldn't auto merge though and that combined with the sub project complexities are well beyond my, very limited, git abilities!

[Bigpie]

Ahh sorry, miss understood. I'll hold off updating my inverter

[Pete9008]

No problem, I'll hopefully sort it tomorrow.

[Pete9008]

Hopefully now all sorted. The stm repos have been deleted and replaced but the links should still be the same. It's now based on Johannes latest and it's a sort of half way house between how it was done last time and how it should be done (it uses a packed structure but doesn't yet use a SOMETHING_LIST to generate it and the JSON header).

I think all the changes we discussed are included. The stm code sends out the binary data OK but I haven't tested the data formatting yet. I'm going to run it from within the simulator to generate some real data but it won't be today.

@Bigpie, all your changes work great so it looks like we may now have a logging solution

I've added a test for the connection to a WiFi network but it is untested so let me know if it fixes it. It also now limits the number of files on the SD card to 200.

Warning - The stm code is completely untested on a real inverter!

It turned out that the problem I had yesterday wasn't the merge. I hadn't noticed that Johannes latest stm32-sine isn't pulling in his latest libopeninv so when I'd inadvertently reset the submodule to the head it had broken things. Unfortunately I didn't realise this until after I had deleted all the repos, re-forked and added my changes only to get to exactly the same point I was at last night. Ohh well, all sorted now and at least I now understand git a bit better!

Edit - One thought on the web interface, how difficult would it be to add a file size for each file on the SD card? It would be useful both to help identify files and to indicate how long a download might take.