Update via wifi fails

Discussion about various user interfaces such as web interface, displays and apps
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Update via wifi fails

Post by johu »

We've had a problem lately that the ESP module would no longer transmit updates to the inverter. Or transmit only part of the update, rendering the inverter "bricked". Not really bricked, as it can be revived with a successful update.

Anyway, the reason was that the ESP had run out of flash memory. When updating, the bin file is stored in the ESPs flash and is transmitted from there. So what happened? The user had chosen to rename the bin files to include version, date or functionality. Which is actually a good idea. But it means the flash memory fills up with files:
Type http://192.168.4.1/list to see this

Code: Select all

[{"type":"file","name":"stm32_foc_457.bin"},{"type":"file","name":"ajax-loader.gif"},{"type":"file","name":"refresh.png"},{"type":"file","name":"gauges.js"},{"type":"file","name":"stm32_foc_485.bin"},{"type":"file","name":"stm32_foc210509.bin"},{"type":"file","name":"index.js"},{"type":"file","name":"stm32_foc (3).bin"},{"type":"file","name":"Leaf 2019-11-10.json"},{"type":"file","name":"stm32_sine.bin"},{"type":"file","name":"stm32_foc (2).bin"},{"type":"file","name":"gauges.html"},{"type":"file","name":"stm32_foc20200217.bin"},{"type":"file","name":"stm32_foc1.bin"},{"type":"file","name":"stm32_foc.bin"},{"type":"file","name":"inverter.js"},{"type":"file","name":"log.js"},{"type":"file","name":"style.css"},{"type":"file","name":"index.html"},{"type":"file","name":"stm32_foc_curki.bin"},{"type":"file","name":"params.json"},{"type":"file","name":"stm32_foc_IDC_adjust.bin"},{"type":"file","name":"chart.min.js.gz"},{"type":"file","name":"stm32_foc_487.bin"},{"type":"file","name":"wifi.html"},{"type":"file","name":"wifi-updated.html"},{"type":"file","name":"log.html"},{"type":"file","name":"gauge.min.js.gz"},{"type":"file","name":"stm32_foc473.bin"}]


Solution: do not rename the file before updating, but afterwards.
To cure the problem, you need to delete the files. Either by reflashing the ESP SPIFS or by deleting with curl like so:

Code: Select all

curl -X "DELETE" http://192.168.4.1/edit?f=/stm32_foc_487.bin
If you're on windows you need to download the curl executable here: https://curl.se/windows/
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
MattsAwesomeStuff
Posts: 871
Joined: Fri Apr 26, 2019 5:40 pm
Has thanked: 278 times
Been thanked: 165 times

Re: Update via wifi fails

Post by MattsAwesomeStuff »

This needs to be in the wiki.. I'm just not sure where.
Lowcoe
Posts: 26
Joined: Sun Oct 04, 2020 8:57 pm
Location: Fallbrook, California

Re: Update via wifi fails

Post by Lowcoe »

I would like to try deleting files and starting over but I’m not following the curl-x example. Can anyone dumb this down??
User avatar
EV_Builder
Posts: 1199
Joined: Tue Apr 28, 2020 3:50 pm
Location: The Netherlands
Has thanked: 16 times
Been thanked: 33 times
Contact:

Re: Update via wifi fails

Post by EV_Builder »

For windows users: make a bat file (a simple text file with the bat extension)

Code: Select all

@echo off
curl -X "DELETE" http://192.168.4.1/edit?f=/stm32_chademo.bin
curl -X "DELETE" http://192.168.4.1/edit?f=/stm32_bms.bin
curl -X "DELETE" http://192.168.4.1/edit?f=/stm32_vcu.bin
curl -X "DELETE" http://192.168.4.1/edit?f=/Stm32_VCU.bin
pause
run it and you will see the output. Not nice but works.
Converting an Porsche Panamera
see http://www.wdrautomatisering.nl for bespoke BMS modules.
User avatar
geduxaz
Posts: 125
Joined: Wed Jun 23, 2021 7:00 am
Has thanked: 3 times

Re: Update via wifi fails

Post by geduxaz »

omg.. thats why i had so many failed updates.... i always rename files to know what version they are..
User avatar
catphish
Posts: 954
Joined: Fri Oct 08, 2021 11:02 pm
Location: Dorset, UK
Has thanked: 93 times
Been thanked: 179 times

Re: Update via wifi fails

Post by catphish »

Just wanted to cross-post here, in case anyone runs into the same problem. I has filed named "stm32_sine(1).bin" and "stm32_sine (1).bin". The space in the latter requires manually URL encoded syntax in the filename %20

Code: Select all

curl -X "DELETE" "http://192.168.4.1/edit?f=/stm32_sine%20(1).bin"
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

I'm getting many reports about failed updates lately. Really bugs me, especially if people have to take apart their drive train.
I plan to change the esp firmware so that
a) the update runs autonumously, i.e. not controlled by the browser
b) delete the file after update
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
catphish
Posts: 954
Joined: Fri Oct 08, 2021 11:02 pm
Location: Dorset, UK
Has thanked: 93 times
Been thanked: 179 times

Re: Update via wifi fails

Post by catphish »

johu wrote: Tue Feb 08, 2022 8:55 am I'm getting many reports about failed updates lately. Really bugs me, especially if people have to take apart their drive train.
I plan to change the esp firmware so that
a) the update runs autonumously, i.e. not controlled by the browser
b) delete the file after update
That sounds like a good idea. I had a failed update yesterday, but luckily it seems the bootloader was still working and I was able to try again. The serial protocol seems to work well, but it's a shame the ESP can't forcefully reset the STM32, or even better, interface directly with the ROM bootloader / boot pins to make it 100% unbrickable.
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

johu wrote: Tue Feb 08, 2022 8:55 am I'm getting many reports about failed updates lately. Really bugs me, especially if people have to take apart their drive train.
I plan to change the esp firmware so that
a) the update runs autonumously, i.e. not controlled by the browser
b) delete the file after update
b) can be done purely in index.js:522 runUpdate.

I just wrote the below, that should work. It isn't really tested, can someone try it?

Code: Select all

512     if (step < totalPages){
513       runUpdate(step, file);
514     } else {
515       document.getElementById("bar").innerHTML = "<p>Update Done!</p>";
516       if(file.endsWith(".bin")){
517         var xhr=new XMLHttpRequest();
518         xhr.open("DELETE", "/edit?f="+file);
519         xhr.send();
520       } 
521     } 
https://github.com/jsphuebner/esp8266-w ... ex.js#L522
- maybe show filesize transmitted, or show something red if filesize transmitted is different to the uploaded filesize
Attachments
index.js
(17.39 KiB) Downloaded 120 times
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

Would be brilliant :)
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

Ok, I am struggling with this again today and bricked an inverter for the moment.

I can download http://192.168.4.1/stm32_foc.bin from the ESP and it has the same SHA256 checksum as the file I uploaded.

runUpdate gets to -1 for reset, returns { "message": "reset", "pages": 48 } , then 0 hangs forever.
I tried manually triggering var file="/stm32_foc.bin";runUpdate(-1,file); but get the same behaviour.

A quick look into FSBrowser.ino reveals that -1 is the reset command, and 0 - 48 copy the file (makes sense). I wonder what goes wrong in the copy process that halts the ESP altogether despite the file being fine?

I am now browsing handleUpdate and am wondering what fails. Serial.write can be assumed to work in principle because it is used in the reset path for -1 already: https://github.com/jsphuebner/esp8266-w ... r.ino#L313

Corresponding line in bootloader sending the 'P' that Arduino Serial.read is waiting for: https://github.com/jsphuebner/tumanako- ... der.c#L199

It looks like the desired response sequence from STM32 is '2' -> 'S' -> 'P' -> 'C' -> 'D' during the upload process, with ( 'P' -> 'T' and 'C' -> 'E' indicating timeout and errors). Now unfortunately I can't see the UART communication.
I think it waits for the 'C' but never receives it?

Edit: Tried flashing stm32_loader.bin and now it times out at step -1.
Edit Edit: Tried flashing another inverter on the bench with a spare ESP, and got to the same place, noteworthy: That one only wanted 28 pages, both now timeout at step -1 already for both
Tried flashing the board directly like described here: https://openinverter.org/forum/viewtopi ... 539#p33539
With stm32_loader.bin and stm32_foc.bin:

Code: Select all

python3 updater.py -f stm32_loader.bin -d /dev/ttyUSB0
File length is 2828 bytes/3 pages
Resetting device...
Sending number of pages...
Sending page 0... b''
Sending page 0... b'\xff'
Sending page 0... b'\xff'
Sending page 0... b'\xff'
Only get binary `?` from the board at baud 115200. Surely I am at the bottom soon!
This one next: https://openinverter.org/forum/viewtopi ... 043#p24043
Attachments
Screenshot from 2022-02-16 12-50-36.png
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

Are you sure the bootloader is actually running and sending its chars? check with a terminal
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

johu wrote: Wed Feb 16, 2022 6:54 pm Are you sure the bootloader is actually running and sending its chars? check with a terminal
I checked earlier, I get ????? with the occasional character, checked different baud rates but that only changed the rate of them appearing. So no, not sure if even the bootloader is there, I will check with the scope to see if I am just getting noise or if it is actually something. I only connected TX & RX, I will try TX & RX & GND tomorrow (powered the board separately with 12V).

Learnt a lot now how the updater is put together, surprised the process failed that badly though, I only started digging when I couldn't get anywhere with the web interface anymore. At least initially there was still a bootloader there, as step -1 gave reasonable output, now I am flashing everything from scratch.
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

oh yeah you want GND!
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

johu wrote: Wed Feb 16, 2022 9:08 pm oh yeah you want GND!
I hit the ground!
bootloader still intact. Now back to the surface!
Needed TX, RX, GND. Do not plug VCCIO in if the OI board is already powered though :)

Code: Select all

09:06:01.562 -> 2D2D2D2D2D2D
Edit: Resurfaced!

Code: Select all

File length is 48580 bytes/48 pages
Resetting device...
Version 2 bootloader, sending magic
Sending number of pages...
Sending page 0... CRC correct!
Sending page 1... CRC correct!
...
Sending page 46... CRC correct!
Sending page 47... CRC correct!
Update done!
Now the only question remains: Why does the ESP fail to update it? I have an open inverter (ha ha) on the bench now where the ESP web upload does not work in the same way it failed yesterday in the car.

Edit: Ok, I will hook up a USART to USB adapter to listen in and print on the terminal what ESP & STM say to each other, hopefully that will help figuring out where it gets stuck and doesn't allow updates when it has half a firmware on it.

Edit Edit:
Ok, there are human readable debug messages between the inverter commands! Surely that is not desired behaviour?

I can see "2SPCPCPCPC..." when it is updating, but at the same time I get output like this:

Code: Select all

handleFileRead: /style.cssUnknown command sequence
handleFileRead: /chart.min.jsUnknown command sequence
So me loading assets from the webinterface sends commands to the inverter. But I sketched the arduino with DebugLevel: None; DebugPort: None and still get handleFileRead: output.
Attachments
output-onlinepngtools(1).png
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

I hd to eliminate all debug output because it essentially cannot be disabled (or I was too stupid). 2SPCPCPC... sounds correct
- 2 "Hello I'm bootloader V2+"
- S "Send the size of the update in pages, i.e. bytes/1024"
- P "Send the page"
- C "Send page CRC"

and back to P
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

johu wrote: Thu Feb 17, 2022 2:04 pm ..2SPC..
Yes, the STM32 side is straightforward I think and makes sense. I think the ESP is spamming and confusing the STM on the other hand, here is an example from the fast baudrate section: I can delete debug messages, but am worried I will delete too many and then lack functionality later on - but I don't know that much about serial comms. I might also be barking up the wrong tree here and the problem is elsewhere.

Code: Select all

handleFileRead: /index.js^MUnknown command sequence
handleFileRead: /style.css^MUnknown command sequence
handleFileRead: /ajax-loader.gif^MUnknown command sequence
handleFileRead: /refresh.png^MUnknown command sequence

json
{ 
 ... snip long json message with all params ...
 }
0^MUnknown command sequence
handleFileUpload Name: /stm32_foc.bin
207577 / 957314
Unknown command sequence
1^MUnknown command sequence
handleFileUpload Data: 2048
Unknown command sequence
handleFileUpload Data: 2048
Unknown command sequence
1^MUnknown command sequence
handleFileUpload Data: 52
2
Unknown command sequence
handleFileUpload Size: 41012^MUnknown command sequence
reset
Edit: Right, I got the inverter out and back in, in case this helps someone in the future:
  1. Communication between ESP & STM broke down when they switched to fastuart, firmware was still running, STM output is 2DOK and then nothing on baud 115200 and 921600. I listened in first with the scope, then I connected my USART-to-USB adapter inbetween STM & ESP: GND & RX only and opened a terminal screen /dev/ttyUSB0 115200 first, then screen /dev/ttyUSB0 921600
  2. I updated ESP firmware from dc8b939.. to b61ccb1.. which has fewer print statements
  3. In Arduino Studio disable Debug output in both "Debug Port" & "Debug Level".
User avatar
Renegate2020
Posts: 50
Joined: Tue Dec 01, 2020 6:20 am
Location: Odessa UA
Has thanked: 1 time
Contact:

Re: Update via wifi fails

Post by Renegate2020 »

how to delete files through android device?
Ken_S
Posts: 26
Joined: Fri Mar 22, 2019 8:58 pm
Location: Clear Creek County, Colorado

Re: Update via wifi fails

Post by Ken_S »

Edit: Right, I got the inverter out and back in, in case this helps someone in the future:
"I updated ESP firmware from dc8b939.. to b61ccb1.. which has fewer print statements"
I believe I have had this issue appear recently as I finally got around to implementing the Tesla PCS controller board that has been sitting on my shelf since mid 2021. It was working well until I tried to update the board firmware using the web interface, after which the board completely stopped responding. I am almost certain that my issue is the same reported here because 1)My ESP8266 firmware was an older version that I hadn't updated, and 2) the controller micro bootloader seems to be intact (sending "2D2D2D" when i connect directly to the serial lines), but 3) any attempt to upload new firmware to the microcontroller via the web interface appeared to succeed but had no effect on restoring normal behavior.

I am in the process of updating the ESP8266 from its originally shipped firmware to the latest version (e87dc18..). I believe I have updated it correctly, but it looks like I deleted the filesystem in the process... I get filenotfound at 192.168.4.1 and [] at 192.168.4.1, however I do get redirected to a very simple update page after a delay.
esp8266 update page.png
esp8266 update page.png (8.75 KiB) Viewed 4865 times
I am now struggling to with how to reload the files required for the web interface. It feels like it should be straightforward (use the update filesystem button on 192.168.4.1/update) but i can't figure out where to find the appropriate .bin file, or how to create it. I am not a software developer, this endeavor to re-flash my esp8266 was my first experience with the Arduino IDE, Github, or TTL serial... what am I doing wrong?
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

Ken_S wrote: Sat Sep 24, 2022 10:07 pm filesystem update
You are nearly there! There is no bin file for assets, you just upload the html/css/js/png assets directly.
The *.bin file is updated via the button once you have index.html & index.js in place.

https://openinverter.org/wiki/Olimex_MO ... bInterface
Ken_S
Posts: 26
Joined: Fri Mar 22, 2019 8:58 pm
Location: Clear Creek County, Colorado

Re: Update via wifi fails

Post by Ken_S »

I am stuck at this step... I am not sure if it is something I am doing wrong?

This is the verbose output of curl when I try to upload index.js.
Screenshot - verbose curl output index_js.png
I am not an expert, but it appears like it worked correctly? However, when I then go to 192.168.4.1/list, I still see an apparently empty filesystem [].

I was using curl to delete extra files from the filesystem at the beginning of this saga, when I thought the problem was too many attempted uploads, and I was able to see files in the list at 192.168.4.1/list, delete them with a curl command, and then refresh 192.168.4.1/list to see that they were gone. What am I doing wrong now?

one note: I wasn't able to get the commands listed in the wiki to work exactly as written, if I left the single apostrophes and the @ symbol I got the following return:
>curl -F 'data=@index.js' http://192.168.4.1/edit
curl: (26) Failed to open/read local data from file/application

but as shown in the screenshot, it works if i delete the @ and ' ' from the statement... could this be related? I am using a windows 10 laptop if it matters.

[EDIT!] I got it to work... I don't understand what the @ symbol does in curl, but adding it back in and forgoing the ' ' apparently caused the files to be uploaded correctly!

curl 192.168.4.1/list
[{"type":"file","name":"index.js"},{"type":"file","name":"log.js"},{"type":"file","name":"inverter.js"},{"type":"file","name":"index.html"},{"type":"file","name":"log.html"},{"type":"file","name":"wifi.html"},{"type":"file","name":"wifi-updated.html"},{"type":"file","name":"ajax-loader.gif"},{"type":"file","name":"refresh.png"},{"type":"file","name":"style.css"},{"type":"file","name":"gauges.html"},{"type":"file","name":"gauges.js"},{"type":"file","name":"gauge.min.js.gz"},{"type":"file","name":"chart.min.js.gz"}]
User avatar
janosch
Posts: 306
Joined: Tue Jun 30, 2020 9:23 am
Location: London, UK
Has thanked: 67 times
Been thanked: 54 times
Contact:

Re: Update via wifi fails

Post by janosch »

Ken_S wrote: Wed Sep 28, 2022 7:06 pm ...
[EDIT!] I got it to work... I don't understand what the @ symbol does in curl, but adding it back in and forgoing the ' ' apparently caused the files to be uploaded correctly!
Neat.

Whats the output from these two commands:

Code: Select all

curl --version
echo "$BASH_VERSION"
I am using curl 7.68.0 (x86_64-pc-linux-gnu) and bash 5.0.17(1)-release.

Why don't you modify the wiki to leave a hint or try to change the script so that it works on both versions? Might help the next person.
Ken_S
Posts: 26
Joined: Fri Mar 22, 2019 8:58 pm
Location: Clear Creek County, Colorado

Re: Update via wifi fails

Post by Ken_S »

janosch wrote: Fri Sep 30, 2022 2:08 pm Neat.

Whats the output from these two commands:

Code: Select all

curl --version
echo "$BASH_VERSION"
I am using curl 7.68.0 (x86_64-pc-linux-gnu) and bash 5.0.17(1)-release.

Why don't you modify the wiki to leave a hint or try to change the script so that it works on both versions? Might help the next person.
curl 7.83.1 (Windows) libcurl/7.83.1 Schannel
Release-Date: 2022-05-13
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI UnixSockets

I will certainly do that! Any suggestions/advice about why the inverter firmware upload function still isn't working? Sometimes it hangs entirely, sometimes it says success, but the activity light on the main controller pcb is still dead, and I am not seeing any change in functionality. I just confirmed that the bootloader is still alive sending its "2D2D2D" ... my next step is to sniff the bus while I attempt a firmware upload and see what happens.
User avatar
jason_arnold
Posts: 21
Joined: Wed Sep 11, 2019 4:42 pm
Location: Canada
Has thanked: 1 time
Been thanked: 2 times

Re: Update via wifi fails

Post by jason_arnold »

johu wrote: Tue Feb 08, 2022 8:55 am I'm getting many reports about failed updates lately. Really bugs me, especially if people have to take apart their drive train.
I plan to change the esp firmware so that
a) the update runs autonumously, i.e. not controlled by the browser
b) delete the file after update
So seeing a lot of great ideas in this thread, but not sure if anything came of them? I did find the Olimex MOD-WIFI-ESP8266 writeup on the OI Wiki, but there's nothing there about OTA updating - given johu's concern about people having to "take apart their drive train" (I've already been there once or twice), I was hoping for something OTA and sort of automagic (i.e. open web interface, flash x.bin, y.bin and done). Something script/CURL-based would do too, though I'd consider that Plan B given the tech literacy required. Anyone able to comment on the above?
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: Update via wifi fails

Post by johu »

The original plan never happened, as in having the ESP upgrade the inverter firmware autonomously.
What has happened in that respect is the new bootloader which allows upgrading via CAN: viewtopic.php?t=2094

Currently working on ESP32 code that connects via CAN rather than the serial link
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Post Reply