ESP32 Based Web Interface & Data Logger

Discussion about various user interfaces such as web interface, displays and apps
Post Reply
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

:D
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

This is what I was thinking but either should work:

Code: Select all

if(haveSDCard && fastLoggingEnabled && (startLogAttempt < 3) && (millis() > DELAY_VAL))
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Still no dice, doesn't even load the params in the car. Will look again tomorrow.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Strange, it should load the parameter regardless of the logging changes :?

How long are the cables between the inverter and the ESP? Although thinking about it the params would be loaded at the old baud rate so shouldn't be affected by the higher speeds during logging.

Which serial ports are you using on the ESP, can you still hook up the ESP debug output with it in the car?

Edit - only other thought is could the ESP32 be more susceptible to interference that the 8266? (that would ba a pain :( ).

Does it connect to the wifi ok, can you access the web page?

Edit2 - if the web page loads then it has to be the serial comms, stupid Q but have to ask, could Rx and Tx be crossed?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I've checked and double checked RX & TX and also tried swapping them over. I've soldered the same header as the olimex so I can just swap over.

It connects and page loads. I've not connected up a usb to the ESP32 while in the car, not sure if I'd need to disconnect the 3.3v from the inverter to do that but will do that in the morning
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Fri Oct 21, 2022 6:31 pm I've checked and double checked RX & TX and also tried swapping them over. I've soldered the same header as the olimex so I can just swap over.

It connects and page loads. I've not connected up a usb to the ESP32 while in the car, not sure if I'd need to disconnect the 3.3v from the inverter to do that but will do that in the morning
Thought you would have checked but had to ask!

Ahh, forgot yours has the built in USB Best to disconnect 3v3. Might be OK with it connected not to but depends how the ESP board is wired and best not to take chances.

Just been checking everything here and all seems to work fine. Have found a bug in the STM code though which causes a corruption in the json header sent as part of the log file (it didn't allow time for the UART DMA to complete before sending data) but this wouldn't have caused your problem (it only showed up when trying to decode logged data). I'll push the changes onto github this evening.

Also tried decoding some longer data files and might need to optimise the decode program a bit, it takes around 10sec per minute of data at the moment, it's also generating some pretty big csv files (0.5GB for ten minutes of data)!

Pulseview does seem to have loaded this file OK though, it's surprisingly quick to load (so quick I thought it hadn't done it at first) and then a bit laggy, but it's not bad at all :) 10min of data is probably about its limit though.

Probably ought to change the ESP code at some point to close the file and open a new one every 10minutes.

Edit - It's not that fast to load, I was just zoomed in while it did it so only saw the first few seconds data load. If you are fully zoomed out then the whole 10min of data takes a little over 20sec.

Edit2 - New decoder code in case it's needed:

Code: Select all

#include <QCoreApplication>
#include <QFile>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <QJsonValue>
#include <QDebug>
#include <QVector>
#include <QtMath>

#define MODMAX (((2U<<15)/1.732050807568877293527446315059) - 200)

struct varDefinitions {
  QString name;
  double scale;
  int bits;
  bool signExtend;
} ;



int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QVector<varDefinitions> varDefs;

    QFile logFile("logfile.bin");
    QFile outFile("decodedfile.csv");

    QByteArray jsonHeader;
    char val;
    int paraCount = -1;
    int messageBits = 0, messageBytes = 0;
    char buffer[25];
    int index = 0;
    int bitsHave = 8;
    uint32_t value = 0;
    uint32_t bitStore = 0;
    uint32_t mask = 0;
    uint32_t messageCount = 0;
    int haveAngle = -1, haveI1 = -1, haveI2= -1;

    logFile.open(QFile::ReadOnly);


    if(logFile.isOpen())
    {
        //read and save parameters
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
        QFile paramFile("paramfile.json");
        paramFile.open(QFile::WriteOnly);
        if(paramFile.isOpen())
        {
            paramFile.write(jsonHeader);
            paramFile.close();
            qDebug("Param file written");
        }
        else
            qDebug("Could not write param file");

        jsonHeader.clear();
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
     }
    else
        qDebug("Could not open input file");

    if(paraCount == 0)
    {
        QJsonDocument jsonResponse = QJsonDocument::fromJson(jsonHeader);
        QJsonObject jsonObject = jsonResponse.object();
        varDefinitions def;
        foreach(const QString& key, jsonObject.keys())
        {
            QJsonObject jsonObject2 = jsonObject[key].toObject();
            foreach(const QString& key2, jsonObject2.keys())
            {
                if(key2 == "name")
                    def.name = jsonObject2[key2].toString();
                else if(key2 == "scale")
                    def.scale = jsonObject2[key2].toDouble();
                else if(key2 == "signed")
                    def.signExtend = jsonObject2[key2].toInt();
                else if(key2 == "size")
                {
                    def.bits = jsonObject2[key2].toInt();
                    messageBits += def.bits;
                }
            }
            if(def.name == "angle")
                haveAngle = varDefs.size();
            else if(def.name == "i1")
                haveI1 = varDefs.size();
            else if(def.name == "i2")
                haveI2 = varDefs.size();

            varDefs.append(def);
        }

        outFile.open(QFile::WriteOnly);

        if(outFile.isOpen())
        {
            QTextStream outStream(&outFile);
            QVectorIterator<varDefinitions> i(varDefs);
            while (i.hasNext())
                outStream << i.next().name << ',';
            if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
            {
                outStream << "iq" << ',' << "id" << ',';
            }
            outStream << "\n";

            messageBytes = (messageBits+7)/8;

            if(messageBytes <=25)
            {
                while(logFile.peek(buffer, messageBytes)==messageBytes)
                {
                    QVector<double> values(varDefs.size());
                    //is csum valid?
                    uint8_t csum = 0;
                    int i;
                    for(i=0;i<messageBytes-1;i++)
                        csum += (uint8_t)buffer[i];
                    if(csum == (uint8_t)buffer[i])
                    { //match, we have a valid message so extract data
                        index = 0;
                        bitsHave = 8;
                        value = 0;
                        bitStore = (uint32_t)((uint8_t)buffer[index++]);
                        for(i=0;i<varDefs.size();i++)
                        {
                            int bitsNeeded = varDefs[i].bits;
                            while(bitsHave < bitsNeeded)
                            {
                                bitStore = (((uint32_t)((uint8_t)buffer[index++]))<<bitsHave) + bitStore;
                                bitsHave = bitsHave + 8;
                            }
                            mask = ~(0xffffffff<<bitsNeeded);
                            value = bitStore & mask;
                            if(varDefs[i].signExtend)
                            {
                                mask = 0x01<<(bitsNeeded-1);
                                if((value & mask) != 0) //msb set so extend
                                {
                                    mask = 0xffffffff<<bitsNeeded;
                                    value = value | mask;
                                    values[i] = ((int32_t)value) * varDefs[i].scale;
                                }
                                else //not set do don't
                                    values[i] = value * varDefs[i].scale;
                            }
                            else
                                values[i] = value * varDefs[i].scale;
                            if(varDefs[i].name == "angle")
                                values[i] = (360.0 * (values[i]/65535));
                            if((varDefs[i].name == "ud") || (varDefs[i].name == "uq"))
                                values[i] = (100.0 * (values[i]/MODMAX));
                            if((varDefs[i].name == "pwm1") || (varDefs[i].name == "pwm2") || (varDefs[i].name == "pwm3"))
                                values[i] = (100.0 * ((values[i]-8192)/8192));
                            bitStore = bitStore >> bitsNeeded;
                            bitsHave = bitsHave - bitsNeeded;
                        }
                        logFile.read(buffer, messageBytes); //discard the data

                        QVectorIterator<double> i(values);
                        while (i.hasNext())
                            outStream << i.next() << ',';
                        if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
                        {
                            double angle = values[haveAngle];
                            double ia = values[haveI1];
                            double ib = ((values[haveI1]+(2.0*values[haveI2]))/qSqrt(3.0));
                            double id = (ia * qCos(qDegreesToRadians(angle))) + (ib * qSin(qDegreesToRadians(angle)));
                            double iq = (-ia * qSin(qDegreesToRadians(angle))) + (ib * qCos(qDegreesToRadians(angle)));
                            outStream << iq << ',' << id << ',';
                        }
                        outStream << "\n";
                        messageCount++;
                        if((messageCount%(8800*60))==0)
                            qDebug("Processed %i minutes of data",messageCount/(8800*60));
                    }
                    else //no match so throw a byte away and try again
                    {
                        logFile.read(buffer, 1);
                        qDebug("Mesage Lost");
                    }
                }
                qDebug("Processing Complete");
            }
            else
                qDebug("Json header message length invalid");
        }
        else
            qDebug("Could not open output file");
    }
    else
        qDebug("Could not find json header");

    logFile.close();
    outFile.close();
    return a.exec();
}
Edit3 - Pulseview is pretty slow to save the files but can do it, the resulting file is around 86MB (down from 464MB for the csv, the SD card file this was generated from was 118MB). The .sr format is probably worth looking into longer term.

Edit4 - github updated (esp and stm code)
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

One other thought. It's possible that the stm has got into the fast baudrate logging mode and the esp isn't able to get it back out (I thought I'd covered all the cases to make sure it can recover but I may have missed one?). If this is the case then a stm reset or power cycle would be needed to get it back to normal comms baud rates. With the esp debug connected you should be able to see whether this is the case or not.
User avatar
mjc506
Posts: 343
Joined: Wed Sep 09, 2020 9:36 pm
Location: Wales, United Kingdom
Has thanked: 30 times
Been thanked: 28 times

Re: ESP32 Based Web Interface & Data Logger

Post by mjc506 »

Just seen this too, not tested it (but I'll forget if I don't write it down :-) )
https://hackaday.io/project/5334-serial ... g-software
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Does your stm32 branch contain any fixes? I cannot get it to cut out on my testing but here's some data.
https://drive.google.com/drive/folders/ ... sp=sharing
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Can't decide if that's good news or not!

I think I put the fix for the PWM overflow in, it's possible that could have done it. See here for details if you want to try removing it viewtopic.php?p=47101#p47101

Edit - What was the cause of the comms problem?
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Here's your logged data, opened no problem at all :)
BigpieLoggedData.png
Not looked at it in detail yet but initial impressions are good.

Edit - there are some odd things going on though:
BigpieLogCurrents.png
There is something odd on the currents il1 and il2, there is a distinct difference between when accelerating (LHS of plot) and decelerating (RHS). The decel data is what I'd expect. Not sure what to make of the left hand acceleration side??

Edit2 - Also something not right in Id and Iq, they should follow IdRef and IqRef.
BigpieIQDRef.png
Not sure whether this is a real problem or a logger problem.

It looks like it is just when you are pushing on a bit more so wondering whether this is the throttle limiter which is in the latest code, might be worth commenting out the following lines in SetTorquePercent to disable it and trying it again (but don't push too hard as the control loops will then be able to go into saturation)?

Code: Select all

    //float maxThrot = throttleLimitController.RunProportionalOnly(Param::GetInt(Param::amp));

   //if (torquePercent > 0)
   //   torquePercent = MIN(maxThrot, torquePercent);
   //else
   //   torquePercent = -MIN(maxThrot, -torquePercent);
Edit - based on Johannes comments in the simulator thread removing the limiter sounds well worth a try.
User avatar
johu
Site Admin
Posts: 5684
Joined: Thu Nov 08, 2018 10:52 pm
Location: Kassel/Germany
Has thanked: 153 times
Been thanked: 960 times
Contact:

Re: ESP32 Based Web Interface & Data Logger

Post by johu »

Wow, cool to see some live data. Left hand currents are super strange indeed
Support R/D and forum on Patreon: https://patreon.com/openinverter - Subscribe on odysee: https://odysee.com/@openinverter:9
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

johu wrote: Sat Oct 22, 2022 3:09 pm Wow, cool to see some live data.
Absolutely!

Edit - See above for comment on the currents (was adding it while you posted), what do you think?
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Another strange bit in the center of the plot. The angle is unchanging so stationary but there is negative DC currents being recorded (-40A):
BogpieAngle.png
Could this be energy stored in the motor being returned to the battery, can it really last for 5sec, stop and then start again!?

@Bigpie - any idea what what happening here?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I dont, I did have to stop at some lights for a few seconds. I was driving pretty erratically trying to get it to cut out. I'll have a go at loading it up in PulseView. I'll get some more logs tomorrow and see if I can get it to cut out.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Sat Oct 22, 2022 3:52 pm I dont, I did have to stop at some lights for a few seconds. I was driving pretty erratically trying to get it to cut out. I'll have a go at loading it up in PulseView. I'll get some more logs tomorrow and see if I can get it to cut out.
Thinking the cut-outs were the PWM overflow, the symptoms would be right. Good if it's fixed :)

What was the problem yesterday getting the ESP to talk to the STM?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.

How do I compile the application to convert to CSV to open in PusleView?
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Sat Oct 22, 2022 4:04 pm I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.

How do I compile the application to convert to CSV to open in PusleView?
I'll have another look at the ESP code and see if I can add something to ensure it recovers from this state.

In Qt create click New Project, select a Qt Console Application, give it a name and location and let the wizard complete and then copy the code into the main.cpp that will have been created. Should then just compile and run. Afraid it uses fixed file names at the moment (the input file needs to be renamed to logfile.bin) and assumes they are in the same directory as the executable (the build-something subdirectory of the Qt project directory if running within Qt).

Edit - did the wifi network test (as opposed to station mode) work OK?

Edit2 - Your'e using a later version of Qt than me, if you find it needs any changes let me know and I'll try to include them as conditional compilations in future versions.
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Sat Oct 22, 2022 4:04 pm I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.
What would be useful in order to try and recreate the problem is a few more details on the network configuration when you had the problem. Are you using the ESP to provide the wifi network or is it connecting to the house or laptop network.
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

It was probably able to connect to house network, but I was connecting to the esp32 AP.

Both powering on at the same time.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Hmm, wondering whether the availability of the house network could have been confusing things. Not looked much at the ESPs wifi settings and have just let it default to AP mode. Do you manually configure which it does?
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

Not too sure, I think this

Code: Select all

  WiFi.mode(WIFI_AP_STA);
causes it to both create an AP and connect to a network if the credentials are stored. I'll have a look into it. More excited about the logging at the moment, and hoping the cutouts are cured.

PulseView doesn't seem to like running on my macbook pro.

EDIT, got pulseview working :D this is awesome. I don't understand some of the graphs, but they look good.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Might be worth clearing the stored credentials for the house and see if it behaves any better?

Afraid I can't help with the mac, never used one :( Does all the Qt stuff run ok on it?

The linux appimage loaded up no trouble at all (which is a bit of a first for my old Linux box which normally never works first time!)
Pete9008
Posts: 1801
Joined: Sun Apr 03, 2022 1:57 pm
Has thanked: 102 times
Been thanked: 347 times

Re: ESP32 Based Web Interface & Data Logger

Post by Pete9008 »

Bigpie wrote: Sat Oct 22, 2022 5:55 pm EDIT, got pulseview working :D this is awesome. I don't understand some of the graphs, but they look good.

Good isn't it :-)

Edit - need to calculate frq in the decoder and add it to the plots, it would make interpreting them much easier!
User avatar
Bigpie
Posts: 1585
Joined: Wed Apr 10, 2019 8:11 pm
Location: South Yorkshire, UK
Has thanked: 74 times
Been thanked: 299 times

Re: ESP32 Based Web Interface & Data Logger

Post by Bigpie »

I'll get some more plots tomorrow, got a couple of trips to make, one of which includes some steep inclines on a NSL road.
'Pushing on a bit' was up to 40mph, but in first gear :D

Brain a bit too frazzled today to make too much sense, you think I hit the throttle limiter?

I think I have FW disabled still, but the motor still goes up to high RPM, the acceleration without cutting out took my by surprise, I did a little swearing.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Post Reply