ESP32 Based Web Interface & Data Logger

[Bigpie]

[Pete9008]

This is what I was thinking but either should work:

Code: Select all

if(haveSDCard && fastLoggingEnabled && (startLogAttempt < 3) && (millis() > DELAY_VAL))

[Bigpie]

Still no dice, doesn't even load the params in the car. Will look again tomorrow.

[Pete9008]

Strange, it should load the parameter regardless of the logging changes

How long are the cables between the inverter and the ESP? Although thinking about it the params would be loaded at the old baud rate so shouldn't be affected by the higher speeds during logging.

Which serial ports are you using on the ESP, can you still hook up the ESP debug output with it in the car?

Edit - only other thought is could the ESP32 be more susceptible to interference that the 8266? (that would ba a pain ).

Does it connect to the wifi ok, can you access the web page?

Edit2 - if the web page loads then it has to be the serial comms, stupid Q but have to ask, could Rx and Tx be crossed?

[Bigpie]

I've checked and double checked RX & TX and also tried swapping them over. I've soldered the same header as the olimex so I can just swap over.

It connects and page loads. I've not connected up a usb to the ESP32 while in the car, not sure if I'd need to disconnect the 3.3v from the inverter to do that but will do that in the morning

[Pete9008]

I've checked and double checked RX & TX and also tried swapping them over. I've soldered the same header as the olimex so I can just swap over.

It connects and page loads. I've not connected up a usb to the ESP32 while in the car, not sure if I'd need to disconnect the 3.3v from the inverter to do that but will do that in the morning

Thought you would have checked but had to ask!

Ahh, forgot yours has the built in USB Best to disconnect 3v3. Might be OK with it connected not to but depends how the ESP board is wired and best not to take chances.

Just been checking everything here and all seems to work fine. Have found a bug in the STM code though which causes a corruption in the json header sent as part of the log file (it didn't allow time for the UART DMA to complete before sending data) but this wouldn't have caused your problem (it only showed up when trying to decode logged data). I'll push the changes onto github this evening.

Also tried decoding some longer data files and might need to optimise the decode program a bit, it takes around 10sec per minute of data at the moment, it's also generating some pretty big csv files (0.5GB for ten minutes of data)!

Pulseview does seem to have loaded this file OK though, it's surprisingly quick to load (so quick I thought it hadn't done it at first) and then a bit laggy, but it's not bad at all 10min of data is probably about its limit though.

Probably ought to change the ESP code at some point to close the file and open a new one every 10minutes.

Edit - It's not that fast to load, I was just zoomed in while it did it so only saw the first few seconds data load. If you are fully zoomed out then the whole 10min of data takes a little over 20sec.

Edit2 - New decoder code in case it's needed:

Code: Select all

#include <QCoreApplication>
#include <QFile>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <QJsonValue>
#include <QDebug>
#include <QVector>
#include <QtMath>

#define MODMAX (((2U<<15)/1.732050807568877293527446315059) - 200)

struct varDefinitions {
  QString name;
  double scale;
  int bits;
  bool signExtend;
} ;



int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QVector<varDefinitions> varDefs;

    QFile logFile("logfile.bin");
    QFile outFile("decodedfile.csv");

    QByteArray jsonHeader;
    char val;
    int paraCount = -1;
    int messageBits = 0, messageBytes = 0;
    char buffer[25];
    int index = 0;
    int bitsHave = 8;
    uint32_t value = 0;
    uint32_t bitStore = 0;
    uint32_t mask = 0;
    uint32_t messageCount = 0;
    int haveAngle = -1, haveI1 = -1, haveI2= -1;

    logFile.open(QFile::ReadOnly);


    if(logFile.isOpen())
    {
        //read and save parameters
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
        QFile paramFile("paramfile.json");
        paramFile.open(QFile::WriteOnly);
        if(paramFile.isOpen())
        {
            paramFile.write(jsonHeader);
            paramFile.close();
            qDebug("Param file written");
        }
        else
            qDebug("Could not write param file");

        jsonHeader.clear();
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
     }
    else
        qDebug("Could not open input file");

    if(paraCount == 0)
    {
        QJsonDocument jsonResponse = QJsonDocument::fromJson(jsonHeader);
        QJsonObject jsonObject = jsonResponse.object();
        varDefinitions def;
        foreach(const QString& key, jsonObject.keys())
        {
            QJsonObject jsonObject2 = jsonObject[key].toObject();
            foreach(const QString& key2, jsonObject2.keys())
            {
                if(key2 == "name")
                    def.name = jsonObject2[key2].toString();
                else if(key2 == "scale")
                    def.scale = jsonObject2[key2].toDouble();
                else if(key2 == "signed")
                    def.signExtend = jsonObject2[key2].toInt();
                else if(key2 == "size")
                {
                    def.bits = jsonObject2[key2].toInt();
                    messageBits += def.bits;
                }
            }
            if(def.name == "angle")
                haveAngle = varDefs.size();
            else if(def.name == "i1")
                haveI1 = varDefs.size();
            else if(def.name == "i2")
                haveI2 = varDefs.size();

            varDefs.append(def);
        }

        outFile.open(QFile::WriteOnly);

        if(outFile.isOpen())
        {
            QTextStream outStream(&outFile);
            QVectorIterator<varDefinitions> i(varDefs);
            while (i.hasNext())
                outStream << i.next().name << ',';
            if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
            {
                outStream << "iq" << ',' << "id" << ',';
            }
            outStream << "\n";

            messageBytes = (messageBits+7)/8;

            if(messageBytes <=25)
            {
                while(logFile.peek(buffer, messageBytes)==messageBytes)
                {
                    QVector<double> values(varDefs.size());
                    //is csum valid?
                    uint8_t csum = 0;
                    int i;
                    for(i=0;i<messageBytes-1;i++)
                        csum += (uint8_t)buffer[i];
                    if(csum == (uint8_t)buffer[i])
                    { //match, we have a valid message so extract data
                        index = 0;
                        bitsHave = 8;
                        value = 0;
                        bitStore = (uint32_t)((uint8_t)buffer[index++]);
                        for(i=0;i<varDefs.size();i++)
                        {
                            int bitsNeeded = varDefs[i].bits;
                            while(bitsHave < bitsNeeded)
                            {
                                bitStore = (((uint32_t)((uint8_t)buffer[index++]))<<bitsHave) + bitStore;
                                bitsHave = bitsHave + 8;
                            }
                            mask = ~(0xffffffff<<bitsNeeded);
                            value = bitStore & mask;
                            if(varDefs[i].signExtend)
                            {
                                mask = 0x01<<(bitsNeeded-1);
                                if((value & mask) != 0) //msb set so extend
                                {
                                    mask = 0xffffffff<<bitsNeeded;
                                    value = value | mask;
                                    values[i] = ((int32_t)value) * varDefs[i].scale;
                                }
                                else //not set do don't
                                    values[i] = value * varDefs[i].scale;
                            }
                            else
                                values[i] = value * varDefs[i].scale;
                            if(varDefs[i].name == "angle")
                                values[i] = (360.0 * (values[i]/65535));
                            if((varDefs[i].name == "ud") || (varDefs[i].name == "uq"))
                                values[i] = (100.0 * (values[i]/MODMAX));
                            if((varDefs[i].name == "pwm1") || (varDefs[i].name == "pwm2") || (varDefs[i].name == "pwm3"))
                                values[i] = (100.0 * ((values[i]-8192)/8192));
                            bitStore = bitStore >> bitsNeeded;
                            bitsHave = bitsHave - bitsNeeded;
                        }
                        logFile.read(buffer, messageBytes); //discard the data

                        QVectorIterator<double> i(values);
                        while (i.hasNext())
                            outStream << i.next() << ',';
                        if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
                        {
                            double angle = values[haveAngle];
                            double ia = values[haveI1];
                            double ib = ((values[haveI1]+(2.0*values[haveI2]))/qSqrt(3.0));
                            double id = (ia * qCos(qDegreesToRadians(angle))) + (ib * qSin(qDegreesToRadians(angle)));
                            double iq = (-ia * qSin(qDegreesToRadians(angle))) + (ib * qCos(qDegreesToRadians(angle)));
                            outStream << iq << ',' << id << ',';
                        }
                        outStream << "\n";
                        messageCount++;
                        if((messageCount%(8800*60))==0)
                            qDebug("Processed %i minutes of data",messageCount/(8800*60));
                    }
                    else //no match so throw a byte away and try again
                    {
                        logFile.read(buffer, 1);
                        qDebug("Mesage Lost");
                    }
                }
                qDebug("Processing Complete");
            }
            else
                qDebug("Json header message length invalid");
        }
        else
            qDebug("Could not open output file");
    }
    else
        qDebug("Could not find json header");

    logFile.close();
    outFile.close();
    return a.exec();
}

Edit3 - Pulseview is pretty slow to save the files but can do it, the resulting file is around 86MB (down from 464MB for the csv, the SD card file this was generated from was 118MB). The .sr format is probably worth looking into longer term.

Edit4 - github updated (esp and stm code)

[Pete9008]

One other thought. It's possible that the stm has got into the fast baudrate logging mode and the esp isn't able to get it back out (I thought I'd covered all the cases to make sure it can recover but I may have missed one?). If this is the case then a stm reset or power cycle would be needed to get it back to normal comms baud rates. With the esp debug connected you should be able to see whether this is the case or not.

[mjc506]

Just seen this too, not tested it (but I'll forget if I don't write it down )
https://hackaday.io/project/5334-serial ... g-software

[Bigpie]

Does your stm32 branch contain any fixes? I cannot get it to cut out on my testing but here's some data.
https://drive.google.com/drive/folders/ ... sp=sharing

[Pete9008]

Can't decide if that's good news or not!

I think I put the fix for the PWM overflow in, it's possible that could have done it. See here for details if you want to try removing it viewtopic.php?p=47101#p47101

Edit - What was the cause of the comms problem?

[Pete9008]

Here's your logged data, opened no problem at all

Not looked at it in detail yet but initial impressions are good.

Edit - there are some odd things going on though:

There is something odd on the currents il1 and il2, there is a distinct difference between when accelerating (LHS of plot) and decelerating (RHS). The decel data is what I'd expect. Not sure what to make of the left hand acceleration side??

Edit2 - Also something not right in Id and Iq, they should follow IdRef and IqRef.

Not sure whether this is a real problem or a logger problem.

It looks like it is just when you are pushing on a bit more so wondering whether this is the throttle limiter which is in the latest code, might be worth commenting out the following lines in SetTorquePercent to disable it and trying it again (but don't push too hard as the control loops will then be able to go into saturation)?

Code: Select all

    //float maxThrot = throttleLimitController.RunProportionalOnly(Param::GetInt(Param::amp));

   //if (torquePercent > 0)
   //   torquePercent = MIN(maxThrot, torquePercent);
   //else
   //   torquePercent = -MIN(maxThrot, -torquePercent);

Edit - based on Johannes comments in the simulator thread removing the limiter sounds well worth a try.

[johu]

Wow, cool to see some live data. Left hand currents are super strange indeed

[Pete9008]

Wow, cool to see some live data.

Absolutely!

Edit - See above for comment on the currents (was adding it while you posted), what do you think?

[Pete9008]

Another strange bit in the center of the plot. The angle is unchanging so stationary but there is negative DC currents being recorded (-40A):

Could this be energy stored in the motor being returned to the battery, can it really last for 5sec, stop and then start again!?

@Bigpie - any idea what what happening here?

[Bigpie]

I dont, I did have to stop at some lights for a few seconds. I was driving pretty erratically trying to get it to cut out. I'll have a go at loading it up in PulseView. I'll get some more logs tomorrow and see if I can get it to cut out.

[Pete9008]

I dont, I did have to stop at some lights for a few seconds. I was driving pretty erratically trying to get it to cut out. I'll have a go at loading it up in PulseView. I'll get some more logs tomorrow and see if I can get it to cut out.

Thinking the cut-outs were the PWM overflow, the symptoms would be right. Good if it's fixed

What was the problem yesterday getting the ESP to talk to the STM?

[Bigpie]

I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.

How do I compile the application to convert to CSV to open in PusleView?

[Pete9008]

I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.

How do I compile the application to convert to CSV to open in PusleView?

I'll have another look at the ESP code and see if I can add something to ensure it recovers from this state.

In Qt create click New Project, select a Qt Console Application, give it a name and location and let the wizard complete and then copy the code into the main.cpp that will have been created. Should then just compile and run. Afraid it uses fixed file names at the moment (the input file needs to be renamed to logfile.bin) and assumes they are in the same directory as the executable (the build-something subdirectory of the Qt project directory if running within Qt).

Edit - did the wifi network test (as opposed to station mode) work OK?

Edit2 - Your'e using a later version of Qt than me, if you find it needs any changes let me know and I'll try to include them as conditional compilations in future versions.

[Pete9008]

I think it's getting in to binaryLogging and the loading the webpage doesn't get it out of it, causing the params to not load so I'd assumed it wasn't working and switched the RX and TX over. Switched them back and the logs had data in.

If I want the params to load, I have to power up get my laptop to connect to the ESP32, then restart the inverter to get the params list to load.

What would be useful in order to try and recreate the problem is a few more details on the network configuration when you had the problem. Are you using the ESP to provide the wifi network or is it connecting to the house or laptop network.

[Bigpie]

It was probably able to connect to house network, but I was connecting to the esp32 AP.

Both powering on at the same time.

[Pete9008]

Hmm, wondering whether the availability of the house network could have been confusing things. Not looked much at the ESPs wifi settings and have just let it default to AP mode. Do you manually configure which it does?

[Bigpie]

Not too sure, I think this

Code: Select all

  WiFi.mode(WIFI_AP_STA);

causes it to both create an AP and connect to a network if the credentials are stored. I'll have a look into it. More excited about the logging at the moment, and hoping the cutouts are cured.

PulseView doesn't seem to like running on my macbook pro.

EDIT, got pulseview working this is awesome. I don't understand some of the graphs, but they look good.

[Pete9008]

Might be worth clearing the stored credentials for the house and see if it behaves any better?

Afraid I can't help with the mac, never used one Does all the Qt stuff run ok on it?

The linux appimage loaded up no trouble at all (which is a bit of a first for my old Linux box which normally never works first time!)

[Pete9008]

EDIT, got pulseview working this is awesome. I don't understand some of the graphs, but they look good.

Good isn't it

Edit - need to calculate frq in the decoder and add it to the plots, it would make interpreting them much easier!

[Bigpie]

I'll get some more plots tomorrow, got a couple of trips to make, one of which includes some steep inclines on a NSL road.
'Pushing on a bit' was up to 40mph, but in first gear

Brain a bit too frazzled today to make too much sense, you think I hit the throttle limiter?

I think I have FW disabled still, but the motor still goes up to high RPM, the acceleration without cutting out took my by surprise, I did a little swearing.