ESP32 Based Web Interface & Data Logger

[Pete9008]

Just working on a program to convert the bin files to csv. In the process I've found a couple of issues with the binary file format (the json header didn't parse nicely and some of the bitfield widths are wrong).

I'll get these sorted in the next day or so but until then there's not much point trying to use the stm code.

[Bigpie]

No problem. Had to do a long journey yesterday, range of speeds, some pretty steep inclines etc. Had quite a few cutouts so looking forward to getting a look at the data. Got one more longer drive tonight, then I'm free to install untested inverter code

[Pete9008]

Stm code should now be sorted and changes have been pushed up to github.

Decoding app is coming on, it can now extract the json header from the binary file and use it to automatically decode and format the data fields. Just need to add some error checking and add the code to actually write the csv file.

[Pete9008]

Couple of plots showing captured data. Note, this is not via a SD card, the OpenInverter logging code is running inside the simulator which is saving the data (that would normally be sent out the serial port to the ESP32) directly to hard disk. It doesn't test the ESP32 side but does check that the data is being correctly compressed by the OpenInverter code and then decoded by the decoding app.

The test run in the simulator that created the log file:

And the result of decoding the logged file to a csv and graphing in a spreadsheet:

Fairly happy with that

The voltages are just the raw values used within the OpenInverter firmware, it should be possible to pull the battery voltage out of the params and scale it but I haven't bothered yet. Also the Iq and Id values are not shown as they need to be calculated from the angle and the phase currents i1 and i2 (will probably do this within the decode app eventually).

What this has shown is that a spreadsheet is not going to be any use at all for processing. The above is only 2 seconds of data (around 264000 measurement points) but must have taken around 10minutes to graph in the spreadsheet!

[mjc506]

Looks pretty though

Got a small portion of the csv available? I might have a go at getting it readable in Pulseview when I get a chance.

[Pete9008]

Here you go

decodedfile.csv

(1.1 MiB) Downloaded 58 times

This is the file plotted above. The first and last columns are the message count and message checksum fields and can be ignored (I'll probably replace the first one with calculated time, each line is 1/8800sec).

Will probably save the parameters as a separate file. Not sure whether to leave them in json format or convert to csv?

[mjc506]

Ta Not promising any time scales mind.. haha

Actually, that was far too easy. Pulseview has a csv import.

Column format string: 1-,5a,1l,1a,1l,6a,- (ignore the first column - can set the samplerate assuming it's fixed, or you can input a timestamp (in seconds), 5 analog columns, 1 logic, 1 ana, 1 logic, 6 ana, ignore last column)

Only 2 secs of data, but opens in a blink of an eye! Need to set axis scales (more work needed here - 'Angle' would be nice to convert to degrees, a few of the variables are larger than Pulseview imagined, so need to increase the range of divisors available. Should be an easy patch). But scrolling and zooming is nice and quick, get a readout of the (approximate) values on mouseover.

Are we getting overflow on ud and uq? Or is that something weird but real happening?

wrt parameters, I can imagine an 'importer' for Pulseview that reads the csv and scales stuff to 'real world' values based on the parameter file. Data and parameters in one file might be 'nicer', but a separate json wouldn't be a problem I don't think.

[Pete9008]

Wow, nice!

Looks like I need to get it installed. Is pulseview part of sigrok or something different?

Not sure on the overflow, it should clip rather than overflow (tbh I didnt look too closely at the data, just ran the simulation with defaults params so I know the loops would have been going unstable at higher speeds). It might just be the space vector modulation, it allows uq and ud to go a little higher than expected. It's also possible that it's overflowing in the logging code, I'll have a look.

Is there a pulseview native format that we could use, or a header that can be added to the csv, to save having to configure anything in pulseview itself? Also is it possible to overlay multiple lines on the same axis (for example the phase currents or iq and id)?

Sorry for all the questions but it does look like a good solution

Edit - one other thought, any idea how easy (or how difficult!) it is to add a custom importer to pulseview?

[mjc506]

Yeah, part of the sigrok 'suite', I guess you'd call it a QT gui for libsigrok? Or a GUI that includes libsigrok in the Windows builds. Either way...

Yeah, would be nice to import from csv without configuring anything. I'll look into it. Already found where to increase the max 'V/div' (https://github.com/sigrokproject/pulsev ... signal.cpp line 92 to '11' would do it) but a custom importer that divides things down into real world units would avoid needing to do that. Not sure if it's possible to overlay.

To be honest, there's so much in Pulseview that isn't needed, might be easiest to start from 'scratch' with a QT graphing library and go from there... What else would be useful? Calculating 3rd phase volts/currents? DC/AC watts? I'll have a play with that too, "how hard can it be" Is it worth logging udc, temps, throttle command etc?

Edit: On the other hand, PV has Android builds, can decode SPI, UART, CAN logic signals etc, might be more useful generally? Will play with importers first...

[Pete9008]

I was thinking more of ditching the separate decoder and trying to import the bin file directly into pulseview. The documentation mentions a python api for adding decoders, any idea whether that can add importers too?

I've used the native qt graphing in the simulator and it isn't as quick as pulseview sounds. It can cope with tens of seconds of simulator data but it would choke on a 10min log. Its also a fair chunk of work to get nice synced scrolling, zooming, hover over, etc. Using the pulseview display routines makes a lot of sense.

Its tricky to log more parameters using this scheme as its using all the serial bandwidth as it is. If we want other parameters the Johannes binary parameter streaming is the way to go. A custom pulseview based app talking over tcp could work for that though?

For the time being Ill add to the standalone decoder to scale uq and ud, calculate i3, iq and id, scale angle to degrees and provide a real time stamp which will give something usable now but longer term having a single app to decode and display would be nice.

Edit - assume the is also a way to tell it that some analogues are amps not volts?

Edit2 - like both the idea of using a standard app (90% of the features, 0% of the work, multi platform support, no on going maintenance and immediately available) but also like the idea of a single custom app that can do everything!

Edit3 - slight issue with scaling ud and uq, udcnom is only for sine builds not for foc so not sure what param to use to scale ud and uq. The average of udcmin and udcmax? Or move udcnom to the common section? Trouble with the later is its default value is 0 and it is unlikely to have been changed in foc installs.

[mjc506]

Ah, gotcha. Sounds like it might be possible to create a '.sr' file with all the data and everything pre-set: https://sigrok.org/wiki/File_format:Sigrok/v2

I don't think the decoders can import (but could do the scaling perhaps). The csv import at least is in libsigrok, but it wouldn't be a massive effort to fork it and do a new importer based on a hacked up csv version.

Ah, of course, I'd forgotten serial bandwidth

Streaming the tcp direct to Pulseview would be lovely... I'm guessing that'll be in libsigrok too, will take a look.

A quick look at the .sr file format seems to say we can define units. Must be doable.

No udcnom in FOC? I was sure it was there... although it's been a while since I've played with an FOC build...

[Pete9008]

The .sr file may be the way to go. What Ill do is get the current csv to a point where it usable. Can then import it and set things up in pulseview and save in .sr format to get a template. Can then have a look to see whether the standalone decoder could generate that directly.

Not sure about udcnom, it's in the MOTOR_PARAMETERS_SINE section of the param file. Having thought about it I'm not going to convert ud and uq to volts, there is just too much scope for confusion (a badly set parameter or variation in voltage with soc). Instead Ill scale them to +-100% to fit with the standard pulseview scaling and hopefully be reasonably intuitive.

Edit, might make sense to scale the PWM values to +-100% too?

[Pete9008]

Are we getting overflow on ud and uq? Or is that something weird but real happening?

You were right on the overflow, the PWM can go over by one bit. Before:

And after fixing:

The change needed is in foc.cpp on line 154. Need to change

Code: Select all

else if (DutyCycles[i] > maxPulse)
      {
         DutyCycles[i] = FP_FROMINT(2);
      }

to

Code: Select all

else if (DutyCycles[i] > maxPulse)
      {
         DutyCycles[i] = (FP_FROMINT(2)-1);
      }

Think there is still potential for overflow on these lines too, not sure whether they are the cause of the above test being true (it should never be)?

Code: Select all

   s32fp ua = (cos * ud - sin * uq) >> CST_DIGITS;
   s32fp ub = (cos * uq + sin * ud) >> CST_DIGITS;

Johannes did suggest a change for these previously which I need to find and include too.

Edit - here is the suggested fix viewtopic.php?p=44583#p44583, removed the -1 above and tried this instead and still get the overflow. @johu - probably safest to include both fixes going forward?

[Pete9008]

Think that's about it for now

The column format spec is now "1a,5a,1l,1a,1l,6a,1-,2a". It still needs the frequency adding. The message count is now included on screen as a sawtooth waveform; it should be visually pretty clear if there is a discontinuity there caused my missing/corrupted messages.

The stm source is updated in github to fix the bugs mentioned above and the decoding app now formats everything as described in the above posts. It also calculates iq and id and includes them in the csv. It will need a LOT of tidying up at some point, and probably a gui adding to allow filenames to be selected, but it will do for now. It's not worth creating a github repo for this code, it's just one qt file, so will just include it here instead:

Code: Select all

#include <QCoreApplication>
#include <QFile>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <QJsonValue>
#include <QDebug>
#include <QVector>
#include <QtMath>

#define MODMAX (((2U<<15)/1.732050807568877293527446315059) - 200)

struct varDefinitions {
  QString name;
  double scale;
  int bits;
  bool signExtend;
} ;



int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QVector<varDefinitions> varDefs;

    QFile logFile("logfile.bin");
    QFile outFile("decodedfile.csv");

    QByteArray jsonHeader;
    char val;
    int paraCount = -1;
    int messageBits = 0, messageBytes = 0;
    char buffer[25];
    int index = 0;
    int bitsHave = 8;
    uint32_t value = 0;
    uint32_t bitStore = 0;
    uint32_t mask = 0;
    uint32_t messageCount = 0;
    int haveAngle = -1, haveI1 = -1, haveI2= -1;

    logFile.open(QFile::ReadOnly);


    if(logFile.isOpen())
    {
        //read and save parameters
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
        QFile paramFile("paramfile.json");
        paramFile.open(QFile::WriteOnly);
        if(paramFile.isOpen())
        {
            paramFile.write(jsonHeader);
            paramFile.close();
            qDebug("Param file written");
        }
        else
            qDebug("Could not write param file");

        jsonHeader.clear();
        do logFile.read(&val, 1); while(val != '{');
        if(val == '{')
        {
            paraCount=1;
            jsonHeader.append(val);
            do
            {
                if(logFile.read(&val, 1)==1)
                {
                    jsonHeader.append(val);
                    if(val == '{')
                        paraCount++;
                    else if(val == '}')
                        paraCount--;
                }
                else
                    break;
            }
            while(paraCount > 0);
        }
     }
    else
        qDebug("Could not open input file");

    if(paraCount == 0)
    {
        logFile.read(&val, 1); //discard json null terminator

        QJsonDocument jsonResponse = QJsonDocument::fromJson(jsonHeader);
        QJsonObject jsonObject = jsonResponse.object();
        varDefinitions def;
        foreach(const QString& key, jsonObject.keys())
        {
            QJsonObject jsonObject2 = jsonObject[key].toObject();
            foreach(const QString& key2, jsonObject2.keys())
            {
                if(key2 == "name")
                    def.name = jsonObject2[key2].toString();
                else if(key2 == "scale")
                    def.scale = jsonObject2[key2].toDouble();
                else if(key2 == "signed")
                    def.signExtend = jsonObject2[key2].toInt();
                else if(key2 == "size")
                {
                    def.bits = jsonObject2[key2].toInt();
                    messageBits += def.bits;
                }
            }
            if(def.name == "angle")
                haveAngle = varDefs.size();
            else if(def.name == "i1")
                haveI1 = varDefs.size();
            else if(def.name == "i2")
                haveI2 = varDefs.size();

            varDefs.append(def);
        }

        outFile.open(QFile::WriteOnly);

        if(outFile.isOpen())
        {
            QTextStream outStream(&outFile);
            QVectorIterator<varDefinitions> i(varDefs);
            while (i.hasNext())
                outStream << i.next().name << ',';
            if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
            {
                outStream << "iq" << ',' << "id" << ',';
            }
            outStream << "\n";

            messageBytes = (messageBits+7)/8;

            if(messageBytes <=25)
            {
                while(logFile.peek(buffer, messageBytes)==messageBytes)
                {
                    QVector<double> values(varDefs.size());
                    //is csum valid?
                    uint8_t csum = 0;
                    int i;
                    for(i=0;i<messageBytes-1;i++)
                        csum += (uint8_t)buffer[i];
                    if(csum == (uint8_t)buffer[i])
                    { //match, we have a valid message so extract data
                        index = 0;
                        bitsHave = 8;
                        value = 0;
                        bitStore = (uint32_t)((uint8_t)buffer[index++]);
                        for(i=0;i<varDefs.size();i++)
                        {
                            int bitsNeeded = varDefs[i].bits;
                            while(bitsHave < bitsNeeded)
                            {
                                bitStore = (((uint32_t)((uint8_t)buffer[index++]))<<bitsHave) + bitStore;
                                bitsHave = bitsHave + 8;
                            }
                            mask = ~(0xffffffff<<bitsNeeded);
                            value = bitStore & mask;
                            if(varDefs[i].signExtend)
                            {
                                mask = 0x01<<(bitsNeeded-1);
                                if((value & mask) != 0) //msb set so extend
                                {
                                    mask = 0xffffffff<<bitsNeeded;
                                    value = value | mask;
                                    values[i] = ((int32_t)value) * varDefs[i].scale;
                                }
                                else //not set do don't
                                    values[i] = value * varDefs[i].scale;
                            }
                            else
                                values[i] = value * varDefs[i].scale;
                            if(varDefs[i].name == "angle")
                                values[i] = (360.0 * (values[i]/65535));
                            if((varDefs[i].name == "ud") || (varDefs[i].name == "uq"))
                                values[i] = (100.0 * (values[i]/MODMAX));
                            if((varDefs[i].name == "pwm1") || (varDefs[i].name == "pwm2") || (varDefs[i].name == "pwm3"))
                                values[i] = (100.0 * ((values[i]-8192)/8192));
                            bitStore = bitStore >> bitsNeeded;
                            bitsHave = bitsHave - bitsNeeded;
                        }
                        logFile.read(buffer, messageBytes); //discard the data

                        QVectorIterator<double> i(values);
                        while (i.hasNext())
                            outStream << i.next() << ',';
                        if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
                        {
                            double angle = values[haveAngle];
                            double ia = values[haveI1];
                            double ib = ((values[haveI1]+(2.0*values[haveI2]))/qSqrt(3.0));
                            double id = (ia * qCos(qDegreesToRadians(angle))) + (ib * qSin(qDegreesToRadians(angle)));
                            double iq = (-ia * qSin(qDegreesToRadians(angle))) + (ib * qCos(qDegreesToRadians(angle)));
                            outStream << iq << ',' << id << ',';
                        }
                        outStream << "\n";
                        messageCount++;
                    }
                    else //no match so throw a byte away and try again
                    {
                        logFile.read(&val, 1);
                        qDebug("Mesage Lost");
                    }
                }
                qDebug("Processed %i messages",messageCount);
            }
            else
                qDebug("Json header message length invalid");
        }
        else
            qDebug("Could not open output file");
    }
    else
        qDebug("Could not find json header");

    logFile.close();
    outFile.close();
    return a.exec();
}

Please don't look too closely at the code, I know it's untidy and could probably be better structured, implemented, commented, etc., but it will do for now (especially if it gets ported to a pulseview importer)

Big thanks to mjc506 for suggesting pulseview, it looks ideal for this

Edit - Ohh, and here is the latest format csv in case anyone wants it for testing.

decodedfile.csv

(1.63 MiB) Downloaded 47 times

Edit2 - just had a look at the pulseview .sr file. It has one metadata file which contains details of the waveforms displayed (can't see any units or graph scaling data though) and then a binary file containing the data for each waveform. The waveform appears to be stored as sequential 32-bit floats, one for each measurement point, so should be fairly easy to create. Also worth noting is that the .sr file is around a third the size of the .csv file!

Edit3 - for reference https://sigrok.org/wiki/File_format:Sigrok/v2

[mjc506]

Awesome, fantastic work

Ah, apologies, might have been the WIP v3 sigrok format that had units and scaling. Or I'm imagining things! Your app fixes the scaling though, and having 'V' floating around isn't the end of the world really. (Just name each 'channel' as "IDC (Amps)" or something worst case )

It doesn't look toooooo hard to create an SR file? I think the metadata file includes the size of the binary one, so a bit of counting etc, but the savings in file size might be worth while...

Edit:
Ah, apparently it's the libsigrok "device" that includes units, scaling etc. Doesn't look like PulseView pays any attention (yet) though, certainly to units. It's either Time, Logic, or Volts haha. The Sigrok wiki claims there is 'an easy API' for new devices/input modules... I'm yet to understand the csv one yet hahah

[Pete9008]

The .sr file looks like the best way to go as it avoids the need to remember configuration strings and set the frequency. Creating the files should be pretty easy, not sure how to do the zip in a platform independent way though?

For now what we've got is perfectly usable. It's also a lot better, and took a lot less time, than I had expected, pulseview works really well for this
The only downside is it makes the graphing on the simulator seem pretty crude by comparison

Edit - there is one other thing that might be worth adding, at the moment it only works with foc builds. It would make sense to add sine support too but I don't have a clue whether there is a real need and if so what fields would be useful to log?

[mjc506]

Does seem to be the route of least resistance. I'm certainly no expert, but my first thought would be lib7zip or similar. Far too powerful, but open and multiplatform.

They've put a lot of effort into showing many many points while keeping things quick! I understand none of that code haha

I've only ever used sine for lightbulb tests to be honest. Not sure what'd be useful, but seems fairly straight forward to add in later if needed?

[Pete9008]

TBH I was expecting the graphing side to be a bit of a nightmare. Going back a few years there was a project at work that needed fast graphing of large datasets, I lost track of how much time we spent on it and how many different commercial libraries we tried before finding a solution. I reckon that pulseview is possibly faster than the one we ended up paying for! Very relieved that just works so thanks again for suggesting and trying it

The sine support would be easy to add, just a new structure definition and function to fill it. Unless anyone expresses an interest will leave it for now though.

Looking forward to trying it on real data!

[Bigpie]

Was hoping to have some data today, esp32 board all works on the desk with a spare inverter board, put it in the car and it doesn't load params or log to sdcard :S will have to troubleshoot over the weekend.

[Pete9008]

Wondering whether its a power up timing issue, if the ESP powers up first it might time out before the stm powers up. It might be worth trying a few second delay at the start of the esp setup function?

Thinking about it my stm board is running without the bootloader which means I wouldnt have noticed it and makes the above seem more likely.

[Pete9008]

Think I can reproduce the problem and changing the delay on line 726 of esp32-webinterface.ino from 100ms to 10000ms seems to fix it. Bit hacky but should do for now. Lower values would probably work too but difficult to test properly without the bootloader on mine.

Sorry about that, should have tested a bit more thoroughly

Edit - does the modded stm code run the motor ok?

[Bigpie]

Seemed to run fine, only did a few short runs. Will test, hopefully shortly.

[Pete9008]

Great, looking forward to seeing some data

Thinking about it a better fix would be an extra if(millis() > DELAY_VAL) test on he start logging condition. That would give the same result but still allow the wifi and web page to come up faster.

[Bigpie]

I've added

Code: Select all

  if((WiFi.softAPgetStationNum() > 0) || (WiFi.status() == WL_CONNECTED || startMillis + DELAY_VAL < millis()))

where startMillis is set in the setup.

[Pete9008]

I've added

Code: Select all

  if((WiFi.softAPgetStationNum() > 0) || (WiFi.status() == WL_CONNECTED || startMillis + DELAY_VAL < millis()))

where startMillis is set in the setup.

Shouldn't that be > millis() if youre putting it there?

You might not need startMillis (I think millis() starts from 0 on boot?)