ESP32 Based Web Interface & Data Logger
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Just working on a program to convert the bin files to csv. In the process I've found a couple of issues with the binary file format (the json header didn't parse nicely and some of the bitfield widths are wrong).
I'll get these sorted in the next day or so but until then there's not much point trying to use the stm code.
I'll get these sorted in the next day or so but until then there's not much point trying to use the stm code.
- Bigpie
- Posts: 1595
- Joined: Wed Apr 10, 2019 8:11 pm
- Location: South Yorkshire, UK
- Has thanked: 75 times
- Been thanked: 304 times
Re: ESP32 Based Web Interface & Data Logger
No problem. Had to do a long journey yesterday, range of speeds, some pretty steep inclines etc. Had quite a few cutouts so looking forward to getting a look at the data. Got one more longer drive tonight, then I'm free to install untested inverter code
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Stm code should now be sorted and changes have been pushed up to github.
Decoding app is coming on, it can now extract the json header from the binary file and use it to automatically decode and format the data fields. Just need to add some error checking and add the code to actually write the csv file.
Decoding app is coming on, it can now extract the json header from the binary file and use it to automatically decode and format the data fields. Just need to add some error checking and add the code to actually write the csv file.
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Couple of plots showing captured data. Note, this is not via a SD card, the OpenInverter logging code is running inside the simulator which is saving the data (that would normally be sent out the serial port to the ESP32) directly to hard disk. It doesn't test the ESP32 side but does check that the data is being correctly compressed by the OpenInverter code and then decoded by the decoding app.
The test run in the simulator that created the log file: And the result of decoding the logged file to a csv and graphing in a spreadsheet: Fairly happy with that
The voltages are just the raw values used within the OpenInverter firmware, it should be possible to pull the battery voltage out of the params and scale it but I haven't bothered yet. Also the Iq and Id values are not shown as they need to be calculated from the angle and the phase currents i1 and i2 (will probably do this within the decode app eventually).
What this has shown is that a spreadsheet is not going to be any use at all for processing. The above is only 2 seconds of data (around 264000 measurement points) but must have taken around 10minutes to graph in the spreadsheet!
The test run in the simulator that created the log file: And the result of decoding the logged file to a csv and graphing in a spreadsheet: Fairly happy with that
The voltages are just the raw values used within the OpenInverter firmware, it should be possible to pull the battery voltage out of the params and scale it but I haven't bothered yet. Also the Iq and Id values are not shown as they need to be calculated from the angle and the phase currents i1 and i2 (will probably do this within the decode app eventually).
What this has shown is that a spreadsheet is not going to be any use at all for processing. The above is only 2 seconds of data (around 264000 measurement points) but must have taken around 10minutes to graph in the spreadsheet!
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Looks pretty though
Got a small portion of the csv available? I might have a go at getting it readable in Pulseview when I get a chance.
Got a small portion of the csv available? I might have a go at getting it readable in Pulseview when I get a chance.
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Here you go
Will probably save the parameters as a separate file. Not sure whether to leave them in json format or convert to csv?
This is the file plotted above. The first and last columns are the message count and message checksum fields and can be ignored (I'll probably replace the first one with calculated time, each line is 1/8800sec).Will probably save the parameters as a separate file. Not sure whether to leave them in json format or convert to csv?
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Ta Not promising any time scales mind.. haha
Actually, that was far too easy. Pulseview has a csv import. Column format string: 1-,5a,1l,1a,1l,6a,- (ignore the first column - can set the samplerate assuming it's fixed, or you can input a timestamp (in seconds), 5 analog columns, 1 logic, 1 ana, 1 logic, 6 ana, ignore last column)
Only 2 secs of data, but opens in a blink of an eye! Need to set axis scales (more work needed here - 'Angle' would be nice to convert to degrees, a few of the variables are larger than Pulseview imagined, so need to increase the range of divisors available. Should be an easy patch). But scrolling and zooming is nice and quick, get a readout of the (approximate) values on mouseover. Are we getting overflow on ud and uq? Or is that something weird but real happening? wrt parameters, I can imagine an 'importer' for Pulseview that reads the csv and scales stuff to 'real world' values based on the parameter file. Data and parameters in one file might be 'nicer', but a separate json wouldn't be a problem I don't think.
Actually, that was far too easy. Pulseview has a csv import. Column format string: 1-,5a,1l,1a,1l,6a,- (ignore the first column - can set the samplerate assuming it's fixed, or you can input a timestamp (in seconds), 5 analog columns, 1 logic, 1 ana, 1 logic, 6 ana, ignore last column)
Only 2 secs of data, but opens in a blink of an eye! Need to set axis scales (more work needed here - 'Angle' would be nice to convert to degrees, a few of the variables are larger than Pulseview imagined, so need to increase the range of divisors available. Should be an easy patch). But scrolling and zooming is nice and quick, get a readout of the (approximate) values on mouseover. Are we getting overflow on ud and uq? Or is that something weird but real happening? wrt parameters, I can imagine an 'importer' for Pulseview that reads the csv and scales stuff to 'real world' values based on the parameter file. Data and parameters in one file might be 'nicer', but a separate json wouldn't be a problem I don't think.
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Wow, nice!
Looks like I need to get it installed. Is pulseview part of sigrok or something different?
Not sure on the overflow, it should clip rather than overflow (tbh I didnt look too closely at the data, just ran the simulation with defaults params so I know the loops would have been going unstable at higher speeds). It might just be the space vector modulation, it allows uq and ud to go a little higher than expected. It's also possible that it's overflowing in the logging code, I'll have a look.
Is there a pulseview native format that we could use, or a header that can be added to the csv, to save having to configure anything in pulseview itself? Also is it possible to overlay multiple lines on the same axis (for example the phase currents or iq and id)?
Sorry for all the questions but it does look like a good solution
Edit - one other thought, any idea how easy (or how difficult!) it is to add a custom importer to pulseview?
Looks like I need to get it installed. Is pulseview part of sigrok or something different?
Not sure on the overflow, it should clip rather than overflow (tbh I didnt look too closely at the data, just ran the simulation with defaults params so I know the loops would have been going unstable at higher speeds). It might just be the space vector modulation, it allows uq and ud to go a little higher than expected. It's also possible that it's overflowing in the logging code, I'll have a look.
Is there a pulseview native format that we could use, or a header that can be added to the csv, to save having to configure anything in pulseview itself? Also is it possible to overlay multiple lines on the same axis (for example the phase currents or iq and id)?
Sorry for all the questions but it does look like a good solution
Edit - one other thought, any idea how easy (or how difficult!) it is to add a custom importer to pulseview?
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Yeah, part of the sigrok 'suite', I guess you'd call it a QT gui for libsigrok? Or a GUI that includes libsigrok in the Windows builds. Either way...
Yeah, would be nice to import from csv without configuring anything. I'll look into it. Already found where to increase the max 'V/div' (https://github.com/sigrokproject/pulsev ... signal.cpp line 92 to '11' would do it) but a custom importer that divides things down into real world units would avoid needing to do that. Not sure if it's possible to overlay.
To be honest, there's so much in Pulseview that isn't needed, might be easiest to start from 'scratch' with a QT graphing library and go from there... What else would be useful? Calculating 3rd phase volts/currents? DC/AC watts? I'll have a play with that too, "how hard can it be" Is it worth logging udc, temps, throttle command etc?
Edit: On the other hand, PV has Android builds, can decode SPI, UART, CAN logic signals etc, might be more useful generally? Will play with importers first...
Yeah, would be nice to import from csv without configuring anything. I'll look into it. Already found where to increase the max 'V/div' (https://github.com/sigrokproject/pulsev ... signal.cpp line 92 to '11' would do it) but a custom importer that divides things down into real world units would avoid needing to do that. Not sure if it's possible to overlay.
To be honest, there's so much in Pulseview that isn't needed, might be easiest to start from 'scratch' with a QT graphing library and go from there... What else would be useful? Calculating 3rd phase volts/currents? DC/AC watts? I'll have a play with that too, "how hard can it be" Is it worth logging udc, temps, throttle command etc?
Edit: On the other hand, PV has Android builds, can decode SPI, UART, CAN logic signals etc, might be more useful generally? Will play with importers first...
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
I was thinking more of ditching the separate decoder and trying to import the bin file directly into pulseview. The documentation mentions a python api for adding decoders, any idea whether that can add importers too?
I've used the native qt graphing in the simulator and it isn't as quick as pulseview sounds. It can cope with tens of seconds of simulator data but it would choke on a 10min log. Its also a fair chunk of work to get nice synced scrolling, zooming, hover over, etc. Using the pulseview display routines makes a lot of sense.
Its tricky to log more parameters using this scheme as its using all the serial bandwidth as it is. If we want other parameters the Johannes binary parameter streaming is the way to go. A custom pulseview based app talking over tcp could work for that though?
For the time being Ill add to the standalone decoder to scale uq and ud, calculate i3, iq and id, scale angle to degrees and provide a real time stamp which will give something usable now but longer term having a single app to decode and display would be nice.
Edit - assume the is also a way to tell it that some analogues are amps not volts?
Edit2 - like both the idea of using a standard app (90% of the features, 0% of the work, multi platform support, no on going maintenance and immediately available) but also like the idea of a single custom app that can do everything!
Edit3 - slight issue with scaling ud and uq, udcnom is only for sine builds not for foc so not sure what param to use to scale ud and uq. The average of udcmin and udcmax? Or move udcnom to the common section? Trouble with the later is its default value is 0 and it is unlikely to have been changed in foc installs.
I've used the native qt graphing in the simulator and it isn't as quick as pulseview sounds. It can cope with tens of seconds of simulator data but it would choke on a 10min log. Its also a fair chunk of work to get nice synced scrolling, zooming, hover over, etc. Using the pulseview display routines makes a lot of sense.
Its tricky to log more parameters using this scheme as its using all the serial bandwidth as it is. If we want other parameters the Johannes binary parameter streaming is the way to go. A custom pulseview based app talking over tcp could work for that though?
For the time being Ill add to the standalone decoder to scale uq and ud, calculate i3, iq and id, scale angle to degrees and provide a real time stamp which will give something usable now but longer term having a single app to decode and display would be nice.
Edit - assume the is also a way to tell it that some analogues are amps not volts?
Edit2 - like both the idea of using a standard app (90% of the features, 0% of the work, multi platform support, no on going maintenance and immediately available) but also like the idea of a single custom app that can do everything!
Edit3 - slight issue with scaling ud and uq, udcnom is only for sine builds not for foc so not sure what param to use to scale ud and uq. The average of udcmin and udcmax? Or move udcnom to the common section? Trouble with the later is its default value is 0 and it is unlikely to have been changed in foc installs.
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Ah, gotcha. Sounds like it might be possible to create a '.sr' file with all the data and everything pre-set: https://sigrok.org/wiki/File_format:Sigrok/v2
I don't think the decoders can import (but could do the scaling perhaps). The csv import at least is in libsigrok, but it wouldn't be a massive effort to fork it and do a new importer based on a hacked up csv version.
Ah, of course, I'd forgotten serial bandwidth
Streaming the tcp direct to Pulseview would be lovely... I'm guessing that'll be in libsigrok too, will take a look.
A quick look at the .sr file format seems to say we can define units. Must be doable.
No udcnom in FOC? I was sure it was there... although it's been a while since I've played with an FOC build...
I don't think the decoders can import (but could do the scaling perhaps). The csv import at least is in libsigrok, but it wouldn't be a massive effort to fork it and do a new importer based on a hacked up csv version.
Ah, of course, I'd forgotten serial bandwidth
Streaming the tcp direct to Pulseview would be lovely... I'm guessing that'll be in libsigrok too, will take a look.
A quick look at the .sr file format seems to say we can define units. Must be doable.
No udcnom in FOC? I was sure it was there... although it's been a while since I've played with an FOC build...
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
The .sr file may be the way to go. What Ill do is get the current csv to a point where it usable. Can then import it and set things up in pulseview and save in .sr format to get a template. Can then have a look to see whether the standalone decoder could generate that directly.
Not sure about udcnom, it's in the MOTOR_PARAMETERS_SINE section of the param file. Having thought about it I'm not going to convert ud and uq to volts, there is just too much scope for confusion (a badly set parameter or variation in voltage with soc). Instead Ill scale them to +-100% to fit with the standard pulseview scaling and hopefully be reasonably intuitive.
Edit, might make sense to scale the PWM values to +-100% too?
Not sure about udcnom, it's in the MOTOR_PARAMETERS_SINE section of the param file. Having thought about it I'm not going to convert ud and uq to volts, there is just too much scope for confusion (a badly set parameter or variation in voltage with soc). Instead Ill scale them to +-100% to fit with the standard pulseview scaling and hopefully be reasonably intuitive.
Edit, might make sense to scale the PWM values to +-100% too?
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
You were right on the overflow, the PWM can go over by one bit. Before: And after fixing: The change needed is in foc.cpp on line 154. Need to change
Code: Select all
else if (DutyCycles[i] > maxPulse)
{
DutyCycles[i] = FP_FROMINT(2);
}
Code: Select all
else if (DutyCycles[i] > maxPulse)
{
DutyCycles[i] = (FP_FROMINT(2)-1);
}
Code: Select all
s32fp ua = (cos * ud - sin * uq) >> CST_DIGITS;
s32fp ub = (cos * uq + sin * ud) >> CST_DIGITS;
Edit - here is the suggested fix viewtopic.php?p=44583#p44583, removed the -1 above and tried this instead and still get the overflow. @johu - probably safest to include both fixes going forward?
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Think that's about it for now
The column format spec is now "1a,5a,1l,1a,1l,6a,1-,2a". It still needs the frequency adding. The message count is now included on screen as a sawtooth waveform; it should be visually pretty clear if there is a discontinuity there caused my missing/corrupted messages.
The stm source is updated in github to fix the bugs mentioned above and the decoding app now formats everything as described in the above posts. It also calculates iq and id and includes them in the csv. It will need a LOT of tidying up at some point, and probably a gui adding to allow filenames to be selected, but it will do for now. It's not worth creating a github repo for this code, it's just one qt file, so will just include it here instead:
Please don't look too closely at the code, I know it's untidy and could probably be better structured, implemented, commented, etc., but it will do for now (especially if it gets ported to a pulseview importer)
Big thanks to mjc506 for suggesting pulseview, it looks ideal for this
Edit - Ohh, and here is the latest format csv in case anyone wants it for testing. Edit2 - just had a look at the pulseview .sr file. It has one metadata file which contains details of the waveforms displayed (can't see any units or graph scaling data though) and then a binary file containing the data for each waveform. The waveform appears to be stored as sequential 32-bit floats, one for each measurement point, so should be fairly easy to create. Also worth noting is that the .sr file is around a third the size of the .csv file!
Edit3 - for reference https://sigrok.org/wiki/File_format:Sigrok/v2
The column format spec is now "1a,5a,1l,1a,1l,6a,1-,2a". It still needs the frequency adding. The message count is now included on screen as a sawtooth waveform; it should be visually pretty clear if there is a discontinuity there caused my missing/corrupted messages.
The stm source is updated in github to fix the bugs mentioned above and the decoding app now formats everything as described in the above posts. It also calculates iq and id and includes them in the csv. It will need a LOT of tidying up at some point, and probably a gui adding to allow filenames to be selected, but it will do for now. It's not worth creating a github repo for this code, it's just one qt file, so will just include it here instead:
Code: Select all
#include <QCoreApplication>
#include <QFile>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <QJsonValue>
#include <QDebug>
#include <QVector>
#include <QtMath>
#define MODMAX (((2U<<15)/1.732050807568877293527446315059) - 200)
struct varDefinitions {
QString name;
double scale;
int bits;
bool signExtend;
} ;
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
QVector<varDefinitions> varDefs;
QFile logFile("logfile.bin");
QFile outFile("decodedfile.csv");
QByteArray jsonHeader;
char val;
int paraCount = -1;
int messageBits = 0, messageBytes = 0;
char buffer[25];
int index = 0;
int bitsHave = 8;
uint32_t value = 0;
uint32_t bitStore = 0;
uint32_t mask = 0;
uint32_t messageCount = 0;
int haveAngle = -1, haveI1 = -1, haveI2= -1;
logFile.open(QFile::ReadOnly);
if(logFile.isOpen())
{
//read and save parameters
do logFile.read(&val, 1); while(val != '{');
if(val == '{')
{
paraCount=1;
jsonHeader.append(val);
do
{
if(logFile.read(&val, 1)==1)
{
jsonHeader.append(val);
if(val == '{')
paraCount++;
else if(val == '}')
paraCount--;
}
else
break;
}
while(paraCount > 0);
}
QFile paramFile("paramfile.json");
paramFile.open(QFile::WriteOnly);
if(paramFile.isOpen())
{
paramFile.write(jsonHeader);
paramFile.close();
qDebug("Param file written");
}
else
qDebug("Could not write param file");
jsonHeader.clear();
do logFile.read(&val, 1); while(val != '{');
if(val == '{')
{
paraCount=1;
jsonHeader.append(val);
do
{
if(logFile.read(&val, 1)==1)
{
jsonHeader.append(val);
if(val == '{')
paraCount++;
else if(val == '}')
paraCount--;
}
else
break;
}
while(paraCount > 0);
}
}
else
qDebug("Could not open input file");
if(paraCount == 0)
{
logFile.read(&val, 1); //discard json null terminator
QJsonDocument jsonResponse = QJsonDocument::fromJson(jsonHeader);
QJsonObject jsonObject = jsonResponse.object();
varDefinitions def;
foreach(const QString& key, jsonObject.keys())
{
QJsonObject jsonObject2 = jsonObject[key].toObject();
foreach(const QString& key2, jsonObject2.keys())
{
if(key2 == "name")
def.name = jsonObject2[key2].toString();
else if(key2 == "scale")
def.scale = jsonObject2[key2].toDouble();
else if(key2 == "signed")
def.signExtend = jsonObject2[key2].toInt();
else if(key2 == "size")
{
def.bits = jsonObject2[key2].toInt();
messageBits += def.bits;
}
}
if(def.name == "angle")
haveAngle = varDefs.size();
else if(def.name == "i1")
haveI1 = varDefs.size();
else if(def.name == "i2")
haveI2 = varDefs.size();
varDefs.append(def);
}
outFile.open(QFile::WriteOnly);
if(outFile.isOpen())
{
QTextStream outStream(&outFile);
QVectorIterator<varDefinitions> i(varDefs);
while (i.hasNext())
outStream << i.next().name << ',';
if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
{
outStream << "iq" << ',' << "id" << ',';
}
outStream << "\n";
messageBytes = (messageBits+7)/8;
if(messageBytes <=25)
{
while(logFile.peek(buffer, messageBytes)==messageBytes)
{
QVector<double> values(varDefs.size());
//is csum valid?
uint8_t csum = 0;
int i;
for(i=0;i<messageBytes-1;i++)
csum += (uint8_t)buffer[i];
if(csum == (uint8_t)buffer[i])
{ //match, we have a valid message so extract data
index = 0;
bitsHave = 8;
value = 0;
bitStore = (uint32_t)((uint8_t)buffer[index++]);
for(i=0;i<varDefs.size();i++)
{
int bitsNeeded = varDefs[i].bits;
while(bitsHave < bitsNeeded)
{
bitStore = (((uint32_t)((uint8_t)buffer[index++]))<<bitsHave) + bitStore;
bitsHave = bitsHave + 8;
}
mask = ~(0xffffffff<<bitsNeeded);
value = bitStore & mask;
if(varDefs[i].signExtend)
{
mask = 0x01<<(bitsNeeded-1);
if((value & mask) != 0) //msb set so extend
{
mask = 0xffffffff<<bitsNeeded;
value = value | mask;
values[i] = ((int32_t)value) * varDefs[i].scale;
}
else //not set do don't
values[i] = value * varDefs[i].scale;
}
else
values[i] = value * varDefs[i].scale;
if(varDefs[i].name == "angle")
values[i] = (360.0 * (values[i]/65535));
if((varDefs[i].name == "ud") || (varDefs[i].name == "uq"))
values[i] = (100.0 * (values[i]/MODMAX));
if((varDefs[i].name == "pwm1") || (varDefs[i].name == "pwm2") || (varDefs[i].name == "pwm3"))
values[i] = (100.0 * ((values[i]-8192)/8192));
bitStore = bitStore >> bitsNeeded;
bitsHave = bitsHave - bitsNeeded;
}
logFile.read(buffer, messageBytes); //discard the data
QVectorIterator<double> i(values);
while (i.hasNext())
outStream << i.next() << ',';
if((haveAngle>=0) && (haveI1>=0) && (haveI2>=0))
{
double angle = values[haveAngle];
double ia = values[haveI1];
double ib = ((values[haveI1]+(2.0*values[haveI2]))/qSqrt(3.0));
double id = (ia * qCos(qDegreesToRadians(angle))) + (ib * qSin(qDegreesToRadians(angle)));
double iq = (-ia * qSin(qDegreesToRadians(angle))) + (ib * qCos(qDegreesToRadians(angle)));
outStream << iq << ',' << id << ',';
}
outStream << "\n";
messageCount++;
}
else //no match so throw a byte away and try again
{
logFile.read(&val, 1);
qDebug("Mesage Lost");
}
}
qDebug("Processed %i messages",messageCount);
}
else
qDebug("Json header message length invalid");
}
else
qDebug("Could not open output file");
}
else
qDebug("Could not find json header");
logFile.close();
outFile.close();
return a.exec();
}
Big thanks to mjc506 for suggesting pulseview, it looks ideal for this
Edit - Ohh, and here is the latest format csv in case anyone wants it for testing. Edit2 - just had a look at the pulseview .sr file. It has one metadata file which contains details of the waveforms displayed (can't see any units or graph scaling data though) and then a binary file containing the data for each waveform. The waveform appears to be stored as sequential 32-bit floats, one for each measurement point, so should be fairly easy to create. Also worth noting is that the .sr file is around a third the size of the .csv file!
Edit3 - for reference https://sigrok.org/wiki/File_format:Sigrok/v2
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Awesome, fantastic work
Ah, apologies, might have been the WIP v3 sigrok format that had units and scaling. Or I'm imagining things! Your app fixes the scaling though, and having 'V' floating around isn't the end of the world really. (Just name each 'channel' as "IDC (Amps)" or something worst case )
It doesn't look toooooo hard to create an SR file? I think the metadata file includes the size of the binary one, so a bit of counting etc, but the savings in file size might be worth while...
Edit:
Ah, apparently it's the libsigrok "device" that includes units, scaling etc. Doesn't look like PulseView pays any attention (yet) though, certainly to units. It's either Time, Logic, or Volts haha. The Sigrok wiki claims there is 'an easy API' for new devices/input modules... I'm yet to understand the csv one yet hahah
Ah, apologies, might have been the WIP v3 sigrok format that had units and scaling. Or I'm imagining things! Your app fixes the scaling though, and having 'V' floating around isn't the end of the world really. (Just name each 'channel' as "IDC (Amps)" or something worst case )
It doesn't look toooooo hard to create an SR file? I think the metadata file includes the size of the binary one, so a bit of counting etc, but the savings in file size might be worth while...
Edit:
Ah, apparently it's the libsigrok "device" that includes units, scaling etc. Doesn't look like PulseView pays any attention (yet) though, certainly to units. It's either Time, Logic, or Volts haha. The Sigrok wiki claims there is 'an easy API' for new devices/input modules... I'm yet to understand the csv one yet hahah
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
The .sr file looks like the best way to go as it avoids the need to remember configuration strings and set the frequency. Creating the files should be pretty easy, not sure how to do the zip in a platform independent way though?
For now what we've got is perfectly usable. It's also a lot better, and took a lot less time, than I had expected, pulseview works really well for this
The only downside is it makes the graphing on the simulator seem pretty crude by comparison
Edit - there is one other thing that might be worth adding, at the moment it only works with foc builds. It would make sense to add sine support too but I don't have a clue whether there is a real need and if so what fields would be useful to log?
For now what we've got is perfectly usable. It's also a lot better, and took a lot less time, than I had expected, pulseview works really well for this
The only downside is it makes the graphing on the simulator seem pretty crude by comparison
Edit - there is one other thing that might be worth adding, at the moment it only works with foc builds. It would make sense to add sine support too but I don't have a clue whether there is a real need and if so what fields would be useful to log?
- mjc506
- Posts: 343
- Joined: Wed Sep 09, 2020 9:36 pm
- Location: Wales, United Kingdom
- Has thanked: 30 times
- Been thanked: 28 times
Re: ESP32 Based Web Interface & Data Logger
Does seem to be the route of least resistance. I'm certainly no expert, but my first thought would be lib7zip or similar. Far too powerful, but open and multiplatform.
They've put a lot of effort into showing many many points while keeping things quick! I understand none of that code haha
I've only ever used sine for lightbulb tests to be honest. Not sure what'd be useful, but seems fairly straight forward to add in later if needed?
They've put a lot of effort into showing many many points while keeping things quick! I understand none of that code haha
I've only ever used sine for lightbulb tests to be honest. Not sure what'd be useful, but seems fairly straight forward to add in later if needed?
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
TBH I was expecting the graphing side to be a bit of a nightmare. Going back a few years there was a project at work that needed fast graphing of large datasets, I lost track of how much time we spent on it and how many different commercial libraries we tried before finding a solution. I reckon that pulseview is possibly faster than the one we ended up paying for! Very relieved that just works so thanks again for suggesting and trying it
The sine support would be easy to add, just a new structure definition and function to fill it. Unless anyone expresses an interest will leave it for now though.
Looking forward to trying it on real data!
The sine support would be easy to add, just a new structure definition and function to fill it. Unless anyone expresses an interest will leave it for now though.
Looking forward to trying it on real data!
- Bigpie
- Posts: 1595
- Joined: Wed Apr 10, 2019 8:11 pm
- Location: South Yorkshire, UK
- Has thanked: 75 times
- Been thanked: 304 times
Re: ESP32 Based Web Interface & Data Logger
Was hoping to have some data today, esp32 board all works on the desk with a spare inverter board, put it in the car and it doesn't load params or log to sdcard :S will have to troubleshoot over the weekend.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Wondering whether its a power up timing issue, if the ESP powers up first it might time out before the stm powers up. It might be worth trying a few second delay at the start of the esp setup function?
Thinking about it my stm board is running without the bootloader which means I wouldnt have noticed it and makes the above seem more likely.
Thinking about it my stm board is running without the bootloader which means I wouldnt have noticed it and makes the above seem more likely.
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Think I can reproduce the problem and changing the delay on line 726 of esp32-webinterface.ino from 100ms to 10000ms seems to fix it. Bit hacky but should do for now. Lower values would probably work too but difficult to test properly without the bootloader on mine.
Sorry about that, should have tested a bit more thoroughly
Edit - does the modded stm code run the motor ok?
Sorry about that, should have tested a bit more thoroughly
Edit - does the modded stm code run the motor ok?
- Bigpie
- Posts: 1595
- Joined: Wed Apr 10, 2019 8:11 pm
- Location: South Yorkshire, UK
- Has thanked: 75 times
- Been thanked: 304 times
Re: ESP32 Based Web Interface & Data Logger
Seemed to run fine, only did a few short runs. Will test, hopefully shortly.
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Great, looking forward to seeing some data
Thinking about it a better fix would be an extra if(millis() > DELAY_VAL) test on he start logging condition. That would give the same result but still allow the wifi and web page to come up faster.
Thinking about it a better fix would be an extra if(millis() > DELAY_VAL) test on he start logging condition. That would give the same result but still allow the wifi and web page to come up faster.
- Bigpie
- Posts: 1595
- Joined: Wed Apr 10, 2019 8:11 pm
- Location: South Yorkshire, UK
- Has thanked: 75 times
- Been thanked: 304 times
Re: ESP32 Based Web Interface & Data Logger
I've added
where startMillis is set in the setup.
Code: Select all
if((WiFi.softAPgetStationNum() > 0) || (WiFi.status() == WL_CONNECTED || startMillis + DELAY_VAL < millis()))
VW Beetle 2003
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
Outlander front generator
Prius Gen 3 inverter (EVBMW logic board)
Outlander charger
3x Golf GTE batteries
Chademo Charging
Outlander water heater
-
- Posts: 1801
- Joined: Sun Apr 03, 2022 1:57 pm
- Has thanked: 102 times
- Been thanked: 347 times
Re: ESP32 Based Web Interface & Data Logger
Shouldn't that be > millis() if youre putting it there?Bigpie wrote: ↑Fri Oct 21, 2022 4:37 pm I've added
where startMillis is set in the setup.Code: Select all
if((WiFi.softAPgetStationNum() > 0) || (WiFi.status() == WL_CONNECTED || startMillis + DELAY_VAL < millis()))
You might not need startMillis (I think millis() starts from 0 on boot?)