My Car tried to Kill me today (1973 Beetle +SDU Conversion), Potential Cruise Control Bug ?

[7yatna]

Hi Guys,

sorry I`m overdue for posting my Car build thread, I Build more than taking pictures and Project #2 is taking a lot of my time now.

So it was a beautiful day here in San Diego, CA so i decided to go to work with my 1973 SDU converted Beetle.

The car has been Super reliable for almost a year now with perfect tune, smooth drive train and power. Even Cruise Control which this incident happened while it was active.

about the setup first:

- VW 1973 Beetle.
- Tesla SDU.
- SDU Logic Board V5
- OI Firmware 5.11R (I didn't update to the recent Revision).
- Cruise Control signal is a Push button on the dash.
- Brake Signal is Going to the SDU through the IO.
- Brake Pressure sensor Connected to Pot 2 to vary Regen.
- Throttle is on Pot1.
- Pot Mode is dual Pot and both are calibrated.
- Battery SOC was around 87% when the issue happened.

Back to the story,
I drove in the morning to work the car was perfectly fine all is working great.
I was driving back home the first couple of miles the car again is flawless I took the High and everything as it should be.

my commute is a little long so i put on the cruise control ( Activated it through the push Button ). The car kept its current speed as expected, the issue started when i was slowing down (Applying the brakes). The brakes should disengage the cruise control right away the issue was the SDU kept the throttle like the Cruise control still engage, and since im pushing the brakes harder the SDU was putting more torque to ramp up the speed.

I reached a point that i locked the front wheels and almost doing a burn out at 60 MPH. I dint freak out taking in consideration that if turn off the ignitions the BMS will open the contactors and that is a BOOM for SDU as it was pumping roughly 300A .

I got the idea of putting the car in neutral from the gear selector switch, which did the trick and the SDU stopped applying torque and i coasted to the side of the highway and put the hazards on.

I didn't turn off the ignition as i wanted to know is it a wiring issue or loose wire and at least gather enough information to write this thread.

the car now is ON, in Neutral and brakes applied like hard.

As soon as I put the car in Drive, A massive Burnout is happening, It was soo cool to see the old beetle doing this Burnout standing ( which theoretically it shouldn't ). the brake light switch is signaling the SDU, and the Pot 2 is reading above the lower threshold so it shouldn't command any torque.

I realized that the issue still there, so i put the car in neutral, waited couple of seconds to stabilize the SDU then turned off the ignition. Couple of seconds later,

Started the car again and Everything was back to normal, including cruise control, Brakes, Regen is proportional to how hard the brakes are pressed, perfectly nothing wrong with the car.

Arrived home safe and checked all wiring related to Brakes light switch, the SDU LV Connector, and anything i can think off all checks out OK.

Further conversation with the guys, someone mentioned some rare beahvior like this was reported before. Un commanded throttle.

I thought to share the story and at least get some opinions on what might be the issue.

I think it is either one of two potential issues:
- The brake light switch somehow didn't work or failed to signal the SDU, which the SDU interpreted as the car is climbing a hill a needed more torque. if this is the case then we should update the cruise control to take and compare two inputs for deactivating cruise control as if one failed that migh be critical, maybe add another switch to stop CC, i know there are CAN Commands for CC and can be switched off.

- the second option is a bit scarry, might be the firmware was stuck in a loop which didnt monitor and react to the I/O change. single thread processors if they are doing some Function and it is intensive they can miss a IO change.

i will get a parameter file from SDU tomorrow and will add it here.

Unfortunately i don't have any can Log or error log.

Now i leave you with some pictures of Betty the wanna be serial Killer.

Thanks,
Karim

[johu]

As discussed on Discord I had a similar incident, or rather (worse), my wife did. It is interesting because my setup is almost orthogonal to yours:
- VW Touran w/ CAN bus vs. VW 1973 Beetle.
- Nissan Leaf sync motor vs. Tesla SDU async
- Nissan drop in board/mini mainboard vs. SDU Logic Board V5
- OI Firmware 5.24R vs. OI Firmware 5.11R (I didn't update to the recent Revision).
- Cruise via CAN vs. Cruise Control signal is a Push button on the dash.
- Brake via CAN vs. Brake Signal is Going to the SDU through the IO.
- Brake pressure via CAN vs. Brake Pressure sensor Connected to Pot 2 to vary Regen.
- preprocessed throttle via CAN vs. Throttle is on Pot1.
- Pot mode CAN vs. Pot Mode is dual Pot and both are calibrated.
- SOC 70% vs. Battery SOC was around 87% when the issue happened.
- Wife never uses cruise control vs. you were on cruise control

I think this rules out the core motor control as yours is the async one, mine the FOC one. It also rules out IO issues as mine come in via CAN. Also 5.11R didn't have the "boat" mods where throttle can command direction and speed rather than torque. Also hill hold is not in that version.
One thing before we dive too deep you should check the reliability of your brake switch.

Here is some relevant code. Cruise control is in two steps, first we determine the speed for cruise control, then we use a P-controller to maintain that speed. Here is the relevant part of determining speed:

Code: Select all

   //Always disable cruise control when brake pedal is pressed
   if (Param::GetBool(Param::din_brake))
   {
      Throttle::cruiseSpeed = -1;
   }
   else
   { .... }
   
   if (Param::GetInt(Param::cruisemode) != CRUISE_CAN)
   {
      Param::SetInt(Param::cruisespeed, Throttle::cruiseSpeed);
   }

So brake always sets cruise speed to -1 rpm, only with no brake is the other code executed.
The code is identical in the newest version and 5.11R. Too late now but would have been interesting to see "cruisespeed" while you were in idle on the hard shoulder.

I'm assuming you have idle mode off. Mine is set to HillHold. So will skip that part for now.

The second stage, in ProcessThrottle() in 5.11R

Code: Select all

   throtSpnt = GetUserThrottleCommand();
   GetCruiseCreepCommand(finalSpnt, throtSpnt);

So GetCruiseCreepCommand() is called unconditionally.

And then in GetCruiseCreepCommand()

Code: Select all

   float cruiseSpnt = Throttle::CalcCruiseSpeed(Encoder::GetSpeed());
   ... idle code ...
   if (Throttle::cruiseSpeed > 0 && Throttle::cruiseSpeed > Throttle::idleSpeed)
   {
      if (throtSpnt <= 0)
         finalSpnt = cruiseSpnt;
      else if (throtSpnt > 0)
         finalSpnt = MAX(cruiseSpnt, throtSpnt);
   }

So Throttle::CalcCruiseSpeed() is always called to keep the filtered value updated, but its output is only used if cruiseSpeed > 0 and > idleSpeed.

So cruiseSpeed must have been > 0, i.e. din_brake failed to go high. I see no possibility of din_brake being high and cruise control not disengaging. Unless cruiseSpeed is set somewhere else, by accident.

In my case there is even some redundancy, the VCU forwards din_brake (taken from CAN bus) to the inverter. Also when it sees brake is sends cruisespeed=0 to the inverter plus the inverter gets the brake signal.

EDIT: for more safety you should use cruise control in "switch" mode. Then you have a second input that kills it.
EDIT2: moved to Inverter Firmware as it seems a generic issue

[johu]

For completeness here is my story:
- Car has driven flawlessly since 2020
- No self acceleration except the well known field weakening bug
- Self acceleration happened once, never before, never after

Wife was driving on highway in somewhat jammy traffic, going between 10-40 kph (far, far away from field weakening). She was cycling between accelerating, foot off regen and brake pedal. Never came to standstill. This went for 3 km. Suddenly, when she came off the brake pedal the car would accelerate hard, allegedly without her touching the throttle pedal. Whenever she touched the brake it would stop accelerating. She came to a halt, turned off the car, turned it back on and it drove normally.

Haven't brought it up so far as I couldn't tell if it was a VCU (stm32-car) or inverter related problem.

VCU has the analog throttle inputs and checks if pot1=2*pot2, otherwise EPC light is triggered. min(pot1, 2*pot2) is send to inverter. Unfortunately, pot1 is allowed to saturate at 4095, so the EPC light code is disabled once pot1 reaches 4090. It means should BOTH dual ground fires come off, this would go undetected.

I have changed is now by halfing the voltage and using the inverter to check throttle validity instead. Now the EPC light code is always active.

[Pete9008]

Now that's not good

Looks a bit like the 100ms scheduler task may not be running as this is where VehicleControl::CruiseControl() is called from and the brake input is read, everything else is done in the 10ms task which is obviously still running.

Not too sure why this is the case though. Have had a look at the main scheduler and it uses round robin so shouldn't allow a higher priority thread to stop a lower priority one. This is the code for reference:

Code: Select all

void Stm32Scheduler::Run()
{
   for (int i = 0; i < nextTask; i++)
   {
      if (timer_get_flag(timer, TIM_SR_CC1IF << i))
      {
         uint16_t start = timer_get_counter(timer);

         TIM_CCR(timer, i) += periods[i];
         functions[i]();
         execTicks[i] = timer_get_counter(timer) - start;
      }
   }
   timer_clear_flag(timer, TIM_SR_CC1IF | TIM_SR_CC2IF | TIM_SR_CC3IF | TIM_SR_CC4IF);
}

I can see a way for it to miss task calls though. If a counter flag is set after the if(timer_get_flag(timer, TIM_SR_CC1IF << i)) line but before the timer_clear_flag(timer, TIM_SR_CC1IF | TIM_SR_CC2IF | TIM_SR_CC3IF | TIM_SR_CC4IF); line it will get cleared without being serviced. A safer way of doing it would be to only clear the flags for tasks that have actually been serviced rather than always clearing them all (the unserviced tasks would then be run when the isr retriggers).

This could cause the 100ms interrupt to me missed if the timing of the 1ms or 10ms task was just right I just find it a little difficult to believe that it could stay synced for long enough to cause the above?

Edit - one other thought, is it possible to limit the amount of throttle that the cruise is allowed to request so that it can never generate more torque than the brakes can handle?

Edit2 - Definitely need more details/pictures of the build too!

Edit3 - This is what I'd suggest and what I think I'm going to change to (not tested yet).

Code: Select all

void Stm32Scheduler::Run()
{
   for (int i = 0; i < nextTask; i++)
   {
      if (timer_get_flag(timer, TIM_SR_CC1IF << i))
      {
         timer_clear_flag(timer,  (TIM_SR_CC1IF << i));
         uint16_t start = timer_get_counter(timer);

         TIM_CCR(timer, i) += periods[i];
         functions[i]();
         execTicks[i] = timer_get_counter(timer) - start;
      }
   } 
}

Edit4 - And got to say it - some data logging would be nice!

Edit5! - Thought of a neater way of doing the code, updated above.

[johu]

Good call. Though NOT calling the Ms100Task for extended periods (like 2s) would trigger a HW watchdog reset. Still, if enough calls were missed, it could just not see the brake signal on time.

[Pete9008]

Good call. Though NOT calling the Ms100Task for extended periods (like 2s) would trigger a HW watchdog reset. Still, if enough calls were missed, it could just not see the brake signal on time.

Literally just realised the watchdog bit and was just about to update the above!

Don't really believe the above is the cause, seems too unlikely for 7yatna's and the 100ms task wouldn't have affected yours but thought it worth mentioning anyway.

Edit - can't help but wonder if this is related to FFMan's issue where the CAN stopped transmitting and the terminal hung while everything else continued working, if so 5.27 might fix it?

[Bigpie]

I've also had a couple of incidents. But my setup is similar to johu.

My stm32-car reads throttle and puts it on the can bus.

I think I have some intermittent can bus problem. When the error occurs, my dash beeps and shows errors, like it's not receiving the can messages. My can log doesn't pickup the messages either.

Pressing brake doesn't work to stop the motor torque, though i can drive with the brake as a inverted throttle to where I can cut the ignition out of the way. I use canoi for brake input too.

I doesn't happen for months, then a couple of times on the same short trip.

[Pete9008]

Need to get the logging sorted!!

Just had a look through the CAN code, can't see anything that would produce the above but can see a way for transmit messages to get corrupted. Messages get sent from the scheduler thread via a call to Can::Send, if a hardware transmit buffer isn't immediately available they get queued in a software buffer. This buffer is then emptied and sent when hardware buffers become available via the can isr. It's possible for the scheduler to call Send again (it's running at a higher priority) while the isr is running and if this happens the message sent could be a combination of part new and and part old message.

This really should be a proper FIFO buffer with separate inbound and outbound pointers rather than using a single pointer.

[johu]

Just pushed the timer_reset_flag() change.
Pete, where can I find your latest logging code with the 1-byte spot value thing? Wouldn't mind integrating it into the PWM routine. I'll refactor it anyway as I already had the "SOMETHING_LIST" thing done.

[Pete9008]

Just pushed the timer_reset_flag() change.
Pete, where can I find your latest logging code with the 1-byte spot value thing? Wouldn't mind integrating it into the PWM routine. I'll refactor it anyway as I already had the "SOMETHING_LIST" thing done.

Pushed it back to the F103 code earlier and just started testing it now. Once it's working I'll push the new auto-setup-withlogging branch up to github.

Not put the above change in mine yet. Instead I've added three spot value counters which are incremented in the three scheduler tasks (1ms, 10ms and 100ms) so that's it's possible to see whether any are being missed in normal operation (a very quick test suggests that they might be but will need a longer test to be sure).

Edit - On github now. Tested but only on dummy hardware. Note - there is something odd going on if the board is powered up while start is high, not looked into it though and no idea whether it's just this build, specific to the bluepill or common to other boards/builds.

[Pete9008]

One more though on this - would it be sensible to make it so that switching to neutral, or changing direction, also cancels cruise?

At the moment the brake input has no redundancy, all you need is a broken wire or switch and cruise can't be cancelled. Adding logic to check for a change in direction would add some redundancy (bit like having cruise disengage when you use the clutch as well as when you use the brake in a manual ICE car).

[johu]

Yes, absolutely

@7yatna : have you checked whether your brake signal was the culprit?

[7yatna]

Yes, absolutely

@7yatna : have you checked whether your brake signal was the culprit?

Sorry Guys got dragged into some life stuff.

I did check the car and the brake signal is 100% perfect no issues at all both same day it happened and every single time i drove the car.

i do not think it was the brake signal but that the only thing make sense.

I will do redundancy on the brake signal.

[7yatna]

One more though on this - would it be sensible to make it so that switching to neutral, or changing direction, also cancels cruise?

At the moment the brake input has no redundancy, all you need is a broken wire or switch and cruise can't be cancelled. Adding logic to check for a change in direction would add some redundancy (bit like having cruise disengage when you use the clutch as well as when you use the brake in a manual ICE car).

that exactly what happened, to really get back in control i put the car in neutral and coasted that disengaged the motor and no torque is requested.

but when i fully stopped and pushing brakes as hard as i could, as soon i pout the car in gear again the car did a burnout on still and acted like it is accelerating.

only to put it back in neutral, dissipate whatever energy in the inverter ignition off and on now we are back to normal.

that extra logic check for cruise control will definately a solid idea.

[MattsAwesomeStuff]

Despite the frustration of intermittent and infrequent errors, I'm kinda shocked this wasn't a higher priority if it's EVER been observed.

On a scale of 1-10, that's a 10... a Pinto-esque crisis that needed solving.

What about all the cloner boards and software out there? Ya think when they ripped it off they tweaked the software, or are there vehicles out there with similar liabilities floating around?

[Pete9008]

Got to agree, I find it surprising, disappointing and fairly demotivating the lack of interest that there appears to be in threads discussing potentially serious problems.

I'm concerned that the lack of apparent interest, and a tendency to blame the install, wiring, parameters, operator, etc all tend to dissuade people from posting about problems. Experience has taught me that developers rarely find the bugs in their own systems as they only tend to thoroughly test their own setup and method of use; it's usually when someone else starts using it in a slightly different way that the issues begin to come to light and at that point it's too easy to think well mine has been working fine for x years, it must be something you've done (it might well be, but it's a dangerous assumption). We need to know about issues if we're to fix them.

It was a similar post by Romale around a year ago about a broken driveshaft (due to unwanted regen) that made me aware of some of the problems with the FOC code and that prompted the simulator development, the aim being to try and understand, debug and solve some of the issues (and I lost count of the number of things it turned up). The trouble is it only covers the core FOC functionality.

This is why I'm keen on SD card logging, the idea is that if something does happen we at least have the data to try and help understand what happened and why and try to ensure that it doesn't happen again. I find the lack of interest surprising, all I can assume is that people would prefer to stick their head in the sand and hope that their install will be fine?

[arber333]

-------
This is why I'm keen on SD card logging, the idea is that if something does happen we at least have the data to try and help understand what happened and why and try to ensure that it doesn't happen again. I find the lack of interest surprising, all I can assume is that people would prefer to stick their head in the sand and hope that their install will be fine?

Way back when Johannes suggested error conditions i suggested eeprom/flash error logging. System would keep 10 latest errors and delete the old ones. The problem is those errors should be predefined.
In your case i would agree. Critical parameters recording would be a must. At least there should be a possibility to turn it ON or OFF if you are driving for testing purposes.

[Pete9008]

The trouble is I doubt that the software was actually in any kind of error state during the above, it was just happily running cruise control.

This is the kind of logging I'm doing viewtopic.php?p=57268#p57268, full details of the implementation are in the ESP32 data logging thread. My plan is to have it running continuously in the background (as problems are rarely considerate enough to only happen when you have remembered to turn logging on )

Edit - although logging can be disabled by just unplugging the ESP32, or removing power to it.

[Bigpie]

In my case, I think it was can bus issues. I was getting a lot of can bus errors and the resistance was only 40ohms between high and low. I'm thinking reflections or something?
I'm very much keen to get the logging working, though I've decided to not build your board (too small for me to attemp), got one of Damiens with CAN, ESP32, SD card, RTC and Serial (already assembled ) so should be able to create logs of pretty much everything going on. I'm also not sure what logging is in the latest release of firmware but I'll be finding out.

I

[Pete9008]

Not seen those, have you got a link or picture?

Glad it seems to be getting a bit more traction than I was aware of

[Bigpie]

He's not released yet.

[Pete9008]

Looks good. The dual CAN ports are a good idea

Does it still have 12V power (can't see the connection point)?

Bit bulky compared to mine but a lot easier to build!

Edit - Interested to see what the software support for CAN logging is like.

[Bigpie]

Not sure what's there software wise at the moment. No 12v either. There's a couple of tweaks to make, needs the pull-ups for the SD card, then I can start looking at getting setup.

[johu]

We have now established that the physical brake signal is most likely not the culprit. I'm assuming you traced din_brake in the web interface and it follows your brake pedal.

Sorry if people are irritated by the inactivity prior to knowing this. This is not the first post mortem analysis I do on rare errors. From the former ones I know it makes no sense to run around in circles and change random things until you have found the root cause. Even the field weakening problem proved how little can be achieved without really understanding what you're up against. 3 years of random stabs remained fruitless. And that was a very repeatable misbehaviour occurring every time you provoked it.

I think we can also exclude that missed cycles of the 100ms task caused the brake signal being missed because switching to neutral is handled by that very same task and worked. Also he pushed the brake for a very long time, so even many missed cycles of 100ms task would have eventually picked it up.

That leaves me without any explanation and with no idea what to do. Please as many people as possible check the code and hardware how this very error can occur.

This includes but isn't limited to answering the following questions:
- In CruiseControl() can a non-zero value end up in Throttle::cruiseSpeed even if the brake pedal is pressed?
- Can a non-zero value end up in Throttle::cruiseSpeed for any other reason than the intended one?
- What could go wrong in GetCruiseCreepCommand() ?
- Could Param::din_brake get stuck at 0 for any reason?
- Could DigIo::brake_in.Get() get stuck at 0?
- Is there a known silicon error leading to wrong sampling of digital inputs?

Another measure would be to remove CruiseMode=Button. Then dropping the cruise signal will disable cruise control as well. But it's a band-aid "fix" and will take years until we know if it helped because the error is so friggin rare. Same goes for dropping cruise control in neutral. Makes perfect sense but only works in setups where you have the possibility to do so and still leaves the driver in the uncomfortable situation of a motor working against the brakes.

[7yatna]

We have now established that the physical brake signal is most likely not the culprit. I'm assuming you traced din_brake in the web interface and it follows your brake pedal.

Sorry if people are irritated by the inactivity prior to knowing this. This is not the first post mortem analysis I do on rare errors. From the former ones I know it makes no sense to run around in circles and change random things until you have found the root cause. Even the field weakening problem proved how little can be achieved without really understanding what you're up against. 3 years of random stabs remained fruitless. And that was a very repeatable misbehaviour occurring every time you provoked it.

I think we can also exclude that missed cycles of the 100ms task caused the brake signal being missed because switching to neutral is handled by that very same task and worked. Also he pushed the brake for a very long time, so even many missed cycles of 100ms task would have eventually picked it up.

That leaves me without any explanation and with no idea what to do. Please as many people A possible check the code and hardware how this very error can occur.

This includes but isn't limited to answering the following questions:
- In CruiseControl() can a non-zero value end up in Throttle::cruiseSpeed even if the brake pedal is pressed?
- Can a non-zero value end up in Throttle::cruiseSpeed for any other reason than the intended one?
- What could go wrong in GetCruiseCreepCommand() ?
- Could Param::din_brake get stuck at 0 for any reason?
- Could DigIo::brake_in.Get() get stuck at 0?
- Is there a known silicon error leading to wrong sampling of digital inputs?

Another measure would be to remove CruiseMode=Button. Then dropping the cruise signal will disable cruise control as well. But it's a band-aid "fix" and will take years until we know if it helped because the error is so friggin rare. Same goes for dropping cruise control in neutral. Makes perfect sense but only works in setups where you have the possibility to do so and still leaves the driver in the uncomfortable situation of a motor working against the brakes.

Hi Johu,

No worried i understand. i will keep an eye and as well dig into the code hopefully i catch something or point into something.

i appreciate the help