Safety of electric heaters

[johu]

I have bad news, the legendary Polo is no more. Lets call it heat damage.

There was no one root cause but a combination of issues. I will explain here so we can all learn something.

First of all, the fire was started by the heater which is designed like this:

It uses the heater core from a cheap hardware store heater blower and a single 1200V IGBT to switch it on and off. The integrated over temperature switch can not be used as is because it would just arc in case of opening. Instead it is wired to turn off the IGBT. The IGBT is turned on by a DC-DC converter. The DC-DC converter only turns on when
a) The main DC-switch is closed
b) The fan is turned on
c) The key is NOT in start position

So it seems there are plenty of safety measures in place. Well, read on.

I lent the car to a friend and after a few weeks he had issues charging the battery. Maybe just a bad comm interconnect. So he decided to turn on the ignition key which indeed will bridge the BMS and charging commences. Forever. They over charged and then looked like this:

They had swollen so badly that they cracked the wooden battery box. Amazingly they still worked and I escorted him 15 km back to my home. I just left it sitting there for a few weeks until I noticed the cells had de-swollen. The BMS showed a pretty high SoC even after all that time.

So next I attempted a recharge and even that worked, only thing I did notice was the inverter showed "battery overvoltage" which means 530V (500V idle). I wasn't really surprised as it was quite cold. I quit charging.
The next day I decided to continue the charging session but this time didn't sit in the car. Well, I better had. Only minutes later my wife complained that her laptop had lost wifi connection. Cause: GFCI trip.
Another minute later she saw smoke gliding past our front window. Oh, bitter taste. I look out the side window to find flames coming out the rear grill (i.e. the one in front of the wind shield). Rushed down, took the fire extinguisher and put an end to that. Well too late, flames had burned their way inside and destroyed it. Wind shield cracked, multiple wiring looms destroyed.

So, I can only speculate what might have happened:
a) a cell interconnect or the inner electrode connection of a cell had come loose because of the bulging
b) The current from the boost mode charger had nowhere to go
c) So the voltage must have risen above the 1200V breakdown voltage of the heater IGBT
d) IGBT failed SHORTED (they usually do)
e) All the energy went into the heater core with no fan running to cool it
f) fire

So there were 3 causes to that incident:
a) the swollen cells caused an interconnect fail
b) The boost mode charger outputted a silly high voltage
c) The single IGBT that stops current from going into the heater core failed

So what do we conclude?

• Do not trust a single IGBT, use a DC rated switch instead or in addition.
• Also make sure the output voltage of the charger can't reach stupid levels. Which is hard because of the heavy low pass filtering. Anyway, I will make sure the charger obeys udcmax (something arber requested a long time ago but I always said it's the BMSes job)
• Use PTC heaters
• Reverse safety: wire the fan to turn on when the heater turns on

Last but not least all components survived the fire, just the actual Polo is damaged. Also the heater circuit had worked flawlessly for 8 years.

[arber333]

Damn! I am sorry to hear that,

Did you salvage the cells at least? I remember killing my first pack in a similar fashion... didnt have the BMS HV alarm though. And i learned the lesson then as well.
That is one of the most important functions of the BMS. So no matter if you use analog or digital system it has to have functional HV alarm connected directly to charger interlock.
The second vital BMS function is to NOT TO ALLOW CHARGING if any signal is missing the loop. That meens AND function.

Anyways i want to share a detail that surprised me...
1. Go and discharge cells down to 3.4V per cell or similar.
2. leave them for a week or more in the cold.
3. Inspect survivors for deformation and slowly charge them in paralell to 3.6V
4. Dischage each cell down to 2.8V with a known current (12V H4 light bulb) and record time until discharge. You will know capacity that way.

I managed to get most of the pack back from being bloated. I wouldnt actually trust them in the car, but they were still good for testing and R&D.

[arber333]

And another though...

You could salvage the car still if your TUV is game .
You could get the VIN plate and the actual pressed VIN (yes they are two usually) from the structure of the Polo and get another Polo chassis.
Then you get the approval from TUV to change the chassis (VIN) from one car to the other. It can be done legally i just dont know your TUV procedures... I have seen this done on oldtimer cars.

That is of course if the rest of Polo chassis is worth the trouble.

A

[Alibro]

That's a bummer Johu.
It goes to show, if it can happen to an experienced EV'er like you it can happen to anyone so I'll take this as advice to be very careful.

[celeron55]

Good breakdown of the fault. The good news is, it isn't difficult or expensive to design a system that fails better.

That could have ended up much worse... imagine a long lasting arc in the battery pack.

Everyone should ask themselves:
1) What happens in your conversion if a cell interconnect fails while charging?
2) What happens if your heater switch fails closed?

[SciroccoEV]

My PTC heater controller, used a relay for isolation in series with mosfets for switching. Gate drive voltage was derived from a linear regulator. It probably wouldn't have survived this kind of overvoltage as the relay contacts may well have flashed over.

I'd like to think I would have used an SCR crowbar circuit on the output of the charger (after a fuse) for overvoltage protection.

[celeron55]

I'll answer my own questions for my conversion, as I find this quite interesting:

1) What happens in your conversion if a cell interconnect fails while charging?
I'm using a modified Leaf PDM as a charger and it has a hardware (and software) output voltage limit, so totally crazy voltages won't happen. The model S modules and BMS practically will not fail inside a module, and between modules the BMS is fully isolated so there shouldn't be much of a problem. If an internal module interconnect fails, I think the individual BMS PCB will burn and hopefully doesn't ignite anything else, after which the charging process will definitely stop as my BMS master requires the correct amount of cells to be visible and ok, and it even has authority to open the main contactor by itself. The charger software will not charge if the BMS isn't continuously telling it the battery is ok. I think I'm good here.

2) What happens if your heater switch fails closed?
The heater switch is a small 1000V contactor with a software rate limit to ensure longevity. It might physically fail closed, especially after let's say 20 years of operation. The heater heats coolant and has a temperature sensor and an overtemperature switch in a single series measurement circuit. The measurement cicuit is monitored by software in an external module that talks on CANbus to another module that actually controls the contactor. So it might fail closed in software too. The heater element is in a cast metal case. I think the worst case here is that the heater element destroys itself, causes an arc, burns through the metal case and sets something on fire outside of the case. This would require the arc to have a current that doesn't trip the heater's 30A fuse, but that's entirely possible. I think this is unsafe and I need to at least add a failsafe in software, specifically to make the BMS open the main contactor if the heater controller is dead or is reporting persistent overtemperature, at least in a charging situation where nobody is there to smell the burning heater and turn the key off.

[johu]

A voltage controlled charger certainly helps. You could still get over voltage from regen but at least then you are in the car and can use your nose to close the control loop.
Putting a relay in series I also thought about but of course that could also just arc over eventually.

The idea I like most is shorting out excessive voltages in a way that blows the respective fuse. So in my example If could have put a properly dimensioned TVS across heater and IGBT (to protect the IGBT). The TVS would have failed shorted and burned the 10A heater fuse. For last resort over temperature protection a normally open temperature switch could be used to short out the current.

[johu]

And another though...

You could salvage the car still if your TUV is game .
You could get the VIN plate and the actual pressed VIN (yes they are two usually) from the structure of the Polo and get another Polo chassis.
Then you get the approval from TUV to change the chassis (VIN) from one car to the other. It can be done legally i just dont know your TUV procedures... I have seen this done on oldtimer cars.

That is of course if the rest of Polo chassis is worth the trouble.

A

I indeed found the two VINs. So you are saying they could just be transferred to another Polo of the same kind? And thereby retain the "type approval"?

[arber333]

I indeed found the two VINs. So you are saying they could just be transferred to another Polo of the same kind? And thereby retain the "type approval"?

I belive so. Second VIN is stamped in sheet metal beam. This would require getting an expert welder to cut and transfer the part of the beam with VIN to the other car. Other VIN plate can simply be drilled and riveted to other car.
In Slovenia we actually have an application for chassis replacement. Usually they use this to salvage oldtimer chasis and they build a car out of two identical models. But i know a guy who changed chassis 2x (yes two times) on an Opel Astra G model electric conversion which had developed terrible rust. It all depends how bad the damage is in your case. Maybe you could get away with only replacing the dash and wiring under it.
It is a lot of work, but since you have TUV and probably a valid licence i think the car is actually worth saving.

Heh... on second note we have an approved Fiat Multipla here in Slovenia which was dissassembled for EV parts, but the car is still approved for EV. I intend to put Outlander motor, inverter, charger and battery in it and make it a daily driver . Worth saving...

[johu]

I still don't quite understand. So I consider the Polo chassis damaged beyond repair. So I would salvage all EV components and put them in a "new" Polo chassis, bolt the "old" VIN plates to the "new" chassis and be done with it?

[arber333]

I still don't quite understand. So I consider the Polo chassis damaged beyond repair. So I would salvage all EV components and put them in a "new" Polo chassis, bolt the "old" VIN plates to the "new" chassis and be done with it?

Yes, but usually the stamped VIN is located on a carrying member near the drivers door and requires care to remove and reweld it not to weaken the chassis too much. Do you have yours stamped under the bonnet? That is the easiest way to transfer them legaly. No heavy welding of car construction...
Country register responsibility and the product plate removal are cause for a TUV application. Because on a police reports from accidents they record both VINs, some cars even have 3!

You would probably have to demonstrate you put the other cars VIN to scrap so it doesnt appear somewhere else in the world .

[LRBen]

In the UK swapping VINs is very much illegal.

[MattsAwesomeStuff]

In the UK swapping VINs is very much illegal.

In the EU apparently with TUV permission you can move them.

IIRC there's more than 2 VINs on a vehicle. There's like a half dozen, squirrelled away in various difficult-to-access places to prevent chop shops from having such an easy go at things.

But if the requirement is the chassis one moves and the inspector finds it where it expects it, no problem.

What I've heard many places is that it is unlawful to remove the VIN from a vehicle, but not unlawful to remove the vehicle from the VIN. I.E. If you cut out a block of metal around the VIN, that block of metal is now "the car", which was formerly surrounded by "parts" (i.e. the entire rest of the car, every piece of which is okay to replace with a newer piece whenever you'd like), and you can replace the "parts".

What they don't want to see is making 2 new cars out of an old one, or someone stealing a car and hiding the VIN from it. Otherwise, not a big deal. There's also other tricks, someone mentioned there are more VINs of a certain classic car, than they ever made of that classic car. Not sure how that one works.

Regardless, as long as TUV is in on it and understands its not theft, follow their instructions and you should be fine.

...

Johannes,

Sorry to hear about your car going up in smoke. There is one fairly consistent thing about people that have a fire, they figured "this kind of thing wouldn't happen to me". I would have never picked you as someone it would happen to.

I myself would have taken lots of shortcuts, not had many safety measures, had lots of "temporary" setups that worked and thus were permanent until not working, etc.

And it kind of shows what can happen even though comedy of errors.

I would rather a hardware solution than a software. Software can have bugs.

Trying to grasp what's actually going on here, in laymen's terms...

Source of the problem is that your charger was not voltage limited, and from its perspective it saw... an open circuit? I.E. One of the battery conductors opened? But meanwhile the rest of the car doesn't know that and it's powered by the charger like it was an HV power supply? But without battery didn't provide enough load to constrict the voltage of the charger given to the rest of the car, so the HV the rest of the car was seeing (versus what is normally just battery voltage) was far higher, and anything in the car that could be turned on electronically (i.e. the 1200V max IGBT that controlled the heater) failed short and turned itself on? The heater wasn't left on, it turned itself on? But didn't turn its fan on, because it wasn't actually on-on?

So some simple things that would avoid this:

1 - A simpler charger. Not this, but for example, if you'd used a dumb variac charger, and the battery suddenly vanished electrically, it's not a problem because its voltage is maximum what the car voltage was supposed to be anyways. Or, just about any commercial charger would have been fine?

2 - Some way to force the fan to be on if the element is on. (heater specific, but, what else inside the car would fail from overvoltage, many other things too, no?).

3 - Isolating the battery from the car during charging (meaning the car is dead, can't communicate or run anything, not very practical).

4 - A software trigger if voltage rises above max battery voltage.

[johu]

Sorry to hear about your car going up in smoke. There is one fairly consistent thing about people that have a fire, they figured "this kind of thing wouldn't happen to me". I would have never picked you as someone it would happen to.

Well thanks But despite being "the expert" aka "trained professional" I never thought about the consequence of the pack going open circuit during charging. I did use a triple throw contactor for charging, two poles for DC +/- and one for AC. So a randomly opening contactor would have not caused that problem.

I myself would have taken lots of shortcuts, not had many safety measures, had lots of "temporary" setups that worked and thus were permanent until not working, etc.

And it kind of shows what can happen even though comedy of errors.

I would rather a hardware solution than a software. Software can have bugs.

Trying to grasp what's actually going on here, in laymen's terms...

Source of the problem is that your charger was not voltage limited, and from its perspective it saw... an open circuit? I.E. One of the battery conductors opened? But meanwhile the rest of the car doesn't know that and it's powered by the charger like it was an HV power supply? But without battery didn't provide enough load to constrict the voltage of the charger given to the rest of the car, so the HV the rest of the car was seeing (versus what is normally just battery voltage) was far higher, and anything in the car that could be turned on electronically (i.e. the 1200V max IGBT that controlled the heater) failed short and turned itself on? The heater wasn't left on, it turned itself on? But didn't turn its fan on, because it wasn't actually on-on?

Exactly. Failed shorted -> turned itself on. Fan off because car off.

1 - A simpler charger. Not this, but for example, if you'd used a dumb variac charger, and the battery suddenly vanished electrically, it's not a problem because its voltage is maximum what the car voltage was supposed to be anyways. Or, just about any commercial charger would have been fine?

Yes on commercial, not sure what a Variac charger is and whether it can create 500V DC.

2 - Some way to force the fan to be on if the element is on. (heater specific, but, what else inside the car would fail from overvoltage, many other things too, no?).

Indeed the heater was the only other device on the DC bus. Not even a DC-DC converter.

3 - Isolating the battery from the car during charging (meaning the car is dead, can't communicate or run anything, not very practical).

That's close to "add a relay in series to the IGBT".

4 - A software trigger if voltage rises above max battery voltage.

Yes, but likely too slow if the interconnect fail is sudden.

[arber333]

Exactly. Failed shorted -> turned itself on. Fan off because car off.

Well i dont know if output DC voltage sensing in a charger would help in your case. I would rather think NOT using wire that glows is a safe thing to have permanently connected to DC bus. Instead i have a covered electrical socket in the cabin which is connected to DC. I can use that to connect a mobile phone, laptop charger, practicaly every PSU that is internaly oscillated. I bet you could just plug in the hairdrier and it will work... for as long as you dont use the AC switch .
I will definitely put a DC relay on it now, just in case.

Yes on commercial, not sure what a Variac charger is and whether it can create 500V DC.

Variac = autotransformer. 3phase units can produce variable 50Vac to 400Vac and i actually use one to charge my car in emergency...
I use variac to make AC voltage and i rectify it with 3phase bridge. I use a diode and 100uF cap to soften the DC ripple on output...
viewtopic.php?p=8196#p8196

[MattsAwesomeStuff]

Variac = autotransformer.

Not by definition, but usually are, yes.

Variac = Variable AC, a brand name I think.

Usually an autotransformer with a dial. More popular back in the day to warm up tube supplies gradually. Old variable power supplies had them.

I've got a 3A and 10A versions, good enough as a bench-test to charge larger batteries if I need to. Current limiting is by slowly turning it up or throwing a huge resistor across it :p

I just chose it as an example of an idiotproof illsuited power supply to contrast the fact that the problem here was specifically the type of charging setup Johannes used flying off to a higher voltage without being held to a heavier load.

[arber333]

I just chose it as an example of an idiotproof illsuited power supply to contrast the fact that the problem here was specifically the type of charging setup Johannes used flying off to a higher voltage without being held to a heavier load.

Well yes, with my variac i need to have BMS allways ON. BMS = myself present.
I dont dare to leave it by itself for more than 20min. Specially at higher voltage...

And yes back in 2016 we discussed his charger and i did not agree with missing output voltage feedback. I think we discussed we could sense input and output voltage by hijacking both throttle inputs for the purpose of charging. But that would only work for standalone charger.
It is why i remained with EMW charger even though i dont think it has all that great record by Damien.... But at least it can shutoff at some predetermined votlage.
And if one uses ACinput and DC output contactors in the event of power outage/thrown fuse etc... energy on IGBT is so low it doesnt cause it to break.

Also now with ISA shunt device charger could actually use CAN data directly from ISA reports.

[johu]

Actually udclim is also in effect in charge mode but I had set it to 1000V