Flying ADC 16 channel BMS 96S test

[skr]

That sucks. Even modules that worked fine for days go bad and even the v2.3 ones with various precautions don't last and even at only 12S, i.e. half the blocking voltage of the FETs.

So far it still seems contained to stacked modules, right? No single module ever broke as far as I'm aware.

I opened my LTC6811 design and realized I really don't feel like spending another week drawing it properly, another week bringing up and another week programming.. summer is coming Johu here has done tremendous amount of work, this is something we can figure out <3
Probably first step would be to repliably replicate this, if you have any ideas, I suppose I have enough parts to kill a few more boards before I run out of parts, so I am all ears.

My only other theory was that somehow my "antistatic" mat may have played a role in pyrofuse failure, but feeding 120V dc and probing it with sharp probes in very close distance to measure current makes this theory very inconclusive.

Alternatively we can always do this:

[PetersonOctavius]

what could be the cause?
i even coated my board to avoid moisture corroding or short-circuiting the pins.
is it a hardware issue? software?
i have no idea. im not that smart lol.
i cant recreate any scenario to trigger the muxshort either.

i'm in the process of building a new battery pack from my model 3 pack. wanted to used the flying adc....but its not promising.

what i can tell is when the flying adc works, i works like a charm. after 2-3 days my cell voltage where getting balanced to the 0.001 of a volt.

right now i only have 4 bms boards connected. i hope they dont go bad. or if they do i hope to notice it in time because muxshorts will thow the whole module out of balance (talking from experience )

what can i do?

[skr]

I managed to kill another board while doing harry potter movements around the the cell taps with a chinese cell internal resistance meter (which as far as I understand measures by AC current injection) probe - only one end. The other probe end just sits on the bench or is touched by me to exxagerate whatever is going on, on oscilloscope the signal looks similar to 50Hz ether background, just much choppier, by doing various magical movements I could get up to about 20Vpp.
I lack many 101s in EE, but as far as I understand:

some transient kicks over VGS and opens the wrong fet at the wrong time, maybe even exceeds the VGSmax and kicks a permanent hole there. The gate capacitance is pretty low so some stray electrons can open it up.
There's long battery balance wires, various power electronics nearby, I don't understand much, but there doesnt seem to be many EMI protection provisoions anywhere on frontend.
Could a simple ceramic capacitor across cell taps aide here if cycling between cells is clocked down from whatever it is to account for the extra delay? Maybe bidirectional TVS diodes to negate anything caught by the long cell taps? Maybe extra zener or tvs near 10k VGS resistor?

I am in a similar situation- I am not far from dropping MEB modules in to the vehicle and I already had to repair fuses in one of them due to my own stupidity. It was long, painful and very sketchy to mig weld battery modules and I'd rather not repeat this.

[Proton]

So far I do not have a charger since I cannot make my Tesla charger work and my cells are arround 3.9V. once I get some money I will have to buy a charger and test with cells above 4.0v. anybody here has tested with cells above 4v?

[PetersonOctavius]

@proton which tesla charger you have? why is it not charging?
i have the model s/x gen 2 charger with Damien's board. works everytime.

[Proton]

I posted the video bere: viewtopic.php?p=82229#p82229

Nothing I did made it start charging.

[johu]

Johu here has done tremendous amount of work, this is something we can figure out <3

Thanks, that is very motivating!

You could have a point with the missing transient protection. It might also be worhtwhile to dampen the gates themselves with a capacitor? Plus the cell taps themselves. The cap on the gates would need to have a sane time constant in conjunction with the 10k pull-down resistor.

Right now a cell voltage is sampled every 30 ms, so all cells are sampled within roughly 0.5s. The dead time is generated by inserting an I2C transaction with the balancer which takes 1 ms on < v2.2 and 100 us on a v2.3 board. Usually that is plenty of dead time.

I also observed that 99% of all failures happen once the boards are chained up AND they happen at startup. I think you are the only one who had a board go bad in the middle of operation. These chain+startup failures date back to the first version boards that we tested in London in 2023. Worked a charme when only one was connected, started blowing up once a second or third one was added.

Since then many aspects of the design changed. To sum it up
- v1.x boards didn't have individual optos per input but a 74-series shift register, adder and decoder behind a digital isolator. We suspected that to behave strangely plus it slowly disbalanced cells over weeks when balancing was turned off
- v2.0 boards then deployed the individual optos together with 74-series decoders to save pins and avoid illegal switch states
- v2.2 boards added pull-down resistors on the decoder inputs
- v2.3 boards replaced the low creepage distance 5V to 5V DC/DC converter by the IS2631 isolator

All these changes have not reliably fixed the problem so we are clearly missing something. The gate and cell caps are worth a try. And we must find out why it is so much more likely to happen when boards are cascaded??

The next production run will revert to the 5V DC/DC as the IS2531 has gone out of stock.

EDIT: a 10 nF cap in parallel to the gate pull-down resistors would add a time constant of 600us. These could just piggy-back the resistors for testing. Likewise caps on the cells taps could just be soldered directly across the inputs.

[skr]

I noticed that the fw checks for passives to see what revision it is.
Does the above timing outline mean that there 2.3 has 10x more time doing nothing for the dead time, or is the overall loop just quicker? I mean that in terms of adding 10nF on gates would still be within the operational window in terms of gate slopes?
If that's not the case, any pointers on fw changes would be welcome, as I now just flashed CI artifacts.

If yes, 10nF or less is within acceptable limits- my plan is to put 1uF or 100nF on cell taps and =<10nF on gates and see if running a taser next to the board or something still kicks the mosfets in to the sunset.

[johu]

It spends less time doing nothing: https://github.com/jsphuebner/FlyingAdc ... ms.cpp#L40 (and #84)
Because the IS2631 has a higher data rate.

And the cycle time is actually 25 ms to accomodate the 17 ms cycle time of the ADC. So it might be worthwhile to have 3 states
1. read ADC, disable mux
2. enable mux
3. start sampling
4. and 5. nothing

So the cycle time would drop to 5 ms and we'd still leave the ADC its 17 ms for sampling. We'd have a loooong dead time of 5 ms

[Jacobsmess]

Apologies for asking again but is this also likely linked to my incorrect Udc readings posted previously?

I've removed a header of one and checked all the connections. They seem fine, ive checked the Fet readings as described several tomes (all at 0.6 with 0.7 for Q1 and Q5) which appears to match my 3 working boards. The lasterr doesn't show anything. If I set the test channel to one with an incorrect reading it reads 0 on spot values but otherwise I'm a bit lost.
I'm running an older board (v2.3?) But otherwise running the latest firmware (0.25.25)....
Any advice on how to proceed/where to look?

Oh, I've also checked all resistors, with all matching working boards.

[skr]

You could have a point with the missing transient protection.

I took one of the muxshort boards and replaced two identified bad fets.

fets replaced, no cells attached, board running, no error.
http://jsonblob.com/1371648109440065536


on live board cells attached, values out of wack (some negative), taps are correct, on other working boards reports normally.
http://jsonblob.com/1371647936345333760

a few json requests after that, muxshort, error
http://jsonblob.com/1371647804015042560

Not sure if it killed it now or I had missed one dead fet at low side, and two poylfuses which had turned in to infinite resistance resistors, not sure before or now, probably before when these polyfuses turned in to partial pyrofuses, IIRC I changed all dead ones, but maybe these were stuck in an undefined state.

now getting values properly. Interestingly on first boot after fixes u0 reported pack voltage and u0 was u1, u11 was u12, after reboot it got back where it should be. I have somewhat forked OIcan, so maybe that's something on my end.
http://jsonblob.com/1371647570421669888

The board is now balancing, only 12S attached to it.


Plan is to use AFG3022 and inject noise simulation with it, increasing apmplitude slowly by +/- 100mVpp until anything happens or not.
AFG3022 is connected to V6 through a 10nF cap + 250 ohm resistor in series, ground ref to V5.

Oscilloscope one channel looking at V6 gates, ground ref to source after 10R. Other channel at V6 voltage.
Balancing turned off otherwise I could not really get the oscilloscope trigger to keep the gate in view.

At 10Vpp noise battery tap looks like shit, gate gets noisier, but BMS keeps chugging along, ADC still shows normal values.


Switching to squarewave 100mVpp -> 10Vpp.
fixed f up to 10Vpp no effect. Sweep from 100kHz to 1MHz no effect, scope completely out of whack.

I had no bafangs to try, but I tried a PI metal detector with coil directly over bms and bms tap cables, IIRC coil peak current was ~1.5A @ 13V. No effect.

Next up: a physics experiment (A universally legal taser)

Located about 20cm from cell taps, can see up to +/-30~60V transients in cell taps, up to +/- 30~40V in gates. My laptop gets random stripes on the display and USB devices randomly reset a meter away.

BMS keeps chugging along.


As a last resort: the chinese cell internal resistance meter on V6:
First one probe on bench, one on tap, then one probe in hand. Then doing all sorts of wavy things.
Nope, BMS has zero fucks to give. My previous theory of my antistatic mat shorting out cell tap TH solderjoints seems more plausible now.

Whats next?
No clue. Maybe Johu has some ideas, the taser should have worked if the issue was transients. Maybe it would have been worth it to be closer to 16S to give those fets something to worry about...maybe for more objective test it should have been done while balancing, but I am too lazy to try and repeat this.

I noticed that ADC values had gotten out of whack showing 3800mV cells as 1000~2800mV, not sure if that was after the taser test or after the internal resistance meter. A power cycle sorted this.

I could try to resort to this regarding those cell tap capacitors + gate capacitors and see if taser tests look better.

image.png (257.88 KiB) Viewed 679 times

what's your opinion Johu?

[skr]

Another thing I noticed- if transients can happen in in cell taps, most batteries may have temp sensors running with the same long wires to a thermistor as well as the current sensor probably goes where some power electronics may cause weird things. Not much current can flow there, but voltage can still cause problems.

For not that many cents more extra protection, in my opinion, should be added here:
1) not to bring 5V line down unexpectedly if something goes wrong with the current sensor or thermistors
2) maybe RC filter the temperature inputs + TVS
3) current sensor also something to protect the STM32 to not get a hole blow in it.

I understand the design goals of this bms by keeping the component count as low as poosible, but it may cause much less headaches down the road with customer support by adding <20 cents to the bom of each board + some assembly cost. This is probably one of the coolest bms designs out there, and I don't question that you will make very many batteries healthspan improvements out there in the long run.

[johu]

That is some extensive tests, cool So it looks like destroying a single board is near impossible, even without any anti-transient measures. I've already placed the gate caps yesterday. I will also remove the 10R input resistors as obviously they fail to protect the circuit but add another failure point. Will use the gained space for cell input caps.

So back to square 1 in fault finding. Here I see the main culprit in cascaded boards. So maybe the coupling capacitance from LV to HV side plays a role here? Like said I'm currently rigging up a test battery that should use a large number of boards.

Regarding the LV side inputs I indeed haven't put too much work into them. I wouldn't worry too much about the current sensor input as that is well isolated and has RC filtering. Has worked for a decade like this on inverter boards.

The temp sensors should receive more protection. A series resistor in the 5V supply and some basic part Zener behind RN3 for protection + capacitor for filtering.
It's not just about low component count but merely about using off the shelve parts. So we can add as many caps and diodes as we want without spoiling the design idea Would just like to keep the dimensions.

That said I replaced the IS2631 by an IS3642 most of all because it isn't out of stock and it is pin compatible with an Adum5402

[skr]

I am afraid about not the circuit protection, but the battery cell tap protection, say a meb module has inaccessible fuses, if anything goes catastrophically wrong on the bms - fuses will work as intended on the battery. I have seen cell taps shorting and instantly using the wire or a pcb trace as a glowing heater in my past. Obviously that's not possible on meb, as it would just ruin the battery, but avoid any fire hazards.

Imo either docs suggestion to do on wire inline fusing or actual inline current protection on board would be very welcome. The actual balancing currents are very low here and protection should be just a notch above

[johu]

Yes, that's a general issue with the topology I guess. Each cell is 3 or 4 FETs away from all others. So if the wrong ones start conducting you have like 50V across the loop. Low value resistors won't help here and high value resistors would turn it into a sensing-only BMS.

Currently undergoing Electrician training. The magic here is: selectivity! So the closer fuse must act before the one further downstream. Can this be implemented here?

FETs conducting when they shouldn't is always bad so we need to make very sure they don't

BTW the first module from 2023 is still working away merrily in my home storage system which to date is - non cascaded. It has seen multiple power cycles. So I think this part needs more thinking

[skr]

Previously you suggested for some of the above posters to check 10R as well, so I presume it may have worked as a fuse in some cases. That 10R should theorethically be enough current limit for meb not to trip (holds 300ish mA, trips instantly at around 750mA, littelfuse G code 1206), at least not instantly.

That same 10R in smaller size would probably work as a fuse proper-ish. 0402/0603 should die at around 400-700mA, but then an actual fuse would make more sense

[skr]

Currently undergoing Electrician training. The magic here is: selectivity! So the closer fuse must act before the one further downstream. Can this be implemented here?

Many years ago at a hackathon we built an internet of shit device, which basically shorted out AC L/N lines as an IoT device. Being aware of the fact that the fuse on our demo stand needs to be fast not to trip anything upwards- we chose most expesive and fastest acting fuses in the nearby diy shop. Needless to say that during our demo it killed stuff way above our initial fuses, leaving the event without livestream and the DJ being very angry with us. So yeah, fuses in general are hard, painfully aware of it.

[johu]

Previously you suggested for some of the above posters to check 10R as well, so I presume it may have worked as a fuse in some cases.

Yes, the resistors but the FETs as well. Ok, you could suspect that more FETs survived because of the resistors. The board that killed your MEB fuses was one with resistors or without?
My thinking was I=U/R, so I=50/10=5A so still too much unless they "blow" before the MEB fuse.

[skr]

My MEB fuses were killed by 8S cell tap cable, which on one end has a loop between two cells. I plugged it in without disassembling the cable completely only to find a loop below the fabric tape and cell taps no longer having voltage on them.

What I did fry was 100/300mA polyfuses before 10R, but I tend to think that was because of conductive desk surface.

Flying ADC BMS has not yet been connected to any MEBs for me. 0805 if it's 1/8 or 1/4 watt is too much, yes. Sizing a fuse proper for max 16s conduction is a different story, I looked at it from max 1S voltages.

My current setup is 100/300mA polyfuse in front of each tap, so approx 11-13R in total before fets.

[johu]

Oh! Ok I thought Flying ADC had killed your MEB fuses. Well now I feel a bit better

So with that I would indeed put the resistors back in as 0603. Quick math: Balancing shunts current over 4 10R resistors. Say we sit at 4.2V then we are looking at 105 mA. So the power in each resistor is P=R*I² = 110 mW

EDIT: back in 0603

[skr]

But imagine how salty one would be if your BMS killed MEB modules which one paid 1k a piece for.
Here is the writeup: viewtopic.php?p=80466#p80466
Most people would not take an angle grinder and a mig welder to a battery module and just cut the loss.
Me, having gone through that on one module, felt very paranoid, thus put those polyfuses to hopefully smoke out before the MEB fuse breaks.

[skr]

I wanted to test the taser with 8200pF 100V fitted on gate resistor + 1uF 50V X5R across cell taps, but repeatedly was getting cell polarity error. Soldered, desoldered caps, checked fets, connections to no avail.
I increased i2cdelay for hw ver 2.3 to 30, 100, could not get past this error unless I reduce cell number below the affected cell. I should probably look at this not during late night sessions, as hw seems ok otherwise...

[johu]

Cell polarity errors can show up when no cell is connected at all or has a poor connection

[PetersonOctavius]

running 4 bms board for almost a week straight, so far no issue.
been balancing away. cell voltages are spot on!!

BOARD 1

BOARD 2

board 2 was the first board to get a muxshort error a while ago, i didnt know better back then, i just left the board unpowered, but connected to the cells, so those 16 cells voltages went all over the place. i had to balance them with my lipo charger which took days, and in the process i charged them higher than the rest of the pack, but flying ADC has been working day and night to bring them down.

BOARD 3

BOARD 4