Odd can issue affects serial comms

[FFMan]

Outlander motor, inverter & charger, home built Teensy zombine controller. Recent convert from Prius openinverter, with maybe 100 miles covered since trouble free until DCDC stopped working, which prompted rewire...

So an issue has reappeared that I have seen before but put down to my prototype wiring, so i set about the long put-off rewire, to reduce the junction boxes and house the master arduino away in the waterproof ecu housing BMW kindly built in.

My master arduino controls the charger and when driving just provides battery gauges, estimated range on a serial Nextion display etc. Its not critical to driving but it does configure the regen etc and i consider it essential really. The zombie vcu, inverter and master arduino all share one can channel on the teensy 4.1 master.

The symptom i see is that under certain conditions the serial feed to the nextion display gets corrupted and words/numbers have random or changed characters in them, and the button presses on the screen fail to register. The processor keeps running in the back ground but the display is inoperable. When it goes wrongs, it is apparent within about 2 seconds as the update takes a while, i assume because some of the update commands get corrupted and the nextion discards them as invalid syntax. I have tried reducing the serial speed and update interval which made no difference.

I spent most of today trying to bottom out the issue and have tried many things. What i can say for sure is that if i disconnect the HV the problem is not there. If i apply the HV but then disconnect the can to either the inverter or the zombie arduino (they're on the same channel) the problem goes away. If i connect the zombie arduino to 1 can channel and the inverter to the other the problem goes away (but this is not a workable configuration without a software bridge).

I can't decide if the issue is air born emi, or cable borne. I have rerouted all cables away from the HV, and tried wrapping the HV cables in grounded aluminium foil to no avail.

the can channel shows 60ohm when power is off so appears to be correctly terminated.

My motor wires between inverter and motor have no shielding (other the foil tried) and no ferrite core, but as the issue is apparent before the inverter is woken by can i do not think emu from these cables is the issue.

I'm not sure where to go next except to use a teensy 4.1 for the VCU and use separate can channels but this seems to be a work around not a solution as such.

Any questions/ideas gratefully received.

[uhi22]

Are you saying that the Arduino still works fine in the error situation, and only the serial communication to the Nextion display is disturbed? And the issue does not appear if there is less traffic on the CAN? This smells like a CAN buffer overflow, so if there is much traffic the Arduino CAN handler overwrites data of the serial display handler. Or a runtime/interrupt issue, e.g. high frequency of CAN interupts are blocking the interrupt-driven serial handler. Just guessing.

[FFMan]

Are you saying that the Arduino still works fine in the error situation, and only the serial communication to the Nextion display is disturbed? And the issue does not appear if there is less traffic on the CAN? This smells like a CAN buffer overflow, so if there is much traffic the Arduino CAN handler overwrites data of the serial display handler. Or a runtime/interrupt issue, e.g. high frequency of CAN interupts are blocking the interrupt-driven serial handler. Just guessing.

> Are you saying that the Arduino still works fine in the error situation

yes it appears so.

> And the issue does not appear if there is less traffic on the CAN?

yes, now you put it like that though i'm not sure why removing the HV would improve matters, perhaps if there is no HV the inverter and charger send fewer messages. i can try looking at my msg/sec counts in each case. Perhaps there is a significant drop.

i did wonder about this, and put in some counts of can msg/sec on each channel and its (from memory) 800/400/500 on channels 1-3 which seemed really high to me. In ethernet terms i have seen switches go faulty that generate spurious traffic that can grind a network, i wonder if something is doing that. I have to say I am not so happy with the can connection to the inverter and its on the list to rework. could electrical noise be creating unnecessary interrupts maybe ?

I did try increasing the rx_256 setting on the can setup to 512 and then 1024, but this was a wild stab in the dark really and didn't appear to help though i didn't pursue it as an option as i wasn't sure what i was doing.

I looked at can filters, though i wasn't sure if this would actually lighten the load as its not clear to me how and where these are applied.

I wondered as a test whether i substitute the display for a laptop running a serial monitor, and actually see if the output is corrupted or whether there might be an issue with the display.

i did try reducing the comms speed to the display and the refresh rate and didn't notice any improvement but this was early on and i might not have notice an improvement if there was still some corruption.

might it help to poll the can channels for messages with a large rx buffer, rather than using the mailbox callback ?

thanks for the thoughts. i'll double check those counts tomorrow and see if no hv reduces them.

[FFMan]

OK so got to the bottom of this.

I put a laptop with a TTL serial to USB on the output from the arduino fully expecting to see corrupted display commands but they were perfect, indicating an issue at the display end. That end contained a relay forming part of an OR arrangement (display awoken by charge mode, or the key being turned) and a small 12v to 5v supply. Now the new main board has a dedicated 5v regulator for the display so i ditched the relay and local supply and ran a new cable for power and all is good.

Still can't be sure what was going on, some odd interaction when the HV is applied - dunno on that one.

Whilst looking through the can logs the inverter emits masses of 0x645 records, like 250/sec so i will work on setting up filters to drop those as in run mode the Teensy is receiving about 1500 msgs/sec, and dropping the not used ones makes sense if I can.

thanks for help, just need to fix the dcdc and we're back on the road.