Re: Tiny Core v17.0 upgrade issues

[Stefann]

Short update (otherwise this ends rather vague)

It ran until last Saturday, june 20 after which I manually stopped it for unrelated reasons. So it did run without crash for 11 days.

Configuration:
- self compiled TC17.0 kernel with included usb-serial and ftdi single port driver (so,.. NOT the full directory)
- using blocking read and select funtions.

I now basically “give up” on finding deepest rootcause. Variation over testcases and lack of logging makes it kind of undoable.
I will again test the published package once TC17.1 beta gets released.

I brought my application to “non-blocking read” and slowly bring all functionality back in the main home utimation computer (during the troubleshooting had my Solaredge energy, battery and PV on the secondary computer to avoid damage from crashes).

Unless this crashes again there will not be followup on this thread until TC17.1

Thanks to all who helped.

[Stefann]

short update.

I'm running TC17.1 beta since Saturday 9:30. It's now running 48hrs without crash
- vmlinuz, modules.gz and roots.gz from TC download area
- usb-serial-6.18.35-tinycore.tcz
- application configured for blocking reads protected by select()
   (recap: nonblocking reads were not crashing on TC17.0)

So.. fingers crossed. I keep it running.

Note: I will post final results also on TC17.1 thread but did not want to pollute it with intermediate updates.

[Stefann]

well....

TC17.1 has been running fort 7 days (168 hrs) now without crash
- vmlinuz, modules.gz and roots.gz from TC download area
- usb-serial-6.18.35-tinycore.tcz
- application configured for blocking reads from serial port protected by select()
  (recap: nonblocking reads were not crashing on TC17.0)

I would say I can declare this issue solved.

I just restarted the application to bring the serial read to "non-blocking". This should even further enhance the stability.

Anyway... unless it crashes again this thread will no longer get updates

Thanks to all who helped. Significantly @Rich. I largely appreciate your keen eye for detail.

[Stefann]

Well….
5 weeks later and an other crash.
Still running the same TC17.1 beta.
Running with nonblocking serial read. So, although non-blocking read fails less than blocking read protected by select(), it still fails.
And also… as the crashing is very sparse, it could be non-serial read related as well.

Anyway
I gave up on finding the Linux rootcause. Now Just concentrate on finding application workaround.
In stead of using the serial port I can get the same date from an http call. I already tested that and it works. So maybe I will walk that path.

[Stefann]

For those still interested,
As I said, I gave up on finding the linux error.
With that said, as the kernel stalls, I definitely consider this a linux error.
If the application does something naughty it would be normal that the application would crash. The OS however should not.

On application side, I made things more robust.
I close and re-open the serial connection if the data package is not read in N attempts.
I close and re-open the serial connection after each successful data package in stead of listening for ever.
I close and re-open the serial connection if I read unexpected characters.

no idea whether it helps. I guess I will have to wait multiple month.

[Stefann]

The story continues...
- Monday augus 17 an other crash (after 7 days)
- today august 21 an other crash (after 4 days)
I adapted a buffer size and give it an other try

If it keeps crashing:
- I will revert to self compiled TC. For some "unknown reason" that seemed to work 2 month ago. Hopwever... it can very well be that it still crashes but simply after "much more weeks". We will see.

[gadget42]

@Stefann, just out of sheer curiosity, what are you planning to use after your current hardware fails?

[Stefann]

@Stefann, just out of sheer curiosity, what are you planning to use after your current hardware fails?

Ha! fun question.

I have several options.
The biggest biggest biggest inconvenience is that I do not have a 100.000% identification of "what is actually the failing thing".
From march to June I tried to do a lot of tests to narrow down the rootcause but last fails are on a configuration I had assumed to be healthy. It only failed after a whopping 5 weeks of spot-free running. This kind of invalidates all conclusions over that trouble-shooting. "assumed safe configurations probably just had a lower fail-rate".

If I really really really get stuck and believe that it is really the serial-over-usb port:
I have a workable option to abandon the serial port completely.

I am using the serial port to read energy data from my energy meter over the so called "P1 port": https://nl.wikipedia.org/wiki/P1-poort

However... Last march I bought a "solaregde homehub battery/solar inverter including a BackUp Interface" This unit can communicate energy data over a modbus interface.

So: in stead of using the serial port I could get the energy data fromthe modbus connection to the solaredge devices.

In fact:
- I already made that modbus connection operational. I did that during the trouble-shoot phase in April as a "plan B"
- during my trouble-shooting I already used it on my 2nd system such that "the pain of frequent crashes was manageable".
- I already made choosing modbus versus serial-usb connection a configuration item in my software
So: the alternative is quite simple.

Having said that...
There are a few downsides on that:
- modbus data is available every 5 seconds while P1 data is available every second
- modbus data is unavailable for 1 minute daily around 23:00 due to "solaredge internal data-maintenance"
These are downsides because I use the energy data to prevent fuse-overload when I'm charging my car. During car charging the home-system monitors the grid-consumption and if the fuse gets overloaded the car-charging is reduced. Fuse-rating requires a response time of 10 seconds so the 5 second interval is acceptable (ia have tested it already). However... the 1 minute gap is NOT acceptable.
Solution for that would be to detect the 1 minute gap and reduce charging during that time anyway. I rarely charge at 23:00 because I charge at times with cheap energy and that is never at 23:00. It would only kick-in when I need a priority charge because I would need the car soon.
- modbus data does not report gas-usage. There is no fix for that. But... My home is "almost" full electric. Only cooking is on gas. My gas-bill is 10euro/month. So... "giving up monitoring on gas" would be ok.
- modbus data is gathered via http "modbus over IP" connection. I tend to feel that to be less reliable as a direct hardwired serial-over-usb connection. But yeah... this serial-over-usb keeps failing. So if I make the extra "1 minute gap robustness" this will also be acceptable.

With al that said....
- the real bad thing that could emerge is that the thing keeps crashing even if serial-over-usb is completely abandoned.
- in that case it would be a hw board-incompatibility with the new linux.
- this cannot be 100.000% excluded as my processor is a Via Eden 1 core 32bit processor. NOT an actual intel.

So.... in short....
- At some point I may switch over to he modbus-over-IP port.
- Hopefully that would fix it
- If it than appears to be the cpu...... I would need a different computer... that would be a real downer.
- 1st thing to try than is using my 2nd system: that's a Via Eden 64bit 2 core processor.

The stupid thing: I never had these crashes on TC15. System could run multiple month without crash.
In theory I could go back to TC15 but I really do not want that. Doing that would probably give me a few extra years but for sure I will get to a point that "extensions I want need a linux update".
I was in that boat. I had been running DSL (damn small linux) sinds 2008 but DSL is long out of support. On DSL I was not able to use https. With all kind of services going for https that became a showstopper. I solved that by going to TC.