Suddenly, the Omega 2+ freezes with the system LED off.
-
I have many installations—around 2,000—that need to run 24/7, 365 days a year. Power relays and RS485 devices are connected to the board.
Occasionally, the system freezes, and the system LED (GPIO44) turns off. The only way to fix it is to physically cut the power and turn it back on.
This can happen after a week or after three months; I can't figure out the reason.
My research suggests a power supply issue or a voltage drop, but I would like to know if there is a way to resolve this state. Whether it is a software or power supply issue, it is as if the board gets stuck during startup.
Do you have any advice?my system:
cat /proc/version
Linux version 4.14.81 (root@c34dd7d43957) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r0+7498-730f2344ba)) #0 Fri Nov 25 04:10:54 2022Thanks.
-
This is very interesting, power cycle suggests boot device read error which is spurious. Due to a transient in the power supply as you are positing. This is a very large deployment. What is the power into the board ? Have you managed to recreate this in the lab ? As the saying goes - at scale things fail.
Without bench recreate this it would be hard to verify, are theres sources of noise around the board that can affect the SPI bus data integrity during boot ? I had this issue when using the Onion in energy monitoring devices and connecting the monitor to the Onion over an SPI bus and sharing the power supply. Occassionally when switching inductive loads, particularly LED or Fluro lamps the Onion would crash with this behaviour.
-
I had an issue where my device would reboot after a couple of weeks, sometimes longer sometimes less but on average it was about 2 weeks. But as it was rebooting it was hanging, so effectively it would fail and never come back up.
This drove me mad for 2 or 3 months but eventually I managed to work it out. I was powering my O2+ via USB so I plugged in a USB power monitor and found that the device was crashing and as the boot sequence brought up the WiFi it would hang. We know that loading the WiFi service increases load while it scans for an AP so this was not unexpected. Eventually I worked out that the WiFi connection was dropping out during normal operation due interference to the AP, typically that was not an issue because the O2+ would automatically reconnect so it would go unnoticed. However when when the O2+ was working hard processing a stream of data from an SPI data source and at the same time the AP disconnected, the O2+ would crash. As it came back up and loaded the WiFi stuff it would then hang.
We did a few things to fix the issue. 1) Move the AP, it was close to a fluorescent light and when it blinked the AP would glitch; 2) Replaced the power supply of the o2+ with higher wattage.
That fixed the issue but didn't fully explain it, but we were busy with other things so I was happy
My guess is that it was a load issue, restarting wifi while processing data required more juice than was available so the device would crash. Then when it was rebooting it was running out of juice starting WiFi because it was already loading another module that was consuming a lot of juice connecting to a sensor that was left in an undefined state due to the initial crash.The moral of the story is ... don't be stingy with your power supply
How did we monitor the $22 USB power monitor? We took an old laptop, pointed its camera at the USB monitor so we had a video of the display, since the laptop was old it had only about 128GB of SSD so video was on a loop of about an hour. Then we had a bash script that pinged the O2+ every minute and if ping failed it sent me an email and stopped the camera so we didn't overwrite the video. A bit like gaffa tape and bits of string but it was what we had to work with at the time
