We have upgraded the community system as part of the upgrade a password reset is required for all users before login in.

Omega2 reboot



  • Hello,

    With firmware v0.1.9-b149, halt (or poweroff) always reboot my Omega2+ 😞.



  • The issues with the reboot flash state and problems with some of the proposed fixes are pretty clear.

    What is more a mystery is how a powerdown used to work. Was this an actual shutdown of the chip, or at least a drastic drop in power consumption by slowing a clock?

    Or was it just a one-way trip to what looked like an "off" state with no file systems mounted, from which externally cutting power would be fully appropriate?

    (Was a hypothetical trip to such a deadend state even a designed behavior, or just an emergent one of the chip trying and eternally failing to boot from a flash stuck in the wrong addressing mode?)

    And was it done by the kernel, by rebooting to U-Boot with flags to trigger special behavior there, or did the other (radio-related) CPU in the chip have a role in it?



  • the question i have when using shutdown is, there is roughly a 3 second period on my omega2+ after the power light goes off and before it comes back on. so if i do a shutdown, wait for the power light to go off, then remove the power plug, does this qualify as a "shutdown" in a legit sense? is it possible that damage to the filesystem will take place.?



  • The best time to pull power would be while the system is sitting in U-Boot; that's easily determined from the serial port, and you can probably correlate it to the state of the LEDs.

    In terms of fileystem corruption, this risk is already substantially lower than it would be on something like an SD card based raspberry pi.

    • The SPI NOR flash memory is used relatively bare; there's no hardware flash translation table that could mess up and scramble one part of the addressable space because another is being written to; if there are bugs, they are in visible, correctable software layers and the modes those configure the chip to. And unlike SD cars, the flash chips have real data sheets.

    • The critical storage - U-Boot, Linux, and the root filesystem are not normally operated in a writable fashion. This combined with the memory architecture should mean they are essentially always almost as safe from power loss as they would be on a system that has been shutdown in preparation for it.

    • The one thing that is writeable is the overalay partition, but this is a journaling file system that makes some degree of effort to make changes only in an interruption-safe manner. Still this is the weak point. However, fortunately the writable filesystem is a place you may be able to tolerate failure - safemode or a factory reset gets you a still bootable system, and you can reconfigure the network and transfer your backups of customizations to the board again.

    So orderly shutdown is a good idea when possible, but the consequences of a disorderly one are lesser than they would be on a different architecture.



  • thanks, Chris. another question, if i enter a dmesg , sometimes, usually after i have used a build for a period of time, the last line will be a warning of ~" vfat, filesystem is corrupt, try using fsck". when i've seen it i've wondered if this business of hacking a shutdown somehow is responsible for the corrupt filesystem that is being reported. do you have any other ideas about how this issue comes about?



  • What vfat filesystem?

    If you have a USB stick or an SD card added to the system, then you're potentially subject the all the various ways those can fail, unless your usage of it is optional enough that failure there is tolerable.

    In essence, with either of these before power loss you need to sync and unmount, just as you'd need to do "safely remove hardware" or drag to the trash or eject or whatever on a larger computer.

    The stock shutdown process (when it works) would hopefully include that, but it's worth being explicitly sure.

    Or just not using comparatively delicate storage, if you can avoid it.



  • i do ocassionally use an sd card. i must have left it in during a shutdown . i guess this means the things i would expect to be handled by the operating sytem are not being done. i need to pay more attention to what i'm doing.



  • @Chris-Stratton said in Omega2 reboot:

    What is more a mystery is how a powerdown used to work. Was this an actual shutdown of the chip, or at least a drastic drop in power consumption by slowing a clock?

    It was neither. The fact that it used to work was actually a bug: the SoC actually does reboot even in the old firmwares, before this fix, but the flash-chip was left in a state that the SoC's bootrom-loader doesn't know how to handle and it got stuck in an internal bootloop. If you attach an oscilloscope to the SPI-pins when the Omega2 looks like it's powered-down you'll see that it's actually periodically trying to boot off of the SPI-flash as there is traffic on the SPI-pins. The drop in power-consumption is simply due to the fact that all the other internal peripherals are off and clocks are at their minimum -- they'd get turned on by u-boot, if the bootrom managed to load it.



  • I'm both surprised... and not. The "emergent" shutdown of being unable to boot due to a flash address mode was actually in my list of guesses, after seeing exactly that (and indeed, the ongoing SPI traffic with mis-positioned response bytes) on another board.

    It might actually be possible to do better intentionally though, or at least put the system into an end state not determined only by a bug. There are mailbox registers that could be used for signaling intention if it ends up being easiest to pass the task to the bootloader.

    On a related point, setting the flash address mode back before an intentional reboot is not a safe fix, as it doesn't handle the unintended reboot case. Effectively, whatever mode the chip needs to be in at boot, is the mode it can never safely leave, or the system will have times when reboot would leave it in a broken state from which only power cycling could recover.



  • @Chris-Stratton said in Omega2 reboot:

    Effectively, whatever mode the chip needs to be in at boot, is the mode it can never safely leave

    It has to, you can only address 16MB of the flash in the default-mode. That's why the mode is switched to the other mode, so that the whole 32MB on the Omega2+ can be addressed.



  • @WereCatf said in Omega2 reboot:

    @Chris-Stratton said in Omega2 reboot:

    Effectively, whatever mode the chip needs to be in at boot, is the mode it can never safely leave

    It has to, you can only address 16MB of the flash in the default-mode. That's why the mode is switched to the other mode, so that the whole 32MB on the Omega2+ can be addressed.

    Then the SoC should be jumpered to start in 4-byte mode so software can always remain in that one.

    Sometimes there is an extended address register that can be used, though that getting cleared could also be an issue.

    I seem to recall from my logic analyzer debugging that there are no preliminaries - just on 0x03 read command with with configured 3 or 4 0x00 address bytes following... correctly aligned vectors and then instructions had better result.

    If the system is really being set by software into a mode that isn't compatible with booting until set back, then the hardware reset input won't work, unless that it also electrically routed to the flash chip.



  • @Chris-Stratton said in Omega2 reboot:

    If the system is really being set by software into a mode that isn't compatible with booting until set back, then the hardware reset input won't work, unless that it also electrically routed to the flash chip.

    Well, without schematics available I really have no idea how they've wired things on the PCB. I don't want to remove the RF-shield just to take a look.



  • @WereCatf said in Omega2 reboot:

    @Chris-Stratton said in Omega2 reboot:

    If the system is really being set by software into a mode that isn't compatible with booting until set back, then the hardware reset input won't work, unless that it also electrically routed to the flash chip.

    Well, without schematics available I really have no idea how they've wired things on the PCB. I don't want to remove the RF-shield just to take a look.

    That is indeed an issue in general. Fortunately though the boot mode settings are relatively available:

    • They are configured by pullup/pulldown of the SPI pins. The difference between 3 and 4 byte addressing is set by /CS1, which is brought out to a module pin. If /CS1 is pulled high, boot mode is 4 bytes, while if low, 3. If done correctly this would seemingly need to differ between a 2 and a 2+, ie an SMT resistor would need to be placed in one of two different places.

    • They are visible as bits [4:2] of the sysconfig register at 0x10000010 (mapped at 0xb0000010 under U-Boot)

    • The flash chip will have its own factory or persistent mode at power-on. Chips such as the MX25L25635 that apparently can't be persistently configured to 4-byte mode at boot may simply be unsuitable, but U-Boot etc can report that on a system that actually has an MX25L25735F which only and ever works in 4-byte mode. In contrast a W25Q256 has a persistent user configurable power-on mode.

    I've been noticing a number of people posting logs of what seem to be power-at-wifi-activity reboots, without U-Boot ever appearing again, if these are uniquely from 2+'s it would seem that the underlying issue remains.



  • @Chris-Stratton said in Omega2 reboot:

    • They are configured by pullup/pulldown of the SPI pins. The difference between 3 and 4 byte addressing is set by /CS1, which is brought out to a module pin. If /CS1 is pulled high, boot mode is 4 bytes, while if low, 3. If done correctly this would seemingly need to differ between a 2 and a 2+, ie an SMT resistor would need to be placed in one of two different places.

    I know, I checked the bootstrap-pins today and it seems that at least CS1 is pulled to GND, ie. 3-byte addressing.

    I've been noticing a number of people posting logs of what seem to be power-at-wifi-activity reboots, without U-Boot ever appearing again, if these are uniquely from 2+'s it would seem that the underlying issue remains.

    Aye. Onion-devs screwed the pooch there. They chose the same flash-chip for both and then decided to keep bootstrap-pins the same and...well, here we are. Kind of a poor decision.



  • Hello All:

    I have managed to reproduce these steps several times and so I'm documenting it. The trick as far as I can tell is that when you visit the spinning wifi-setup screen while you run the wifisetup steps in a ssh terminal window, it somehow manages to persist the changes correctly.

    If you run into a changing WiFi issue... try this.

    1. Power on your Onion Omega +2.
    2. On your computer or laptop - switch your wireless connection to your device omega-####.local where #### represents the last four numbers of MAC address.
    3. Once you are connected, you should be able to SSH onto the device:
    ssh root@omega-####.local
    
    1. Now you want to switch to a web browser at this time and open the following address:
    http://omega-####.local
    
    1. You will be presented with a login, go ahead and login.
    2. On the web-browser console, open "Settings" and then click on "WiFi-Settings", you will get the usual spinning circle (as described on this thread above).
    3. While this is spinning, switch back to the SSH terminal and type
    wifisetup 
    

    Follow instruction and pick the correct WiFi you want.
    8. At the end, it will say "> Restarting wifimanager for changes to take effect".
    9. Reconnect on your computer or laptop WiFi back to "omega-####.local" device.
    10. SSH in if you're disconnected. If you use a keep-alive, you will be reconnected on the SSH terminal.
    11. On the SSH terminal, issue the following command:

    ifconfig | grep "inet addr"
    

    This will display all the IP's used by your various devices on your network. You should an additional one like this.

         inet addr:10.11.12.186  Bcast:10.11.12.255  Mask:255.255.255.0
         inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
         inet addr:127.0.0.1  Mask:255.0.0.0
    

    In this example, 10.11.12.186 is the IP under which my Onion connected with the local WiFi router.
    12. At this point, you can safely switch your laptop or computer WiFi to your router and you should be able to "ping omega-####.local" on the network.

    Rebooting your Onion Omega should promptly reconnect to the WiFi without any snag.

    Cheers!

    If this works for you please visit my blog at https://mythinkpond.com/ and subscribe!


Log in to reply
 

Looks like your connection to Community was lost, please wait while we try to reconnect.