We have upgraded the community system as part of the upgrade a password reset is required for all users before login in.

Pro keeps rebotting, some files gone, and filesystem in read only



  • My 2 pro has been working great since I got it a while back (had preordered it). Yesterday it crashed. Now it is in a boot loop where it will boot and be fine for a minute or 2 and then restart again. Was running my thermostat, so I can live with loosing the data that was on it if I need to. When I get into it it says

    BusyBox v1.28.3 () built-in shell (ash)
    
       ____       _             ____
      / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
     / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
     \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
     W H A T  W I L L  Y O U  I N V E N T ?  /___/
     -----------------------------------------------------
       Ω-ware: 0.3.1 b211
     -----------------------------------------------------
    Your JFFS2-partition seems full and overlayfs is mounted read-only.
    Please try to remove files from /overlay/upper/... and reboot!
    root@Omega-58B6:~#
    

    My script to kill the LED is there, but the rest of my stuff is not and my cronjob is gone. All my stuff was stored in /root/.

    Every time before it does a restart it has
    Removing MTD device #6 (rootfs_data) with use count 1
    and then
    reboot: Restarting system followed by restarting (see serial output from COM3 below).

    COM3
    https://pastebin.com/1NTK4jmJ

    How do I access the files to back them up. And how to stop the reboot. Or is doing an update my only option and my stuff is just gone? It is not the power supply-has been running off a samsung fast charger since got it, and tried a couple other supplies just to check and they all behave the same way.



  • @josh-hightower I had a similar issue with a 2+ which kept rebooting since the file system was full. My solution was to boot from external USB device (the details are in the docs) then I was able to access the filesystem and back up my files. You will need a second omega in order to create the external bootable device however.



  • @crispyoz Used a new 2+ to do this https://docs.onion.io/omega2-docs/boot-from-external-storage.html and then plugged the usb into the pro and it behaved the same way (on the 2+ it showed full ~16 gigs).

    The only file that I could find that was large (217.7 MB) was swap.page, which by pasting the following in was fast enough to copy it off before reboot (like seconds to spare). Last line to confirm it had finished.

    mkdir /tmp/mounts
    mount /dev/sda1 /tmp/mounts/
    cd /overlay/upper/mnt
    cp swap.page /tmp/mounts/swap.page
    ls -lah /tmp/mounts/
    

    After that reboot deleted swap.page which allowed it to boot normally and stop the loop. With and without the USB, it behaved the same way-like a normal 2/2+ with no additional storage added. The 8 GB eMMC worth of storage is not showing up tho. Wondering if the eMMC burned out causing all the issues, but if that were the case, why has my little shell script been there the whole time?

    If able, adding eMMC to the tags since it might be related. In the mean time, I'll run my stuff off the 2+ with a flash drive, which is kinda ironic because the whole reason this was switched to pro was to not need to have a USB off the side.



  • @josh-hightower Boot off the USB stick and run df -hT so we can see what has mounted and where.



  • Can probably skip this at the top as the problem seems to be with a command filling and growing swap.page with null.

    @crispyoz
    factory reset 2+

    root@Omega-3183:~# df -hT
    Filesystem           Type            Size      Used Available Use% Mounted on
    /dev/root            squashfs        6.0M      6.0M         0 100% /rom
    tmpfs                tmpfs          61.4M     92.0K     61.3M   0% /tmp
    /dev/mtdblock6       jffs2          24.4M    768.0K     23.6M   3% /overlay
    overlayfs:/overlay   overlay        24.4M    768.0K     23.6M   3% /
    tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
    

    https://docs.onion.io/omega2-docs/boot-from-external-storage.html completed on 2+

    root@Omega-3183:~# df -hT
    Filesystem           Type            Size      Used Available Use% Mounted on
    /dev/root            squashfs        6.0M      6.0M         0 100% /rom
    tmpfs                tmpfs          61.4M    216.0K     61.1M   0% /tmp
    /dev/sda1            ext4           14.6G     47.1M     13.8G   0% /overlay
    overlayfs:/overlay   overlay        14.6G     47.1M     13.8G   0% /
    tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
    /dev/sda1            ext4           14.6G     47.1M     13.8G   0% /tmp/run/mountd/sda1
    

    pro no USB (not in read only mode, but not 8 gigs either)

    root@Omega-58B6:~# df -hT
    Filesystem           Type            Size      Used Available Use% Mounted on
    /dev/root            squashfs        7.8M      7.8M         0 100% /rom
    tmpfs                tmpfs          29.3M    208.0K     29.1M   1% /tmp
    /dev/mtdblock6       jffs2           6.6M      1.1M      5.4M  17% /overlay
    overlayfs:/overlay   overlay         6.6M      1.1M      5.4M  17% /
    tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
    

    USB plugged in

    root@Omega-58B6:~# df -hT
    Filesystem           Type            Size      Used Available Use% Mounted on
    /dev/root            squashfs        7.8M      7.8M         0 100% /rom
    tmpfs                tmpfs          29.3M    208.0K     29.1M   1% /tmp
    /dev/mtdblock6       jffs2           6.6M      1.8M      4.8M  27% /overlay
    overlayfs:/overlay   overlay         6.6M      1.8M      4.8M  27% /
    tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
    

    mkdir /mnt/<device name>
    mount /dev/<device name> /mnt/

    root@Omega-58B6:~# df -hT
    Filesystem           Type            Size      Used Available Use% Mounted on
    /dev/root            squashfs        7.8M      7.8M         0 100% /rom
    tmpfs                tmpfs          29.3M    208.0K     29.1M   1% /tmp
    /dev/mtdblock6       jffs2           6.6M      2.3M      4.3M  35% /overlay
    overlayfs:/overlay   overlay         6.6M      2.3M      4.3M  35% /
    tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
    /dev/sda1            ext4           14.6G     47.1M     13.8G   0% /mnt
    

    Did not duplicate /overlay as the stuff that would be moved to the flash drive is not showing up, so no reason to duplicate it

    ===================================================================================

    /overlay/upper/mnt/swap.page is continuously growing, which reached ~200 meg again causing reboot loop and read-only file system.

    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       35.3M May 16 11:23 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       38.3M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       40.3M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       42.0M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       42.7M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       43.4M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       43.9M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       44.6M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       45.1M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       45.9M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       46.6M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       47.2M May 16 11:24 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    root@Omega-58B6:~# ls -lah /overlay/upper/mnt/
    drwxr-xr-x    4 root     root           0 May 16 11:20 .
    drwxr-xr-x    7 root     root           0 May 16 11:14 ..
    lrwxrwxrwx    1 root     root           1 May 16 11:14 .fs_state -> 2
    -rw-r--r--    1 root     root       47.6M May 16 11:25 swap.page
    drwxr-xr-x    6 root     root           0 May 16 11:14 upper
    drwxr-xr-x    3 root     root           0 Jan  1  1970 work
    

    Assumed this was swap for ram, but apparently not.

    root@Omega-58B6:~# cat /proc/meminfo  | grep Swap
    SwapCached:            0 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    

    Found the source of the growth, but do not understand why it is doing it. The 2+ is not running that command and cant find that file on 2+ at all.

    Mem: 48172K used, 11776K free, 208K shrd, 752K buff, 21984K cached
    CPU:   0% usr  99% sys   0% nic   0% idle   0% io   0% irq   0% sirq
    Load average: 2.08 1.92 1.33 2/57 1977
      PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
     1907  1863 root     R     2240   4%  98% dd if /dev/zero of /mnt/swap.page bs 1M count 384
     1977  1691 root     R     1212   2%   1% top
      204     2 root     RW       0   0%   1% [spi0]
     1669  1176 root     S     1132   2%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
        7     2 root     SW       0   0%   0% [ksoftirqd/0]
     1272     2 root     SW       0   0%   0% [RtmpCmdQTask]
     1831     1 mosquitt S     3320   6%   0% mosquitto -c /etc/mosquitto/mosquitto.conf
     1421     1 root     S     1968   3%   0% /usr/sbin/onion-helper
     1771     1 root     S     1796   3%   0% /sbin/rpcd
     1052     1 root     S     1724   3%   0% /sbin/netifd
     1351     1 nobody   S     1568   3%   0% avahi-daemon: running [Omega-58B6.local]
        1     0 root     S     1556   3%   0% /sbin/procd
    

    ===================================================================================

    Was going to copy the old swap.page back on and see how it behaves. Did not try that as the problem seems to be a growing zeroed out swap.page file



Looks like your connection to Community was lost, please wait while we try to reconnect.