Pro keeps rebotting, some files gone, and filesystem in read only
-
My 2 pro has been working great since I got it a while back (had preordered it). Yesterday it crashed. Now it is in a boot loop where it will boot and be fine for a minute or 2 and then restart again. Was running my thermostat, so I can live with loosing the data that was on it if I need to. When I get into it it says
BusyBox v1.28.3 () built-in shell (ash) ____ _ ____ / __ \___ (_)__ ___ / __ \__ _ ___ ___ ____ _ / /_/ / _ \/ / _ \/ _ \ / /_/ / ' \/ -_) _ `/ _ `/ \____/_//_/_/\___/_//_/ \____/_/_/_/\__/\_, /\_,_/ W H A T W I L L Y O U I N V E N T ? /___/ ----------------------------------------------------- Ω-ware: 0.3.1 b211 ----------------------------------------------------- Your JFFS2-partition seems full and overlayfs is mounted read-only. Please try to remove files from /overlay/upper/... and reboot! root@Omega-58B6:~#My script to kill the LED is there, but the rest of my stuff is not and my cronjob is gone. All my stuff was stored in /root/.
Every time before it does a restart it has
Removing MTD device #6 (rootfs_data) with use count 1
and then
reboot: Restarting systemfollowed by restarting (see serial output from COM3 below).COM3
https://pastebin.com/1NTK4jmJHow do I access the files to back them up. And how to stop the reboot. Or is doing an update my only option and my stuff is just gone? It is not the power supply-has been running off a samsung fast charger since got it, and tried a couple other supplies just to check and they all behave the same way.
-
@josh-hightower I had a similar issue with a 2+ which kept rebooting since the file system was full. My solution was to boot from external USB device (the details are in the docs) then I was able to access the filesystem and back up my files. You will need a second omega in order to create the external bootable device however.
-
@crispyoz Used a new 2+ to do this https://docs.onion.io/omega2-docs/boot-from-external-storage.html and then plugged the usb into the pro and it behaved the same way (on the 2+ it showed full ~16 gigs).
The only file that I could find that was large (217.7 MB) was swap.page, which by pasting the following in was fast enough to copy it off before reboot (like seconds to spare). Last line to confirm it had finished.
mkdir /tmp/mounts mount /dev/sda1 /tmp/mounts/ cd /overlay/upper/mnt cp swap.page /tmp/mounts/swap.page ls -lah /tmp/mounts/After that reboot deleted swap.page which allowed it to boot normally and stop the loop. With and without the USB, it behaved the same way-like a normal 2/2+ with no additional storage added. The 8 GB eMMC worth of storage is not showing up tho. Wondering if the eMMC burned out causing all the issues, but if that were the case, why has my little shell script been there the whole time?
If able, adding eMMC to the tags since it might be related. In the mean time, I'll run my stuff off the 2+ with a flash drive, which is kinda ironic because the whole reason this was switched to pro was to not need to have a USB off the side.
-
@josh-hightower Boot off the USB stick and run df -hT so we can see what has mounted and where.
-
Can probably skip this at the top as the problem seems to be with a command filling and growing swap.page with null.
@crispyoz
factory reset 2+root@Omega-3183:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 6.0M 6.0M 0 100% /rom tmpfs tmpfs 61.4M 92.0K 61.3M 0% /tmp /dev/mtdblock6 jffs2 24.4M 768.0K 23.6M 3% /overlay overlayfs:/overlay overlay 24.4M 768.0K 23.6M 3% / tmpfs tmpfs 512.0K 0 512.0K 0% /devhttps://docs.onion.io/omega2-docs/boot-from-external-storage.html completed on 2+
root@Omega-3183:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 6.0M 6.0M 0 100% /rom tmpfs tmpfs 61.4M 216.0K 61.1M 0% /tmp /dev/sda1 ext4 14.6G 47.1M 13.8G 0% /overlay overlayfs:/overlay overlay 14.6G 47.1M 13.8G 0% / tmpfs tmpfs 512.0K 0 512.0K 0% /dev /dev/sda1 ext4 14.6G 47.1M 13.8G 0% /tmp/run/mountd/sda1pro no USB (not in read only mode, but not 8 gigs either)
root@Omega-58B6:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 7.8M 7.8M 0 100% /rom tmpfs tmpfs 29.3M 208.0K 29.1M 1% /tmp /dev/mtdblock6 jffs2 6.6M 1.1M 5.4M 17% /overlay overlayfs:/overlay overlay 6.6M 1.1M 5.4M 17% / tmpfs tmpfs 512.0K 0 512.0K 0% /devUSB plugged in
root@Omega-58B6:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 7.8M 7.8M 0 100% /rom tmpfs tmpfs 29.3M 208.0K 29.1M 1% /tmp /dev/mtdblock6 jffs2 6.6M 1.8M 4.8M 27% /overlay overlayfs:/overlay overlay 6.6M 1.8M 4.8M 27% / tmpfs tmpfs 512.0K 0 512.0K 0% /devmkdir /mnt/<device name>
mount /dev/<device name> /mnt/root@Omega-58B6:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 7.8M 7.8M 0 100% /rom tmpfs tmpfs 29.3M 208.0K 29.1M 1% /tmp /dev/mtdblock6 jffs2 6.6M 2.3M 4.3M 35% /overlay overlayfs:/overlay overlay 6.6M 2.3M 4.3M 35% / tmpfs tmpfs 512.0K 0 512.0K 0% /dev /dev/sda1 ext4 14.6G 47.1M 13.8G 0% /mntDid not duplicate /overlay as the stuff that would be moved to the flash drive is not showing up, so no reason to duplicate it
===================================================================================
/overlay/upper/mnt/swap.page is continuously growing, which reached ~200 meg again causing reboot loop and read-only file system.
root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 35.3M May 16 11:23 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 38.3M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 40.3M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 42.0M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 42.7M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 43.4M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 43.9M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 44.6M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 45.1M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 45.9M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 46.6M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 47.2M May 16 11:24 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 work root@Omega-58B6:~# ls -lah /overlay/upper/mnt/ drwxr-xr-x 4 root root 0 May 16 11:20 . drwxr-xr-x 7 root root 0 May 16 11:14 .. lrwxrwxrwx 1 root root 1 May 16 11:14 .fs_state -> 2 -rw-r--r-- 1 root root 47.6M May 16 11:25 swap.page drwxr-xr-x 6 root root 0 May 16 11:14 upper drwxr-xr-x 3 root root 0 Jan 1 1970 workAssumed this was swap for ram, but apparently not.
root@Omega-58B6:~# cat /proc/meminfo | grep Swap SwapCached: 0 kB SwapTotal: 0 kB SwapFree: 0 kBFound the source of the growth, but do not understand why it is doing it. The 2+ is not running that command and cant find that file on 2+ at all.
Mem: 48172K used, 11776K free, 208K shrd, 752K buff, 21984K cached CPU: 0% usr 99% sys 0% nic 0% idle 0% io 0% irq 0% sirq Load average: 2.08 1.92 1.33 2/57 1977 PID PPID USER STAT VSZ %VSZ %CPU COMMAND 1907 1863 root R 2240 4% 98% dd if /dev/zero of /mnt/swap.page bs 1M count 384 1977 1691 root R 1212 2% 1% top 204 2 root RW 0 0% 1% [spi0] 1669 1176 root S 1132 2% 0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 7 2 root SW 0 0% 0% [ksoftirqd/0] 1272 2 root SW 0 0% 0% [RtmpCmdQTask] 1831 1 mosquitt S 3320 6% 0% mosquitto -c /etc/mosquitto/mosquitto.conf 1421 1 root S 1968 3% 0% /usr/sbin/onion-helper 1771 1 root S 1796 3% 0% /sbin/rpcd 1052 1 root S 1724 3% 0% /sbin/netifd 1351 1 nobody S 1568 3% 0% avahi-daemon: running [Omega-58B6.local] 1 0 root S 1556 3% 0% /sbin/procd===================================================================================
Was going to copy the old swap.page back on and see how it behaves. Did not try that as the problem seems to be a growing zeroed out swap.page file