We use OM2+ in a few of our devices for commercial purposes. Recently, we encountered a scenario where the OM2+ got factory reset to vanilla OpenWRT. While looking through WinSCP, we found there are no files or folders at all. I have attached a screenshot of the console below.

I have no answers to how the hostname is retained, i.e. AirSENCE-06233B5B40356. We tried flashing the Omegaware-0.3.3 b253 image, but it didn't allow us to. We have had situations where the device got restored to factory settings but this is something extraordinary that we cannot find answers for.

After running, firstboot -y && sync && reboot, we could get the default OM2+ console page.
Any feedback/thoughts would be highly appreciated.
Thank you.

Hi @crispyoz & @Lazar-Demin, hope you guys are doing great.

The error you see when booting now makes sense: Linux expects /etc/hotplug.json to exist, but since the build system couldn't find that package it's not included in the firmware.

Should we see the same behavior if we use the binary downloaded from the Onion FW Repo?

Lately, we've started seeing the same behavior again with b255 where it gets stuck in the boot process. On checking, it is the same error related to the hotplug.json file not being found. We're not creating and flashing custom images for now, all this is using the binaries from the FW Repo. I saw that there is a new build release, b259, that we are planning to try.

One way we recover from this stage is by accessing the device through failsafe mode and doing firstboot. I'll try flashing the new build using sysupgrade -N <filename> and share the behavior here. I also wanted to know if there's any difference between the way an image is flashed on the devices. There are methods to upgrade or re-install images like through ethernet, SD Card/USB, and also there's sysupgrade. However, we only use two methods, i.e. through ethernet or syupgrade to upgrade or re-flash a device on bricking. How different are these two processes?

Thanks again for all the responses and suggestions. Really looking forward to get to the core of this.

Hi @crispyoz & @Lazar-Demin, thank you so much for your input. I've been working on some immediate deliverables, so I won't be able to share what you asked for at least the next few days. I'll post the logs and observations as soon as I try what you both have suggested. Thanks again!

I'm using sysupgrade to flash the custom image.

@Lazar-Demin, I've compared the commits too, and agree that there aren't many significant changes. Flashing b255 from the Onion Firmware Repo works fine. Talking about the custom image creation process, the issue was observed even when there weren't any changes done to the build system before creating the image.

I cloned the b255 commit from the repo, and did no changes. I created the image using the same cloned build system, AS-IS, and when flashed with the image created, the onion still goes in the loop of the booting process. I'm ok to share more details if you want, please let me know what, and I'll share accordingly.

Thanks.

No build other than b255 are working on these devices. And I don't think trying builds lower than b255 would be sensible on these devices. I tested this with the images downloaded from the Onion FW Repo.

However, as far as the custom build goes, I only created a custom image using the b255 build system.

The default build system is not changed. I believe this is a built-in script that we don't make any changes to. The only customization is adding binaries, scripts, and supporting packages required by our FW. Also, the same build works fine on some devices. However, they all brick, eventually, after there is a power cycle.

The same image worked fine with our older batch of devices, in which we used the build system b252 to create the custom image. Since we came across this notice, we upgraded the build system to b255, as suggested and we started to see this error.

Thanks for the quick response @crispyoz I tried what you suggested but looks like it's not recognizing any inputs. I tried all 4 debug levels but the messages didn't change. Everytime time the logs are same. See the logs below

   ____       _             ____
  / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
 / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
 \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
 W H A T  W I L L  Y O U  I N V E N T ? /___/"

Board: Onion Omega2 APSoC DRAM:  128 MB
relocate_code Pointer at: 87f60000
flash manufacture id: ef, device id 40 19
find flash: W25Q256FV
*** Warning - bad CRC, using default environment

============================================
Onion Omega2 UBoot Version: 4.3.0.3
--------------------------------------------
ASIC 7628_MP (Port5<->None)
DRAM component: 1024 Mbits DDR, width 16
DRAM bus: 16 bit
Total memory: 128 MBytes
Flash component: SPI Flash
Date:Nov 18 2022  Time:19:21:40
============================================
icache: sets:512, ways:4, linesz:32 ,total:65536
dcache: sets:256, ways:4, linesz:32 ,total:32768
CPU freq = 575 MHZ
Estimated memory size = 128 Mbytes
Resetting MT7628 PHY.
Initializing MT7688 GPIO system.
wifi mac address = 881E59010029.


**************************************
* Hold Reset button for more options *
**************************************



Boot Linux from Flash NO RESET PRESSED.
## Booting image at bc050000 ...
   Image Name:   MIPS OpenWrt Linux-4.14.81
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    1617506 Bytes =  1.5 MB
   Load Address: 80000000
   Entry Point:  80000000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
No initrd
## Transferring control to Linux (at address 80000000) ...
## Giving linux memsize in MB, 128

Starting kernel ...

[    0.000000] Linux version 4.14.81 (mrahul@DESKTOP-UIQA19L) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r0+7496-be7777b369)) #0 Fri Nov 18 20:23:14 2022
[    0.000000] Board has DDR2
[    0.000000] Analog PMU set to hw control
[    0.000000] Digital PMU set to hw control
[    0.000000] SoC Type: MediaTek MT7688 ver:1 eco:2
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019655 (MIPS 24KEc)
[    0.000000] MIPS: machine is Onion Omega2+
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] random: get_random_bytes called from start_kernel+0x8c/0x474 with crng_init=0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Writing ErrCtl register=00018359
[    0.000000] Readback ErrCtl register=00018359
[    0.000000] Memory: 124580K/131072K available (3708K kernel code, 207K rwdata, 920K rodata, 212K init, 205K bss, 6492K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 256
[    0.000000] intc: using register map from devicetree
[    0.000000] CPU Clock: 580MHz
[    0.000000] timer_probe: no matching timers found
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6590553264 ns
[    0.000012] sched_clock: 32 bits at 290MHz, resolution 3ns, wraps every 7405115902ns
[    0.007673] Console: colour dummy device 80x25
[    0.011931] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216)
[    0.073505] pid_max: default: 32768 minimum: 301
[    0.078194] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.084537] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.098565] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.108059] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.114006] pinctrl core: initialized pinctrl subsystem
[    0.119578] NET: Registered protocol family 16
[    0.155795] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.161479] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.167002] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.173623] i2c-mt7621 10000900.i2c: clock 100 kHz
[    0.183173] clocksource: Switched to clocksource MIPS
[    0.195416] NET: Registered protocol family 2
[    0.200547] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.207294] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.213398] TCP: Hash tables configured (established 1024 bind 1024)
[    0.219659] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.225293] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.231586] NET: Registered protocol family 1
[    0.250226] Crashlog allocated RAM at address 0x3f00000
[    0.256838] workingset: timestamp_bits=30 max_order=15 bucket_order=0
[    0.269153] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.274771] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.294884] io scheduler noop registered
[    0.298615] io scheduler deadline registered (default)
[    0.304654] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    0.311994] console [ttyS0] disabled
[    0.315495] 10000c00.uartlite: ttyS0 at MMIO 0x10000c00 (irq = 28, base_baud = 2500000) is a 16550A
[    0.324217] console [ttyS0] enabled
[    0.324217] console [ttyS0] enabled
[    0.331212] bootconsole [early0] disabled
[    0.331212] bootconsole [early0] disabled
[    0.340014] 10000d00.uart1: ttyS1 at MMIO 0x10000d00 (irq = 29, base_baud = 2500000) is a 16550A
[    0.349688] 10000e00.uart2: ttyS2 at MMIO 0x10000e00 (irq = 30, base_baud = 2500000) is a 16550A
[    0.359202] cacheinfo: Failed to find cpu0 device node
[    0.364450] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.371420] spi-mt7621 10000b00.spi: sys_freq: 193333333
[    0.393126] m25p80 spi0.0: w25q256 (32768 Kbytes)
[    0.398001] 5 fixed-partitions partitions found on MTD device spi0.0
[    0.404455] Creating 5 MTD partitions on "spi0.0":
[    0.409321] 0x000000000000-0x000000030000 : "u-boot"
[    0.415306] 0x000000030000-0x000000040000 : "u-boot-env"
[    0.421509] 0x000000040000-0x000000050000 : "factory"
[    0.427556] 0x000000050000-0x000001f80000 : "firmware"
[    0.522794] 2 uimage-fw partitions found on MTD device firmware
[    0.528842] 0x000000050000-0x0000001daea2 : "kernel"
[    0.534819] 0x0000001daea2-0x000001f80000 : "rootfs"
[    0.540628] mtd: device 5 (rootfs) set to be root filesystem
[    0.547873] 1 squashfs-split partitions found on MTD device rootfs
[    0.554216] 0x000000e30000-0x000001f80000 : "rootfs_data"
[    0.560627] 0x000001f80000-0x000002000000 : "user"
[    0.567397] libphy: Fixed MDIO Bus: probed
[    0.583352] rt3050-esw 10110000.esw: link changed 0x00
[    0.590231] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5
[    0.598931] i2c /dev entries driver
[    0.604385] NET: Registered protocol family 10
[    0.613296] Segment Routing with IPv6
[    0.617135] NET: Registered protocol family 17
[    0.621690] 8021q: 802.1Q VLAN Support v1.8
[    0.635528] VFS: Mounted root (squashfs filesystem) readonly on device 31:5.
[    0.643889] Freeing unused kernel memory: 212K
[    0.648394] This architecture does not have kernel memory protection.
[    1.502472] init: Console is alive
[    1.506263] init: - watchdog -
[    2.223186] random: fast init done
[    5.165744] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    5.450996] usbcore: registered new interface driver usbfs
[    5.456739] usbcore: registered new interface driver hub
[    5.462230] usbcore: registered new device driver usb
[    5.473864] exFAT: Version 1.2.9
[    5.512981] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.529372] SCSI subsystem initialized
[    5.538863] ehci-platform: EHCI generic platform driver
[    5.554560] phy phy-10120000.usbphy.0: remote usb device wakeup disabled
[    5.561362] phy phy-10120000.usbphy.0: UTMI 16bit 30MHz
[    5.566698] ehci-platform 101c0000.ehci: EHCI Host Controller
[    5.572558] ehci-platform 101c0000.ehci: new USB bus registered, assigned bus number 1
[    5.580749] ehci-platform 101c0000.ehci: irq 26, io mem 0x101c0000
[    5.613218] ehci-platform 101c0000.ehci: USB 2.0 started, EHCI 1.00
[    5.620679] hub 1-0:1.0: USB hub found
[    5.624952] hub 1-0:1.0: 1 port detected
[    5.635058] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    5.642945] ohci-platform: OHCI generic platform driver
[    5.648638] ohci-platform 101c1000.ohci: Generic Platform OHCI controller
[    5.655606] ohci-platform 101c1000.ohci: new USB bus registered, assigned bus number 2
[    5.663734] ohci-platform 101c1000.ohci: irq 26, io mem 0x101c1000
[    5.738258] hub 2-0:1.0: USB hub found
[    5.742536] hub 2-0:1.0: 1 port detected
[    5.761516] MTK MSDC device init.
[    5.823406] mtk-sd: MediaTek MT6575 MSDC Driver
[    5.835017] sdhci: Secure Digital Host Controller Interface driver
[    5.841308] sdhci: Copyright(c) Pierre Ossman
[    5.847413] sdhci-pltfm: SDHCI platform and OF driver helper
[    5.859313] usbcore: registered new interface driver usb-storage
[    5.871232] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    5.889141] init: - preinit -
[    6.146398] mmc0: new high speed SDHC card at address aaaa
[    6.166433] mmcblk0: mmc0:aaaa SK32G 29.7 GiB
[    6.316622]  mmcblk0: p1
[    6.621932] random: procd: uninitialized urandom read (4 bytes read)
[    7.122444] rt3050-esw 10110000.esw: link changed 0x00
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
1
[    8.009329] mount_root: loading kmods from internal overlay
[    8.085787] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[    8.099192] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[    9.308077] jffs2: notice: (525) jffs2_build_xattr_subsystem: complete building xattr subsystem, 24 of xdatum (2 unchecked, 22 orphan) and 51 of xref (22 dead, 0 orphan) found.
[    9.333593] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[    9.355189] block: extroot: not configured
[    9.516086] jffs2: notice: (523) jffs2_build_xattr_subsystem: complete building xattr subsystem, 24 of xdatum (2 unchecked, 22 orphan) and 51 of xref (22 dead, 0 orphan) found.
[    9.534502] mount_root: loading kmods from internal overlay
[    9.608432] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    9.624605] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[   10.435361] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[   10.450089] block: extroot: not configured
[   10.455833] mount_root: switching to jffs2 overlay
[   11.085676] rt3050-esw 10110000.esw: link changed 0x01
[   12.090812] overlayfs: upper fs does not support tmpfile.
[   12.101469] urandom-seed: Seed file not found (/etc/urandom.seed)
[   12.242934] procd: - early -
[   12.246061] procd: - watchdog -
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such file or directory
json_object_from_file: error opening file /etc/hotplug.json: No such fijson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromopen: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory

Looks like the file didn't got attached. See the below snippet.

   ____       _             ____
  / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
 / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
 \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
 W H A T  W I L L  Y O U  I N V E N T ? /___/"

Board: Onion Omega2 APSoC DRAM:  128 MB
relocate_code Pointer at: 87f60000
▒

   ____       _             ____
  / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
 / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
 \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
 W H A T  W I L L  Y O U  I N V E N T ? /___/"

Board: Onion Omega2 APSoC DRAM:  128 MB
relocate_code Pointer at: 87f60000
flash manufacture id: ef, device id 40 19
find flash: W25Q256FV
*** Warning - bad CRC, using default environment

============================================
Onion Omega2 UBoot Version: 4.3.0.3
--------------------------------------------
ASIC 7628_MP (Port5<->None)
DRAM component: 1024 Mbits DDR, width 16
DRAM bus: 16 bit
Total memory: 128 MBytes
Flash component: SPI Flash
Date:Nov 18 2022  Time:19:21:40
============================================
icache: sets:512, ways:4, linesz:32 ,total:65536
dcache: sets:256, ways:4, linesz:32 ,total:32768
CPU freq = 575 MHZ
Estimated memory size = 128 Mbytes
Resetting MT7628 PHY.
Initializing MT7688 GPIO system.
wifi mac address = 881E59010440.


**************************************
* Hold Reset button for more options *
**************************************

Boot Linux from Flash NO RESET PRESSED.
## Booting image at bc050000 ...
   Image Name:   MIPS OpenWrt Linux-4.14.81
   Image Type:   MIPS Linux Kernel Image (lzma compressed)
   Data Size:    1617506 Bytes =  1.5 MB
   Load Address: 80000000
   Entry Point:  80000000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
No initrd
## Transferring control to Linux (at address 80000000) ...
## Giving linux memsize in MB, 128

Starting kernel ...

[    0.000000] Linux version 4.14.81 (mrahul@DESKTOP-UIQA19L) (gcc version 7.3.0                                                                                                                                                              (OpenWrt GCC 7.3.0 r0+7496-be7777b369)) #0 Fri Nov 18 20:23:14 2022
[    0.000000] Board has DDR2
[    0.000000] Analog PMU set to hw control
[    0.000000] Digital PMU set to hw control
[    0.000000] SoC Type: MediaTek MT7688 ver:1 eco:2
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019655 (MIPS 24KEc)
[    0.000000] MIPS: machine is Onion Omega2+
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 byt                                                                                                                                                             es
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] random: get_random_bytes called from start_kernel+0x8c/0x474 with                                                                                                                                                              crng_init=0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jff                                                                                                                                                             s2
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Writing ErrCtl register=00024bf4
[    0.000000] Readback ErrCtl register=00024bf4
[    0.000000] Memory: 124580K/131072K available (3708K kernel code, 207K rwdata                                                                                                                                                             , 920K rodata, 212K init, 205K bss, 6492K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 256
[    0.000000] intc: using register map from devicetree
[    0.000000] CPU Clock: 580MHz
[    0.000000] timer_probe: no matching timers found
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_i                                                                                                                                                             dle_ns: 6590553264 ns
[    0.000012] sched_clock: 32 bits at 290MHz, resolution 3ns, wraps every 74051                                                                                                                                                             15902ns
[    0.007673] Console: colour dummy device 80x25
[    0.011932] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216)
[    0.073505] pid_max: default: 32768 minimum: 301
[    0.078194] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.084537] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.098565] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, ma                                                                                                                                                             x_idle_ns: 19112604462750000 ns
[    0.108059] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.114006] pinctrl core: initialized pinctrl subsystem
[    0.119578] NET: Registered protocol family 16
[    0.155795] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.161479] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.167003] mt7621_gpio 10000600.gpio: registering 32 gpios
[    0.173623] i2c-mt7621 10000900.i2c: clock 100 kHz
[    0.183174] clocksource: Switched to clocksource MIPS
[    0.195416] NET: Registered protocol family 2
[    0.200548] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.207295] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.213399] TCP: Hash tables configured (established 1024 bind 1024)
[    0.219660] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.225294] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.231587] NET: Registered protocol family 1
[    0.250227] Crashlog allocated RAM at address 0x3f00000
[    0.256838] workingset: timestamp_bits=30 max_order=15 bucket_order=0
[    0.269152] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.274771] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORIT                                                                                                                                                             Y) (c) 2001-2006 Red Hat, Inc.
[    0.294884] io scheduler noop registered
[    0.298616] io scheduler deadline registered (default)
[    0.304655] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    0.311995] console [ttyS0] disabled
[    0.315495] 10000c00.uartlite: ttyS0 at MMIO 0x10000c00 (irq = 28, base_baud                                                                                                                                                              = 2500000) is a 16550A
[    0.324218] console [ttyS0] enabled
[    0.324218] console [ttyS0] enabled
[    0.331212] bootconsole [early0] disabled
[    0.331212] bootconsole [early0] disabled
[    0.340015] 10000d00.uart1: ttyS1 at MMIO 0x10000d00 (irq = 29, base_baud = 2                                                                                                                                                             500000) is a 16550A
[    0.349688] 10000e00.uart2: ttyS2 at MMIO 0x10000e00 (irq = 30, base_baud = 2                                                                                                                                                             500000) is a 16550A
[    0.359203] cacheinfo: Failed to find cpu0 device node
[    0.364451] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.371421] spi-mt7621 10000b00.spi: sys_freq: 193333333
[    0.393246] m25p80 spi0.0: w25q256 (32768 Kbytes)
[    0.398079] 5 fixed-partitions partitions found on MTD device spi0.0
[    0.404540] Creating 5 MTD partitions on "spi0.0":
[    0.409406] 0x000000000000-0x000000030000 : "u-boot"
[    0.415392] 0x000000030000-0x000000040000 : "u-boot-env"
[    0.421595] 0x000000040000-0x000000050000 : "factory"
[    0.427642] 0x000000050000-0x000001f80000 : "firmware"
[    0.522885] 2 uimage-fw partitions found on MTD device firmware
[    0.528933] 0x000000050000-0x0000001daea2 : "kernel"
[    0.534907] 0x0000001daea2-0x000001f80000 : "rootfs"
[    0.540716] mtd: device 5 (rootfs) set to be root filesystem
[    0.547961] 1 squashfs-split partitions found on MTD device rootfs
[    0.554304] 0x000000e30000-0x000001f80000 : "rootfs_data"
[    0.560715] 0x000001f80000-0x000002000000 : "user"
[    0.567485] libphy: Fixed MDIO Bus: probed
[    0.583351] rt3050-esw 10110000.esw: link changed 0x00
[    0.590230] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0                                                                                                                                                             100000, irq 5
[    0.598930] i2c /dev entries driver
[    0.604383] NET: Registered protocol family 10
[    0.613295] Segment Routing with IPv6
[    0.617135] NET: Registered protocol family 17
[    0.621690] 8021q: 802.1Q VLAN Support v1.8
[    0.635529] VFS: Mounted root (squashfs filesystem) readonly on device 31:5.
[    0.643888] Freeing unused kernel memory: 212K
[    0.648393] This architecture does not have kernel memory protection.
[    1.502393] init: Console is alive
[    1.506184] init: - watchdog -
[    2.233187] random: fast init done
[    5.163269] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    5.447933] usbcore: registered new interface driver usbfs
[    5.453681] usbcore: registered new interface driver hub
[    5.459170] usbcore: registered new device driver usb
[    5.470686] exFAT: Version 1.2.9
[    5.509876] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.526289] SCSI subsystem initialized
[    5.535915] ehci-platform: EHCI generic platform driver
[    5.551587] phy phy-10120000.usbphy.0: remote usb device wakeup disabled
[    5.558399] phy phy-10120000.usbphy.0: UTMI 16bit 30MHz
[    5.563739] ehci-platform 101c0000.ehci: EHCI Host Controller
[    5.569599] ehci-platform 101c0000.ehci: new USB bus registered, assigned bus                                                                                                                                                              number 1
[    5.577790] ehci-platform 101c0000.ehci: irq 26, io mem 0x101c0000
[    5.613219] ehci-platform 101c0000.ehci: USB 2.0 started, EHCI 1.00
[    5.620672] hub 1-0:1.0: USB hub found
[    5.624948] hub 1-0:1.0: 1 port detected
[    5.635021] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    5.642890] ohci-platform: OHCI generic platform driver
[    5.648582] ohci-platform 101c1000.ohci: Generic Platform OHCI controller
[    5.655550] ohci-platform 101c1000.ohci: new USB bus registered, assigned bus                                                                                                                                                              number 2
[    5.663679] ohci-platform 101c1000.ohci: irq 26, io mem 0x101c1000
[    5.738259] hub 2-0:1.0: USB hub found
[    5.742540] hub 2-0:1.0: 1 port detected
[    5.761461] MTK MSDC device init.
[    5.823405] mtk-sd: MediaTek MT6575 MSDC Driver
[    5.835151] sdhci: Secure Digital Host Controller Interface driver
[    5.841443] sdhci: Copyright(c) Pierre Ossman
[    5.847575] sdhci-pltfm: SDHCI platform and OF driver helper
[    5.859496] usbcore: registered new interface driver usb-storage
[    5.871413] kmodloader: done loading kernel modules from /etc/modules-boot.d/                                                                                                                                                             *
[    5.889317] init: - preinit -
[    6.196335] mmc0: new high speed SDHC card at address aaaa
[    6.294432] mmcblk0: mmc0:aaaa SK32G 29.7 GiB
[    6.306725]  mmcblk0: p1
[    7.107856] rt3050-esw 10110000.esw: link changed 0x00
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[    7.305586] random: procd: uninitialized urandom read (4 bytes read)
[   10.485125] mount_root: loading kmods from internal overlay
[   10.558033] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[   10.571499] kmodloader: done loading kernel modules from //etc/modules-boot.d                                                                                                                                                             /*
[   11.054705] rt3050-esw 10110000.esw: link changed 0x01
[   11.778627] jffs2: notice: (526) jffs2_build_xattr_subsystem: complete buildi                                                                                                                                                             ng xattr subsystem, 9 of xdatum (2 unchecked, 7 orphan) and 36 of xref (7 dead,                                                                                                                                                              0 orphan) found.
[   11.795231] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[   11.816017] block: extroot: not configured
[   11.972418] jffs2: notice: (524) jffs2_build_xattr_subsystem: complete buildi                                                                                                                                                             ng xattr subsystem, 9 of xdatum (2 unchecked, 7 orphan) and 36 of xref (7 dead,                                                                                                                                                              0 orphan) found.
[   11.990545] mount_root: loading kmods from internal overlay
[   12.068078] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/mo                                                                                                                                                             dules-boot.d/*
[   12.084420] kmodloader: done loading kernel modules from /tmp/overlay/upper/e                                                                                                                                                             tc/modules-boot.d/*
[   12.890642] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[   12.905336] block: extroot: not configured
[   12.910773] mount_root: switching to jffs2 overlay
[   14.549628] overlayfs: upper fs does not support tmpfile.
[   14.560131] urandom-seed: Seed file not found (/etc/urandom.seed)
[   14.701247] procd: - early -
[   14.704387] procd: - watchdog -
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such file or dir                                                                                                                                                             ectory
json_object_from_file: error opening file /etc/hotplug.json: No such fijson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje                                                                                                                                                             ct_fromopen: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory
open: No such file or directory

[0_1722361906780_587.txt](Uploading 100%) We use OM2+ in lot of our commercial devices, say 700+. Most of our devices are from the 88:1E:59. We came across this notice from Onion that there is compatibility issue with wifi-warp-core driver with builds lower than b255.

Firstly, no builds other than b255 works on these devices. Most of them had b256 and even if we download and flash the same, b256, image from Onion Firmware Repo, it crashes and gets stuck into a continuous booting process.

We embed some custom binaries and scripts required by our devices with the build system to create an OS image. We flashed a couple of devices with this image. Initially, it worked fine and we let it run for 12+ hours. We power cycled these devices today morning and some of them got bricked. I have attached a log file of one of the device's boot process where you can see I'm getting some file missing error.

I personally feel that there could be some other dependencies involved other than wifi-warp-core driver for this build. I'm still looking into it.

Any thoughts? @crispyoz or @Lazar-Demin This is really affecting our production line making the whole process to come to a halt. Any feedback would be greatly appreciated. Thanks in advance.

[0_1722361887022_587.txt](Uploading 100%)

Thanks, @crispyoz & @Lazar-Demin. I'll try these suggestions and get back with the logs.

In the meantime, apart from filesystem corruption, are there any other possible scenarios where the device can get reset?