OM2+ getting stuck in boot stage with custom firmware using b255
-
[0_1722361906780_587.txt](Uploading 100%) We use OM2+ in lot of our commercial devices, say 700+. Most of our devices are from the
88:1E:59. We came across this notice from Onion that there is compatibility issue withwifi-warp-coredriver with builds lower thanb255.Firstly, no builds other than
b255works on these devices. Most of them hadb256and even if we download and flash the same,b256, image from Onion Firmware Repo, it crashes and gets stuck into a continuous booting process.We embed some custom binaries and scripts required by our devices with the build system to create an OS image. We flashed a couple of devices with this image. Initially, it worked fine and we let it run for 12+ hours. We power cycled these devices today morning and some of them got bricked. I have attached a log file of one of the device's boot process where you can see I'm getting some file missing error.
I personally feel that there could be some other dependencies involved other than
wifi-warp-coredriver for this build. I'm still looking into it.Any thoughts? @crispyoz or @Lazar-Demin This is really affecting our production line making the whole process to come to a halt. Any feedback would be greatly appreciated. Thanks in advance.
[0_1722361887022_587.txt](Uploading 100%)
-
Looks like the file didn't got attached. See the below snippet.
____ _ ____ / __ \___ (_)__ ___ / __ \__ _ ___ ___ ____ _ / /_/ / _ \/ / _ \/ _ \ / /_/ / ' \/ -_) _ `/ _ `/ \____/_//_/_/\___/_//_/ \____/_/_/_/\__/\_, /\_,_/ W H A T W I L L Y O U I N V E N T ? /___/" Board: Onion Omega2 APSoC DRAM: 128 MB relocate_code Pointer at: 87f60000 ▒ ____ _ ____ / __ \___ (_)__ ___ / __ \__ _ ___ ___ ____ _ / /_/ / _ \/ / _ \/ _ \ / /_/ / ' \/ -_) _ `/ _ `/ \____/_//_/_/\___/_//_/ \____/_/_/_/\__/\_, /\_,_/ W H A T W I L L Y O U I N V E N T ? /___/" Board: Onion Omega2 APSoC DRAM: 128 MB relocate_code Pointer at: 87f60000 flash manufacture id: ef, device id 40 19 find flash: W25Q256FV *** Warning - bad CRC, using default environment ============================================ Onion Omega2 UBoot Version: 4.3.0.3 -------------------------------------------- ASIC 7628_MP (Port5<->None) DRAM component: 1024 Mbits DDR, width 16 DRAM bus: 16 bit Total memory: 128 MBytes Flash component: SPI Flash Date:Nov 18 2022 Time:19:21:40 ============================================ icache: sets:512, ways:4, linesz:32 ,total:65536 dcache: sets:256, ways:4, linesz:32 ,total:32768 CPU freq = 575 MHZ Estimated memory size = 128 Mbytes Resetting MT7628 PHY. Initializing MT7688 GPIO system. wifi mac address = 881E59010440. ************************************** * Hold Reset button for more options * ************************************** Boot Linux from Flash NO RESET PRESSED. ## Booting image at bc050000 ... Image Name: MIPS OpenWrt Linux-4.14.81 Image Type: MIPS Linux Kernel Image (lzma compressed) Data Size: 1617506 Bytes = 1.5 MB Load Address: 80000000 Entry Point: 80000000 Verifying Checksum ... OK Uncompressing Kernel Image ... OK No initrd ## Transferring control to Linux (at address 80000000) ... ## Giving linux memsize in MB, 128 Starting kernel ... [ 0.000000] Linux version 4.14.81 (mrahul@DESKTOP-UIQA19L) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r0+7496-be7777b369)) #0 Fri Nov 18 20:23:14 2022 [ 0.000000] Board has DDR2 [ 0.000000] Analog PMU set to hw control [ 0.000000] Digital PMU set to hw control [ 0.000000] SoC Type: MediaTek MT7688 ver:1 eco:2 [ 0.000000] bootconsole [early0] enabled [ 0.000000] CPU0 revision is: 00019655 (MIPS 24KEc) [ 0.000000] MIPS: machine is Onion Omega2+ [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 08000000 @ 00000000 (usable) [ 0.000000] Initrd not found or empty - disabling initrd [ 0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes. [ 0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 byt es [ 0.000000] Zone ranges: [ 0.000000] Normal [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] random: get_random_bytes called from start_kernel+0x8c/0x474 with crng_init=0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 32512 [ 0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jff s2 [ 0.000000] PID hash table entries: 512 (order: -1, 2048 bytes) [ 0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) [ 0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) [ 0.000000] Writing ErrCtl register=00024bf4 [ 0.000000] Readback ErrCtl register=00024bf4 [ 0.000000] Memory: 124580K/131072K available (3708K kernel code, 207K rwdata , 920K rodata, 212K init, 205K bss, 6492K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS: 256 [ 0.000000] intc: using register map from devicetree [ 0.000000] CPU Clock: 580MHz [ 0.000000] timer_probe: no matching timers found [ 0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_i dle_ns: 6590553264 ns [ 0.000012] sched_clock: 32 bits at 290MHz, resolution 3ns, wraps every 74051 15902ns [ 0.007673] Console: colour dummy device 80x25 [ 0.011932] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216) [ 0.073505] pid_max: default: 32768 minimum: 301 [ 0.078194] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.084537] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.098565] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, ma x_idle_ns: 19112604462750000 ns [ 0.108059] futex hash table entries: 256 (order: -1, 3072 bytes) [ 0.114006] pinctrl core: initialized pinctrl subsystem [ 0.119578] NET: Registered protocol family 16 [ 0.155795] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.161479] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.167003] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.173623] i2c-mt7621 10000900.i2c: clock 100 kHz [ 0.183174] clocksource: Switched to clocksource MIPS [ 0.195416] NET: Registered protocol family 2 [ 0.200548] TCP established hash table entries: 1024 (order: 0, 4096 bytes) [ 0.207295] TCP bind hash table entries: 1024 (order: 0, 4096 bytes) [ 0.213399] TCP: Hash tables configured (established 1024 bind 1024) [ 0.219660] UDP hash table entries: 256 (order: 0, 4096 bytes) [ 0.225294] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes) [ 0.231587] NET: Registered protocol family 1 [ 0.250227] Crashlog allocated RAM at address 0x3f00000 [ 0.256838] workingset: timestamp_bits=30 max_order=15 bucket_order=0 [ 0.269152] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.274771] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORIT Y) (c) 2001-2006 Red Hat, Inc. [ 0.294884] io scheduler noop registered [ 0.298616] io scheduler deadline registered (default) [ 0.304655] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled [ 0.311995] console [ttyS0] disabled [ 0.315495] 10000c00.uartlite: ttyS0 at MMIO 0x10000c00 (irq = 28, base_baud = 2500000) is a 16550A [ 0.324218] console [ttyS0] enabled [ 0.324218] console [ttyS0] enabled [ 0.331212] bootconsole [early0] disabled [ 0.331212] bootconsole [early0] disabled [ 0.340015] 10000d00.uart1: ttyS1 at MMIO 0x10000d00 (irq = 29, base_baud = 2 500000) is a 16550A [ 0.349688] 10000e00.uart2: ttyS2 at MMIO 0x10000e00 (irq = 30, base_baud = 2 500000) is a 16550A [ 0.359203] cacheinfo: Failed to find cpu0 device node [ 0.364451] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 0.371421] spi-mt7621 10000b00.spi: sys_freq: 193333333 [ 0.393246] m25p80 spi0.0: w25q256 (32768 Kbytes) [ 0.398079] 5 fixed-partitions partitions found on MTD device spi0.0 [ 0.404540] Creating 5 MTD partitions on "spi0.0": [ 0.409406] 0x000000000000-0x000000030000 : "u-boot" [ 0.415392] 0x000000030000-0x000000040000 : "u-boot-env" [ 0.421595] 0x000000040000-0x000000050000 : "factory" [ 0.427642] 0x000000050000-0x000001f80000 : "firmware" [ 0.522885] 2 uimage-fw partitions found on MTD device firmware [ 0.528933] 0x000000050000-0x0000001daea2 : "kernel" [ 0.534907] 0x0000001daea2-0x000001f80000 : "rootfs" [ 0.540716] mtd: device 5 (rootfs) set to be root filesystem [ 0.547961] 1 squashfs-split partitions found on MTD device rootfs [ 0.554304] 0x000000e30000-0x000001f80000 : "rootfs_data" [ 0.560715] 0x000001f80000-0x000002000000 : "user" [ 0.567485] libphy: Fixed MDIO Bus: probed [ 0.583351] rt3050-esw 10110000.esw: link changed 0x00 [ 0.590230] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0 100000, irq 5 [ 0.598930] i2c /dev entries driver [ 0.604383] NET: Registered protocol family 10 [ 0.613295] Segment Routing with IPv6 [ 0.617135] NET: Registered protocol family 17 [ 0.621690] 8021q: 802.1Q VLAN Support v1.8 [ 0.635529] VFS: Mounted root (squashfs filesystem) readonly on device 31:5. [ 0.643888] Freeing unused kernel memory: 212K [ 0.648393] This architecture does not have kernel memory protection. [ 1.502393] init: Console is alive [ 1.506184] init: - watchdog - [ 2.233187] random: fast init done [ 5.163269] kmodloader: loading kernel modules from /etc/modules-boot.d/* [ 5.447933] usbcore: registered new interface driver usbfs [ 5.453681] usbcore: registered new interface driver hub [ 5.459170] usbcore: registered new device driver usb [ 5.470686] exFAT: Version 1.2.9 [ 5.509876] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.526289] SCSI subsystem initialized [ 5.535915] ehci-platform: EHCI generic platform driver [ 5.551587] phy phy-10120000.usbphy.0: remote usb device wakeup disabled [ 5.558399] phy phy-10120000.usbphy.0: UTMI 16bit 30MHz [ 5.563739] ehci-platform 101c0000.ehci: EHCI Host Controller [ 5.569599] ehci-platform 101c0000.ehci: new USB bus registered, assigned bus number 1 [ 5.577790] ehci-platform 101c0000.ehci: irq 26, io mem 0x101c0000 [ 5.613219] ehci-platform 101c0000.ehci: USB 2.0 started, EHCI 1.00 [ 5.620672] hub 1-0:1.0: USB hub found [ 5.624948] hub 1-0:1.0: 1 port detected [ 5.635021] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.642890] ohci-platform: OHCI generic platform driver [ 5.648582] ohci-platform 101c1000.ohci: Generic Platform OHCI controller [ 5.655550] ohci-platform 101c1000.ohci: new USB bus registered, assigned bus number 2 [ 5.663679] ohci-platform 101c1000.ohci: irq 26, io mem 0x101c1000 [ 5.738259] hub 2-0:1.0: USB hub found [ 5.742540] hub 2-0:1.0: 1 port detected [ 5.761461] MTK MSDC device init. [ 5.823405] mtk-sd: MediaTek MT6575 MSDC Driver [ 5.835151] sdhci: Secure Digital Host Controller Interface driver [ 5.841443] sdhci: Copyright(c) Pierre Ossman [ 5.847575] sdhci-pltfm: SDHCI platform and OF driver helper [ 5.859496] usbcore: registered new interface driver usb-storage [ 5.871413] kmodloader: done loading kernel modules from /etc/modules-boot.d/ * [ 5.889317] init: - preinit - [ 6.196335] mmc0: new high speed SDHC card at address aaaa [ 6.294432] mmcblk0: mmc0:aaaa SK32G 29.7 GiB [ 6.306725] mmcblk0: p1 [ 7.107856] rt3050-esw 10110000.esw: link changed 0x00 Press the [f] key and hit [enter] to enter failsafe mode Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level [ 7.305586] random: procd: uninitialized urandom read (4 bytes read) [ 10.485125] mount_root: loading kmods from internal overlay [ 10.558033] kmodloader: loading kernel modules from //etc/modules-boot.d/* [ 10.571499] kmodloader: done loading kernel modules from //etc/modules-boot.d /* [ 11.054705] rt3050-esw 10110000.esw: link changed 0x01 [ 11.778627] jffs2: notice: (526) jffs2_build_xattr_subsystem: complete buildi ng xattr subsystem, 9 of xdatum (2 unchecked, 7 orphan) and 36 of xref (7 dead, 0 orphan) found. [ 11.795231] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab [ 11.816017] block: extroot: not configured [ 11.972418] jffs2: notice: (524) jffs2_build_xattr_subsystem: complete buildi ng xattr subsystem, 9 of xdatum (2 unchecked, 7 orphan) and 36 of xref (7 dead, 0 orphan) found. [ 11.990545] mount_root: loading kmods from internal overlay [ 12.068078] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/mo dules-boot.d/* [ 12.084420] kmodloader: done loading kernel modules from /tmp/overlay/upper/e tc/modules-boot.d/* [ 12.890642] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab [ 12.905336] block: extroot: not configured [ 12.910773] mount_root: switching to jffs2 overlay [ 14.549628] overlayfs: upper fs does not support tmpfile. [ 14.560131] urandom-seed: Seed file not found (/etc/urandom.seed) [ 14.701247] procd: - early - [ 14.704387] procd: - watchdog - json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such file or dir ectory json_object_from_file: error opening file /etc/hotplug.json: No such fijson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_obje ct_fromopen: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory
-
@mrahul Note in you log above, it says: "Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level", select 4 and enter then we will have more info.
-
Thanks for the quick response @crispyoz I tried what you suggested but looks like it's not recognizing any inputs. I tried all 4 debug levels but the messages didn't change. Everytime time the logs are same. See the logs below
____ _ ____ / __ \___ (_)__ ___ / __ \__ _ ___ ___ ____ _ / /_/ / _ \/ / _ \/ _ \ / /_/ / ' \/ -_) _ `/ _ `/ \____/_//_/_/\___/_//_/ \____/_/_/_/\__/\_, /\_,_/ W H A T W I L L Y O U I N V E N T ? /___/" Board: Onion Omega2 APSoC DRAM: 128 MB relocate_code Pointer at: 87f60000 flash manufacture id: ef, device id 40 19 find flash: W25Q256FV *** Warning - bad CRC, using default environment ============================================ Onion Omega2 UBoot Version: 4.3.0.3 -------------------------------------------- ASIC 7628_MP (Port5<->None) DRAM component: 1024 Mbits DDR, width 16 DRAM bus: 16 bit Total memory: 128 MBytes Flash component: SPI Flash Date:Nov 18 2022 Time:19:21:40 ============================================ icache: sets:512, ways:4, linesz:32 ,total:65536 dcache: sets:256, ways:4, linesz:32 ,total:32768 CPU freq = 575 MHZ Estimated memory size = 128 Mbytes Resetting MT7628 PHY. Initializing MT7688 GPIO system. wifi mac address = 881E59010029. ************************************** * Hold Reset button for more options * ************************************** Boot Linux from Flash NO RESET PRESSED. ## Booting image at bc050000 ... Image Name: MIPS OpenWrt Linux-4.14.81 Image Type: MIPS Linux Kernel Image (lzma compressed) Data Size: 1617506 Bytes = 1.5 MB Load Address: 80000000 Entry Point: 80000000 Verifying Checksum ... OK Uncompressing Kernel Image ... OK No initrd ## Transferring control to Linux (at address 80000000) ... ## Giving linux memsize in MB, 128 Starting kernel ... [ 0.000000] Linux version 4.14.81 (mrahul@DESKTOP-UIQA19L) (gcc version 7.3.0 (OpenWrt GCC 7.3.0 r0+7496-be7777b369)) #0 Fri Nov 18 20:23:14 2022 [ 0.000000] Board has DDR2 [ 0.000000] Analog PMU set to hw control [ 0.000000] Digital PMU set to hw control [ 0.000000] SoC Type: MediaTek MT7688 ver:1 eco:2 [ 0.000000] bootconsole [early0] enabled [ 0.000000] CPU0 revision is: 00019655 (MIPS 24KEc) [ 0.000000] MIPS: machine is Onion Omega2+ [ 0.000000] Determined physical RAM map: [ 0.000000] memory: 08000000 @ 00000000 (usable) [ 0.000000] Initrd not found or empty - disabling initrd [ 0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes. [ 0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes [ 0.000000] Zone ranges: [ 0.000000] Normal [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff] [ 0.000000] random: get_random_bytes called from start_kernel+0x8c/0x474 with crng_init=0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 32512 [ 0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2 [ 0.000000] PID hash table entries: 512 (order: -1, 2048 bytes) [ 0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) [ 0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) [ 0.000000] Writing ErrCtl register=00018359 [ 0.000000] Readback ErrCtl register=00018359 [ 0.000000] Memory: 124580K/131072K available (3708K kernel code, 207K rwdata, 920K rodata, 212K init, 205K bss, 6492K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS: 256 [ 0.000000] intc: using register map from devicetree [ 0.000000] CPU Clock: 580MHz [ 0.000000] timer_probe: no matching timers found [ 0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6590553264 ns [ 0.000012] sched_clock: 32 bits at 290MHz, resolution 3ns, wraps every 7405115902ns [ 0.007673] Console: colour dummy device 80x25 [ 0.011931] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216) [ 0.073505] pid_max: default: 32768 minimum: 301 [ 0.078194] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.084537] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.098565] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.108059] futex hash table entries: 256 (order: -1, 3072 bytes) [ 0.114006] pinctrl core: initialized pinctrl subsystem [ 0.119578] NET: Registered protocol family 16 [ 0.155795] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.161479] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.167002] mt7621_gpio 10000600.gpio: registering 32 gpios [ 0.173623] i2c-mt7621 10000900.i2c: clock 100 kHz [ 0.183173] clocksource: Switched to clocksource MIPS [ 0.195416] NET: Registered protocol family 2 [ 0.200547] TCP established hash table entries: 1024 (order: 0, 4096 bytes) [ 0.207294] TCP bind hash table entries: 1024 (order: 0, 4096 bytes) [ 0.213398] TCP: Hash tables configured (established 1024 bind 1024) [ 0.219659] UDP hash table entries: 256 (order: 0, 4096 bytes) [ 0.225293] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes) [ 0.231586] NET: Registered protocol family 1 [ 0.250226] Crashlog allocated RAM at address 0x3f00000 [ 0.256838] workingset: timestamp_bits=30 max_order=15 bucket_order=0 [ 0.269153] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 0.274771] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc. [ 0.294884] io scheduler noop registered [ 0.298615] io scheduler deadline registered (default) [ 0.304654] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled [ 0.311994] console [ttyS0] disabled [ 0.315495] 10000c00.uartlite: ttyS0 at MMIO 0x10000c00 (irq = 28, base_baud = 2500000) is a 16550A [ 0.324217] console [ttyS0] enabled [ 0.324217] console [ttyS0] enabled [ 0.331212] bootconsole [early0] disabled [ 0.331212] bootconsole [early0] disabled [ 0.340014] 10000d00.uart1: ttyS1 at MMIO 0x10000d00 (irq = 29, base_baud = 2500000) is a 16550A [ 0.349688] 10000e00.uart2: ttyS2 at MMIO 0x10000e00 (irq = 30, base_baud = 2500000) is a 16550A [ 0.359202] cacheinfo: Failed to find cpu0 device node [ 0.364450] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 0.371420] spi-mt7621 10000b00.spi: sys_freq: 193333333 [ 0.393126] m25p80 spi0.0: w25q256 (32768 Kbytes) [ 0.398001] 5 fixed-partitions partitions found on MTD device spi0.0 [ 0.404455] Creating 5 MTD partitions on "spi0.0": [ 0.409321] 0x000000000000-0x000000030000 : "u-boot" [ 0.415306] 0x000000030000-0x000000040000 : "u-boot-env" [ 0.421509] 0x000000040000-0x000000050000 : "factory" [ 0.427556] 0x000000050000-0x000001f80000 : "firmware" [ 0.522794] 2 uimage-fw partitions found on MTD device firmware [ 0.528842] 0x000000050000-0x0000001daea2 : "kernel" [ 0.534819] 0x0000001daea2-0x000001f80000 : "rootfs" [ 0.540628] mtd: device 5 (rootfs) set to be root filesystem [ 0.547873] 1 squashfs-split partitions found on MTD device rootfs [ 0.554216] 0x000000e30000-0x000001f80000 : "rootfs_data" [ 0.560627] 0x000001f80000-0x000002000000 : "user" [ 0.567397] libphy: Fixed MDIO Bus: probed [ 0.583352] rt3050-esw 10110000.esw: link changed 0x00 [ 0.590231] mtk_soc_eth 10100000.ethernet eth0: mediatek frame engine at 0xb0100000, irq 5 [ 0.598931] i2c /dev entries driver [ 0.604385] NET: Registered protocol family 10 [ 0.613296] Segment Routing with IPv6 [ 0.617135] NET: Registered protocol family 17 [ 0.621690] 8021q: 802.1Q VLAN Support v1.8 [ 0.635528] VFS: Mounted root (squashfs filesystem) readonly on device 31:5. [ 0.643889] Freeing unused kernel memory: 212K [ 0.648394] This architecture does not have kernel memory protection. [ 1.502472] init: Console is alive [ 1.506263] init: - watchdog - [ 2.223186] random: fast init done [ 5.165744] kmodloader: loading kernel modules from /etc/modules-boot.d/* [ 5.450996] usbcore: registered new interface driver usbfs [ 5.456739] usbcore: registered new interface driver hub [ 5.462230] usbcore: registered new device driver usb [ 5.473864] exFAT: Version 1.2.9 [ 5.512981] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 5.529372] SCSI subsystem initialized [ 5.538863] ehci-platform: EHCI generic platform driver [ 5.554560] phy phy-10120000.usbphy.0: remote usb device wakeup disabled [ 5.561362] phy phy-10120000.usbphy.0: UTMI 16bit 30MHz [ 5.566698] ehci-platform 101c0000.ehci: EHCI Host Controller [ 5.572558] ehci-platform 101c0000.ehci: new USB bus registered, assigned bus number 1 [ 5.580749] ehci-platform 101c0000.ehci: irq 26, io mem 0x101c0000 [ 5.613218] ehci-platform 101c0000.ehci: USB 2.0 started, EHCI 1.00 [ 5.620679] hub 1-0:1.0: USB hub found [ 5.624952] hub 1-0:1.0: 1 port detected [ 5.635058] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 5.642945] ohci-platform: OHCI generic platform driver [ 5.648638] ohci-platform 101c1000.ohci: Generic Platform OHCI controller [ 5.655606] ohci-platform 101c1000.ohci: new USB bus registered, assigned bus number 2 [ 5.663734] ohci-platform 101c1000.ohci: irq 26, io mem 0x101c1000 [ 5.738258] hub 2-0:1.0: USB hub found [ 5.742536] hub 2-0:1.0: 1 port detected [ 5.761516] MTK MSDC device init. [ 5.823406] mtk-sd: MediaTek MT6575 MSDC Driver [ 5.835017] sdhci: Secure Digital Host Controller Interface driver [ 5.841308] sdhci: Copyright(c) Pierre Ossman [ 5.847413] sdhci-pltfm: SDHCI platform and OF driver helper [ 5.859313] usbcore: registered new interface driver usb-storage [ 5.871232] kmodloader: done loading kernel modules from /etc/modules-boot.d/* [ 5.889141] init: - preinit - [ 6.146398] mmc0: new high speed SDHC card at address aaaa [ 6.166433] mmcblk0: mmc0:aaaa SK32G 29.7 GiB [ 6.316622] mmcblk0: p1 [ 6.621932] random: procd: uninitialized urandom read (4 bytes read) [ 7.122444] rt3050-esw 10110000.esw: link changed 0x00 Press the [f] key and hit [enter] to enter failsafe mode Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level 1 [ 8.009329] mount_root: loading kmods from internal overlay [ 8.085787] kmodloader: loading kernel modules from //etc/modules-boot.d/* [ 8.099192] kmodloader: done loading kernel modules from //etc/modules-boot.d/* [ 9.308077] jffs2: notice: (525) jffs2_build_xattr_subsystem: complete building xattr subsystem, 24 of xdatum (2 unchecked, 22 orphan) and 51 of xref (22 dead, 0 orphan) found. [ 9.333593] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab [ 9.355189] block: extroot: not configured [ 9.516086] jffs2: notice: (523) jffs2_build_xattr_subsystem: complete building xattr subsystem, 24 of xdatum (2 unchecked, 22 orphan) and 51 of xref (22 dead, 0 orphan) found. [ 9.534502] mount_root: loading kmods from internal overlay [ 9.608432] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/* [ 9.624605] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/* [ 10.435361] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab [ 10.450089] block: extroot: not configured [ 10.455833] mount_root: switching to jffs2 overlay [ 11.085676] rt3050-esw 10110000.esw: link changed 0x01 [ 12.090812] overlayfs: upper fs does not support tmpfile. [ 12.101469] urandom-seed: Seed file not found (/etc/urandom.seed) [ 12.242934] procd: - early - [ 12.246061] procd: - watchdog - json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such file or directory json_object_from_file: error opening file /etc/hotplug.json: No such fijson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromjson_object_fromopen: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory open: No such file or directory
-
@mrahul I just noticed yo mentioned this is custom firmware, it looks like it is having an issue executing procd scripts, because you're missing hotplug.json. Do you have hotplug enabled in your configurtion?
-
The default build system is not changed. I believe this is a built-in script that we don't make any changes to. The only customization is adding binaries, scripts, and supporting packages required by our FW. Also, the same build works fine on some devices. However, they all brick, eventually, after there is a power cycle.
The same image worked fine with our older batch of devices, in which we used the build system
b252to create the custom image. Since we came across this notice, we upgraded the build system tob255, as suggested and we started to see this error.
-
-
No build other than
b255are working on these devices. And I don't think trying builds lower thanb255would be sensible on these devices. I tested this with the images downloaded from the Onion FW Repo.However, as far as the custom build goes, I only created a custom image using the
b255build system.
-
@mrahul What happens when b258 firmware is loaded on the device?
Note that not too much has changed in the OnionIoT/source repo between b255 and b258.
For details see: https://github.com/OnionIoT/source/compare/be7777b369c55c53b3582e868fd268ff4bf31318...928cad900fd7ecbf37af5ab6804e1af161e89430The most notable difference is the change of the default repo from
http://downloads.openwrt.org/releases/18.06-SNAPSHOTtohttp://downloads.openwrt.org/releases/18.06.1inpackage/base-files/image-config.inIf the b255 fw released by Onion works on these devices and the custom image based on the b255 build system does not, this points to an issue introduced in the custom image building process.
Can you share more details about that?
-
@Lazar-Demin, I've compared the commits too, and agree that there aren't many significant changes. Flashing
b255from theOnion Firmware Repoworks fine. Talking about the custom image creation process, the issue was observed even when there weren't any changes done to the build system before creating the image.I cloned the
b255commit from the repo, and did no changes. I created the image using the same cloned build system, AS-IS, and when flashed with the image created, the onion still goes in the loop of the booting process. I'm ok to share more details if you want, please let me know what, and I'll share accordingly.Thanks.
-
@mrahul Can I ask how you are updating your devices, are you sysupgrade or flash via u-boot option?
-
I'm using
sysupgradeto flash the custom image.
-
@mrahul said in OM2+ getting stuck in boot stage with custom firmware using b255:
I created the image using the same cloned build system, AS-IS, and when flashed with the image created, the onion still goes in the loop of the booting process. I'm ok to share more details if you want, please let me know what, and I'll share accordingly.
Yes please share more on your process of making the image.
Did you use the instructions from the OnionIoT/source readme?EDIT: actually, the b255 commit of OnionIoT/source shouldn't successfully compile anymore since it points to the OpenWRT
18.06-SNAPSHOTpackage repos. These package repos have since been deleted.
The error you see when booting now makes sense: Linux expects/etc/hotplug.jsonto exist, but since the build system couldn't find that package it's not included in the firmware.You will need to use b257 (and up) to successfully compile firmware. See the b257 commit.
-
@mrahul try sysupgrade -N so the existing configuration is not preserved. I'm wondering if there is something in your configuration creating the issue.
-
Hi @crispyoz & @Lazar-Demin, thank you so much for your input. I've been working on some immediate deliverables, so I won't be able to share what you asked for at least the next few days. I'll post the logs and observations as soon as I try what you both have suggested. Thanks again!
-
Hi @crispyoz & @Lazar-Demin, hope you guys are doing great.
The error you see when booting now makes sense: Linux expects /etc/hotplug.json to exist, but since the build system couldn't find that package it's not included in the firmware.Should we see the same behavior if we use the binary downloaded from the Onion FW Repo?
Lately, we've started seeing the same behavior again with
b255where it gets stuck in the boot process. On checking, it is the same error related to the hotplug.json file not being found. We're not creating and flashing custom images for now, all this is using the binaries from the FW Repo. I saw that there is a new build release,b259, that we are planning to try.One way we recover from this stage is by accessing the device through
failsafemode and doingfirstboot. I'll try flashing the new build usingsysupgrade -N <filename>and share the behavior here. I also wanted to know if there's any difference between the way an image is flashed on the devices. There are methods to upgrade or re-install images like through ethernet, SD Card/USB, and also there'ssysupgrade. However, we only use two methods, i.e. through ethernet orsyupgradeto upgrade or re-flash a device on bricking. How different are these two processes?Thanks again for all the responses and suggestions. Really looking forward to get to the core of this.
-
@mrahul said in OM2+ getting stuck in boot stage with custom firmware using b255:
Should we see the same behavior if we use the binary downloaded from the Onion FW Repo?
No, my previous comment does not apply to firmware in the Onion firmware repo - these firmware images are already compiled and should contain everything needed.
So devices with mac addresses that start with 40:A3:6B should boot and operate properly when flashed with any firmware from the Onion firmware repo.
Devices with mac address addresses starting with 88:1E:59 will boot and operate correctly when flashed with firmware b255 and higher.
I've confirmed this with devices I have on hand
Device with 40:A3:6B mac address running firmware b254:
Device with 40:A3:6B mac address running firmware b259:
Device with 88:1E:59 mac address running firmware b256:
@mrahul said in OM2+ getting stuck in boot stage with custom firmware using b255:
However, we only use two methods, i.e. through ethernet or syupgrade to upgrade or re-flash a device on bricking. How different are these two processes?
For a device that boots into linux successfully, upgrading the firmware thru sysupgrade and thru ethernet with the bootloader will have the same effect. As long as sysupgrade is run with the
-noption to overwrite the existing configuration and filesystem.For a device that cannot boot into Linux, we recommend reflashing the firmware with ethernet thru the bootloader.
@mrahul said in OM2+ getting stuck in boot stage with custom firmware using b255:
Lately, we've started seeing the same behavior again with b255 where it gets stuck in the boot process.
Does this mean the firmware sometimes works and sometimes doesn't?
This could point to something in the hardware design impacting the boot sequence. Do you use any SPI devices? Do you follow the bootstrapping pins guidelines in your hardware?
My recommendation
Since you mentioned you're using the through-hole Omega2+, we can try to isolate where the issue is coming from.
I recommend doing the following:
- Remove an Omega2+ device from your custom board
- Plug it into an Onion Expansion Dock with an Ethernet Expansion
- Use the bootloader + ethernet to flash firmware b259 from the Onion firmware repo
- Observe the outcome, given my testing I expect this device should boot properly
- Plug the device back into your custom board and attempt to boot it again
Let me know how it goes!
