Moderator

@mayur_ingle For my 2c I'd run your device on a standard dock and see if you experience the same issue, I'd move yoru write to an SD Card and would also add a regular file system check that reports to the console to your script.

@Lazar-Demin If you have ipv6 installed it will default a link local address on each interface. Same as 169.254.x.x ipv4 addresses. An easy test is to use ping 6 command:

ping6 -c 2 -i 4 -I wlp0s20f3 ff02::1

Where "wlp0s20f3" is the name of my notebook's interface and the ff02::1 is the link local "all nodes" address, like broadcast address (but that term upsets the ipv6 professors). It will respond with the ipv6 address of all nodes on link local. Here 's what's on my development network:

ping6 -c 2 -i 4 -I wlp0s20f3 ff02::1 
ping6: Warning: source address might be selected on device other than: wlp0s20f3
PING ff02::1 (ff02::1) from :: wlp0s20f3: 56 data bytes
64 bytes from fe80::b06d:1d07:966:4c65%wlp0s20f3: icmp_seq=1 ttl=64 time=0.160 ms
64 bytes from fe80::feec:daff:fe31:e1c6%wlp0s20f3: icmp_seq=1 ttl=64 time=2.94 ms
64 bytes from fe80::8bb4:297b:abb:7968%wlp0s20f3: icmp_seq=1 ttl=64 time=3.27 ms
64 bytes from fe80::46d9:e7ff:feb8:53a7%wlp0s20f3: icmp_seq=1 ttl=64 time=3.54 ms
64 bytes from fe80::6ad7:9aff:fe02:b2b8%wlp0s20f3: icmp_seq=1 ttl=64 time=3.58 ms
64 bytes from fe80::c251:7eff:fe5d:7c73%wlp0s20f3: icmp_seq=1 ttl=64 time=3.58 ms
64 bytes from fe80::76ac:b9ff:fe4e:29db%wlp0s20f3: icmp_seq=1 ttl=64 time=3.58 ms
64 bytes from fe80::b6a3:82ff:fe4b:20b6%wlp0s20f3: icmp_seq=1 ttl=64 time=3.58 ms
64 bytes from fe80::f64d:30ff:fe6b:8b4a%wlp0s20f3: icmp_seq=1 ttl=64 time=3.59 ms
64 bytes from fe80::188a:822d:9b56:843c%wlp0s20f3: icmp_seq=1 ttl=64 time=3.93 ms
64 bytes from fe80::42a3:6bff:fec4:1d8e%wlp0s20f3: icmp_seq=1 ttl=64 time=4.24 ms
64 bytes from fe80::d8f4:6b58:8b5e:cd9f%wlp0s20f3: icmp_seq=1 ttl=64 time=73.7 ms
64 bytes from fe80::a2dc:4ff:fe2a:8411%wlp0s20f3: icmp_seq=1 ttl=64 time=181 ms
64 bytes from fe80::a91:15ff:fefe:8897%wlp0s20f3: icmp_seq=1 ttl=64 time=304 ms
64 bytes from fe80::10b7:9243:cdf4:873e%wlp0s20f3: icmp_seq=1 ttl=64 time=304 ms
64 bytes from fe80::b6fb:e4ff:fec9:3b8%wlp0s20f3: icmp_seq=1 ttl=64 time=308 ms
64 bytes from fe80::f29f:c2ff:fec5:9ab2%wlp0s20f3: icmp_seq=1 ttl=64 time=356 ms
64 bytes from fe80::f29f:c2ff:fe09:fe1%wlp0s20f3: icmp_seq=1 ttl=64 time=878 ms
64 bytes from fe80::b06d:1d07:966:4c65%wlp0s20f3: icmp_seq=2 ttl=64 time=0.081 ms

--- ff02::1 ping statistics ---
2 packets transmitted, 2 received, +17 duplicates, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.081/128.421/877.572/217.804 ms

You can see they all start with "fe80" which indicates these addresses are created by the Stateless Auto Address Configuration function of ipv6 (SLACC). So even if you set an ipv6 address on an interface ipv6 will still create the link local address.

There is a lot to ipv6, but this is one of the nice little features.

@MK Correct. OWRT 19 was really not much more than a kernel and toolchain upgrade. OWRT 20 was never officially released. OWRT 21 included a load of networking changes so it was the most logical to change up to but then OWRT 22 came along pretty quickly and fixed the 2038 date issue and upgraded firewall technology so it was nice step forward. Alas it broke the GPIO numbering so many of us needed to rejig things and that takes time. Hence 23 is the logical step forward.

The custom firmware build I use is mainly to debloat things and add in my own software and configuration, I try to stay as close as possible to the Onion build.

@MK When I switched from OpenWrt 18 to 22 is was quite smooth, but it depends on what functionality you are using. Initially the network setup was a pain, then changing MUX setting of a couple of pins was an issue. Then an issue with ledchain, then an issue requiring conversion of sqlite3 db. Mostly these were easily resolved so now I run 23 on my devices without issue. Also initially I started experimenting with OWRT 19, jumped to 22 then went into production on 23.

Thanks @luz ! I've was offered these LEDs here locally in Oz and was wondering about compatibility.

@mayur_ingle I can see you're using a standard Onion release of the firmware so the SD Card requirements are pre-installed. You can insert an SD card into the slot and you'll see in the log something like this:

[63130.024501] mmc0: new high speed SDHC card at address 59b4
[63130.041132] mmcblk0: mmc0:59b4 SD16G 14.6 GiB 
[63130.048468]  mmcblk0: p1 p2

So we can see the device is mmcblk0 (the first device) and it has two partitions, p1 and p2. You set it to automount using these commands:

uci set fstab.@global[0].auto_mount='1'
uci commit fstab

Re-insert the card and you should see the card is mounted automagically. Use the mount command to see where it was mounted:

/dev/mmcblk0p2 on /mnt/mmcblk0p2 type ext4 (rw,relatime)
/dev/mmcblk0p1 on /mnt/mmcblk0p1 type vfat (rw,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

This card I used above is from a Raspberry Pi 4B so it has two partitions, you probably will only see /dev/mmcblk0p1, you can see it is mounted at /mnt/mmcblk0p1, you can see it's contents using the command:

ls -la /mnt/mmcblk0p1

To set the default mountpoint for the card to my preferred location of /etc/myappname/data we can change this by adding an entry to fstab.First we need the UUID assigned to the SD Card device using the command:

block info

The output will be something like this:

/dev/mtdblock5: UUID="188c96f5-f6939c36-1805def7-637d9f6c" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock6: MOUNT="/overlay" TYPE="jffs2"
/dev/mtdblock7: MOUNT="/mnt/mtdblock7" TYPE="jffs2"
/dev/mmcblk0p1: UUID="3537-3964" LABEL="NO NAME" VERSION="FAT32" MOUNT="/mnt/mmcblk0p1" TYPE="vfat"

You can see the UUID of the mmc device at the bottom is "3537-3964" and the card is formatted as FAT32. Now I can add the default mount point using the following commands:

uci add fstab mount
uci set fstab.@mount[0].uuid='3537-3964'  <--- UUID you found above
uci set fstab.@mount[0].target='/etc/myappname/data'
uci set fstab.@mount[0].enabled='1'
uci commit fstab

Remove the card then reinsert it, you should now be able to see the card is mounted at /etc/myappname/data

You can point your database or script output to this directory and everything will be written to your SD Card.

@mayur_ingle

A few other users have reported file system instability when programs are running that frequently write to the flash storage

I can echo @Lazar-Demin's observation re regular flash writes leading to file system issues. I have a custom PCB based on Omega2S+ that maintains a sqlite3 database of network traffic on a specific set of ports, it then pushes counters via MQTT to our central server every few seconds. The upshot is that it is writing to FLASH sometimes every second or more. After about 3 months of normal use I started to see devices failing with file system issues. JFSS2 was able to fix many of the issues on restart and the sqlite3 db could be rebuilt to largely recover the historical data, but it was not a good long term solution.

I added an SD Card to my design and all my problems went away. Other than my hardware costs I use Kingston 16GB SDHC U1 C10 which you can buy for a few shekels each. A symlink or mountpoint negates the need to modify any software.

@mayur_ingle A custom PCB makes it more tricky. Just so I am clear, you are using the Omega2+ (through hole version) not the Omega2S+ (surface mount)? Do you have a power dock and an ethernet expansion?

If the device can't boot how did you "Verified the /etc/rc.local configuration for any errors." I assume you mean you checked it from version control system, not the device directly.

I have a few thoughts on possible causes:

1. Have you checked your design against Hardware Design Guide? I read that your design is well tested already but I had a similar issue a few years ago where a pin should have been pulled/pushed (I forget) and had accidentally been so, but every once in a while my devices wouldn't boot or started to boot then stopped. I think this is less likely in your case, but worth checking.

2. My guess is the RAM is getting corrupted by something because we can see the correct relocation address, it reads the environment then stops. I recall a similar issue on another custom PCB, the Onion guys looked into it and adding some shielding resolved the issue.

3. You mentioned you are running some scripts, it is possible to corrupt your file system programmatically. Can you provide more detail on what the script(s) are doing.

If you are using an Omega2+ (through hole) my next step would be to insert it into a standard dock and view the boot process using minicom or some other terminal software. I looked at DockLight but haven't used it, but a raw terminal would remove any potential issues of handshaking or such causing the issue.

You can access the flash by removing the cover and accessing the chip directly (if you're adventurous) @luz provided some instructions on this. Check his posts for more details.

@mayur_ingle Thanks for providing the detailed information.

Is the Omega2+ in a dock? If so which one?
Have you tried holding the reset button while turning on the Omega, continue to hold it while it starts up.
Have you checked your power source?

You provided an example of one of your Omega2 that is booting, are you using the exact same dock and power source when testing both Omega2?

@MK thanks, yes I saw your time clock gizmo, nice piece of kit. Congratulations.

@Lazar-Demin The reason why I think a PoE version would be useful is as @MK mentioned, getting power to devices is often a pain, I think especially in a development of IoT devices I want to test devices in various locations and honestly I prefer to cable my stuff for reliability as WiFi is not often reliable in some locations. Especially when I'm fixing things to walls or up in the ceiling of a building where cables, metal and fluorescent lights may be found. Getting an ethernet cable to some of these locations can be a challenge, but getting power there as well can get really complicated and expensive.

I also think a solid reference implementation for use with Omega2S+ would be a bonus, or even better with the Omega3S+