Community

magge

@crispyoz Thanks for responding again!
It should be 2nd bullet above, there I get a bunch of JFFS2 errors, so I'm not sure if maybe the file is just too big without upx, it's about 9.4 MB in that case.
Looks to me like the 3rd bullet is slightly bettery (there the binary is not stripped but compressed with upx).

magge

Short update.

Unfortunately I don't have a Omega2P that is known to be good, I only have my custom hardware. Is there a prototyping board I should order when working with this? I tried to google for it but did not get any wiser...

I have been working on this, trying to be more structured and changing less things at once, rebooting often and going through different variations.
New observations when using the official omega2p-v0.3.3-b256.bin firmware

The binary used above (go build -ldflags "-s -w" and upx --brute, filesize < 2MB) can be transferred to Omega2P and MD5 does not change over reboot. No logs of filesystem corruption.
However! If the app has been started at least once, the next reboot will log xz decompression failed data is probably corrupt, and next reboot the system is not well. My application fails and other things like opkg update will log errors, and there will be errors logged by squashfs . It was my understanding that squashfs was read-only on the Omega2P, so I'm not sure how it can fail.
Recovery seems consistent by going to failsafe and wiping out rootfs_data (for example firstboot or mtd erase rootfs_data).

Alternative binary (no stripping, no upx, filesize 9.4 MB) I start getting JFFS2 warnings (errors?) in the log, like:

[ 1179.029279] jffs2: warning: (703) jffs2_do_read_inode_internal: no data nodes found for ino #134
[ 1179.038223] jffs2: iget() failed for ino #134

The MD5 of the app binary changes over reboot (somehow it still worked though!).

# before reboot
$ md5sum /isys/isysctrl /mnt/sda1/isys-3/isysctrl-full
ef7936ff1a4030871fd47687d9306965  /isys/isysctrl
ef7936ff1a4030871fd47687d9306965  /mnt/sda1/isys-3/isysctrl-full

# after reboot
$ md5sum /isys/isysctrl /mnt/sda1/isys-3/isysctrl-full
05300935ae776926d7ce20f145682392  /isys/isysctrl
ef7936ff1a4030871fd47687d9306965  /mnt/sda1/isys-3/isysctrl-full

/etc/banner got corrupted:

BusyBox v1.28.3 () built-in shell (ash)

▒local▒▒JSON_PREFIX=▒1=▒=▒"▒$___val▒"▒eval_json_set_var▒1 .bp▒a_json_set_var1▒btb▒b▒b▒b▒blb,b▒bb$bb_a_value=▒▒2=▒local▒▒JSON_PREFIX=▒1=▒=▒"▒${▒JSON_PREFIX=▒1=} ▒$_a_value▒"▒eval_jroot@Omega-266F:/#

No xz decompression errors logged when rebooting after app has been running.

JFFS2 seems to be in trouble , fex opkg fails:

root@Omega-266F:/# opkg update
[   86.107495] jffs2: notice: (1925) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x220558.
[   86.118551] jffs2: notice: (1925) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x2204fc.
[   86.129229] jffs2: notice: (1925) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x22006c.
[   86.139536] jffs2: warning: (1925) jffs2_do_read_inode_internal: no data nodes found for ino #113
[   86.148627] jffs2: iget() failed for ino #113
[   86.155510] jffs2: warning: (1925) jffs2_get_inode_nodes: Eep. No valid nodes for ino #113.
[   86.164064] jffs2: warning: (1925) jffs2_do_read_inode_internal: no data nodes found for ino #113
[   86.173074] jffs2: iget() failed for ino #113
[   86.178849] jffs2: warning: (1925) jffs2_get_inode_nodes: Eep. No valid nodes for ino #113.
[   86.187422] jffs2: warning: (1925) jffs2_do_read_inode_internal: no data nodes found for ino #113
[   86.196445] jffs2: iget() failed for ino #113
Collected errors:
 * opkg_conf_parse_file: /etc/opkg/distfeeds.conf:1: Ignoring invalid line: `▒@'

The best working option I have found (no stripping, upx --brute, filesize 4MB).
No errors after transfer.
No errors first reboot after transfer.
No xz decompression errors first reboot after app has been running.
Still got /etc/banner corruption, opkg update still fails.

magge

@Lazar-Demin Thanks!

My app logic only reads a config file, and that's it for FS access. It will then do GPIO to control motor directions and read buttons, I2C to PWM chip for motor speeds, I2C to AD converter chip for current measurements.
Running the app does not seem to trigger issues from what I can tell. The trouble seems to start after uploading the files and rebooting.

magge

@crispyoz Thanks again!

I've seen the issue on more than one power supply, but I will try to check with oscilloscope if I can see something.
Yes, that was suspect initially. However, FS is still corrupted for some reason even just adding a small /etc/init.d/ now (for the case where I run the app from USB storage). df -h reports still free space after I upload my files but my understanding is that JFFS2 is a compressing FS and that remaining space is a guesstimate.
EMI. I read another article here on community that shielding was a thing. There is no radio on our device. Stick a tin foil hat on my Omega2S+ and see if it helps?

I will try check the power supply and get back to you. There are PWM'ed motors connected on the device, so I could see there being some noise coming back from that. That being said I'm almost sure I've seen FS go bad even without running the motors. I.e. uploading the software, rebooting without even running it, getting bad FS.

If there is noise from motors/supply could that permanently damage the flash so that it's just broken now and that's why "nothing works"?
Could a damaged flash operate like I have described due to caching smartness or similar; i.e. I upload the file and it looks okay, trouble only starts after a reboot?

magge

@crispyoz Hi and thank you for responding. Sorry I was unclear there; the Omega2s is on custom hardware. What I meant was just that I grabbed a device that I had not messed with, i.e. a device with an Omega2S+ that should be original / as-delivered.

A bit more; Our device is quite simple (warning, I'm not an electrical engineer). It's voltage regulation and DC motor controllers (hooked up to GPIO for direction control), multi channel PWM controller (i2c), multi channel AD converter for motor current measurement (i2c). Buttons, connectors, LEDs. I'm powering my device with a lab/bench power.

Are there any know electrical issues that I can try to look for? I have an oldskool Tekntronix oscilloscope that I might be able to investigate with...

magge

Product being used: Omega2S+ on custom hardware
Firmware version: omega2p-v0.3.3-b256, omega2p-v0.3.2-b246, also custom builds of latest https://github.com/OnionIoT/source

Description of the issue and context:

With from-factory firmware, or with one of the official firmwares above, the Omega2S+ seems to work fine.
I put my Golang application files on it. Application runs fine
Reboot and a lot of JFFS2 errors appear in the console log, the filesystem is broken, app cannot start, system binaries are also broken so I need to recover with uboot's flash from USB.

Putting my application on ramdisk (/tmp), running the app, reboot, no errors from JFFS2.

Putting my application on USB stick (/mnt/sda1), running the app, reboot, no errors from JFFS2.

Putting my application files on the controller and rebooting before running the app, also breaks JFFS2.

As a desperate workaround I tried to have the application stored on USB stick and only add the procd script to start it on flash (/etc/init.d).
Again my FS becomes corrupt!

Is there something residing on flash after I do sysupgrade -n -p so that I'm not getting back to a good state from that? The FS looks like from factory after that... Maybe it's not.

Expected Behavior

Filesystem stays intact.

Observed Behavior

Filesystem corrupted on reboot.

Log extracts:

[   54.861857] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x76ca80.
[   54.872424] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x76ba3c.
[   54.882971] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x76a9f8.
[   54.893538] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x7699b4.
[   54.904196] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x768970.
[   54.914809] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x76792c.
[   54.925408] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x7668e8.
[   54.935955] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x7658a4.
[   54.946488] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x764860.
[   54.957084] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x76381c.
[   54.967637] jffs2: notice: (1699) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x7627d8.

[   55.240899] jffs2: iget() failed for ino #104
[   60.268864] jffs2: warning: (1904) jffs2_get_inode_nodes: Eep. No valid nodes for ino #104.
[   60.277413] jffs2: warning: (1904) jffs2_do_read_inode_internal: no data nodes found for ino #104
[   60.286463] jffs2: iget() failed for ino #104
[   65.314331] jffs2: warning: (1908) jffs2_get_inode_nodes: Eep. No valid nodes for ino #104.
[   65.322816] jffs2: warning: (1908) jffs2_do_read_inode_internal: no data nodes found for ino #104
[   65.331917] jffs2: iget() failed for ino #104
[   70.359425] jffs2: warning: (1912) jffs2_get_inode_nodes: Eep. No valid nodes for ino #104.
[   70.367973] jffs2: warning: (1912) jffs2_do_read_inode_internal: no data nodes found for ino #104
[   70.377039] jffs2: iget() failed for ino #104
[   75.404384] jffs2: warning: (1916) jffs2_get_inode_nodes: Eep. No valid nodes for ino #104.
[   75.412868] jffs2: warning: (1916) jffs2_do_read_inode_internal: no data nodes found for ino #104
[   75.422003] jffs2: iget() failed for ino #104

Trying to list application files, most files are gone:

root@Omega-266F:/# cd /isys/
root@Omega-266F:/isys# ls -l
[  142.705947] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x742450.
[  142.716377] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x7422b0.
[  142.727048] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x74216c.
[  142.737619] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x74201c.
[  142.747913] jffs2: warning: (1939) jffs2_do_read_inode_internal: no data nodes found for ino #105
[  142.756959] jffs2: iget() failed for ino #105
ls: ./config.jso[  142.762123] jffs2: warning: (1939) jffs2_get_inode_nodes: Eep. No valid nodes for ino #104.
n: I/O error
[  142.771902] jffs2: warning: (1939) jffs2_do_read_inode_internal: no data nodes found for ino #104
[  142.782132] jffs2: iget() failed for ino #104
ls: ./isysctrl: [  142.787342] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x8d642c.
I/O error
[  142.799085] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x8d619c.
[  142.810262] jffs2: notice: (1939) jffs2_get_inode_nodes: Wrong magic bitmask 0x0000 in node header at 0x8cd2f0.
[  142.820553] jffs2: warning: (1939) jffs2_do_read_inode_internal: no data nodes found for ino #98
[  142.829491] jffs2: iget() failed for ino #98
ls: ./www: I/O error
-rwx------    1 root     root           202 Nov 29 10:25 start.sh

Steps to reproduce the issue

Grab unused hardware (from-factory hardware) or flash firmware omega2p-v0.3.3-b256 with sysupgrade -n -p . Waiting until led stops blinking (overlay FS has been mounted from mtdblock6).
Avahi is always stuck at "registering" this point and we want MDNS to discover the device, so we need to reboot or restart avahi.
```
root@Omega-266F:/# ps | grep avahi
 2000 nobody    1588 S    avahi-daemon: registering [Omega-266F.local]
```
Reboot. All looks good, no JFFS2 errors, avahi is now in "running" status.

Transfer application.

$ scp -r ../_testpackage/app root@10.150.201.26:/isys
The authenticity of host '10.150.201.26 (10.150.201.26)' can't be established.
RSA key fingerprint is SHA256:bmae8Lq0YSZqo/b83DFdxrquEs4fi6X1/xPYDtBKh20.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.150.201.26' (RSA) to the list of known hosts.
root@10.150.201.26's password:
config.json                                                                           100% 9260    94.1KB/s   00:00
isysctrl                                                                              100% 1457KB 123.2KB/s   00:11
start.sh                                                                              100%  202    27.3KB/s   00:00
index.072ba125.css                                                                    100%  231KB 120.6KB/s   00:01
index.093d5efa.js                                                                     100%  118KB 134.0KB/s   00:00
index.html                                                                            100%  397    46.9KB/s   00:00

Test application works OK on device

root@Omega-266F:/# du -sh /isys
1.8M    /isys

root@Omega-266F:/# cd /isys/
root@Omega-266F:/isys# ./isysctrl

Reboot (using reboot command, I've also tried to use sync before rebooting).
See JFFS2 errors or SQUASHFS errors during next boot.

What's been attempted to resolve the issue

Full golang binaries. Golang binaries that have reduced size (LDFLAGS) and compressed with upx (because initially I was thinking the file size was the problem).

Various firmware versions. The very latest (v0.3.3-b256) and what I consider "the most stable previous version" (v0.3.2-b246).
Also tried building the firmware myself, a minimal version, from the git repo (https://github.com/OnionIoT/source, openwrt-18.06 branch).
Even tried building OpenWRT 22 (https://github.com/openwrt/openwrt/tree/openwrt-22.03 , supported as per https://openwrt.org/toh/start?dataflt[Brand*~]=onion).

What kind of help are you looking for?

I must be missing something, please help me understand why it breaks.

Attachments (if any)

None.

magge

@crispyoz Thanks for all your help!

magge

@crispyoz Sorry, yes I am referring to GPIO-26 or SD.CLK. I have been looking at this diagram https://miro.medium.com/max/1100/0*TL9bMpprfW5sp_b_ .
You are right that it's pin 57, I guess.

That being said I have found 2 solutions:

1:
The mmc_core, mmc_block, sdhci* kmods are the ones causing interference. Building a custom firmware without them fixes the problem!

2:
The omega2-ctrl gpiomux could also have fixed this if it was updated for the Omega2S+ (S meaning SD card, right?).
One of the MT7688 (https://cdn.hackaday.io/files/279051193887520/MT7688_Datasheet_v1_4.pdf) registers that this tool writes (GPIO1_MODE) actually has SD_MODE (bits 11:10) which allows controlling what the pins are used for!

I have not found a nice way to read the register and bitwise-and/or with the only the bits we want to set, however we can write the whole register as shown below. This sets all modes in GPIO1_MODE to GPIO except for I2C, which is what I need in this particular case, including the invisible SD_MODE...

# devmem 0x10000060 32 0x55055554
# omega2-ctrl gpiomux get
Group i2c - [i2c] gpio
Group uart0 - uart [gpio]
Group uart1 - uart [gpio] pwm01
Group uart2 - uart [gpio] pwm23
Group pwm0 - pwm [gpio]
Group pwm1 - pwm [gpio]
Group refclk - refclk [gpio]
Group spi_s - spi_s [gpio] pwm01_uart2
Group spi_cs1 - spi_cs1 [gpio] refclk
Group i2s - i2s [gpio] pcm
Group ephy - ephy [gpio]
Group wled - wled [gpio]

Setting the gpio mux correctly allows it to work even with the standard kmods running.

# lsmod | grep mmc
mmc_block              21446  0
mmc_core               88573  3 mmc_block,sdhci,mtk_sd
# gpioctl dirin 26
Using gpio pin 26.
# gpioctl get 26
Using gpio pin 26.
Pin 26 is LOW
# devmem 0x10000060 32 0x55055554
# gpioctl get 26
Using gpio pin 26.
Pin 26 is HIGH

magge

@crispyoz

Was not able to avoid mmc_core and mmd_block loading on boot. I think they are maybe loaded from the read-only /rom, seems it's the only place they are left now...

# find / | grep mmc_core
/overlay/upper/root/backup/mmc_core.ko.bak
/overlay/upper/lib/modules/4.14.81/mmc_core.ko
/rom/lib/modules/4.14.81/mmc_core.ko
/root/backup/mmc_core.ko.bak

However I figured out that mmc_core was not possible to rmmod because of dependent modules, but removing all deps first allows also removing mmc_core.
Still getting pin 26 as low though, so maybe this is not about SD after all...

# rmmod sdhci_pltfm
# rmmod sdhci
# rmmod mmc_block
# rmmod mmc_core
# lsmod | grep mmc

# gpioctl dirin 26
Using gpio pin 26.
# gpioctl get 26
Using gpio pin 26.
Pin 26 is LOW

magge

@crispyoz Thanks!

Somehow it can still load those modules even when I moved the files into /root/backup and even added .bak on their filename (and rebooted).

# lsmod | grep mmc
mmc_block              21446  0
mmc_core               88573  2 mmc_block,sdhci
# ls -l /lib/modules/4.14.81/ | grep mmc
#

I will try to read up on building a kernel without mmc...

magge

Product being used : Omega2S+

Firmware version : 0.3.3 b247

Description of the issue and context:

GPIO pin 26 always returns LOW, unlike all other GPIO pins (also including pin 27 also in the
SD-card reader block of pins).

  # omega2-ctrl gpiomux get
  Group i2c - [i2c] gpio
  Group uart0 - uart [gpio]
  Group uart1 - uart [gpio] pwm01
  Group uart2 - uart [gpio] pwm23
  Group pwm0 - pwm [gpio]
  Group pwm1 - pwm [gpio]
  Group refclk - refclk [gpio]
  Group spi_s - spi_s [gpio] pwm01_uart2
  Group spi_cs1 - spi_cs1 [gpio] refclk
  Group i2s - i2s [gpio] pcm
  Group ephy - ephy [gpio]
  Group wled - wled [gpio]

  # gpioctl dirin 26
  Using gpio pin 26.
  # gpioctl dirin 27
  Using gpio pin 27.
  # gpioctl get 26
  Using gpio pin 26.
  Pin 26 is LOW
  # gpioctl get 27
  Using gpio pin 27.
  Pin 27 is HIGH

Expected Behavior

Pin 26 is wired similar to 27 and should be HIGH.

Observed Behavior

Pin 26 is LOW.

Steps to reproduce the issue

Connection:

  3.3V --> R100K --> Omega2S+ pin 26
                  |  
   GND --> C10p --/

What kind of help are you looking for?

I've been searching without luck what I need to do to make sure the software knows to use
pins 22-29 for the SD-card reader as GPIO pins. Or; what I need to uninstall to make
sure those pins are never disturbed by SD-card related software/drivers.

My only guess is that pin 26 reads low because it's somehow used as SD.CLK.

If you can please help me get pin 26 as a plain GPIO pin, very much appreciated.

Posts made by magge