Posts made by Lazar Demin

Wanted to weigh in here - getting an Expansion Dock + Ethernet Expansion or an Evaluation Board will save you lots of time and headaches!

This was an oversight on my end. It went unnoticed because the p44 ledchain package is not part of our default firmware. So we never ran into the issue.

Resolution:
We've updated the image builder profiles in the OnionIoT/openwrt-imagebuilder-wrapper and OnionIoT/openwrt-packages (openwrt-23.05 branch) repos to include the new repo (http://repo.onioniot.com/omega2/packages/openwrt-23.05.3/core/).

Now firmware images that include the kmod-p44-ledchain package can be built out of the box.

Why was this an issue?

• The kmod-p44-ledchain package is no longer being built with the build system, so it's no longer available in the http://downloads.onioniot.com/releases/23.05.3/targets/ramips/mt76x8/packages/ package repo, as @DocHardinger noticed above
• Now, the OpenWRT SDK is being used to build the kmod-p44-ledchain package, this involved creating a new package repo
• The new package repo was included in the firmware and in the OpenWRT SDK wrapper configuration, but not the image builder configuration

See the relevant thread for more details

@DocHardinger said in 23.05.3 iwinfo scan disconnects AP:

It seems there has something changed inside the wifi driver? @Lazar-Demin

Yes, there have definitely been changes in the wifi driver between the 22.03 and 23.05 firmware releases:

• The 22.03.5 firmware uses the mt76 wifi driver with datecode 2022-09-06
  (https://github.com/openwrt/openwrt/blob/v22.03.5/package/kernel/mt76/Makefile#L11)

• While the 23.05.3 firmware uses mt76 wifi driver datecode 2023-09-11 (https://github.com/openwrt/openwrt/blob/v23.05.3/package/kernel/mt76/Makefile#L11)

We're not as up to date on the internal changes since the new beta firmware uses the open source mt76 driver.
But since it's open source, the good news is you can investigate this directly and open an issue if needed: https://github.com/openwrt/mt76

Yep, production is complete and the units are all set to be shipped out to Crowd Supply. Just ironing out the final details with the Crowd Supply fulfillment team.

@MK I agree with @crispyoz that rc.local is the lowest impact file.

To make it even more low impact, I would suggest adding a uci defaults file that changes rc.local instead of overwriting the default rc.local.

UCI defaults are generally used to pre-configure the UCI configuration (/etc/config) in your images. But we also use this mechanism to pre-configure other things, like the login banner for example: https://github.com/OnionIoT/OpenWRT-Packages/blob/openwrt-23.05/omega2-base/files/etc/uci-defaults/14_banner

More info on uci defaults here: https://openwrt.org/docs/guide-developer/uci-defaults

@Mark-C We've actually been working with our distributors to make the Omega2 LTE Global variant more accessible!

We're happy to report both Mouser and Digikey have reduced their minimum order quantity for the OM-O2LTE-G. Find the purchase links here: https://onion.io/buy?p=OM-O2LTE-G

@crispyoz DT overlay packages unfortunately won't work in vanilla 24.10

DT overlay depends on a few patches to the kernel that we add to the build system (See patches 251-253 at https://github.com/OnionIoT/openwrt-buildsystem-wrapper/tree/openwrt-23.05/onion/patches)

@crispyoz I don't remember the exact reasoning, but in one kernel release all of the bitbang implementations of hw buses were taken out.
I really liked how flexible these modules were - you could specify which GPIOs did what in the insmod command that loaded the kernel module.

Like @luz mentioned, DT overlays are the best game in town for this at the moment.
We have the onion-dt-overlay-sw-spi package available, and we also recently put out a onion-dt-overlay-sw-spi-alt package that uses more readily accessible GPIOs:

sck -> gpio3
cs0 -> gpio2
mosi -> gpio1
miso -> gpio0

(coming soon to the documentation) See the Software SPI bus article in the new documentation for more details.

It's not as easy as using kmod-spi-gpio-custom , but if you're comfortable using the OnionIoT/openwrt-sdk-wrapper, you can work off onion-dt-overlay-sw-spi to spin your own software spi package that uses GPIOs that work for you!

@luz said in u-boot binary too large on compile:

having cross-links between build environments, pre-built x86 binaries etc. feels a bit brittle to me, so what I would like to do is getting uboot for omega2 built as an openwrt package.

I agree, it is brittle. It was something we put in place back when x86 was the only game in town for desktop computing

We also noticed the OpenWRT build system has an option to build the bootloader, but haven't explored that path yet.

Like I mentioned in the Omega2S+ sys_led pin thread, we're contemplating moving to a more modern u-boot. I think this would give us more impact for the time spent.
And we could set it up to be less brittle from the get go.

@luz @crispyoz let me know your thoughts and if you're interested in collaborating on this!

@luz said in Omega2S+ sys_led pin:

I would be much more nice to have:
//gpio44 gpio_dclr_1 644 clear bit12
RALINK_REG(RT2880_REG_PIORESET+0x04)=1<<12;

Agreed! There's many instances of this type of thing in the omega2-bootloader codebase, among other quirky code.
There were many different contributors and we didn't have coding standards or best practises defined at the time.

Our philosophy on this bootloader is currently "if it ain't broke, don't fix it." That said, we do welcome PRs if you're inspired.

Since we're on the topic of bootloaders:
We're also contemplating moving to modern u-boot. The issue there is the default bootloader binary is over 512kB, while the Omega2/2+ device definition only partitions 192kB for the bootloader.
So any devices with the new bootloader would be incompatible with the existing firmware images.
Alternatively, we could add a new variant/SKU that is the same hardware (as the Omega2S+) but runs the new bootloader. This involves logistical challenges.

We're open to suggestions on this front. Let us know what works for you, Onion Community!

Today, debugging a device, I found that GPIO44 is getting configured as output and set to low actively somehow very early, like ~0.5 seconds after reset.

@luz I would agree with @crispyoz's suggestion to take a closer look at the bootloader.

In 0.5 seconds after reset, Linux should be booting up, but it's possible the device is still in the bootloader and the bootloader is making some gpio or gpio muxing changes.

Another thing to look out for:
Since the GPIO44 pin is multiplexed with WLED wifi activity LED, it's also possible it gets temporarily flipped to the WLED mode - This is controlled by the GPIO2_MODE register

Let me know how it goes and if you need any pointers!

@Dennis-German can you elaborate a little on this? Do you mean having the pin layout on 8-pin header of the Omega2 Eval Board match the first 4 rows of the Omega2S Eval Board?

p44-ledchain driver v9 is now available as an installable package from the Onion repo!

Upgrade to firmware 23.05.3-20250121, then run opkg update; opkg install kmod-p44-ledchain

Some technical details:

• The previously available p44-ledchain driver package was implemented through a patch to the OnionIoT/openwrt-buildsystem-wrapper, this patch has been removed
• The new p44-ledchain v9 driver package is now built using the OnionIoT/openwrt-sdk-wrapper
• To make this package installable on the device, we added new package repo to the firmware. This repo holds kernel modules compiled using the OpenWRT SDK, making it easy for Onion to publish custom kernel modules
  • The repo can be found at http://repo.onioniot.com/omega2/packages/openwrt-23.05.3/core/

Today, we’re excited to introduce the Omega2 Eval Boards!

The new Omega2 Eval Boards are built to streamline your experience with the Omega2 IoT modules – from the very first prototype to a final, production-ready device. By bringing nearly all of the Omega2’s capabilities together on a single, compact board, these Eval Boards make it easier than ever to prototype, develop, and ship your connected projects.

For you, this means less time spent juggling different boards and expansions, and more freedom to focus on what matters most: bringing your ideas to life. By streamlining the entire process, the Eval Boards help you stay motivated, creative, and productive—whether you’re a seasoned engineer or a first-time builder.

Available in two variants – one with the classic through-hole Omega2+ module and another featuring the surface-mount Omega2S+ – these Eval Boards offer immediate, hands-on access to all of the Omega2’s I/O pins, an integrated Ethernet port for easy networking, USB-C for power and serial command line access, a MicroSD slot for expanded storage, and more. This combination of features eliminates the need for multiple add-ons and expansions. In particular, the Omega2S Eval Board delivers the expanded functionality of the Omega2S at a more accessible price than any previous solution.

We’ve also introduced new firmware based on OpenWRT 23.05, providing a modern, lightweight foundation that’s easier to customize, and a new documentation site to go along with it. With faster builds and less time spent on the getting the basics working, you’ll enjoy a smoother, more direct path to getting your ideas running in the real world.

Learn more and get your Omega2 Eval Board at Crowd Supply

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

Initially, it performed as expected, but after 14 days, the device entered a stuck mode, similar to previous occurrences.

The fact that it now takes 14 days instead of 8-10 days is strange. If the cause of modules getting stuck was indeed too many writes to flash, then I would expect the problem to be resolved. If this was not the cause, I would expect the time it takes to get stuck to be the same as before. It's strange that it changed by a few days.

From a high-level, what else does the script do? Are there any other interactions with the filesystem?
How often do you move the files from /tmp to the flash storage?

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

The script is designed to automatically reboot and restart the system if Python is detected as not running,

How are you starting the python program in the first place? And is there a specific reason why you're rebooting the whole device if the python program stops running?

A more straight forward solution would be to run the python program as a service: the system will take care of restarting it if it stops executing, without the need for a full reboot and downtime.
This is the recommended approach.

See this blog post for more information on running a program as service: https://onion.io/2bt-custom-initd-service/

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

Do you have any further suggestions for recovering the stuck devices, given that the bootloader menu doesn’t appear to be accessible?

This is a very strange situation. The bootloader is in a completely separate partition, I'm not sure how anything done in linux can impact the bootloader partition. Especially since the bootloader partition is set to read-only from Linux...

Suggestions from my previous post:

For the stuck devices your next step should be trying them on a standard Dock from Onion, and using a simple terminal program like screen, minicom, or putty to try to activate the bootloader menu.
Otherwise, these 3 devices might be write-offs. You can try to recover them by using an external device to rewrite the flash but we (Onion) don't recommend this procedure as a lot can go wrong.

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

Could there be an underlying hardware issue contributing to this behavior?

I find this highly unlikely. Otherwise this issue would be seen on all Omega devices eventually.
Since other users have seen a similar issue and successfully resolved it by moving the writes to /tmp, I would venture a guess that it has something to do with how the compressed filesystem reacts to very frequent writes to files.

There's one more avenue you could explore. Try running the same program (but as a service and all file writes going to /tmp, and your reboot script removed) on the new beta firmware.
The beta firmware is based on kernel 5.15, there may be updates to how the kernel interacts with the flash that could resolve the issue you're seeing.

More info and installation instructions for the beta firmware found here: https://onion.io/embracing-the-future-new-omega2-beta-firmware-and-documentation-site/

Let me know how it goes!

Cool! Thanks for the heads up @luz

I've made a github issue to track this update: https://github.com/OnionIoT/OpenWRT-Packages/issues/106

For now the plan is to make this updated driver available in the new 23.05 firmware.

Does anybody have a specific need to have this available in the older v0.3.4 firmware??

@mayur_ingle ok, glad to hear you're now seeing the expected behaviour with the working devices.

The situation is a little unusual with the stuck devices. I didn't expect the bootloader to be impacted by the file system issue.

I agree with what @crispyoz said above:

If you are using an Omega2+ (through hole) my next step would be to insert it into a standard dock and view the boot process using minicom or some other terminal software. I looked at DockLight but haven't used it, but a raw terminal would remove any potential issues of handshaking or such causing the issue.

For the stuck devices your next step should be trying them on a standard Dock from Onion, and using a simple terminal program like screen, minicom, or putty to try to activate the bootloader menu.

Otherwise, these 3 devices might be write-offs. You can try to recover them by using an external device to rewrite the flash but we (Onion) don't recommend this procedure as a lot can go wrong.

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

a) On a working device:
I followed the recommended steps: powering on while holding the FW_RST pin (GPIO38) active.
Observation: This process erases the existing firmware on the working device, resets it, and allows me to re-upload the firmware. After this, folders become accessible, and the hostname is visible.
I did not see a bootloader menu during this process; it directly erased the firmware and enabled reconfiguration. In short working fine

This is not expected.
Can you elaborate on your observations? What do you mean by it erases the existing firmware on the device? What were the steps you had to do to make this happen? Can you post a log of the terminal?

Expected behaviour

If you have Omega2 devices manufactured in the last ~7 years, you should see a bootloader menu if the device is powered on with the FW_RST active:

You then need to select an option from the menu. See the Firmware Flashing With Web Recovery Mode docs article for the full process.

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

Following your suggestion, I updated the file writes to /tmp and the device is under observation for 8 to 10 days (as my old device is got stucked after 8 days).

Great! I suspect this will resolve the issue. Let us know how it goes!

@mayur_ingle this is a great bug report, very detailed!

Since you're using the through-hole Omega2+, I agree with @crispyoz 's suggestion to try out your stuck devices on a standard dock. Just to rule out any hardware issues.

IMO the issue is more likely to be file system corruption than RAM corruption

Avoiding File System Corruption

@mayur_ingle said in Firmware Boot Failure on Onion Omega2+: Steady Orange LED, Stuck Bootloader Mode and Missing Hostname:

The main_app.py script captures Modbus data packets (32 packets of 105 bytes each) and appends them to an Excel file every 2 seconds.

I responded to your colleague on a GitHub Issue but I will post this here for visibility:

A few other users have reported file system instability when programs are running that frequently write to the flash storage. To get around this, we recommend moving any file writes to the /tmp directory (as this is actually on the RAM, not the flash).

In this case, data that should persist indefinitely should be copied over from /tmp to the flash filesystem (anything else on /) at some longer interval, perhaps daily. Cron is solid tool for this copy job.

Recovering Stuck Devices

How many stuck devices do you currently have?
I'd like to confirm if the bootloader can be accessed on a stuck device.

On a working device, the bootloader menu can be enabled by powering on the device while holding the FW_RST pin (GPIO38) active. This reset pin is active-high, and this is the pin used by the reset button on the Omega2 Docks.
Keep in mind pressing the enter or space keys will not activate the bootloader menu.

Please try this first on a working device, and then try it on a "stuck" device. Report back how it goes.

@crispyoz Your PoE suggestion is interesting! Can you give us a little more detail on why, when, and how PoE is helpful?

This goes for all users!

Who knows? Maybe we make a variant with onboard PoE