Posts made by luz

hi @mauriziomeucci, I understand the necessity for efficient factory programming via Ethernet, but I am genuinely curious to learn what you need avahi for in this scenario.

I am factory programming omega2s via ethernet for many years now, but my script is using IPv6 neighbour discovery, which automatically provides the full MAC address of the omegas, as the MAC is part of the IPv6 link local address. Of course that requires IPv6 to be initially enabled, which was the case so far with omega2 in factory state. @Lazar-Demin77 I hope this will remain the case also for future factory firmware…

hi @baby-Onioneer, doesn't look so "baby" any more what you're doing here

Just to make sure: is this a Omega2+ with 128MB RAM? Because you definitely try to boot a "plus" kernel expecting 128MB:

[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]

0x0000000007ffffff = 128MB

The point in the startup sequence where it crashes is where a reference log of one of my omegas initializes (allocates?) some IP stack related tables, and if it tries to do that in the upper half of the supposedly installed RAM -> boom.

I guess you should look at mtd rather than dd. mtd is the tool sysupgrade uses to write the firmware partition.

At the uboot level, only 4 partitions exist, uboot, uboot_env, factory and firmware, the last one covering the entire remaining space of the flash.

firmware contains the uimage, which consists of a header, the kernel, and the compressed (squashfs) rootfs. It's is the mtd kernel driver which only later creates the kernel, rootfs and rootfs_data partitions on the fly, and also makes kernel and rootfs read-only (because these don't even align with erase blocks, you can see this in the output of dmesg)

I guess you'll be able to write something arbitrary into firmware using mtd, but I doubt just a 1:1 copy of the uboot partition will do, because as far as I understand, uboot needs to find that uimage header at the beginning of firmware to instruct uboot what to do with the contents (decompress, load into ram, call,...). So you probably need to prefix your second copy of uboot with a suitable uimage header such that it gets properly loaded and started.

Anyway, for such experiments, personally I'd try to do this from the uboot console, not from within linux, because the latter will work only one time (and then you'd need to recover via uboot http server or similar anyway).

[Update] If you have the openwrt sdk installed, you'll also have the mkimage tool in ./staging_dir/host/bin/mkimage which can generate uimage headers.

You might want to look at the preinit mechanism. This is similar to the /etc/init.d scripts, but runs much earlier.

The preinit scripts are in /lib/preinit/ and, according to my dmesg outputs (search for "init: - preinit -" line), start running around 3.5 seconds into the boot process.

However, when these start running, you have almost nothing ready yet. In particular, loading of additional kernel modules, such as the i2c driver, seems to happen only after preinit is done. Most of the preinit runtime seems to be mounting the jffs2 overlay, taking around 20 seconds. Only then, procd starts running, and kernel module autoload starts.

The main challenge might be: before the jffs overlay is up, you only have the files that are baked into the firmware image. So whatever custom things you want to do early, need to be part of a custom-built OpenWrt image and can't be just installed later on top.

@Lazar-Demin said in ️ Beta Firmware Update: NodeJS v16.19, OpenWRT 22.03.3, and more! ️:

[...] we'll be look at your patches to see how they line up with what we've currently got in our backlog [...]

The "Possibility of using One Wire again" item in the backlog hints to a general problem. Not only w1-gpio-custom was removed, but all other of those *-gpio-custom drivers (spi, i2c and maybe others).

It does not seem re-establishing (and later maintaining) those would be a simple task.

That's why I took the route towards runtime loadable device tree overlays (patches 251..253). While these are not mainline linux, these patches are in use in RaspberryPi for a long time, offering the same as, and much more than, the *-gpio-custom drivers did: configuring standard hardware drivers at runtime, in general.

The overlay directory in the RaspberryPi kernel tree on github gives a lot of examples - there's also one for the w1-gpio driver we could glean from

I did not try W1 yet, but I did try software SPI. This works on device running a build with patches 251..253 and dtc installed (the latter is a menuconfig option):

# create overlay definition
cd /tmp
cat >sw-spi.dts <<ENDOVERLAY
/dts-v1/;
/plugin/;
/ {
  fragment@0 {
    target-path = "/";
    __overlay__ {
      sw-spi {
        compatible = "spi-gpio";
        #address-cells = <0x1>;
        #size-cells = <0x0>;
        sck-gpios = <&gpio 11 0>;
        mosi-gpios = <&gpio 5 0>;
        miso-gpios = <&gpio 4 0>;
        cs-gpios = <&gpio 23 0>;
        num-chipselects = <1>;

        spi-max-frequency = <100000>;
        spi-cs-high;

        /* clients */
        spidev@0 {
          compatible = "linux,spidev";
          reg = <0>;
          spi-max-frequency = <10000000>;
        };

      };

    };
  };
};
ENDOVERLAY

# compile overlay definition
dtc -b 0 -@ -O dtb sw-spi.dts -o sw-spi.dtbo

# apply overlay definition
mkdir /sys/kernel/config/device-tree/overlays/swspi
cat sw-spi.dtbo >/sys/kernel/config/device-tree/overlays/swspi/dtbo

# have a look at kernel messages for errors
dmesg

# check overlay status (should return "applied")
cat /sys/kernel/config/device-tree/overlays/swspi/status

The patchset also includes a startup script that will automatically load compiled device tree overlays placed in /lib/firmware/device-tree/overlays/*.dto at startup.

So future hardware configuration for the Omega2 might become as simple as choosing an overlay file from a library, and copying it to /lib/firmware/device-tree/overlays/...

@Lazar-Demin Hi Lazar, I'm exited to see OpenWrt-buildsystem-wrapper essentially replicating p44build - available for years on github and mentioned several times online and offline, when discussing openwrt level patches, e.g here!

I guess the need for this approach was more urgent for me, having not one, but several different targets on top of openwrt, where maintaining a fork for each of them was simply a nightmare

So I'm very happy to see OnionOS is becoming a set of patches, config and custom packages on top of the original OpenWrt tree, too!

It seems to me from the first quick look, that while the workflows your build.sh and my p44b support are a bit different, the assets (patches, configs) they work with are probably 100% compatible, so using the Onion patchset under p44b and my patches under build.sh should work

So, to contribute more than words, I just created a new p44b-onion "umbrella" package in the plan44 feed aimed at becoming a way to build the Onion Firmware with p44b. It does not work for that, yet, but it contains hopefully useful patches which I created over the years for my Omega2 + OpenWrt projects and should be easily portable/usable in OpenWrt-buildsystem-wrapper:

• hardware PWM (204)
• I2S sound (210)
• runtime loadable device tree overlays (251..253)
• Modern GPIO via cdev support (252)
• re-introducing mt7621 GPIO "base" of-property to keep SoC pin based GPIO numbering (257, 281)

I am really excited about this, because I see great potential for the Onion community to share openwrt level work more easily than in the past, where the hurdle of diverging trees made it hard to "just try" the latest changes of someone else.

@huntc said in Secured uboot image:

@luz Is this the case if there is no additional external flash i.e. just the flash integrated with the Omega device itself?

Yes. The pins needed to access the internal flash are exposed (the SPI pins of the Omega), as is the VDD of the flash chip, so it can be accessed easily. Which is, for non-backbox applications, mostly an advantage. It allows for efficient factory flashing and debricking even of the bootloader.

I'd be genuinely interested in understanding the background of wanting to prevent reading out firmware in the first place.

Does it contain real secrets such as key material? Hopefully not. Or is it just not disclosing some algorithms? For both I cannot imagine an attacker for whom that kind of low level information is valuable/useful but who would not be capable of surmounting any protection short of a completely secured (iPhone-style) platform...

@Jim-W Not exactly the same pins as those on the devboard, but I got really well working ones called FlexyPins from solder.party from their lectronz store.

I needed them for a small programmer board for flashing O2S via Ethernet.

@nsmith said in Secured uboot image:

So there is no way to prevent the code (either the one in the flash, or the one running in memory) from getting read out and recreated?

No.

I completely agree with @crispyoz that all security is relative!

But I‘d like to point out that reading out the flash in a SoC designed like the MT7688 (as a cost effective network router chip) is particularily easy. It does not even require unsoldering or opening the Omega. You might need to cut a single track on the PCB and contact a few SPI pins, then you can access the flash contents with anything that can speak SPI, like another Omega or Raspberry or CH431A ultra-cheap SPI programmer…

@crispyoz ok, but who else but a at least moderately technically skilled snoop would want to get at a embedded device's firmware binary in the first place? Just wondering - without knowing what @huntc's threat model actually is and what type of secrets are to be proteced, it's hard to guess...

While I understand the goal, I don't see how securing an MT7688 based system, which has no anchor for the needed chain of trust, could work at all.

For an attacker who has physical access to the device, reflashing or reading out the SPI chip directly is about as hard or easy as getting access to the uboot console. So just removing flash access commands from uboot does not help much, IMHO...

To protect against attackers without physical access, it is sufficient to have an update mechanism which checks signatures, like yours probably already does.

Am I missing something?

When you say "bricking", you mean not destroying it due to some HW mishap, but a software issue that prevents it from booting?

I have a few 1000 Omega2S out in the field and so far only ever had to software-level unbrick one (I actually burnt a few more due to PSU issues in one production charge, and soldering parameter errors in another, but these are different stories).

From that one I learnt that all you need to unbrick from any level of software brick, including ruined u-boot (which IMHO can't be repaired in the devkit) is

• a way to apply power only to VDD_FLASH (A7) but not to the other VDD pins (C17, C18), to activate the SPI flash chip but not the MT7688 SoC (nor any other part of your circuitry)
• access to the SPI lines (A8..A12)
• a SPI flash programmer (anything from cheap $3 aliexpress up to Segger - in my case, a colleague had the latter and knew how to use it)

Only the first bullet point might be difficult (but usually manageable by cutting some tracks) if your PCB design does not provide for it. Accessing A8..A12 should always be possible; if you have to do it more than once, building an apdapter with 6 pogo-pins helps a lot.

With such a setup, you can write every single byte of the SPI flash chip inside the Omega2S, and with that, reflash any part of it that might be bricked.

That way, you don't need to unsolder the Omega2S completely to unbrick it

PS: if you can pull up GPIO38, have Ethernet and the serial console, and not bricked u-boot, but only openwrt, then unbricking is even simpler via the built-in recovery http server, see docs.

@DumTux the p44b script works from the information in the p44build directory you specify at init. It seems you tried with an older project that was still on 18.06.4. p44b works with 22.03 but I don't have a complete setup yet exactly because of the various detail issues I am still having.

But it's not easy for me as the gpio-mt7621.c seems like changed.

I agree , it takes an awful lot of time to dig into such things and get it right.

But by now, gpio-mt7621.c seems to be Linux mainline

Looking at the commit history of that file in Linux mainline, it seems that there were several changes related to IRQ handling in the past 2 years.

Maybe it would make more sense to try to use that latest mainline version and see if the changed IRQ handling now works correctly. Hoping this seems not too unrealistic to me - all the changes mostly seem to be generalisations, i.e. replacing mt7621 specific things by generic mechanisms that work for many other drivers, too.

@Gerhard-Bertelsmann said in CAN Bus: using MCP2515 with Omega2:

I switched to 22.03 but the problem is still present

Ok, interesting! So the rework did not catch that. Maybe I mixed it up in hindsight with the initialisation of GPIO output's initial value (did not work as expected in 19.07, I had to patch that too back then, but definitely this now works in 22.03 as is)

I would be interested in this patch

Here's the patch.

Note that this is a patch on top of a pristine openwrt v19.07.8 tree, which consists of adding kernel patches, which then are applied in the process of building the kernel. So the patch is kind of a "meta-patch", doing the following:

• (up to line 607) updates the gpio-mt7621.c to a newer version from upstream. I had forgotten about that - it seems this newer version is quite near to today's 22.03 version.
• (lines 649..717) patches that new driver to fix the IRQ problem. This patch is ugly because it needs to swap the private irq_ops with a modified copy to implement the select callback.
• (lines 608..648) patches OMEGA2.dtsi and (lines 718..) adds a new version for mt7628an.dtsi called mt7688newgpio.dtsi because the new driver now addresses the gpios by number, no longer by bank+offset.

I package such changes into openwrt tree (meta)patches rather than forking the openwrt tree, because different projects need different patches, and I don't want a multitude of trees or branches, and keep the changes versioned per project in my own feeds, not openwrt.

The p44build script helps with that workflow in case you are interested.

@rdot said in Installing OpenWrt 22.03 on an Omega2S+:

We have a custom setup too and we've been seeing odd failures where we let the device run overnight and at some point it appears to factory reset itself.

This does not sound like something that could happen because of the "broken flash reset" to me.

To get a factory reset, active and quite specific write operations to the flash must happen (wiping the overlay partition). I cannot see how this could possibly be caused by the "broken reset" problem.

The "broken reset" issue is simply that the SPI flash chip must be in 3-byte address mode at the moment the MT7688 SoC resets, otherwise it won't reboot.

To access flash beyond 16MB, the SPI flash must be set to 4-byte mode. On systems without the "broken reset" problem, the SPI flash driver keeps the chip in that mode all the time because it is more efficient. On Omega2 and similar systems, the driver switches back to 3-byte as soon as possible, to minimize the periods of time when a reset would lead to a stuck system. Still, the problem cannot be avoided entirely, if you get a hardware reset exactly in the one of the remaining (short) periods of time when the SPI is in 4-byte mode, you're out of luck.

So if you don't need the ability to hardware-reset the SoC, you also don't need the extra circuitry which removes power from the SPI flash chip during reset (and thus resets it to 3-byte mode).

I'm not sure if it can happen that the MT7688 SoC internally hard-resets (brown-out, watchdog, maybe?) - if it can and does, not even the external circuit could avoid the hang.

But during normal operation, with no hardware reset involved, all this can't be what's causing your problem, IMHO.

@Gerhard-Bertelsmann Yes, the 19.07 version of gpio-mt7621.c interrupt routing implementation is broken for GPIO interrupts assigned via DT.

I made ugly patches to fix that back then, I can dig them out if someone wants to use them, but I can't exactly recommend (it was kind of an emergency hack for an urgent project).

As far as I know, the issue should be fixed in the 22.03 version of gpio-mt7621.c, as the entire driver has been reworked to align with other modern GPIO drivers and handles the banks differently.

But I haven't had the chance to test it so far, and the link to the OpenWrt forum you posted saying it is not yet fixed in 21.02 makes me doubt a bit, because I thought the rework already happened in the 21.02 time frame, but I can't verify or test that right now.

@Rob-Dautel of course I can share the patches. As the forum does not allow to attach .zip, please download the patches from my website here, these are diffs against a completely unmodified OpenWrt 22.03.2 checkout.

These work in my builds, but I can't test right now if these work with the standard Omega2(p) build target settings (as I use different ones for my targets), but I hope so.

• 251..253 add support for modern gpio cdev and for DT overlays (but the respective options need to be enabled in menuconfig to actually enable these features)

• 256 is cosmetic only, it reduces the inevitable message about the "broken flash reset" to a single line, because, yes, that's how things are on the omega2(s)+, it may not recover from unexpected reboots without powercycling the SPI flash chip (that's why reference schematics do exactly that at hardware reset). But I don't need a multiline catastrophy trace about this in every boot log...

• 257 and 258 finally restore the normal GPIO numbering starting at 0.

@nsmith said in Multiple devices on same SPI bus:

Do i need to do insmod command three times with three different chip select pins?

No, you can't insmod the same kernel module multiple times.

But you can just append another three numbers for each additional CS you need, the first being the mode, the second the speed, and the third the GPIO for the CS.

Like this, which adds a second CS with SPI mode=0, speed=100kHz on GPIO19, and a third on GPIO44:

insmod spi-gpio-custom bus0=1,11,5,4,0,100000,18,0,100000,19,0,100000,44

You can repeat that up to a total of 8 CS lines!

Then do fopen on the three resulting devices in /dev?

Yes, these will be named /dev/spidev1.0, /dev/spidev1.1 and /dev/spidev1.2, resp.

BTW: if you need multiple buses, i.e. another set of SCK,MISO,MOSI plus some CS, you can append bus1=2,... to the same command line, separated with a space. Up to 4 buses are supported.

@nsmith said in spidev fails:

I think the HW SPI is only used during boot up to read the OpenWRT kernel code from the flash into RAM, after that i believe its not used at all, otherwise the SPI would be un-usable. I really hope this is true becuase I'm already lacking 4 GPIO pins, if I have to give up more pins to a bitbanged SPI then my project will be over.

hopefully not, but:

The SPI is used for loading the bootloader from flash (without any driver, this part is hard-wired in the MT7688), then the bootloader (u-boot) will load the Linux kernel from flash using its own flash driver.

But from then onwards the linux kernel will still access the flash whenever it needs to read or write to a file that is not on the ramdisk (which is - pretty everything except from /tmp and /var). Of course, there is some caching, so not every tiny file access will always access flash, but still in many cases.

Depending on what your overall application does, there might be very little file accesses, but avoiding them entirely (from linux services running in the background, responding to network, keeping WiFi connections, serving web requests or ssh etc.) would mean you need to strip down OpenWrt severely, essentially killing all services except for your own app, or transplant everything to the ramdisk. Maybe possible, but far from a no-brainer.
(Btw: sysupgrade needs to do the ramdisk transplant thing when it replaces the firmware, so one could glean some ideas from there - complicated beast though, too.)

Unfortunately, it seems to me not even the "switch everything to SW-SPI" trick you posted can help here, because after that you'll still be sharing the SPI bus with the flash chip, only that flash accesses now take 40 times more time

Maybe, when you run out of GPIOs, a I/O extender chip like MCP23S17 might be far easier than too much struggle trying to re-use those HW SPI pins in any way. These are good for half-duplex SPI devices that need high bandwidth but no real time bus availability, such as a LCD display (and the internal flash), but for nothing else.

What these suggested device tree changes do is entirely replacing hardware SPI with software (bit banged) SPI.

Yes, this will make full duplex work, but it will also make flash rom (file system) access muuuch slower (because software SPI max speed is ~1Mbit/s, while hardware SPI is ~40Mbit/s if I correctly recall).

However, no need for that - you can use software SPI with the Omega2 without changing the device tree (which would require building your own OpenWrt image), thanks to the spi-gpio-custom module. You just can't use the same pins (7,8,9) as the hardware SPI does, so you'd need to reserve 3 other GPIOs for it (you can re-use GPIO6, SPI_CS1, however).

The only reason to switch to software SPI entirely would be when you absolutely can't spare those 3 separate GPIOs and slowing down file system access by a factor 40 is not an issue.

Otherwise, just use spi-gpio-custom to create a separate SPI interface for your device(s) that need full duplex, it's as simple as:

# insmod spi-gpio-custom bus<Y>=<devid>,<SCK>,<MOSI>,<MISO>,<mode>,<speedInHz>,<CS> 
insmod spi-gpio-custom bus0=2,11,5,4,0,100000,18

• bus0=2 means that the software SPI device appears as /dev/spidev2.0 (first CS, second would be /dev/spidev2.1 etc.)
• GPIO11 is the SPI clock SCK
• GPIO5 is MOSI
• GPIO4 is MISO
• mode is 0, which means standard SPI
• speed is set to 100000 which would mean 100kHz, but it is irrelevant because software SPI always runs as fast as it can, ~1MHz on the MT7688, which is still slow in general SPI terms.
• GPIO18 finally will be CS for the device.

You can also have more than one device, by adding more (mode, speed, CS) triples, or you can run without CS by leaving it unspecified. The full syntax is:

# insmod spi-gpio-custom bus<Y>=<devid>,<SCK>,<MOSI>,<MISO>,<mode0>,<speedInHz0>[,<CS0>[,<mode1>,<speedInHz1>,<CS1>[,<mode2,...]]]