Onion Memory structure
-
Hi Community,
I'm looking more in detail into the onion disk storage layout. And I must admit....I'm really confused....
First: I'm working with on onion omega 2S+ which is given with 32MB of flash storage. If I run a df -h command, I get this:
I do not understand where I can see the total available disk space of 32MB.
Second: running a du -h -d 1 / command, will output this:
I really don't get the link between both as the df command indicates a usage of 11.6MB in /overlay and the du command indicates a size of 20.7MB for overlay
Third (and this is where it get's really difficult for me to understand): If I just dump a 2967KB text file into /root (via ssh) I get this if I run the du command:
I do not see where the additionnal 2967KB went...Moreover, the du command shows:
According to this, adding this 2,967MB file used about 5,8MB on disk...
Any explanation for this? I'm just making these trials in order to make sure my application will not run out of storage. However, I'm not able to understand this.
Thanks in advance,
MK
-
@MK If your application needs sufficient reliable storage, then is it a possibility for you to use the SD Card interface? That might be a better option.
Does your app write frequently to the storage? Does it over write frequently?
I don't know how much wear and tear the built in flash chip can take.
-
@MK Take a look at this link that explains how file systems work on OpenWrt and pay particular attention to how the overlay partition works.
-
@SpiderKenny
Thanks for your reply.
My app will write quite a lot. The device is intended to collect external data (every minute or so) and transfer the data to an MQTT broker. However, the device will always keep an internal Database (sqlite) in order to backup the sent data. Database will be about 2MB at it's maximum size.
Not sure the internal flash really is an issue with frequent writings. I always understood sd cards are more prone to errors over time. My device has a built in sd card slot if needed. If really really necessary I could therefore switch to the sd card. However, if possible, I'd like to avoid this.@crispyoz
Thanks for the link. This is exactly what I was looking for. I'll dive into this content.
-
Hi Community,
Just an update on this. Thanks to @crispyoz , I had A LOT of really interesting documentation to read. So this is how I would summarize my investigations:
-
Point 1 was that after dumping a 2,967MB file into /root, the du -h command showed a total additional storage usage of 5,8MB (3MB in /root and 2,9MB in /overlay). As I understand, the overlay FS is a union of 2 filesystems. In OpenWRT the two filesystems will be /rom containing the rootfs and /overlay which will be the actual storage space. As in my understanding /root is uncluded in /overlay, this explains why the du command counts the used storage as double.
-
Point 2 was that the df command however only returns an additional storage usage of 0.8MB after dumping my 2.967MB file. I couldn't find any explanation for this in the OpenWRT specific flash layout. However, some dark memory popped into my mind regarding JFFS2 and compression. Following Stackoverflow post seems to confirm my thoughts:
https://stackoverflow.com/questions/31622039/jffs2-filesystem-storage-usage-inconsistent-with-real-file-size
To conclude:
1 - It seems du and df commands do not calculate storage the same way. du seems to add the individual file sizes, whereas df seems to take into account the "real" used flash storage taking into account JFFS2 compression algorithms.
2- This makes it quite difficult to evaluate remaining storage. For my projects I think I'll go over to practical use cases with max file sizes (+ some safety) in order to make sure my applications do not run out of storage.Open Points:
1 - Does anybody know where I could find the Omega specific flash layout (just out of curiosity in order to cross check it with the standard OpenWRT flash layout found here: https://openwrt.org/docs/techref/flash.layout#explanations)?
2 - I think the Omega Flash in of NOR type. I however was unable to find any official datasheet confirming this. Does anyone have this kind of ressource?
-
-
@MK Omega2 uses NOR flash, there's a discussion here:
https://community.onion.io/topic/4987/enabling-reset-hack-on-boot?_=1714985949781
You can also check some of my posts for an image of the Omega without the top cover so you can see the actual chip.
Not sure about the specifics of the difference between du and df however all file systems waste space, as a simple example imagine your file requires 2.5bytes of data, ie 20bits, ie 2 full 8bit bytes and half the other byte, the 4 remaining bits of the 8 bit byte end up as slack on most file systems. Also consider JFSS etc mark bad bytes, so the actual capacity of the FLASH is less than specified. Not a definitive answer but pointers for further research.
-
@MK said in Onion Memory structure:
Does anybody know where I could find the Omega specific flash layout (just out of curiosity in order to cross check it with the standard OpenWRT flash layout
You can find this in the DTS files for the Omega2 devices:
- General Omega2: https://github.com/OnionIoT/source/blob/openwrt-18.06/target/linux/ramips/dts/OMEGA2.dtsi
- Omega2/Omega2S specific: https://github.com/OnionIoT/source/blob/openwrt-18.06/target/linux/ramips/dts/OMEGA2.dts
- Omega2+/Omega2S+ specific: https://github.com/OnionIoT/source/blob/openwrt-18.06/target/linux/ramips/dts/OMEGA2P.dts
Here's how you would read it:
So it would be like this:
- u-boot: 192kb
- u-boot-env: 64kb
- factory: 64kb
- firmware: remainder of flash
