Node-Red on Omega2+: Segfault on low memory even /w swap active
-
Hello,
I have been having a lot of fun with my Omega2+. I own an off-grid cabin with lots of automation currently done with Ubiquity mFi products. That product line is EOL, so I'm having some fun with building my own replacement.
I've updated the firmware to the latest version, and might have actually gone one version to far, don't know if that is an issue?
oupgrade -check
Device Firmware Version: 0.1.9 b150
Checking latest version online...
Repo Firmware Version: 0.1.9 b149
Comparing version numbers
Device firmware is up to date!I've added a 64G microSD and moved the overlay filesystem and enabled swap, as a result I now have:
#free
total used free shared buffers cached
Mem: 125760 63924 61836 104 34416 12540
-/+ buffers/cache: 16968 108792
Swap: 262140 0 262140I've installed nodejs, npm and node-red following the manual at:
https://wiki.onion.io/Tutorials/node-redI'm unable to redirect the IP to the non AP one, but that will be a question for another time
It starts and run fine, but after a few minutes node-red segfaults:The host is: 10.1.0.79
28 Jan 18:55:05 - [info]Welcome to Node-RED
28 Jan 18:55:05 - [info] Node-RED version: v0.16.1
28 Jan 18:55:05 - [info] Node.js version: v4.3.1
28 Jan 18:55:05 - [info] Linux 4.4.42 mipsel LE
28 Jan 18:55:13 - [info] Loading palette nodes
28 Jan 18:56:03 - [warn] ------------------------------------------------------
28 Jan 18:56:03 - [warn] [rpi-gpio] Info : Ignoring Raspberry Pi specific node
28 Jan 18:56:03 - [warn] ------------------------------------------------------
28 Jan 18:56:06 - [info] Settings file : /root/.node-red/settings.js
28 Jan 18:56:06 - [info] User directory : /root/.node-red
28 Jan 18:56:06 - [info] Flows file : /root/.node-red/flows_Omega-6F23.json
28 Jan 18:56:06 - [info] Server now running at http://192.168.3.1:1880/
28 Jan 18:56:07 - [info] Starting flows
28 Jan 18:56:07 - [info] Started flows<--- Last few GCs --->
112746 ms: Mark-sweep 27.5 (46.5) -> 26.5 (46.5) MB, 118.6 / 0 ms (+ 889.4 ms in 5 steps since start of marking, biggest step 511.0 >ms) [allocation failure] [GC in old space requested].
113668 ms: Mark-sweep 27.1 (46.5) -> 26.5 (46.5) MB, 869.3 / 0 ms [last resort gc].
114617 ms: Mark-sweep 26.5 (46.5) -> 26.5 (46.5) MB, 949.7 / 0 ms [last resort gc].<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x56424d59 <JS Object>
2: parse(aka urlParse) [url.js:84] [pc=0x496a0214] (this=0x5239d0a5 <an Object with map 0x4e356705>,url=0x5646db6d <String[6]: /comms>,parseQueryString=0x56408099 <undefined>,slashesDenoteHost=0x56408099 <undefined>)
3: arguments adaptor frame: 1->3
4: handleUpgrade [/usr/bin/onion-node-red/node_modules/node-red/node_modules/ws/lib/WebSocketServer.js:163] [pc=0x4969fb5c] >(this=0x2687928...FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Segmentation faultThis was only seconds after starting the service and trying to connect from a client. On previous runs I have been able to execute and run a flow (the basic timestamp -> debug test)
I've kept an eye on free, and only a small amount of swap is used.
Any suggestions on how to get Node-red to run on the Omega2?
Tx
Bas
-
@Bas-Rijniersce I think that you need to have the mipsel architecture version for omega2 to work. I believe your running node-red version meant for original omega (which was just MIPS).
-
I would think / hope that the install using the opkg tool would only deliver a version of the code that is for the Omega2? The opkg update shows it's talking to Onion repositories.
I did follow the instructions from that page, downloaded and extracted the package. Found the location that was the best match (/overlay/upper/usr/bin/onion-node-red/node_modules/node-red/node_modules/ws/) and replaced the contents.
Node-red started OK, but failed in the exact same way
Tx!
Bas
-
@Bas-Rijniersce now that I actually look at the error messages it seems that you need to reduce the size of your base image. Even still, node red might just be taking up the remainder of your 64MB of memory despite that.
-
@Bas-Rijniersce
http://nodered.org/docs/hardware/raspberrypi.htmlThe article above talks about starting node red on a pi such that memory is culled at 256MB! There's also mention of how you can (and should) set the limit at which node.js and node-red begin memory reclamation.
@Lazar-Demin @Boken-Lin : do you all intend to set the node.js memory limit in the omega openwrt package itself?
-
@Theodore-Borromeo I am reading this message correctly as RAM, correct? We are not talking about free space on the root partition?
There was plenty of SWAP available to allocate memory for the process. Will give the other article a read later.TX!
-
@Bas-Rijniersce no problem. I did mean ram, as I don't know what the swap space size is set at for the omega 2. also, turns out that node red is only built out for omega2, so it's def not an architectural issue. This all makes me wonder why yours is not working when the tutorial seems to imply this should work out of the box. . .
-
@Bas-Rijniersce also, swap space only moves data out of ram based on LRU, it doesn't take away the fact that while in operation, an entire program generally needs to be resident in memory. this would mean that node.js, libv8, and node red at least ( and that's not counting whatever lib you are going to put on to run the web front-end, if at all)
-
@Theodore-Borromeo That is actually untrue. Swapping is per-page, and a program of any size to be an issue is going to have multiple code pages as well as data pages. If execution reaches a page that is not currently in memory, a page fault occurs and execution pauses while it is loaded, the same as would happen if access were attempted to a data page that was not resident in memory.
This is important, because a large program will have portions which are rarely needed. Going a step further, it varies by implementation, but ideally the program won't even all necessarily be loaded from disk into RAM at the start, but rather portions may only be loaded as needed, even though to the process itself, it will look as if it is all there - it's just those parts won't actually be there until an attempt is made to access them.
However swapping, especially writeable memory data to an SD scard (or even worse, the build-in SPI flash) is a poor solution. Likely the task is either being done with unnecessary inefficiency, or the wrong platform has been selected, or both.
-
@Chris-Stratton That how I have always understood the use of SWAP by the OS. I can report however that starting node red with:
node --max-old-space-size=56 red.jsHas now kept it up for several hours of playing. The 56 was picked by looking at the output of
freeWith regards to picking the wrong platform for what I'm trying to do. Completely possible
I had not heard of Node Red until a few days ago. I'm trying see if can use the Omega2+ as a cheap sensor platform that can:- Interaction with sensors for temp, humidity, voltage, current. It seems that with I2C and ControlEverything products I can do all that
- Report said data to a central control system for graphing etc
- Trigger relay closures from a central control system easily
- Operate without internet connection
Between Node Red and MQTT it looks like I can do most of that without spending a minute in writing something custom.
Bas
-
IMHO, more RAM is always a good thing, say 512KB or even 1MB.
Don't know if there is a production plan and hope the price won't jump too much.ccs_hello
-
@Chris-Stratton I have never seen swap pushed that far, or assumed to be that efficient on any embedded system. I think, though, that you've seemed to answer the question with your suspicion on SD card performance. Have you tried class 10 cards on a 2plus? the other option is the max size command, which seems to work for some.
-
@ccs-hello said in Node-Red on Omega2+: Segfault on low memory even /w swap active:
IMHO, more RAM is always a good thing, say 512KB or even 1MB.
Don't know if there is a production plan and hope the price won't jump too much.it so happens that the smaller and larger configurations of these modules ship with 128 times the memory ranges you propose
This is plenty to do reasonable things efficiently; but it won't be enough for software stacks that assume unlimited resources and consume them even for simple tasks.
-
I must have been sleep typing :} I meant 512MB or 1GB RAM.
ccs_hello
-
Then you really want a fundamentally different system - the point of these is that router chips have a lot of additional usage possibilities.
-
Per MT7688AN datasheet (yes, it's an AN chip, not KN), it can support up to 256MB DDR2 RAM max.
Currently it's using a single 16M x 16 bit/word DDR2 SDRAM.(which is equivalent to 128MB RAM.)
A redesign can use two pieces of MT5TU128M8HE chips in parallel to get to 256MB.
Naturally the board will be slightly longer.ccs_hello
-
@Bas-Rijniersce thanks .. work with me
