Omega2 Dash Segfault

[David Kerr]

I love this new device! Makes for a very powerful screen.

I wrote an LVGL (Rev 7.0.1) app for the Dash to monitor aspects of two Lithium battery packs (SOC, BMS data) plus inverter control. Works great in the Simulator and runs for at least a week without problems.

However, on the real Dash hardware, it runs for about 40mins then dies with a seg fault.

I have done some investigating and found that the number of file handles on the Dash is increasing quickly at a rate of about 30 per second. As evidenced by:

cat /proc/sys/fs/file-nr

When the Dash gets to 65,000+ file handles (Omega2 limit is 12442), the Seg Fault occurs and the extra file handles get released. At first I suspected sockets because I listen (non blocking) on a UDP socket. However I commented out the socket code and it makes no difference. I have no file system code in the App.

The same code on the simulator (running under Ubuntu 19.10) does not consume file handles. The extremely simple Onion "Hello World" example does not consume file handles. I had THOUGHT I saw someone else had a similar problem (this was before I received my actual Dash hardware) but I can no longer find the information. However, I recall there might have been a problem with the lv_tick_inc(5); call every 5msecs.

Any clues on where to look next would be much appreciated!

Best Regards,
David

[David Kerr]

The Onion supplied demo program has the SAME problem!

Omega2-Dash/bin/o2-dash-lvgl-demo

Consumes file handles at the rate of about 22 per second. My project is about 30 per second.

If you remove the call to task handler in the main loop, the problem goes away but of course the Apps no longer function properly. If you increase the delay in the main loop, the creation of file handles slows down.

This problem does not happen in the simulator (at least, not in mine- Eclipse with SDL2).

Regds,
David

[crispyoz]

@David-Kerr here is my suggestion for how to monitor where your file hanldes are going.

Edit /etc/opkg/distfeeds.conf and uncomment lines 2 and 5

opkg update
opkg install lsof

Now you can use lsof to see what process is using the file handles.

lsof produces a a lot of data that is not so useful, use these commands to get a list of the number of file handles by process in a sorted list:

lsof > ~/file-handles.log
cat ~/file-handles.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20

I would add this to your crontab so you can get a regular update of the file handle usage by process. To do this make sure you update the script so it reads:

lsof > ~/file-handles.log
cat ~/file-handles.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20 >> /var/log/progessive-file-handles.log

Let it run for a while then you can get the incremental details using the command:

cat /var/log/progessive-file-handles.log

By examining the log you will see which processes are increasing their file handle usage over time.

[Lazar Demin]

@David-Kerr Responded in your GitHub issue, but reposting here so the community at large can see:

Yep, there was a memory leak in the code for the Omega2 Dash touch IC. It's now been fixed and the binaries updated.
See the terminal output below that shows the file handles holding steady at 322 five minutes after the program was launched.

The code in the Omega2-Dash repo under demo/littlev-demo is meant to be a quick and dirty demo for showing how lvgl works and runs on the Omega2 Dash.
We don't recommend building on top of it. If you're looking for a solid foundation for building an LVGL program for the Omega2 Dash, please see the lvgl quickstart template: https://github.com/OnionIoT/lv_example

And more info on the memory leak fix here: https://github.com/OnionIoT/lv_drivers/pull/1

---

root@Omega-47E2:~# date ; cat /proc/sys/fs/file-nr
Wed Jun 10 18:34:57 GMT 2020
317	0	12442
root@Omega-47E2:~# date ; /tmp/o2-dash-lvgl-demo &
Wed Jun 10 18:35:05 GMT 2020
root@Omega-47E2:~# lv_init
The framebuffer device was opened successfully.
320x240, 16bpp
The framebuffer device was mapped to memory successfully.
lv_test_theme_2
starting loop

root@Omega-47E2:~# date ; cat /proc/sys/fs/file-nr
Wed Jun 10 18:35:09 GMT 2020
322	0	12442
root@Omega-47E2:~# date ; cat /proc/sys/fs/file-nr
Wed Jun 10 18:35:37 GMT 2020
322	0	12442
root@Omega-47E2:~# date ; cat /proc/sys/fs/file-nr
Wed Jun 10 18:37:39 GMT 2020
322	0	12442
root@Omega-47E2:~# date ; cat /proc/sys/fs/file-nr
Wed Jun 10 18:40:13 GMT 2020
322	0	12442

[David Kerr]

Thanks Lazar for the quick response.

Yes, all fixed now with the latest drivers.

I had downloaded them about 12 days ago which meant I got the bug. Clearly the demo also had the bug. Great to have all that cleared up.

I had not used the demo as a basis for my code. Just coincidence that both projects had the older driver. It did not affect me until I received my real hardware a few days ago.

Best Regards,
Dave

[Lazar Demin]

@David-Kerr Good to hear! Would love to see the final product when it's done!

[David Kerr]

Thanks Lazar and also to crispyoz for the lsof info.

Project is finished but not yet in a box. I'm investigating battery power/charging for it and will report back when the whole thing is done.

I'm actually using it now but with the Dash hanging off the end of a USB cable.

Cheers,
Dave