@James-Penick Do you mind if I contact you outside the Onion forums? You can send me a Linked-In page and we can work it out from there, if it makes sense. I want to ask you a few things that are outside the scope of this forum.
Best posts made by JP Norair
RE: Omega2+ serial not working as expected
Hi James: I've had enormous trouble getting Onion Omega 2+ UART to work properly. There's clearly a bug somewhere, although it's hard to say where. We think it's in musl, but, regardless, it's not something easy to fix. The latest builds of the Onion firmware are especially bad. The present stable build works somewhat. This bug is actually preventing me from deploying Onions commercially.
Here's the thread from August that details this very issue.
RE: How to debug?
@mawg You should see if it's possible to run GDB on your onion -- probably you'll want the Omega2 that has the most RAM, i.e. 2+. GDB offers things like breakpoints, stack-trace, read variables, capture on fault, etc.
Midway down this page, there's description on how to install gdb.
RE: How to debug?
Do you chuckle at people using electric and pneumatic tools to build things? We have these tools now, so you probably should use them. I don't suspect anyone is "unable to manage without [GDB]," but it makes the job a whole lot easier.
As for my development methodology, most-often the huge majority of time is spent on the mac, and it's much faster to iterate on a desktop than it is on an embedded target. One example of a problem appearing in a late stage of this process is my recent struggle with the tty driver on the Omega2, which wasn't obvious until running on the omega2, but this wouldn't have been any easier to deal-with had I been running directly on the target the whole time. There's simply a bug in musl.
Anyway, the theme is pretty simple: the best strategy is the one that yields the best result in the shortest time.
RE: How to debug?
Why would GDB be a last resort? Obviously, you would use it with a debug build. It will show thread calls and allow trace-back to the point of any crashes. If you're programming in C, that's helpful.
Anyway, here's how I write C programs for Linux. Most of which are multithreaded, via pthreads, and many of which access devices.
Set up an environment on mac using the libraries you'll need, but for the mac platform. Mac uses the XNU kernel, which is based on BSD, and it's far more reliable than most linuxes. Controversial, yes, but from my experience very true. Especially for multithreaded apps.
Code the software in XCode. XCode has the best linting and real-time error/warning detection of any tool I've ever seen. It runs LLVM/Clang underneath, which is has a compatible interface with GCC. You'll detect most bugs before you even build your app.
Run the software on mac. You can use lldb or gdb (I use both). lldb is somewhat more capable IMO.
Once it works on mac, get it running on a Linux desktop. Use GDB for debugging.
Once it works on Linux desktop, get it running on the Linux target platform (i.e. Onion). A nice I/O-driven test framework for this purpose is TAP (Test Anything Protocol).
RE: Request for improvement on driver for ttyS_ (or help to do it myself)
Here is the list of issues I've encountered with the Omega2+ serial port. I'm a big fan of the Onion project, and I'd like to help if possible. I can send you hardware and software to replicate this, if you'd like.
- Do not use poll() or tcflush()!
- Use read(fd, buf, 1) only. That is, size param = 1.
Receiver on UART0 seems to have a lot of capacitance on it. This can be overcome with a strong pull-up, but I thought I'd mention this in any case.
I have a half-duplex binary packet protocol. It's much like SLIP or PPP. If I send a Request TX packet to the endpoint, and it generates a Response RX packet, problems can occur during the RX.
A. If poll() is used during reception of the Response RX Packet, the serial port will spew a bunch of data on the TX line (see behavior in logic analyzer capture).
B. If tcflush(..., TCIFLUSH) is used after the packet is fully received and read (via read()), similar behavior as in (2A) will ensue.
C. Usage of tcflush(), in any way, anywhere in the program seems to cause issues with the fd it is used-on.
There is another issue with reception. When I'm receiving a binary packet, the 16th byte is often dropped.
A. When I scope the lines, I can verify that the 16th byte is always on the line.
B. Here's an example of a correct packet. Bytes 0:1, sync. Bytes 2:7, header. Bytes 8:..., payload. This packet has 0x39 payload size (57 bytes).
[FF 55] [F9 F4 00 39 08 00] [C0 35 01 35 01 00 00 00 30 00 00 36 00 4A 30 06 ...]
C. Here's the same packet, with byte 16 missing. The last byte of the payload will be read as FF, because this is the idle state of the serial lines.
[FF 55] [F9 F4 00 39 08 00] [C0 35 01 35 01 00 00 30 00 00 36 00 4A 30 06 ... FF]
D. If I change the way read() is done (i.e. changing the size boundary to smaller values), this often causes more bytes to be dropped. HOWEVER, if I use size=1, and only size=1, reception generally works without dropping bytes. This is one workaround I'm using.
tcdrain(...) does not appear to cause any problems, although I'm skeptical if it actually blocks program flow as it should.
tcdrain() and tcflush() both return 0 (no errors)
I've run the original SW over USB-serial (ttyUSBx) and USB-CDCACM (ttyACMx) interface without issue.
I've run the original SW on ttySx, with a 4.x kernel (might have been 4.4.xxx) on an OrangePi without issue. Not sure what HW they have, but definitely it is a 32 bit ARM Cortex A7, and definitely it's NOT running musl. So the libc is different. I expect the problems are probably not so much in the driver but in musl, but that's just my guess.
Request for improvement on driver for ttyS_ (or help to do it myself)
Hi folks. I have a situation using poll() in C code, with /dev/ttyS0.
I'd like to either make poll() a lot faster (seems unlikely), or find a way to increase the size of the read buffer for each ttyS_ device. I figure this means rebuilding the kernel.
I'm sending packets over ttyS0 -- not characters. Serial packet protocols are not uncommon, but they won't work very well with Omega2/LEDE because poll() can't really be used.
By the time poll() returns, the tty buffer is already long gone.
Based on what I can glean from testing dozens of different poll/read configurations, I think the tty driver keeps a buffer of only 8 bytes (which is probably just the hardware FIFO).
Anyway, that's it. I have a fully working buildroot cross-compilation setup for the Omega2 platform, if that helps. Thanks.