@James-Penick Do you mind if I contact you outside the Onion forums? You can send me a Linked-In page and we can work it out from there, if it makes sense. I want to ask you a few things that are outside the scope of this forum.

Hi James: I've had enormous trouble getting Onion Omega 2+ UART to work properly. There's clearly a bug somewhere, although it's hard to say where. We think it's in musl, but, regardless, it's not something easy to fix. The latest builds of the Onion firmware are especially bad. The present stable build works somewhat. This bug is actually preventing me from deploying Onions commercially.

Here's the thread from August that details this very issue.
https://community.onion.io/topic/3654/request-for-improvement-on-driver-for-ttys_-or-help-to-do-it-myself

@mawg You should see if it's possible to run GDB on your onion -- probably you'll want the Omega2 that has the most RAM, i.e. 2+. GDB offers things like breakpoints, stack-trace, read variables, capture on fault, etc.

https://docs.onion.io/omega2-docs/c-compiler-on-omega.html

Midway down this page, there's description on how to install gdb.

@crispyoz

Do you chuckle at people using electric and pneumatic tools to build things? We have these tools now, so you probably should use them. I don't suspect anyone is "unable to manage without [GDB]," but it makes the job a whole lot easier.

As for my development methodology, most-often the huge majority of time is spent on the mac, and it's much faster to iterate on a desktop than it is on an embedded target. One example of a problem appearing in a late stage of this process is my recent struggle with the tty driver on the Omega2, which wasn't obvious until running on the omega2, but this wouldn't have been any easier to deal-with had I been running directly on the target the whole time. There's simply a bug in musl.

Anyway, the theme is pretty simple: the best strategy is the one that yields the best result in the shortest time.

@crispyoz

Why would GDB be a last resort? Obviously, you would use it with a debug build. It will show thread calls and allow trace-back to the point of any crashes. If you're programming in C, that's helpful.

Anyway, here's how I write C programs for Linux. Most of which are multithreaded, via pthreads, and many of which access devices.

1. Set up an environment on mac using the libraries you'll need, but for the mac platform. Mac uses the XNU kernel, which is based on BSD, and it's far more reliable than most linuxes. Controversial, yes, but from my experience very true. Especially for multithreaded apps.

2. Code the software in XCode. XCode has the best linting and real-time error/warning detection of any tool I've ever seen. It runs LLVM/Clang underneath, which is has a compatible interface with GCC. You'll detect most bugs before you even build your app.

3. Run the software on mac. You can use lldb or gdb (I use both). lldb is somewhat more capable IMO.

4. Once it works on mac, get it running on a Linux desktop. Use GDB for debugging.

5. Once it works on Linux desktop, get it running on the Linux target platform (i.e. Onion). A nice I/O-driven test framework for this purpose is TAP (Test Anything Protocol).

@Lazar-Demin
Here is the list of issues I've encountered with the Omega2+ serial port. I'm a big fan of the Onion project, and I'd like to help if possible. I can send you hardware and software to replicate this, if you'd like.

Current workarounds:

• Do not use poll() or tcflush()!
• Use read(fd, buf, 1) only. That is, size param = 1.

1. Receiver on UART0 seems to have a lot of capacitance on it. This can be overcome with a strong pull-up, but I thought I'd mention this in any case.

2. I have a half-duplex binary packet protocol. It's much like SLIP or PPP. If I send a Request TX packet to the endpoint, and it generates a Response RX packet, problems can occur during the RX.
   A. If poll() is used during reception of the Response RX Packet, the serial port will spew a bunch of data on the TX line (see behavior in logic analyzer capture).
   B. If tcflush(..., TCIFLUSH) is used after the packet is fully received and read (via read()), similar behavior as in (2A) will ensue.
   C. Usage of tcflush(), in any way, anywhere in the program seems to cause issues with the fd it is used-on.

3. There is another issue with reception. When I'm receiving a binary packet, the 16th byte is often dropped.
   A. When I scope the lines, I can verify that the 16th byte is always on the line.
   B. Here's an example of a correct packet. Bytes 0:1, sync. Bytes 2:7, header. Bytes 8:..., payload. This packet has 0x39 payload size (57 bytes).
   [FF 55] [F9 F4 00 39 08 00] [C0 35 01 35 01 00 00 00 30 00 00 36 00 4A 30 06 ...]
   C. Here's the same packet, with byte 16 missing. The last byte of the payload will be read as FF, because this is the idle state of the serial lines.
   [FF 55] [F9 F4 00 39 08 00] [C0 35 01 35 01 00 00 30 00 00 36 00 4A 30 06 ... FF]
   D. If I change the way read() is done (i.e. changing the size boundary to smaller values), this often causes more bytes to be dropped. HOWEVER, if I use size=1, and only size=1, reception generally works without dropping bytes. This is one workaround I'm using.

4. tcdrain(...) does not appear to cause any problems, although I'm skeptical if it actually blocks program flow as it should.

5. tcdrain() and tcflush() both return 0 (no errors)

Final notes.
I've run the original SW over USB-serial (ttyUSBx) and USB-CDCACM (ttyACMx) interface without issue.
I've run the original SW on ttySx, with a 4.x kernel (might have been 4.4.xxx) on an OrangePi without issue. Not sure what HW they have, but definitely it is a 32 bit ARM Cortex A7, and definitely it's NOT running musl. So the libc is different. I expect the problems are probably not so much in the driver but in musl, but that's just my guess.

Hi folks. I have a situation using poll() in C code, with /dev/ttyS0.

I'd like to either make poll() a lot faster (seems unlikely), or find a way to increase the size of the read buffer for each ttyS_ device. I figure this means rebuilding the kernel.

I'm sending packets over ttyS0 -- not characters. Serial packet protocols are not uncommon, but they won't work very well with Omega2/LEDE because poll() can't really be used.

By the time poll() returns, the tty buffer is already long gone.

Based on what I can glean from testing dozens of different poll/read configurations, I think the tty driver keeps a buffer of only 8 bytes (which is probably just the hardware FIFO).

Anyway, that's it. I have a fully working buildroot cross-compilation setup for the Omega2 platform, if that helps. Thanks.

That works, thanks. Would be nice if this link were in the documentation!

Does anyone know where to find this?

It's not in the archives.
https://docs.onion.io/omega2-docs/manual-firmware-installation.html

However, it's the last known firmware build I've used that kind of works with ttyS0. Newer builds simply don't work with UART over ttyS0 -- I think it's a conflict of something in the musl library and I desperately wish for a uClib variant of the FW -- but that's beside the point. Right now I just want b202 so I can have again an almost-working solution.

Mosquitto is on the default Omega 2 image. You may be able to use it rather than Paho, depending on your workflow.

I'm fiddling with the onion firmware buildroot, trying to solve some issues I believe are due to usage of musl as libc. I'll come back to this in a future post.

Anyway, I've noticed that MT7688 is in fact a MIPS24KEC, not a MIPS24KC, with the E meaning it contains the DSPr1 ASE from MIPS. Therefore, we should be able to compile with the following flags in gcc:

Target Optimizations:
-Os -pipe -mno-branch-likely -march=mips32r2 -mtune=24kec -mdsp -mfix-24k

Extra Optimizations
-fno-caller-saves -fno-plt -ftree-vectorize

I'm curious if anyone has tried these options, and if there are caveats. Code compiled with 24kec (and the dsp instructions) should run and link fine with code compiled for 24kc, but just broadcasting this out there.

On the onion, you need to put the keys in /etc/dropbear/authorized_keys. Maybe you're doing this -- it's not clear. I can say is that I use rsync to upload my distro to Omega2+, and it works. So whatever you're doing is fixable.

@James-Penick Do you mind if I contact you outside the Onion forums? You can send me a Linked-In page and we can work it out from there, if it makes sense. I want to ask you a few things that are outside the scope of this forum.

Hi James: I've had enormous trouble getting Onion Omega 2+ UART to work properly. There's clearly a bug somewhere, although it's hard to say where. We think it's in musl, but, regardless, it's not something easy to fix. The latest builds of the Onion firmware are especially bad. The present stable build works somewhat. This bug is actually preventing me from deploying Onions commercially.

Here's the thread from August that details this very issue.
https://community.onion.io/topic/3654/request-for-improvement-on-driver-for-ttys_-or-help-to-do-it-myself

You can try to re-deploy using the latest Onion build, which is running on the 4.14 kernel.

$ oupgrade -l

Be warned, though, I do not have good results running UART on this version. 0.2.2 b202 has other problems, but it is more stable for me.

@mawg You should see if it's possible to run GDB on your onion -- probably you'll want the Omega2 that has the most RAM, i.e. 2+. GDB offers things like breakpoints, stack-trace, read variables, capture on fault, etc.

https://docs.onion.io/omega2-docs/c-compiler-on-omega.html

Midway down this page, there's description on how to install gdb.

@crispyoz

Do you chuckle at people using electric and pneumatic tools to build things? We have these tools now, so you probably should use them. I don't suspect anyone is "unable to manage without [GDB]," but it makes the job a whole lot easier.

As for my development methodology, most-often the huge majority of time is spent on the mac, and it's much faster to iterate on a desktop than it is on an embedded target. One example of a problem appearing in a late stage of this process is my recent struggle with the tty driver on the Omega2, which wasn't obvious until running on the omega2, but this wouldn't have been any easier to deal-with had I been running directly on the target the whole time. There's simply a bug in musl.

Anyway, the theme is pretty simple: the best strategy is the one that yields the best result in the shortest time.