[Project] Docker Omega2 SDK for Cross Compilation + CMake support
-
Omega2 Docker for Cross Compilation
In my repositories ( GitHub and DockerHub ) I have an automated Dockerfile build based on borromeotlhs', but his is currently broken, and doesn't automate all the process.
Now you only need to pull from my Docker Hub with the command:
docker pull jlcs/omega2-sdk
and then run
make
to finish the build (will take a long time).Or use this other one with the toolchain already built (see Edit 2 below):
docker pull jlcs/omega2-docker-built
To run it use:
docker run -it --name omega2-sdk-app -v /my_host_dir:/remote jlcs/omega2-sdk bash
or the bigger one (see Edit 2)
docker run -it --name omega2-sdk-app -v /my_host_dir:/remote jlcs/omega2-docker-built bash
where the container is given a name, shared directory and opens a terminal.
How to compile C code
Once you are in the Docker container, some environment variables are set (maybe not depending on current dockerfile) for you to use:
mipsel-openwrt-linux-gcc [options]
The compiler binaries are generated inside the directory:
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin/
Note: The 'el' in mipsel indicates it is a Little Endian version of MIPS.
Another option is to create a package, and install with opkg, as explained here (step 6b).
Edits
Edit 1: What to change for Omega2+
If you read the Dockerfile you will notice the lines
# Set SDK environment for Omega2 # For Omega2+ change the third echo line with: (notice the 'p' for plus) # echo "CONFIG_TARGET_ramips_mt7688_DEVICE_omega2p=y" > .config && \ RUN echo "CONFIG_TARGET_ramips=y" > .config && \ echo "CONFIG_TARGET_ramips_mt7688=y" >> .config && \ echo "CONFIG_TARGET_ramips_mt7688_DEVICE_omega2=y" >> .config && \ make defconfig
So before running make, you can run
echo "CONFIG_TARGET_ramips=y" > .config && \ echo "CONFIG_TARGET_ramips_mt7688=y" >> .config && \ echo "CONFIG_TARGET_ramips_mt7688_DEVICE_omega2p=y" >> .config && \ make defconfig
And then run
make
.Edit 2: Added docker image with prebuilt SDK (Omega2)
In this second Docker Hub repository I simply run
make tools/install
andmake toolchain/install
from my previous image.The download is about 2GB.
Use
docker pull jlcs/omega2-docker-built
to download it.
Edit 3: Added CMake support.
You can update your image with CMake 3.7.2, and use this
Toolchain-omega2-mipsel.cmake
file to configure your cross-compilation:# this one is important SET(CMAKE_SYSTEM_NAME Linux) #this one not so much SET(CMAKE_SYSTEM_VERSION 1) # specify the cross compiler SET(CMAKE_C_COMPILER /lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin/mipsel-openwrt-linux-gcc) SET(CMAKE_CXX_COMPILER /lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin/mipsel-openwrt-linux-g++) # where is the target environment SET(CMAKE_FIND_ROOT_PATH /lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl) # search for programs in the build host directories SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) # for libraries and headers in the target directories SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) # From: # http://www.vtk.org/Wiki/CMake_Cross_Compiling # Use the commands: # mkdir build # cd build # cmake -DCMAKE_TOOLCHAIN_FILE=Toolchain-omega2-mipsel.cmake .. # make
Edit 4: A Docker micro-reference
Check this cheat sheet and this other one.
Some definitions:
- Image – a read-only layer that is the base of your container. It can have a parent image to abstract away the more basic filesystem snapshot. So a Java image would inherit from a linux image with the preinstalled utilities. A tomcat image will have a Java image as the parent because it depends on Java to run Tomcat.
- Container – a runnable instance of the image, basically it is a process isolated by docker that runs on top of the filesystem that an image provides.
Some commands:
- Download an image, and all its parents, from the registry:
docker pull image-name
- Run a shell command inside a freshly created and started container:
docker run -ti --name container-name image-name /command
- Start and stop a container, duh!
docker start container-name docker stop container-name
- Run a command inside a container from the image and remove the container when command is done.
docker run --rm -ti image-name /command
Simplified vision of Docker:
Sharing host folders as a mounted volume:
Images from dotnetcurry
-
Hello José. As I mentioned, I have limited linux skills so please bear with me (Back when I did hardware development I came from a windows environment where I used Keil-C51 and asm51to develop 8031/51). I installed Docker and then ran
<code>docker run -it --name omega2-sdk-app -v /my_host_dir:/remote jlcs/omega2-sdk bash</code> and this resulted in....
..
..
..
<code>
ac6fd0b9cf61: Pull complete
Digest: sha256:5ef396085b6de58353f94c22f038b75dd41adaaefa35a799fcb02a7c66e36360
Status: Downloaded newer image for jlcs/omega2-sdk:latest
docker: Error response from daemon: Mounts denied:
The path /my_host_dir
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
..
xxxxxx:~ xxxx$ make
xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.
</code>
Then, I was not sure how to run the make command (I remember with the 8051's you would list your source and all the libraries you included) so figured I'd just enter it and it may give me a usage report. My computer then wants me to load XCODE. Is this the route you took? And, what parameters did you feed MAKE?I appreciate your support.
Thank you.
PS: Voting wise, even though it is bigger, I think for the un-initated like myself, the two heavy Docker repositories would probably be more suitable.
-
@Jo-Kritzinger
First, the commaddocker run
was not for copy&paste directly, you must change after the-v
option the directory you want to share, like /Users/kritzinger/Documents/myProgramCodedocker run -it --name omega2-sdk-app -v /CHANGE_THIS:/remote jlcs/omega2-sdk bash
The -v option tells docker to mount the directory on the left of the colon on the container
/remote
directory. Think of it like aE:/
drive in windows when you plug a pendrive, but it's just another directory inside the container.Once you run the
docker run
command and it works, a different prompt will appear, with something like:omega@490016bda699:/lede$
That is the terminal inside the container, like another computer. In that terminal you can write:
omega@490016bda699:/lede$ make
And then you go for a coffee, 15 minutes later come back for everything built. Notice this is if you have Omega2, if you have Omega2+ read my new edit (give me a minute).
Also, you can run only:
omega@490016bda699:/lede$ make tools/intall omega@490016bda699:/lede$ make toolchain/install
which won't compile packages, but will give you the gcc cross-compiler, and save a couple GB of space (not much).
Edit1
On Mac to share some directories you have to add them before using the GUI. By default /Users/ should work, but here you have more documentation.
-
Thank you, will give it a go
-
@José-Luis-Cánovas Thank you, the installation went well. However, I am a little lost here, like a paper plane enthusiast in a Space Shuttle cockpit. Oooh, sooo many buttons! When you have a moment, can you provide a bit of hand holding here please?
I have a omega@05518b3b34b9 prompt but when I call mipsel-openwrt-linux-gcc it says command not found. However, under
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.16
/bin$
I see mipsel-openwrt-linux-gcc and it seems to be a link (?) to mipsel-openwrt-linux-musI-gcc. Neither runs when entered on the command line. gcc on it's own does run but obviously the target then is x86-64. I assumed it may be an environment varible that was not setup so I entered env to see if I can glean anything and I get as follows
HOSTNAME=05518b3b34b9
TERM=xterm
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.axa=00;36:.oga=00;36:.spx=00;36:.xspf=00;36:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/lede
SHLVL=1
HOME=/home/omega
no_proxy=*.local, 169.254/16
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env
OLDPWD=/So, assuming the environment variables are setup correctly, I use the HOME folder for my source code right? How do I link the Omega libraries? What options should be specified as parameters for the compile?
Thank you in advance.
-
@Jo-Kritzinger said in [Project] Docker Omega2 SDK for Cross Compilation:
I assumed it may be an environment varible that was not setup
You assumed well. Because in my dockerfile I don't run
make
, I also don't run:ENV PATH "$PATH:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin"
Don't copy and paste this code, because this is Dockerfile syntax, but you get that your path must include the
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.15/bin
directory.About linking libraries, this thread answers it.
And finally, you can use any directory for your source. It is a good practice to have it on /home/omega/myProject/, but you can also use the
/remote
directory created whendocker run
, and have your code in your mac. If you didn't use the-v
option to mount the/remote
directory, it can only be done with adocker run
, but don't do it, because you will create another container (one image creates on docker run as many containers as you want). Read aboutdocker commit
to create a new image with the SDK (warning: the image will need another 17GB of disk), and then do adocker run -v ...
with the new image.
-
Thank you, I will try to get my head around it. Thank you for your patience.
-
@José-Luis-Cánovas Yay! I got a hello world compiled, transferred and run on the omega. Thank you. seems there is light at the end of the tunnel.
But...
There are still some problems. I can't seem to set PATH and the environment variable STAGING_DIR. The workaround I've used is to run the export command at the container command line.
i.e. export PATH=$PATH:/lede/staging_dir/.... and so on
and then
export STAGING_DIR=/lede/staging_dir
Is there a way of setting this without having to run the export commands so I don't have to do it every time?I don't understand what you said about the -v option and not to do it. Docker is still a bit of a mystery to me. I use the /remote folder to copy the bin to so I can then connect via a terminal from the host machine to transfer it to the omega. I'ts a bit of a pain but I can live with it.
Also, lets say you want to use the i2c library. So, you specify the #include <onion-i2c.h> in the source, but how do you compile/link and generate the binary file? Or, how would you get to the gpio's? Would you use a pointer to the addresses in memory directly? A long * ?
What are the most common options you specify on the mipsel-openwrt-linux-gcc line?
Then, lastly: I have both Omega2 and Omega2+, how can I set this up so I can develop for both of them? I noticed you mentioned editing some files but does that mean I have to run a make eveyrtime I change hardware platform?
Thank you José. I really appreciate your help.
-
@Jo-Kritzinger said in [Project] Docker Omega2 SDK for Cross Compilation:
@José-Luis-Cánovas Yay! I got a hello world compiled, transferred and run on the omega. Thank you. seems there is light at the end of the tunnel.
Then, lastly: I have both Omega2 and Omega2+, how can I set this up so I can develop for both of them? I noticed you mentioned editing some files but doe that mean I have to run a make overtime I change hardware platform?
You shouldn't have to do much of anything different between the two when building ordinary userspace programs; only the bootloader and system builds really need to know about flash partitions, address width and memory size.
-
@Chris-Stratton Thank you Chris
-
@Jo-Kritzinger said in [Project] Docker Omega2 SDK for Cross Compilation:
export PATH=$PATH:/lede/staging_dir/.... and so on
and then
export STAGING_DIR=/lede/staging_dir
Is there a way of setting this without having to run the export commands so I don't have to do it every time?Add it to the /home/omega/.bashrc line, at the end, and those two
export
lines will execute everytime.I don't understand what you said about the -v option and not to do it. Docker is still a bit of a mystery to me. I use the /remote folder to copy the bin to so I can then connect via a terminal from the host machine to transfer it to the omega. I'ts a bit of a pain but I can live with it.
That's actually a use for the -v option, to have a shared directory (/remote) between the container and your host machine. The same way you copy your bin to /remote and then you can access it from your mac, use the other way around to copy your source code to the machine. That way you can edit your code with Atom/Sublime text/XCode/etc. from your mac, and access them from the container in /remote.
Also, lets say you want to use the i2c library. So, you specify the #include <onion-i2c.h> in the source, but how do you compile/link and generate the binary file? Or, how would you get to the gpio's? Would you use a pointer to the addresses in memory directly? A long * ?
I haven't used the I2C library, but here is the official documentation. In it you can read:
Library for Linker In your project’s makefile, you will need to add the following dynamic libraries to the linker command: -loniondebug -lonioni2c The dynamic libraries are stored in /usr/lib on the Omega.
The two
-l
options you add to the Makefile are the same you add to your gcc command. It tells to find dynamic libraries namedoniondebug
andonioni2c
, but doesn't need them now (dynamic). The library files are already inside your omega2, so you don't need to copy any files.What are the most common options you specify on the mipsel-openwrt-linux-gcc line?
The same I would use with common gcc. Like before, if I need a dynamic library, use the
-l
option, etc. Depends on your project. You can search for an introduction to Makefile for "good practices", but it can became something too big for what you want, so stay in the basis of make.Then, lastly: I have both Omega2 and Omega2+, how can I set this up so I can develop for both of them? I noticed you mentioned editing some files but does that mean I have to run a make eveyrtime I change hardware platform?
I don't know exactly what processor has the Omega2+, but I don't think it's different from the MIPSel of the Omega2. To build executable binaries the same compiler should do fine, try to run the same binary hello_world in both and check for yourself. To build your own firmware, then you must proceed with caution not to break your omega.
Thank you José. I really appreciate your help.
No worries. This messages can help other people too, and we all started from the bottom.
-
Thank you José.
-
@Jo-Kritzinger
if you are still interested, I added (in the OP edits) the image with all themake
commands to build the SDK already executed, and also added CMake support, very easy to use and works like a charm for me.If you use CLion projects for example, you only have to copy your source code to the shared directory (between your mac and docker), copy the
Toolchain-omega2-mipsel.cmake
file I posted in theEdit 3
and run cmake from docker as said in the last lines.
If any errors happen, delete the CMake cache files and try again.
-
@José-Luis-Cánovas thank you José
-
@José-Luis-Cánovas José, me again. I used the -lonioni2c etc. but it says it can not find the libraries.
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.16/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -loniondebug
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl-1.1.16/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -lonionspi
collect2: error: ld returned 1 exit statusThen I added in the -L/usr/lib. Same deal. I added a -v on the line and, somewhere in the heap of messages, I see "Ignoring nonexistent directory "/lede/staging_dir/usr/include". So, from my limited experience, it seems that the linker needs
- to see the actual libraries at compile/link time
- is looking at a directory in docker, i.e. /lede/staging_dir
Is this because of using Docker and does it mean I will have to download the actual source files and compile those, ignoring the actual library?
By the by, there is something odd in this whole thing as the h files are missing so I was forced to download those as well. Logically (again in my limited experience in the Linux world) I would expect the see the h files in the same (or similar) directory as the actual library .so file? e.g. /usr/include or something? I mean, the pre-processor and compiler must have access to them right? Well, really, they (h and so files) should be part of Docker?
I feel I am missing something here.
Probably that I do not understand the whole Docker setup properly as Docker is (obviously) not aware of the actual hardware/firmware/software that is on the Omega meaning referencing any directories are not going to work as expected because the compilation/linking is not taking place on the target system?
Does using docker mean that using shared object libraries are off the cards?
In the interest of completeness, this is the last gcc instruction I used:
mipsel-openwrt-linux-gcc -L/usr/lib -Wl,-rpath=/usr/lib -Wall testSpi.c -loniondebug -lonionspi -o testspi.out
-
@Jo-Kritzinger
Think of Docker as a virtual machine (doesn't work like that, but for the sake of simplicity), in particular, an Ubuntu 14.04 with some files added (/lede/*
).Anything you run on the Docker container, doesn't affect your host machine, only your shared directories, and those are for copying code.
Now that we think of the container as a machine on its own, let's see the SDK.
Ubuntu comes with gcc that compiles C code to its own architecture, and it can run the compiled binaries with its shared libraries, e.g. in /usr/lib. All that is fine, if you compile for the PC.
Omega2 can't compile anything because the compiler is too big for it to run on the Omega2. Therefore, we need a cross compiler, including the gcc binary and cross compiled libraries aiming for the same MIPSel arquitecture as the Omega2 has.
The gcc binary and cross compiled libraries are not in your Ubuntu 14.04 /usr/lib directory, because Ubuntu doesn't need any of them. They are inside the/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/
(the compiler) and/lede/staging_dir/target-mipsel_24kc_musl/
(the include and library files) directories. There is wheremipsel-openwrt-linux-gcc
must look for the headers and libraries (see below).Your errors come from the linker not finding suitable files for your specified libraries, because the directory is wrong.
One bug I find is that your toolchain dir has a
-1.1.16
in the end. It is now gone in the new builds of the docker image.Another thing you can do is pasting these lines in your
~/.bashrc
file, or exec adocker pull
from the last image built, where the env vars are already set:export PATH "$PATH:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin" export STAGING_DIR "/lede/staging_dir" export CFLAGS=-I/lede/staging_dir/target-mipsel_24kc_musl/usr/include export LDFLAGS=-L/lede/staging_dir/target-mipsel_24kc_musl/usr/lib
Notice that
PATH
no longer has-1.1.16
, and the include and libraries directories are set for the compiler.Finally, once you compile successfully and copy it to the Omega2, the binary that uses shared libraries will find the .so objects in the Omega2 too, but the lede SDK already have them all in your host machine.
-
@José-Luis-Cánovas thank you José, I will download the new image and try again. Will it install overvthe old image or must I use docker to delete the old one?
-
@José-Luis-Cánovas Still having issues. I pulled the jlcs/omega2-docker-built and ran with
run -it --name omega2-sdk-app-f -v /<inserted my host dir here>:/remote jlcs/omega2-sdk bash.
(NOTE the --name omega2-sdk-app-f. Without it I got a error saying it exists already and I was not to keen to delete because I'm not familiar with the system and was worried I'd delete something important so I just gave it a new name)This gave me a omega@a1e6ec1ab9d3: prompt.
Then I tried a compile and it failed. So, I investigated a bit further:
printenv results in the following:
LDFLAGS=-L/lede/staging_dir/target-mipsel_24kc_musl/usr/lib
HOSTNAME=a1e6ec1ab9d3
TERM=xterm
LS_COLORS(====Edit all the colour info out=====)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin
PWD=/lede/staging_dir/target-mipsel_24kc_musl/lib
SHLVL=1
HOME=/home/omega
CFLAGS=-I/lede/staging_dir/target-mipsel_24kc_musl/usr/include
no_proxy=*.local, 169.254/16
STAGING_DIR=/lede/staging_dir
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %sThen I did this to see if I could spot something:
pwd
result ==> /lede/staging_dir/target-mipsel_24kc_muslls
result ==> bin include lib stamp
*NOTE, There is no /usr directorycd include/
ls
result ==>cd ..
cd lib/
ls -a
Result ==> . ..Here is the compile instruction:
/lede/staging_dir/target-mipsel_24kc_musl/lib$ mipsel-openwrt-linux-gcc -Wall /remote/testSpi.c -lonionspi -o testSpi.o
Result==>
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -lonionspi
collect2: error: ld returned 1 exit statusSo I ran the compile with the -v again and the same line came up again in the whole list:
ignoring nonexistent directory "/lede/staging_dir/usr/include"Lastly I did:
find /lede -name *.h | grep libonionResult==>
What am I doing wrong?
-
@Jo-Kritzinger The image will update, but you will have to run a new container from it with
docker run ...
.
-
@Jo-Kritzinger said in [Project] Docker Omega2 SDK for Cross Compilation + CMake support:
I got a error saying it exists already and I was not to keen to delete because I'm not familiar with the system and was worried I'd delete something important so I just
It is the old container. In Docker we have
images
andcontainers
, images are static, and containers dynamic, what you run. With your command you are telling docker to create a new container with a fixed name; because it already exists (the container you have been working with) it shows an error.I pulled the jlcs/omega2-docker-built and ran with
run -it --name omega2-sdk-app-f -v /<inserted my host dir here>:/remote jlcs/omega2-sdk bash.Here is the error. The penultimate argument in your command should be
jlcs/omega2-docker-built
, the name of the new image, but you usedjlcs/omega2-sdk
, the one that didn't run themake
commands.