[Project] Docker Omega2 SDK for Cross Compilation + CMake support

[José Luis Cánovas]

@Jo-Kritzinger
Think of Docker as a virtual machine (doesn't work like that, but for the sake of simplicity), in particular, an Ubuntu 14.04 with some files added (/lede/*).

Anything you run on the Docker container, doesn't affect your host machine, only your shared directories, and those are for copying code.

Now that we think of the container as a machine on its own, let's see the SDK.
Ubuntu comes with gcc that compiles C code to its own architecture, and it can run the compiled binaries with its shared libraries, e.g. in /usr/lib. All that is fine, if you compile for the PC.
Omega2 can't compile anything because the compiler is too big for it to run on the Omega2. Therefore, we need a cross compiler, including the gcc binary and cross compiled libraries aiming for the same MIPSel arquitecture as the Omega2 has.
The gcc binary and cross compiled libraries are not in your Ubuntu 14.04 /usr/lib directory, because Ubuntu doesn't need any of them. They are inside the /lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/ (the compiler) and /lede/staging_dir/target-mipsel_24kc_musl/ (the include and library files) directories. There is where mipsel-openwrt-linux-gcc must look for the headers and libraries (see below).

Your errors come from the linker not finding suitable files for your specified libraries, because the directory is wrong.

One bug I find is that your toolchain dir has a -1.1.16 in the end. It is now gone in the new builds of the docker image.

Another thing you can do is pasting these lines in your ~/.bashrcfile, or exec a docker pull from the last image built, where the env vars are already set:

export PATH "$PATH:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin"
export STAGING_DIR "/lede/staging_dir"
export CFLAGS=-I/lede/staging_dir/target-mipsel_24kc_musl/usr/include
export LDFLAGS=-L/lede/staging_dir/target-mipsel_24kc_musl/usr/lib

Notice that PATH no longer has -1.1.16, and the include and libraries directories are set for the compiler.

Finally, once you compile successfully and copy it to the Omega2, the binary that uses shared libraries will find the .so objects in the Omega2 too, but the lede SDK already have them all in your host machine.

[Jo Kritzinger]

@José-Luis-Cánovas thank you José, I will download the new image and try again. Will it install overvthe old image or must I use docker to delete the old one?

[Jo Kritzinger]

@José-Luis-Cánovas Still having issues. I pulled the jlcs/omega2-docker-built and ran with
run -it --name omega2-sdk-app-f -v /<inserted my host dir here>:/remote jlcs/omega2-sdk bash.
(NOTE the --name omega2-sdk-app-f. Without it I got a error saying it exists already and I was not to keen to delete because I'm not familiar with the system and was worried I'd delete something important so I just gave it a new name)

This gave me a omega@a1e6ec1ab9d3: prompt.

Then I tried a compile and it failed. So, I investigated a bit further:

printenv results in the following:
LDFLAGS=-L/lede/staging_dir/target-mipsel_24kc_musl/usr/lib
HOSTNAME=a1e6ec1ab9d3
TERM=xterm
LS_COLORS(====Edit all the colour info out=====)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/bin
PWD=/lede/staging_dir/target-mipsel_24kc_musl/lib
SHLVL=1
HOME=/home/omega
CFLAGS=-I/lede/staging_dir/target-mipsel_24kc_musl/usr/include
no_proxy=*.local, 169.254/16
STAGING_DIR=/lede/staging_dir
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s

Then I did this to see if I could spot something:
pwd
result ==> /lede/staging_dir/target-mipsel_24kc_musl

ls
result ==> bin include lib stamp
*NOTE, There is no /usr directory

cd include/
ls
result ==>

cd ..
cd lib/
ls -a
Result ==> . ..

Here is the compile instruction:
/lede/staging_dir/target-mipsel_24kc_musl/lib$ mipsel-openwrt-linux-gcc -Wall /remote/testSpi.c -lonionspi -o testSpi.o
Result==>
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -lonionspi
collect2: error: ld returned 1 exit status

So I ran the compile with the -v again and the same line came up again in the whole list:
ignoring nonexistent directory "/lede/staging_dir/usr/include"

Lastly I did:
find /lede -name *.h | grep libonion

Result==>

What am I doing wrong?

[José Luis Cánovas]

@Jo-Kritzinger The image will update, but you will have to run a new container from it with docker run ....

[José Luis Cánovas]

@Jo-Kritzinger said in [Project] Docker Omega2 SDK for Cross Compilation + CMake support:

I got a error saying it exists already and I was not to keen to delete because I'm not familiar with the system and was worried I'd delete something important so I just

It is the old container. In Docker we have images and containers, images are static, and containers dynamic, what you run. With your command you are telling docker to create a new container with a fixed name; because it already exists (the container you have been working with) it shows an error.

I pulled the jlcs/omega2-docker-built and ran with
run -it --name omega2-sdk-app-f -v /<inserted my host dir here>:/remote jlcs/omega2-sdk bash.

Here is the error. The penultimate argument in your command should be jlcs/omega2-docker-built, the name of the new image, but you used jlcs/omega2-sdk, the one that didn't run the make commands.

[Jo Kritzinger]

@José-Luis-Cánovas Thanks José, I will give it a go tomorrow. I shouls have spotted that. (Embarrased grin)

[Philipp Zeitschel]

@José-Luis-Cánovas
thank you for your work, the container runs perfect and you saved me a lot of time!

[José Luis Cánovas]

@Philipp-Zeitschel you are welcome!

[Jo Kritzinger]

This post is deleted!

[Jo Kritzinger]

@José-Luis-Cánovas Hi José. I have been doing other stuff and am just getting back into this.
I'm still missing something. I think I have the right one running, checked by doing a docker ps that resulted in

628a8ef6b111 jlcs/omega2-docker-built "bash" 27 minutes ago Up 3 minutes omega2-sdk-app-built

Also /lede/staging_dir/target-mipsel_24kc_musl/include and /lib are both empty.

This is the result from the compiler:

mipsel-openwrt-linux-gcc   -Wall testSpi.c -loniondebug -lonionspi -o testspi.out
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -loniondebug
/lede/staging_dir/toolchain-mipsel_24kc_gcc-5.4.0_musl/lib/gcc/mipsel-openwrt-linux-musl/5.4.0/../../../../mipsel-openwrt-linux-musl/bin/ld: cannot find -lonionspi
collect2: error: ld returned 1 exit status

Any advise?

Thank you.

[José Luis Cánovas]

@Jo-Kritzinger yes, silly me, I saw the directories errors, but not that you wanted onion libraries.

[Run inside the docker container]
You must add the onion feeds (work in progress for my docker image, in the meantime do it manually) to the file feeds.conf.default (check WereCatf's first_time_setup.sh), update the feed and add the onion packages:
scripts/feeds update -a
scripts/feeds install -a
Then, from make menuconfig select the onion libraries (Onion -> Libraries/Utilities -> select as [M]odule).
Don't worry about a worning with recursive dependencies.
Run this chunk of code to avoid adding an SSH certificate to github only to clone some Onion packages:

(Read Edit 1)

# From https://github.com/WereCatf/source/blob/image/first_time_setup.sh
SCRIPTDIR=`pwd`
#Ugly fix for several packages
#Onion-devs don't give a fuck
onionNeedCommit=0
find feeds/onion -iname "Makefile" | while read filename
do
githubUrl=$(grep "git@github.com" "$SCRIPTDIR/$filename")
if [ $? -eq 0 ]
then
    githubUrl=$(echo "$githubUrl"|sed 's/git@github.com:/https:\/\/github.com\//'|sed 's/\.git//'|cut -d '=' -f 2-)
    curl --output /dev/null --silent --head --fail "$githubUrl"
    if [ $? -eq 0 ]
    then
        echo "Patching $filename..."
        sed -i 's/git@github.com:/https:\/\/github.com\//' "$SCRIPTDIR/$filename"
        onionNeedCommit=1
    fi
fi
done

Then run make -j#, where # is your number of cores +1, to speed things up (check again the end of first_time_setup.sh).

Whenmake finishes, an ls staging_dir/target-mipsel_24kc_musl/usr/lib/ (note the penultimate /usr/ directory) shows:

omega@dee42fad6a75:/lede$ ls staging_dir/target-mipsel_24kc_musl/usr/lib/
[...]    liboniondebug.so   [...]    libonioni2c.so  [...]    libonionspi.so  [...]

and many more onion and non-onion libraries.

And inside staging_dir/target-mipsel_24kc_musl/usr/include/:

omega@dee42fad6a75:/lede$ ls staging_dir/target-mipsel_24kc_musl/usr/include/                                         
[...] onion-spi.h   [...]    onion-debug.h    [...]      onion-i2c.h      [...]

Edit 1: Install curl before running the script:

First, we need to login as root in the container to gain privileges to install new packages.
With the docker container stopped, run:
[Run in your host's terminal (mac, windows, linux)]

$   docker start omega2-sdk-app-built
$   docker exec -ti --user root omega2-sdk-app-built bash

Now you are root user inside the container. Install the curl package.
[Run inside the docker container]

#   apt update && apt install -y curl
#   exit

Then, go back to the container as omega user with the usual:
[Run in your host's terminal (mac, windows, linux)]

$   docker start omega2-sdk-app-built
$   docker attach omega2-sdk-app-built

Run the script above.

[Jo Kritzinger]

@José-Luis-Cánovas Haha, no worries. When I have a block of time I'll sit down and give this a go. I'm assuming this is all run (esp the first chunk of code) from within docker, i.e. at the omega@xxxxxx prompt?

By the way, I really appreciate your help.

[José Luis Cánovas]

@Jo-Kritzinger yes, everything inside the docker container except for the docker commands in Edit 1.
The omega2 will have the headers and libraries already inside, and if not, use opkg update and opkg install <package> where package is your desired onion library. In the SDK, selecting with make menuconfig and building with make compiles the source code of these libraries, generating the necessary files for the linker (ld).

[Jo Kritzinger]

@José-Luis-Cánovas So, the code from edit 1 just at a normal prompt on my Mac?

[José Luis Cánovas]

@Jo-Kritzinger yup, except apt because you will be inside the container. I will edit so it is clearer.

[Richard Berg]

Trying to use this and wanted to install other base packages to the shell with apt-get and getting prompted for the user Omega password. I tried to review all the replies here and original post but didn't see it listed anywhere.

@José-Luis-Cánovas

[José Luis Cánovas]

@Richard-Berg
check 5 posts above, the Edit1, changing curl with your package.

@José-Luis-Cánovas said in [Project] Docker Omega2 SDK for Cross Compilation + CMake support:

Edit 1: Install curl before running the script:

First, we need to login as root in the container to gain privileges to install new packages.
With the docker container stopped, run:
[Run in your host's terminal (mac, windows, linux)]

$   docker start omega2-sdk-app-built
$   docker exec -ti --user root omega2-sdk-app-built bash

Now you are root user inside the container. Install the curl package.
[Run inside the docker container]

#   apt update && apt install -y curl
#   exit

Then, go back to the container as omega user with the usual:
[Run in your host's terminal (mac, windows, linux)]

$   docker start omega2-sdk-app-built
$   docker attach omega2-sdk-app-built

Run the script above.

This is because omega user haven't got sudo permissions, and I prefer not to run anything inside docker with root permissions. Because the --user root option in docker is an easy and explicit way to use root, I don't plan to change this.

If you need to, create your own Dockerfile with FROM jlcs/omega2-docker-built at the start, and you will have ready to go image for your project.

Also, tell me what packages you need, because I plan to add many others.

[Richard Berg]

@José-Luis-Cánovas
Doh! I figured you would have had something and sorry I missed it mate!

For me, who has some experience in Linux and other older *nix systems I tend to have my 'creature comforts' like always having apt-get updated and nano for an editor (not that vi/vim is bad; I just use nano easier, lol) and things like that. Please don't add anything like that because what you have is great. I just wanted to install nano this round and couldn't but with the above start then exec as root; I was able to update and install.

Also, I am working on another SBC named VoCore2 which also employs OpenWRT (not LeDe yet) and I was working with a similar environment but within VMWare. Docker might make that compilation easier as well so may take note of your Docker setup (which I've never really used) and make my own for the VoCore2 board!

Cheers

[Richard Berg]

@José-Luis-Cánovas

Thanks again for you work on the Omega2 dockerfile! It has given me some good examples to work from and pilfer a bit of your work on another board I'm working with; the VoCore2

Here is my new dockerfile, albeit in constant tweak/modification, for said work.

https://hub.docker.com/r/ictinike/vocore2-default/

Thanks again!
~Icky

[Peter Buelow]

Reply to an old topic, but I thought I would share. This cmake definition works, but ignores the actual sysroot and causes workarounds needed to do linking. To fix this, you need to tell cmake to define SYSROOT with the following. I create a new variable CMAKE_FIND_TARGET_PATH and point it to the sysroot path in the lede build area. The compiler needs to be told that sysroot is not the compiler path, but the actual target filesystem area. This is an important distinction, and avoids having to use -rpath and other linker options.

SET(CMAKE_FIND_TARGET_PATH <path to>/staging_dir/target-mipsel_24kc_musl)
SET (CMAKE_SYSROOT ${CMAKE_FIND_TARGET_PATH})

Finally, you need to tell the compiler where the sysroot include directories are

ADD_DEFINITIONS ("-isystem <path to>/staging_dir/target-mipsel_24kc_musl/usr/include")

Then, you can use the following logic in your local CMakeLists.txt to find Onion libraries.

add_library (onioni2c SHARED IMPORTED)
set_target_properties(onioni2c PROPERTIES IMPORTED_LOCATION "${CMAKE_FIND_TARGET_PATH}/usr/lib/libonioni2c.so")

This will make it work with the compiler std paths (so std::cout will be found and work correctly), and it will allow you to find <onion-i2c.h> and the others correctly.

Hope this helps everyone.