Cross compile Mono from source for the Omega2 and Omega2+

[Jackdaw]

Background info

In this thread I will explain how to compile the latest Mono from source for the Onion Omega2 and Omega2+, including some things to customize Mono for your needs.
Mono compilation consists of two things

• Native code (C Code). The main program, virtual execution system, garbage collector etc.
• Managed code (C# Code). The Mono Library.

The first thing you will need to compile is the main program written in C. After you have done this, you can compile the Library (C#). Under normal circumstances you can do both in one make command. That only works when you are compiling on the device itself. In this tutorial I will use a cross toolchain to compile the main program for the Omega2. Mono does not support compiling managed code when cross compiling. For that you will need the Linux host on which you are cross compiling or a different Linux host to compile the managed code. After compilation is done you can just copy the C# library from the Linux host.

Cross compile distribution

Ubuntu Server 18.04 LTS

Step 0. Compiling the cross toolchain

I used the following cross toolchain for cross compiling. Build the cross toolchain explained there.
https://github.com/OnionIoT/source
Note: Make sure the toolchain's usr/lib has libraries like libz. Otherwise the mono build will fail while cross compiling.

Step 1. Compile Mono on the Linux host

I used the following working directory

• /home/project/

The first step is to compile Mono on the Linux host. Go to the working directory

cd /home/project/

Download and install all the necessary packages

sudo apt-get install subversion g++ zlib1g-dev build-essential git python libncurses5-dev gawk gettext unzip file libssl-dev wget libtool-bin autoconf cmake

Clone the mono source

git clone –recursive https://github.com/mono/mono

Change to mono directory

cd mono

Configure and make mono

./autogen.sh --prefix=/ && make

Create directory for exporting mono installation

mkdir ../BuildHost

Make install Mono

make install DESTDIR=/home/project/BuildHost

Now you have a full working Mono installation for your Linux host. We only need the library /home/project/BuildHost/lib/mono

Clean up source directory

make clean

Step 2. Cross Compile mono for Omega2

Then we will build the main program for the Omega2.
Create the cross toolchain environment. For that you will need the cross toolchain binaries from the Openwrt SDK directory. In my case /home/project/openwrt-sdk-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/

export CROSS=/home/project/openwrt-sdk-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64/staging_dir/toolchain-mipsel_24kc_gcc-7.3.0_musl/bin/mipsel-openwrt-linux
export CC=$CROSS-gcc
export CXX=$CROSS-g++
export CPP=$CROSS-cpp
export AR=$CROSS-ar
export AS=$CROSS-as

Ok so the cross toolchain environment is set up. At this point you will need to tweak the configure script so that it forces to use __thread as Thread Local Storage for Mono. The configure script automatically selects pthread instead of __thread. The reason for doing this is because the standard garbage collector for mono, sgen does not support pthread. Normally you can select boehm as the alternative garbage collector, but since boehm doesn’t support the musl library used for the Omega2 there is no other choice then to tweak the configure script. Also using the option --with-tls=__thread with the configure script doesn’t work. So tweaking it is. Fortunately __thread works just fine on the Omega2.

Tweak the configure script

sed -i -- 's/with_tls=pthread/with_tls=__thread/g' configure configure.ac 

What I could not figure out was this error when I was cross compiling Mono

In file included from mono-context.c:10:0:
mono-context.c: In function 'mono_sigctx_to_monoctx':
../../mono/utils/mono-sigcontext.h:480:73: error: request for member 'fp_r' in something not a structure or union
 # define UCONTEXT_FPREGS(ctx) (((ucontext_t *)(ctx))->uc_mcontext.fpregs.fp_r.fp_dregs)
                                                                         ^
mono-context.c:448:24: note: in expansion of macro 'UCONTEXT_FPREGS'
   mctx->sc_fpregs[i] = UCONTEXT_FPREGS (sigctx) [i];
                        ^
mono-context.c: In function 'mono_monoctx_to_sigctx':
../../mono/utils/mono-sigcontext.h:480:73: error: request for member 'fp_r' in something not a structure or union
 # define UCONTEXT_FPREGS(ctx) (((ucontext_t *)(ctx))->uc_mcontext.fpregs.fp_r.fp_dregs)
                                                                         ^
mono-context.c:460:3: note: in expansion of macro 'UCONTEXT_FPREGS'
   UCONTEXT_FPREGS (sigctx) [i] = mctx->sc_fpregs[i];
   ^

If you have suggestions on how to fix this, comment it please. I have been working on this error for quite some time. It has to do with the musl library used with the Mips architecture. It compiles normally when glibc library is used.

For now just remove the lines which gives the make error.

sed -i '/# define UCONTEXT_FPREGS(ctx)/c\' mono/utils/mono-sigcontext.h
sed -i '/UCONTEXT_FPREGS (sigctx)/c\' mono/utils/mono-context.c

Then you can configure Mono. Because boehm doesn’t work we can disable it. At this point you can select which features you want to use with Mono. I highly recommend to create a small footprint since the Omega2 does not have much space. You can of course expand the root partition with a USB flash drive or a MicroSD card and use a completer version of Mono. In this tutorial I will explain how to build a small footprint http://www.mono-project.com/docs/compiling-mono/small-footprint/

Configure mono with small footprint

./autogen.sh --prefix=/ --enable-minimal=aot,profiler,debug,logging --host=mipsel-openwrt-linux --enable-small-config=yes --with-shared_mono=no --with-static_mono=yes --disable-mcs-build --without-mcs-docs --with-moonlight=no --disable-boehm

Then make mono with optimization for size

make CFLAGS=-Os

Create directory for exporting Omega2 Mono installation

mkdir ../BuildOmega

Export Omega2 Mono installation

make install DESTDIR=/home/project/BuildOmega

Change to BuildOmega directory

cd ../BuildOmega

Remove include and share. We don’t need those

rm -r include share

Change to bin directory

cd bin

Remove all binaries except mono and mono-sgen

mv mono mono-sgen ../
rm *
mv ../mono ../mono-sgen .

Then strip the mono binary

$CROSS-strip mono-sgen

Change to lib directory

cd ../lib

Strip some libraries

$CROSS-strip lib*

Remove unnecessary files and directories

rm *.la *.a
rm -r pkgconfig mono

Now we can copy the .NET library to the Mono Omega build directory
If size doesn’t matter (chuckle), just merge the whole directory /home/project/BuildHost/lib/mono with /home/project/BuildOmega/lib/mono so that it has the complete mono library.
Or just choose which .NET features you want to use and copy those to /home/project/BuildOmega/lib/mono

I will explain how to create a very small library, almost equal to Debian’s mono-runtime

cd /home/project/BuildHost/lib/mono
mv 4.5 gac ../
rm -r *
mv ../4.5 ../gac .
cd 4.5
mv cert-sync.exe gacutil.exe I18N.dll I18N.West.dll Mono.Security.dll mscorlib.dll System.Configuration.dll System.dll System.Security.dll System.Xml.dll ../
rm -r *
mv ../cert-sync.exe ../gacutil.exe ../I18N.dll ../I18N.West.dll ../Mono.Security.dll ../mscorlib.dll ../System.Configuration.dll ../System.dll ../System.Security.dll ../System.Xml.dll .
cd ../gac
mv I18N I18N.West Mono.Security System System.Configuration System.Security System.Xml ../
rm -r *
mv ../I18N ../I18N.West ../Mono.Security ../System ../System.Configuration ../System.Security ../System.Xml .
rm I18N/4.0.0.0*/*.pdb I18N.West/4.0.0.0*/*.pdb Mono.Security/4.0.0.0*/*.pdb System/4.0.0.0*/*.pdb System.Configuration/4.0.0.0*/*.pdb System.Security/4.0.0.0*/*.pdb System.Xml/4.0.0.0*/*.pdb

Then copy the library to the BuildOmega directory

cp -R /home/project/BuildHost/lib/mono /home/project/BuildOmega/lib/mono

You can then copy the contents of /home/project/BuildOmega to the root of your Omega2(+).

This is how you compile Mono for the Omega2. If there are any questions, please ask them in the comments.

Timezone error

Some .NET apps requires /etc/localtime which is not available by standard. To fix this download and install zoneinfo. Before you do this you need to select your contintent. In this example zoneinfo-europe

opkg install http://downloads.lede-project.org/snapshots/packages/mipsel_24kc/packages/zoneinfo-europe_2017b-1_mipsel_24kc.ipk

Copy timezone to /etc/localtime (In this example London)

cp /usr/share/zoneinfo/Europe/London /etc/localtime

[paaablo80]

Thanks for sharing this info, very valuable!

Is there a chance to share also some more info regarding your C# projects that you run with this mono (.NET version, environment, compiling, library compatibility issues, etc)?
Would love to hear more on C# on Omega Onion!

I wonder is it achievable to build a .NET app using Visual Studio (.NET Core?) that could handle HTTP requests (Kestrel/WebListener or even HttpServer for non-core), manage Omega GPIO and I2C bus as well as managing other native OpenWRT processes outside the mono (e.g. madplay).

Cheers!

[Jackdaw]

So far I didn't have any problems running a few .NET applications on the Omega. Will look further on this subject. I also looked at .NET core but could not manage to compile it for the Omega. As far as I know and described in the git repository, Core supports only a few CPU architectures but not MIPS unfortunately.

[Jackdaw]

Updated thread with timezone fix.

[Martin Frederiksen]

Hi.

Thank you for the great work regarding mono on omega2.
I have tried skipping the whole "build from source" step, and go directly to "install from your ipkg file" - simply to see if I can get that part working before I start customizing my own, but it seems, that if i run "opkg install mono-runtime_4.3.0-1_mips_24kc.ipk" from a path where I have downloaded your "mono-runtime 4.3.0 30-04-2017", then it fails, saying the package is not for Mips... am I missing something?

Kind regards, and again, thank you for your work.

[Jackdaw]

@Martin-Frederiksen Yeah I also had this problem with the .ipk file. You can extract the ipk and then the data.tar.gz file and copy all the contents of data.tar.gz to "/". That precompiled mono package you mentioned is old. Will compile a newer version for you. ;). Maybe that one will work with opkg.

UPDATE: here you go:
http://51.255.195.37/mono-runtime_5.0-1_mipsel_24kc.ipk

Small note: I could not check which version this one is. I compiled the latest mono source and named it 5.0. You can check mono --version which version it really is.

So if opkg doesn't work then extract the files and copy those to root:

tar xf mono-runtime_5.0-1_mipsel_24kc.ipk
tar xf data.tar.gz
cp -R ./bin ./lib /

[Martin Frederiksen]

@Jackdaw Thanks man.. That is so awesome, and yes, I actually had to extract, as opkg did not work. For anyone following along though, I had to use tar with xzf parameters, to uncompress from zip.... Great work, and it is version 5.7.0 in this package.

Cheers

/M

[Jackdaw]

@Martin-Frederiksen Glad that it worked! Will have to take a look at the ipk file. Shame that opkg doesn't work.

[Rainer Müller]

Hi @Jackdaw!

I tried to compile mono myself, it seems there is an environment variable STAGING_DIR missing. But this did not completely solve my problems. Can you perhaps provide a working link with your mono version?

[Jackdaw]

@Rainer-Müller

You can ignore the STAGING_DIR. If you do all the things written in my instructions, then you have a fully working mono package :).

I see that the precompiled package is not there anymore. Will create one somewhere this weekend.

[Irawan Fardiawan]

Hi Jack can you reupload the ipk compiled for mono you build. Thanks a lot

[Jackdaw]

Wow, sorry I forgot to add a package. That took me a time to notice.

Do people still have problems compiling mono for the Omega? I will try again this weekend and update the tutorial. Please remind me again. :).

[Jackdaw]

I could not cross compile the recent version of mono. Will see if I will be more successfull at the end of this week.

[paaablo80]

This post is deleted!

[paaablo80]

@Jackdaw I stuck on setting crosscompiled environment. I mean - after all setup for crosscopile (I used also recent openwrt-sdk-18.06.1-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64.tar.xz) BuildMono/bin/mono is build for Ubuntu (not for Omega2). Both BuildHost/bin/mono and BuildMono/bin/mono works on Ubuntu but the latter not works on Omega2 (wrong ELF format). So I suspected that cross-tools are wrongly setup and in the second stage still Ubuntu host tools are used (in /Makefile commands CC = gcc)

I tried your solution with export CC=$CROSS-gcc
and also alising it manually before automake and make but it still build host-platform version

Any idea how to properly setup cross-toolsets before compiling for ramips-mt76x8 ?

[paaablo80]

I think I have ignored that autogen.sh ends with an error around checking shm_open:

checking for inotify_rm_watch with unsigned wd... no
checking for shm_open that works well enough with mmap... configure: error: in `/home/project/mono':
configure: error: cannot run test program while cross compiling
See `config.log' for more details

How to workaround it? Obviously it can not run crosscompiled test on build machine

[paaablo80]

...ok, I figure it out (more or less):
There's on open regressed issue on Mono project for this:
https://github.com/mono/mono/issues/9951

I had to switch back to 2018-04 branch on Mono to make it work.
Compiled with openwrt-sdk-18.06.1-ramips-mt76x8_gcc-7.3.0_musl.Linux-x86_64.tar.xz

Cheers!

[eimantas duda]

Is it possible to get new link for package download?

[Jackdaw]

Sorry for the long wait.

You can download the precompiled package here. Use at your own risk.
It contains the mono binary and the full .net library, so you need to extend your root partition with a microsd card.

http://207.180.192.51/mono-5.8.1.0-1_mipsel_24kc.ipk

Could someone check if the package will install with opkg this time? Thanks.

I will update this thread in a while. Some things has been changed.

[Danny Hacker]

@Jackdaw

Is there an updated valid link for mono-5.8.1.0-1_mipsel_24kc.ipk? I'm not able to get it to download.