[Solved, more or less] OpenWRT C++ CrossCompile uClibc setup

Alexander Pitzer

Hiho,
I want to compile a "helloWorld"-package with the openwrt standart setup found on: https://github.com/OnionIoT/OpenWRT-Packages/wiki/Setting-Up-the-Cross-Compile-Environment

The problem is, that with this setup a normal program links dynamically against: "ld-musl-mips-sf.so.1" but the onion has "ld-uClibc.so.0" installed. I know that a static linked program works, but in the end I don't want to get hello World to work

$ readelf -l helloWorld

Elf file type is EXEC (Executable file)
Entry point 0x400600
There are 9 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00400034 0x00400034 0x00120 0x00120 R E 0x4
  INTERP         0x000154 0x00400154 0x00400154 0x0001a 0x0001a R   0x1
      [Requesting program interpreter: /lib/ld-musl-mips-sf.so.1]
  REGINFO        0x000170 0x00400170 0x00400170 0x00018 0x00018 R   0x4
  LOAD           0x000000 0x00400000 0x00400000 0x008f0 0x008f0 R E 0x10000
  LOAD           0x000fdc 0x00410fdc 0x00410fdc 0x00050 0x00074 RW  0x10000
  DYNAMIC        0x000188 0x00400188 0x00400188 0x00110 0x00110 RWE 0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10
  GNU_RELRO      0x000fdc 0x00410fdc 0x00410fdc 0x00024 0x00024 R   0x1
  NULL           0x000000 0x00000000 0x00000000 0x00000 0x00000     0x4

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .interp 
   02     .reginfo 
   03     .interp .reginfo .dynamic .hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.plt .init .text .MIPS.stubs .fini .rodata .eh_frame .plt 
   04     .ctors .dtors .got.plt .rld_map .got .sdata .bss 
   05     .dynamic 
   06     
   07     .ctors .dtors .got.plt 
   08

After reading "https://community.onion.io/topic/9/how-to-install-gcc" I believe that this dynamic link is what caused the problem for Kit Bishop:

root@Omega:~# ls -l
-rwxrwxrwx    1 root     root          8994 Nov 18 12:26 testc
root@Omega:~# ./testc
-ash: ./testc: not found
root@Omega:~#

(Sry but I can't find a link to his post option :-()

I see that the tool chain uses the wrong stdLib, that path name of the tool chain looks like: "...toolchain-mips_34kc_gcc-5.2.0_musl-1.1.11/

I have the kernel settings on "Preferred standard C++ library (uClibc++) --->" but the other uClibc settings I can't activate because mips is set and I don't know where openwrt configures the tool chain std lib.

So the question is: How to configure the openwrt setup to compile programs for the existing onion os? How to use uClibc instead of musl?

Just for completness here are my helloWorld Makefile:

# build helloworld executable when user executes "make"

helloWorld: helloWorld.o
	@echo "\n\n----------------------\n\n"
	@echo "CXX: $(CXX)"
	@echo "CFLAGS: $(CFLAGS)"
	@echo "LDFLAGS: $(LDFLAGS)"
	@echo "LIB: $(LIB)"
	$(CXX) $(LDFLAGS) helloWorld.o -o helloWorld
helloWorld.o: helloWorld.c
	$(CXX) $(CXXFLAGS) -c helloWorld.c

all: helloWorld
clean:
	rm *.o helloWorld

and the openWRT project Makefile:

include $(TOPDIR)/rules.mk

PKG_NAME:=helloWorld
PKG_VERSION:=1
PKG_RELEASE:=1

# This specifies the directory where we're going to build the program.  
# The root build directory, $(BUILD_DIR), is by default the build_mipsel 
# directory in your OpenWrt SDK directory
PKG_BUILD_DIR := $(BUILD_DIR)/$(PKG_NAME)

include $(INCLUDE_DIR)/package.mk 

define Package/helloWorld
	SECTION:=alex
	CATEGORY:=alex
	TITLE:=HalloWorld
endef

define Package/helloWorld/description
	Hello World
endef

# Specify what needs to be done to prepare for building the package.
# In our case, we need to copy the source files to the build directory.
# This is NOT the default.  The default uses the PKG_SOURCE_URL and the
# PKG_SOURCE which is not defined here to download the source from the web.
# In order to just build a simple program that we have just written, it is
# much easier to do it this way.
define Build/Prepare
	mkdir -p $(PKG_BUILD_DIR)
	$(CP) /home/apit/projects/openwrt/helloWorld/* $(PKG_BUILD_DIR)/
endef

MAKE_OPTS:= \
    ARCH="$(LINUX_KARCH)" \
    CROSS_COMPILE="$(TARGET_CROSS)" \
    SUBDIRS="$(PKG_BUILD_DIR)"

define Package/helloWorld/Compile
	cd $(PKG_BUILD_DIR)

	#$(MAKE) -C $(PKG_BUILD_DIR) \
	#	LDFLAGS="$(EXTRA_LDFLAGS)" \
	#	CXXFLAGS="$(TARGET_CFLAGS) $(EXTRA_CPPFLAGS) -nostdinc++" \
	#	$(TARGET_CONFIGURE_OPTS) \
	#	CROSS="$(TARGET_CROSS)" \
	#	ARCH="$(ARCH)" \
	#	$(1);
	$(MAKE) -C $(PKG_BUILD_DIR) $(MAKE_OPTS)
	
endef

define Package/helloWorld/install
	$(INSTALL_DIR) $(1)/bin/
	$(INSTALL_BIN) $(PKG_BUILD_DIR)/helloWorld $(1)/bin/helloWorld
endef
$(eval $(call BuildPackage,helloWorld))

Of course alot of that lines are from openwrt hello world tutorials or from onion omega projects. I also remove some lines, so if it not compile its because I erased to much ... I tried to keep it clean here... sry.

PS: thanks alot for your help!

PSS: I found this statistic: http://www.etalabs.net/compare_libcs.html
Is it possible that compiling the os with musl is faster and smaller then with uClibc?

Johan Simons

Hi Alexander,

Below are in a few steps my way to success in building my cross platform apps:

I run a CentOS 6.5 linux box (32bit)
download the 32bits mips toolchain for Onion to /usr from this link https://s3-us-west-2.amazonaws.com/onion-downloads/openwrt/OpenWrt-Toolchain-ar71xx-generic_gcc-4.8-linaro_uClibc-0.9.33.2.Linux-i686.tar.bz2
unpack this toolchain using:
cd /usr
tar -xvf OpenWrt-Toolchain-ar71xx-generic_gcc-4.8-linaro_uClibc-0.9.33.2.Linux-i686.tar.bz2
move the extracted dir to a shorter name:
mv OpenWrt-Toolchain-ar71xx-generic_gcc-4.8-linaro_uClibc-0.9.33.2.Linux-i686 onion
in your app directory create Makefile.onion with following contents:

CC = /usr/onion/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-uclibc-g++
LD = /usr/onion/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-uclibc-ld
CFLAGS = -I /usr/onion/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/include
LDFLAGS = -L /usr/onion/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/lib

EXEC= hello.onion

THREADLIB = -lpthread
RTLIB = -lrt
DLLIB = -ldl

IFLAGS = -I..
LDFLAGS += ${THREADLIB} $(RTLIB) $(DLLIB)

all: clean $(EXEC)

OBJS=
hello.o

$(EXEC): $(OBJS)
$(CC) -o $(EXEC) $(OBJS) $(LDFLAGS)

clean:
rm -rf $(OBJS) $(EXEC)

.c.o:
$(CC) -c -o $@ $(DEBUG) $(CFLAGS) $(IFLAGS) $<

.cpp.o:
$(CC) -c $(CFLAGS) $(DEBUG) $< -o $@ $(IFLAGS)

hello.o: ../hello.cpp
<<<
6) make your hello app: make -f Makefile.onion

copy hello.onion to Onion and give it exec rights

Alexander Pitzer

Thank you for your suggestion,

I know that a lot of people just use the SDK to compile programs, but I want to get the openWRT package CrossCompile environment to work. I want to create packages that I can install on the onion (.ipk packages if I'm remember correctly). I don't know how to get the toolchain you suggested into the openWRT environment, so if you have an Idea how to do that, it would be perfect

I mean someone compiles their packages with the correct toolchain/setup and provide this packages to us, don't they?

Boken Lin

Hi @Alexander-Pitzer, In the make menuconfig system, go into Advanced configuration options (for developers) > Toolchain Options > C Library implementation. You just need to switch from musl to uClibc.

Alexander Pitzer

@Boken-Lin

upload-05844751-a882-4d38-af1e-fcc5199d0191

Yeah I tried this before, too. But there is no option for uClibc....
May it be possible to poste a .conf with the right kernel settings?

Because the dependency for uClibc contains : .... && !mips [=y] && ....
and mips is set by: .... || TARGET_ar71xx_generic [=y] && <choice> && TARGET_ar71xx [=y] || ....

upload-213bc76c-4737-4f13-abac-464d6630c7ad

Boken Lin

@Alexander-Pitzer that's quite strange. Where did you get the buildroot files?

Alexander Pitzer

@Boken-Lin

https://github.com/OnionIoT/OpenWRT-Packages/wiki/Setting-Up-the-Cross-Compile-Environment#step-2-get-openwrt-source

I believe I found the problem, in the current version of openWRT uClibc is available if broken packages are allowed. RIght now I compile the toolchain: toolchain-mips_34kc_gcc-5.2.0_uClibc-1.0.9
They switched to uClibc-ng-1.0.9

8e493a4b7c0749bf65cfd545ca18b384bc1d4f32    12/03/2015 03:47 PM
uClibc-ng: update to 1.0.9

Update to 1.0.9, switch to XZ tarball.

Signed-off-by: Waldemar Brodkorb <wbx@uclibc-ng.org>

git-svn-id: svn://svn.openwrt.org/openwrt/trunk@47715 3c298f89-4303-0410-b956-a3cf2f4a3e73

b70a36d1d969f5aa9ad081c9a395633793582a5a  11/02/2015 07:12 PM

uclibc: remove version 0.9.33

Latest uClibc-ng is now the only supported option

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

git-svn-id: svn://svn.openwrt.org/openwrt/trunk@47357 3c298f89-4303-0410-b956-a3cf2f4a3e73

Boken Lin

@Alexander-Pitzer Oh i see. Were you using a version of the buildroot that did not include uClibc?

Alexander Pitzer

Sry I updated my post... so yes uClibc was kicked and uClibc-ng is in but only as broken package

Boken Lin

@Alexander-Pitzer Bummer. I guess the only way to do it right now is to check out f14d574f3082577a4489f778d0c8cf7117d33cee commit, and come back to the new one once the broken package has been fixed...

Alexander Pitzer

@Boken-Lin I try to compile the new version and will play around with the toolchain a bit. If nothing helps I will go back. Thank's alot for your help. The Onion Omega is a great product and I really like to play around with it alot!

Boken Lin

@Alexander-Pitzer I'm glad you like it If there's anything else we can help you with just let us know!