Take a look at this thread on setting up CMake for Docker and Omega, maybe it helps.

nosys.specs is used in a cross compiling environment where the host is not available such as when you are compiling on linux to deploy on OpenWrt. When openmodbus is being linked it seems it cannot find the required library stubs. You need to look to see what it is linking with, I don't use CMake nor OpenModbus so I cannot check.