Ideal solution—Is it possible to accomplish the omega2-ctrl gpiomux set spi_cs1 gpio command in a C program?
Yes, if you want to learn how, have a look at the omega2-ctrl sources on github, in particular gpiomux.c and gpiomux.h
What it does is setting and clearing bits in the GPIO1_MODE and GPIO2_MODE registers of the MT7688 SoC (see MT7688 Datasheet page 61ff).
These registers are at 0x10000060 and 0x10000064 in the physical address space - so to access these from user space (any regular program runs in user space), omega2-ctrl needs to map that part of physical addresses into user space.
The gpiomux_mmap_open function in gpiomux.c maps 0x10000000..0x100003FF into the gpio_mmap_reg pointer, using the linux mmap function. After that, SoC registers can be accessed via that pointer (i.e. gpio_mmap_reg+0x60 points to GPIO1_MODE).
Would a system() call work (something like system(omega2-ctrl gpiomux set spi_cs1 gpio))? Is that the best way to accomplish it?
Yes, using system works too, and is much less code. So that might be the best way in your case
However, I mentioned all of the above because via mmap you could access all SoC registers, and do many more things (superpower, be careful!) not exposed by omega2-ctrl.
Another reason to directly access SoC registers and bypass all Linux tools would be speed - system() is orders of magnitude slower (several milliseconds) because it involves forking a shell which in turn forks omega2-ctrl. That's fine for a one-time setup operation, but sometimes one needs nanosecond order speed, which is what mmap-ed direct access to SoC registers can deliver.