@György-Farkas looks like I only thought about automating that setting and then promptly forgot it ever existed...
That was it, thank you!
What still bugs me is that there is no mention of the LTE board in the documentation whatsoever. A getting started guide is fine for the first setup and that's what I used, but after that I'd like to be able to find answers in the documentation. Or at least a link to the getting started guide, telling me thats where I need to look.