Okay, so @rsx comment sort of pushed me into the right direction. It seems that the upgrade from 2.10 to 2.11 (or rather: the wrong configuration of 2.10) was the issue here, which I didn’t notice until I tried to use the API on one of the agents.
Simply moving the agent configuration to zones.d/* wasn’t enough, because I still had global configs in conf.d. I had to move the appropriate configuration items to zones.d/global-templates as well.
I assume that the other agents didn’t fail the same way as the one I moved around for testing and subsequently reinstalled because they still had proper configurations sitting in /var/lib/icinga2/api/zones/ locally.
Anyway, it works again and makes sense now.
Thanks to @nhilverling for your patience as well!