Ah, gotcha. Definitely a tough situation. Do you know if other coworkers are uncomfortable with it? If you could get together with a number of them to push back, at least it wouldn't look like you alone are a troublemaker.

For a while we had a policy that high risk personnel weren't going to be called back into the office unless something major happened (our job can't be performed remotely), but it looks like that policy has been rescinded at high levels of our org. It's still up to individual managers to make the schedules, though, so I think unofficially many are not putting them on the schedule. Still, it's very frustrating.