I think the most important thing about the class vectors is their norm. The model was trained on one-hots, which are 2-norm 1. So I just used a Gaussian prior for them. It doesn't seem to matter that much that the model hadn't seen negative classes.

To keep my parameters well-scaled I made their prior distribution N(0, I) and sampled their initial values from it. Then I multiplied them by 1 / sqrt(1000) - 1000 classes in ImageNet - before feeding them to the model, so its soft class mixes would have a 2-norm around 1.

This was successful at de-dogging the model! At first I had used @advadnoun's initial distribution for the prior and sigmoided them before feeding to the model, following Big Sleep. But that resulted in the model not being able to get away from dogs!