I just discovered a stylegan surprise. In stylegan2, the discriminator learning rate can't be set. G_lrate is used for both the generator *and* the discriminator's rate. Everyone's been fooling themselves. It's currently impossible to set the discriminator's learning rate.
Peter Baylies @pbaylies
30 Jan 2020
Replying to @5agado @Norod78 @gpt2ent @gwern @Veqtor
My tips are, try the two-time update rule - set the generator learning rate to half of the discriminator learning rate - try increasing the minibatch size - and, yes, if all else fails, also try lowering the learning rate.

Feb 19, 2020 · 6:54 AM UTC

4
4
2
29
To confirm this, you can grep the stylegan2 codebase for D_lrate and see that it's not used anywhere.
2
3
It was possible in stylegan1. Looks like stylegan2 broke the ability to set the discriminator's learning rate. Here's stylegan1 (left) vs stylegan2 (right). Notice that in stylegan2, G_lrate is incorrectly used as the learning rate for D_train_op:
3
1
5
Replying to @theshawwn
In all the Colab notebooks I used for training, both run-training and training-loop are either %writefile’ed after being edited or heavily edited in the fork (like @pbaylies’s WikiArt fork). Will certainly have a double check
3
Replying to @theshawwn
Good catch. Thought I was going insane the other day trying to tweak my learning rates.
1