I haven’t experimented with balancing the loss, it kinda worked well out of the box. Although pixel smoothing did help to get rid of some noisy pixels.
As an improvement, I was thinking of weighing the alpha channel with a Gaussian that would penalise opaque pixels at the border more than at the center.