“Systems That Defy Detailed Understanding § Deep Reinforcement Learning”, 2020-02-22 ():
…Deep Learning: A few years ago, during a 3-month sabbatical, I spent some time playing with deep learning toolkits and algorithms. I was particularly interested in reinforcement learning, so after reading a bunch of papers and blog posts, I wrote a toy implementation of an actor-critic system to play pong on an emulated Atari, a pretty well-known task for deep reinforcement learning.
My system would make progress and I could watch its progress improving slowly … for a little while, and then it would diverge, with all of my parameters going to infinity and everything blowing up.
I spent literal months (on and off) trying to debug that ~200-line program. My experience with systems engineering lead me to go deeper into trying to understand its behavior, adding more and more metrics and summaries to my TensorBoard setup, trying to comb over parameters and gradients and potentials step by step, hoping that maybe I could find the first moment where things started going wrong, and thereby somehow deduce where my problem was.
Eventually—somehow—I found the bug, mostly by accident: I had an off-by-one error when feeding frames into the training step. Because deep neural networks are so d—n good at learning, they were still able to extract signal from the misaligned frames and start learning; but eventually the bad data won out and the system went off the rails. Fixing that data error resolved the problem and the network continued learning much longer and more effectively.
This experience taught me some really important lessons about deep learning. While working with deep learning “looks” like other programming in a lot of ways—certainly you’re writing and running programs—it’s also deeply different in a lot of important and practical ways. For one, actually no one really understands why these deep learning systems work. There are certainly many people with much better understanding and intuition than I have, but ultimately trying to understand these systems’ behavior too closely, especially as a novice to the field, is deeply the wrong way to approach it.
[Karpathy’s law: “neural nets want to work.”]
What works instead: The kind of approach I settled on instead—which some friends who work professionally in deep learning echoed—is a far more incrementalist, empirical approach. Start with the simplest possible version of a system, demonstrate that it works as a baseline, and iterate in small steps from there. Make small changes, verify that it still works, and repeats. If it ever stops working, you know it was the most-recent change, and you can try to figure out what went wrong. If you’re reimplementing a paper or published algorithm, try to find their code and first make sure you are matching their behavior as precisely as possible, and only then try to modify from there.
Oh, and also double & triple-check your data and your data infrastructure! This part of a system is “just plain-old code”, and your usual engineering strategies work, so double down on it. Write lots of assertions and checks that you’re feeding the right data in to your model and sanity-check it. Once you hit the neural networks errors become incredibly hard to trace, so try to enforce as much rigor as you can before then.