distillation is going fine now after i stopped hand tweaking values and literally asked chatgpt to suggest values because the ones present in most papers were doing very poorly
zephyr distill into my 44m textbooks model for a full epoch on textbooks as a continued pretraining, the textbook model only did 1 epoch and i raised dropout to 30% for this
![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="73" height="73"><rect fill-opacity="0"/></svg>)
