Excellent and informative article--and a good bit of brand-building, I might say :-). One thing I'd love to see more writing about is prototyping and iterative development in these contexts--deep NNs are notoriously hard to get "right", and there seems to be a constant tension between model architecting, tuning hyperparameters, etc.--for example, you presumably don't want to have to wait a couple of weeks (and burn through thousands of dollars) seeing if one choice of hyperparameters works well for your chosen architecture.
Of course, some development practices, such as ensuring that your loss function works in a basic sense, are covered in many places. But I'd love to see more in-depth coverage of architecture development & development best practices. Does anyone know of any particularly good resources / discussions there?
I would like to second this. Thanks for linking this. As someone starting out in deep learning and noticing that a lot of things are still more art than science this seems great for avoiding some footguns!
Of course, some development practices, such as ensuring that your loss function works in a basic sense, are covered in many places. But I'd love to see more in-depth coverage of architecture development & development best practices. Does anyone know of any particularly good resources / discussions there?