I'm surprised it doesn't use Character Aware Neural Language Models (CNN -> LSTM RNN) but instead a layered RNN. Interesting!