Are you sure about the padding? On page 1746, bottom right it says "padded as necessary". And intuitively it makes sense that all your inputs need to be the same size for a CNN.