Exactly. What version of Photoshop dear HN reader was your first? Think about all the features that have been added since then and how hard it must be to maintain even a modicum of backwards compatibility.
Another thing to consider: a file format as popular as PSD probably had a great deal of interest from competitors. Sometimes file formats get obfuscated simply for the purpose of breaking the other guys' import filters.
I've seen this behavior in the file format of an extremely popular application. IIRC, in one section, there were magic bytes that told you the alignment of other blocks that told you how to look up the version of the algorithms that you were supposed to use to unpack the binary data blobs that held resource data. By jiggling around the various forms of indirection, it was possible to make the file formats nigh indecipherable from version to version, while still maintaining backwards compatibility.
Naturally, the parsing code was a total nightmare.