I was thinking the same thing: I would use HTML as an intermediate, targeting PDF through weasyprint.
In fact I quite often go .md -> .html with pandoc, but write the .md in such a way that, when translated, it is the kind of html that weasyprint will be able to turn into the PDF that I want.
Might have to use a headless chromium wrapper (maybe pandoc has this anyway) to then get to pdf but that may not be too bad