The fallacy fallacy is when you dismiss a conclusion based on a faulty argument. Him not addressing the argument isn't that. Him dismissing the conclusion of the argument is that.