I can attest that these guys are known to post bogus benchmark numbers. The claim on their home page ("4x faster than Protocol Buffers in this test!") is incredibly misleading; their benchmark is set up in such a way that Protocol Buffers are copying the strings but Message Pack is just referencing them. It's true that the open-source release of Protocol Buffers doesn't have an easy way of referencing vs. copying, but it is still highly misleading not to mention that you're basically benchmarking memcpy().
One of these days I'll re-run their benchmark with my protobuf parser which can likewise just reference strings. I am pretty sure I will trounce their numbers.
I pointed out the Protocol Buffers comparison to a friend on IRC. He came back with:
"They rolled their own "high-level" serializer and deserializer for PB, built on top the lower-level stuff the documentation advises you not to use. Using the recommended interfaces, PB is faster than in their test. It is still slower than msgpack. Not sure why they'd make their test look cooked when they win in a fair test anyway. Further examination shows that the test is _mainly_ a test of std::string, for protobuf. std::string is required to use contiguous storage, whilst msgpack's test uses a msgpack-specific rope type."
I can attest that these guys are known to post bogus benchmark numbers. The claim on their home page ("4x faster than Protocol Buffers in this test!") is incredibly misleading; their benchmark is set up in such a way that Protocol Buffers are copying the strings but Message Pack is just referencing them. It's true that the open-source release of Protocol Buffers doesn't have an easy way of referencing vs. copying, but it is still highly misleading not to mention that you're basically benchmarking memcpy().
One of these days I'll re-run their benchmark with my protobuf parser which can likewise just reference strings. I am pretty sure I will trounce their numbers.