ZeroMQ is a library, it is embedded in the application code and can support a peer-to-peer messaging topology. It also has support for Broker patterns but you have to develop the Broker application yourself; you could build a solution such as this one using it, but the zmq library itself really handles lower-level concerns with a high degree of flexibility but no out-of-the-box functionality. zmq can be thought of as something as simple as an improved socket library but with support for async, storing & high-water marks when there are no attached consumers, various distribution strategies, queues and lots more.

Right, I guess my question more specifically is: ZeroMQ allows you to built applications like this in (seemingly) just a few lines of code. Did you guys consider building it on top of ZeroMQ, and if so, what made your decide to not use it? Nothing against what you've built btw, it looks great, but am just curious

The truth is we wanted to use ZeroMQ initially. It would have been really awesome to have all that flexibility on the client side.

The ZeroMQ documentation is fantastic as well. It inspired and helped shape some of the design choices we made.

The further along we got from design to implementation it became obvious that it would be important to "own" the socket. Generally speaking, this is exactly what ZeroMQ prevents you from doing (and rightly so, it aims to abstract all of that away).

The choice to use Go had an impact here as well. Language features like channels and the breadth of the standard library made it really easy to translate our NSQ design into working code, offsetting the benefit of ZeroMQ's abstractions.

Makes a lot of sense, thanks for sharing!

So - this has nothing to do with NSQ - just my $0.02. Maybe zeromq has changed a lot since I looked at it quite a while back. The trouble we constantly ran into was zeromq is really just a fancy socket library - this means that you have to program all the logic yourself if you want certain things to happen if messages cannot be delivered (e.g. log the fact at a minimum), or get queued abnormally long, or get acks from the other side etc. Which is fine - you have to do all this yourself with plain sockets unless you just need a fire & forget system(albeit with the smarts of retrying and failover and some convenience of pub-sub). However we ran into problems where we could not get the info out of zmq that we wanted, such as the fact a peer had failed, the current rtt, and other things - leaving the gain we'd get from zmq at practically nothing.