I found this post to be somewhat disappointing. It doesn't really have much depth to it and the claims have little backing. It's not that I don't believe the claims, they have been made before with evidence to support them, by other people. It's that I don't really see how this adds anything to the discussion.
I am not presenting breakthrough material here. I wrote this because there is still a lot of confusion about concurrency and people looking for hot new solutions without understanding the basic tradeoffs.
You have not done much to clarify the confusion, though.
The tone of your blog post indicates that absolute truth and universal advice is to follow — and then it doesn't. What I got from your blog post is that one needs to consider various concurrency primitives and solutions (duh) and that you used Java locking, Java frameworks and Hadoop, and haven't found much use for Clojure's STM. Well, good for you, but I guess people expected more from a blog post titled "Concurrency, State, and Functional Programming".
There are two aspects worth mentioning:
1) It doesn't matter how fast you get the wrong answer, so your concurrency approach had better be correct. This is much easier to achieve with Clojure STM than with manual locking and mutable data structures.
2) Cost/benefit ratio: assuming each concurrency approach gives you some benefits (usually execution speed is what you're after), what is the cost? Cost meaning your effort to write the code, debug it, test it, maintain it, document it, cost for others to understand it later?
For what it's worth, I am getting very good mileage out of Clojure's STM. For scaling on SMP machines it gets me good results with minimal effort. I get correct code (#1) and the cost/benefit ratio is excellent (#2).
>. This is much easier to achieve with Clojure STM than with manual locking and mutable data structures.
For easy stuff yes. For hard stuff that is such a grandiose claim to make, if you could do it with any certainty you would certainly be published. Personally, I don't think it's any easier.
Agreed. The author supports going outside of Clojure for larger scale processing, e.g. Hadoop. This is sensible, but it's some leap to call concurrency a fad.
It's likely that Clojure will target some other runtime (in the future). Concurrency primitives in Clojure (var, ref, atom, agent, future, promise etc.) are practical, elegant, and focused tools. Why work around them?
I am not calling concurrency a fad, I am calling the rush to try hot new concurrency primitives without understanding these basic tradeoffs a fad.
Clojure's concurrency primitives are just wrappers around the same java libs I mention in the post.
For locking, it just matters which code you depend on based on your need. If you need to deal with it yourself, using Clojure's STM is good, since Rich has done the work to wrap the locking. But if you just need a data structure with synchronized access, then you don't even need Clojure's STM. Similar arguments hold for Agents vs Executors.
I am not saying Clojure's concurrency primitives are bad, I am saying we need to understand the basics, and Clojure's primitves are not always necessary or the right solution.
> I am not saying Clojure's concurrency primitives are bad, I am saying we need to understand the basics, and Clojure's primitves are not always necessary or the right solution.
Your post seemed very light on actually discussing this point, which I think would have been appreciated. There are always a lot of trade-offs to be made in writing concurrent code. The shallowness of your post left me wondering why you made some of your choices, or if you even thought particularly hard about them.
Not using transactions effectively forces anyone who wants to call into your code to also not use transactions, or they'll be bitten by the retry behavior.
This may not apply to your library (it seems like maybe it only calls code, rather than being called by it), but the approach you're suggesting can greatly reduce the utility of a library for other people if you're not careful.
