Thanks for the bullets. This is a good list but some of the points like "Easily scalable" and "DB support" are not cheaply available in any runtime and require careful attention to detail as well as domain-specific thinking. IMHO, the JVM already does a lot of heavy-lifting in this regard.
It comes down to this: are those developing the language thinking about how to add features just to try to hone and hold on to the enterprise developers that still use it, or are they thinking about what would make it more fun, productive, and practical?
Java's going to be around a long time. Those that stick with it will be fine. COBOL programmers made a lot of money in 1999, and some people still use Fortran.
But, Java's original big mantra was "write once, run anywhere." Such idealism then. Cool things have been done in the past few years, but can't we do more?
I work for Pivotal so I'm biased. But I'd take a long look at Sprint Boot in terms of "fun , productive , practical" Java: https://projects.spring.io/spring-boot/
In particular:
- curated and tested open source library dependencies to the point that you can generate a single JAR for anything you might want to build on the server: http://start.spring.io/
- Annotations and APIs to make REST service development a breeze
- Cloud connectors to make it easy to run your app on Heroku, Kubernetes, Mesos or Cloud Foundry, or to leverage NetflixOSS components, or to build stream processing applications (Spring cloud stream / data flow)
It is being downloaded and used at a very high rate (a couple million a month).
Spring Boot is awesome, thank you guys for your massive contribution. All of my co-workers are .Net guys and keep trying to say that Java is dead now that .Net Core is out, but I still haven't found a server-side solution better than Spring yet :)
Cool, but much less useful than the full suite of Spring projects (which is why I love Spring in the first place, it's basically batteries included but everything remains modular so I can pick and choose what I want). .Net Core is a great stepping stone, but between deployment headaches (I can just run yum install java-1.8.0-openjdk to get a JVM and run a spring-boot app, .Net Core forces me to bundle the runtime right now) and the lack of a cohesive integrated suite of components keeps my heart with Spring (even if I'm forced to use .Net at work more often than not).
This looks very good. At the moment, for the jvm, there are not a lot of features above simply building services using netty and the protobufdecoder/encoder. Similarly, http2 goodness is also already available in netty.
This translates exceptionally well to fields other than basketball, as well. I wish this "magically gifted engineer just needing to work 3hrs a day making e'thing look so easy" profile goes away.
One big challenge is the expense that is otherwise avoided via usage of connection pooling. Without the hack to keep the backing containers alive, tearing down and restarting them is expensive and wipes out all hot path optimizations made by JIT compilers besides having to reestablish connections and pay the handshake prices. Apart from the simplest of use-cases at this time, imho this is not an efficient model if sustained performance and low latency are of interest.
I've been wondering this same thing. We use gunicorn to handle our web requests and it pre-loads our code so it's ready at the drop of a hat to serve a request. It takes a few seconds to load up our site, so wouldn't Lambda have to pay that cost with every web request?
I've been going under the assumption that it's just not the right use case for Lambda, though I'm hoping I'm wrong.
If you have any kind of real load, there skills always be instances running. If you have less load than that, you are not a case worth optimizing for, because you pay less than a few hundred per month.
Also, small instances are cheap to keep around, and as you pay per request, it's in the providers best interest to make that efficient.
Does anyone have real prod scale numbers - even order of magnitude concurrency would be helpful. We're obviously very wary of trying out new and young data stores.
Does someone have any benchmarks or comparisons with Zk? We have run it for many yrs without a problem and are very happy with it. Would be interested in hearing from anyone who switched from zk->etcd for distributed locking, presence, leader election type idioms and ran it in prod for a few months/yrs and their takeaways.
etcd3 has similar performance compared to ZK for small scale. For large dataset, etcd3 does better since it does incremental snapshot/ smaller memory footprint, when ZK does full snapshot that takes a lot of resources. For watches, etcd3 does streaming + TCP multiplexing to save memory.
I too have been using ZK for many years now, and it's pretty great.
Etcd can provide faster election notifications but it comes at the cost of etcd still being pretty new, so be prepared to get cut by the bleeding edge :)
Apples all the way. Etcd is pretty much a clone of ZooKeeper in Go; they both support hiearchical keys, atomic autoincrement, watches, though Etcd uses the Raft consensus algorithm, whereas ZooKeeper uses its own homegrown algorithm, and there are other minor differences. Both are intended for configuration management and coordination. (Of course, both are ultimately clones of Google's internal tool, Chubby.)
As I understand it, both Paxos and Zab will result in similar performance characteristics, since writes need, by design, to be coordinated with peers and serialized in a strict manner. In this sense, Etcd and ZK are very much alike, irrespective of how they are implemented internally. I wouldn't be surprised if Etcd was found to be faster and more scalable than ZK, however.
It really depends on how you view the problem. Yes, the latency of agreeing a proposal is similar, which is limited by physical (network latency + disk io). However, there are ways to put more stuff into one proposal (batching) and submitting proposals continuously (pipelining). These optimizations highly depend on the implementations.
Your point is well-taken. This qn comes up often enough for similar use cases and hence I asked if someone has data to support an arg. Considering the criticality of Zk in the larger stack and our success and operational expertise with it, it has to take significant convincing to switch to something like etcd.
The following two paragraphs from the etcd project site seem to hint that they're trying to target overlapping use cases:
"Your applications can read and write data into etcd. A simple use-case is to store database connection details or feature flags in etcd as key value pairs. These values can be watched, allowing your app to reconfigure itself when they change.
Advanced uses take advantage of the consistency guarantees to implement database leader elections or do distributed locking across a cluster of workers."
I worked with Eclipse for a few years and found it was a constant source of stress, refusing to stay synchronized with maven, crashing, having to restart it, etc.
I was hating maven, until I started using IntelliJ. I don't remember ever Eclipse going to zero error count on a maven project whereas the project would just compile if maven was used outside of Eclipse.
IntelliJ just works. FTW it has Eclipse key-mappings as well.
IntelliJ lets you do a lot more from the keyboard. It's suggestions for code completion are very useful. It's amazing to see the code almost writing itself, several times faster than you can type!
Go-lang is good but not as mature. Other than that, there's C++ and Erlang/Elixir. Not a whole lot of mature choices out there. For most of the critical production grade systems that need to be secure, have to scale and perform predictably, you don't want to blaze the trail.
I think your last point hits the nail on the head. There's a large amount of risk (especially at Twitter's size), and nobody wants to be saddled with an enormous and embarrassing failure. In addition, you need many hands to build the system, and finding developers for <favorite lang> that ALSO have relevant experience could be difficult. You could make arguments that developers in <favorite lang> are 10x as productive as your typical Java dev, but few people are going to take that chance. I'd say even Scala is still considered bleeding-edge at a lot of companies.
Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.
General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on.
Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.
Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.
As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.
There are (IMHO, superior) alternatives like Etcd and Consul that could be used.
Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.
