When I read this, all I can think about is the enormous amount of man-hours people have put in to make Python the go-to for Julias target market.
As the author says, the biggest drawback is libraries, and that is (1 of) the big attraction(s) towards languages.
Nobody wants to have to switch between C, Python, Assembly, etc. to make their code faster, but even fewer people want to write bindings to underlying C code. This also goes exactly against what the author doesn't want to do, which is to yak-shave around the language to get things working.
It's extremely common to prototype in a language like python or R to be able to bang out a draft idea before diving into making it fast. If I'm not sure how awesome something is going to be, there's no sense in wasting weeks getting a C implementation working. And prototyping in a higher level language often reveals a lot of the architectural complexity you'l have to deal with in a lower level implementation. So I think this guy's use-case is more common than you think.
Python addresses a good deal of the problem with Cython, which lets you declare variables and compile to get ~90% of the speed gain you would get from a pure C implementation. And then you can just link to a C implementation if you still find you need to build one...
>It's extremely common to prototype in a language like python or R to be able to bang out a draft idea before diving into making it fast.
I think what's far more common is prototyping something in Python or R, and then realizing that 99% of its CPU time is already spent in wrapped C code. And then you call it a day and move on to other things :)
This is why Julia hasn't taken off nearly as quickly as I initially expected - Python/R/etc are highly optimized for their common use-cases.
> I think what's far more common is prototyping something in Python or R, and then realizing that 99% of its CPU time is already spent in wrapped C code. And then you call it a day and move on to other things.
Someone has to write those libraries.
> This is why Julia hasn't taken off nearly as quickly as I initially expected
Not sure what you were expecting, but consider:
- the most conservative userbase estimate I would believe is 20k users (based on mailing list subscriptions, website stats, and download numbers).
Yes, but the discussion is about the "extremely common" case of prototyping, which definitely should not require library building, and usually should not require veering much from established libraries.
>Not sure what you were expecting
The data community can coalesce around a tool extremely quickly, in the matter of a year or two. Spark is about the same age as Julia, and has a thriving ecosystem around it. In 2004 R was a fairly esoteric analysis tool, but 2008-2010 it was the de facto data science language. Python made similar advancements in just a few years in the data science community.
Julia? I don't know a single person who uses it day-to-day, but I know a lot of people who tried very hard (myself included). The critical mass simply is not there: people aren't building packages because the users don't exist, and they don't exist because the packages don't exist. Your chart shows a linear growth in packages, which implies a constant amount of development work. This means it's not a growing language. This is what a growing language looks like: http://blog.revolutionanalytics.com/2010/01/r-package-growth...
Numba accelerates Python loopy code using an LLVM backend in the same spirit as Julia, so there's not much reason to look at Julia just for speed reasons
You can use python libraries in Julia. And it is really good at binding to C and Fortran.
Python is old and stuck with some serious design flaws which can not easily be fixed. The question is, can Python get fixed in shorter time than it takes to give Julia a competitive set of libraries.
I would love to know what fundamental design flaws are there for Python in your opinion...GIL? No problem with Cython. Unicode? It is ugly but it could be done. Other stuff like static type system, is about language itself, and I don't think it needs to be addressed as flaw to be fixed or rather just being different preference.
For me, there is not something I can think of being such a deal breaker for me to switch to a another language. More likely it might not be ideal but could addressed by the ecosystem of Python itself.
The question is not "can these problems be more or less solved" with Cython or Numpy, but: we shouldn't have these problems in the first place. We haven't talked about deployement yet, which is another wart of Python. I guess it can also be kind of solved with another lib/tooling yet-to-come, but how many of these things are we going to pile, whereas a newer language could get it right, or at least, better, from the word go?
But I don't think Julia is going to so well-rounded in face of those problems. Why deployment is hard?
Because Python has so many C/Fortran external dependencies.
Is Julia going to change it? By itself maybe, but currently it still borrows from Python in order to enrich its ecosystem, so basically adding even more layers. That for me, speaks to the volume that language is not really the pain here, library is.
Real-world problems are ugly deep down to its core and the requirements and constraints are every changing. Who thought GPU programming will be such a big deal just 5 years ago, if not DNN revolution completely turning thing around? And I don't think Julia's improvement is really prepared for this. Even more, if Tensorflow style data flow programming is better accepted, then probably any language could be used to just draw some the computation graph and the heavy lifting will be done by the framework anyway. It is going to be harder to persuade programmers to adapt a language just for the perks that is only peripheral.
Julia's GPU capabilities are _far_ better than python's. There are near complete, free wrappers for all the CUDA libraries. The same is not true for python (e.g. CUSPARSE).
This is a consequence of how easy it is to interface with languages like C in Julia. I.e. whenever a new CUDA library or change happens, it will take the Julia devs less effort to update their wrappers.
Add to this Julia's HPC support and it is by far the best offering if you want to do REPL style programming on a GPU cluster.
Wow! Thanks for the CUSPARSE.jl shoutout (I'm the maintainer of that package)!
I must take issue with "near complete" - CUSOLVER.jl, my other CUDA wrapper, is missing a lot of the RF (refactorization) functionality and doesn't really have a pretty high level API yet. The doc and testing situation is also pretty bad. Everyone else's packages in JuliaGPU (the overarching GPU org on GitHub) are in a much better state.
You are right, though, that writing CUDA bindings in Julia is very easy - so easy that I can do it! It's also thanks to packages like Clang.jl, which make it easy for us to automate the procedure of wrapping the low-level interfaces.
Finally I must add the caveat that although I should test CUSPARSE.jl and CUSOLVER.jl with MPI (and GPUDirect) I've been extremely busy recently and not able to do so. If anyone wants to help out in this regard I would be very appreciative!
This is a good argument. But that is not a market where majority Python programmers are.
Once the wrappers are done, for most developers, it becomes the matter to layering those build blocks building their own functionality. It is not a strong enough argument that people should use Julia because it is a better language to write wrapper. Let alone that Python wrappers for data science stuff is already like a de-facto builtin.
Not saying Python is perfect, but IMHO, Julia is not hitting the right spot in order to really stand up against Python.
You have to some extent switched to a different language when you're using Cython. Do you teach Cython to newcomers when you're introducing them to Python? If no, why not? Performance and parallelism shouldn't require hurdles.
I love Python (despite the community). Sorry to be that guy but Python is always the second best and for me that usually means I end up using Python less and less. There isn't one area that Python is the best tool and sadly that is why it just isn't taking over the programming world.
As the author says, the biggest drawback is libraries, and that is (1 of) the big attraction(s) towards languages.
Nobody wants to have to switch between C, Python, Assembly, etc. to make their code faster, but even fewer people want to write bindings to underlying C code. This also goes exactly against what the author doesn't want to do, which is to yak-shave around the language to get things working.