This entire article has dog in burning house vibes, but I can’t tell if it was intentional or not. I’ll give them credit for being a good sport about it at any rate.
Homebrew is an interesting beast. I was debugging something architecture-related recently during one of my first forays into rust land and quickly learned that my brew-installed rust wasn’t going to cut it because I needed rustup. Meanwhile, I don’t think I’ve used brew-installed node since at least v12 because I use nvm to support and debug multiple projects with different node versions. Is this kind of thing just becoming par for the course?
For the pain with the Django project, I guess I’m also willing to cut Rye slack because the underlying goal seems admiral and sadly the problem the author encountered seems common in Python land in general.
Yeah I think we’ve all independently concluded that you use brew for everything except those things complicated enough that you need to switch between versions.
I always want the latest version of git or ripgrep, no questions asked. Brew does that well. But for Rust, Node and others where you’re switching between versions, I prefer rustup, nvm etc.
Go is an exception though, I find I never want to go to a previous version of Go so brew is fine.
It seems most package managers were built with the assumption that you will stick to a single, officially sanctioned version of the language runtime and core libraries.
Which is a reasonable assumption to make, for example, if the language in question is also used by system utilities in a traditional Linux distribution. You really don't want yum invoking the wrong version of python.
For newer languages that change fast, though, we need something different. I don't think "every project brings in its own language runtime and the kitchen sink" is a sustainable situation, either. The dependency graph is crazy enough already.
> It seems most package managers were built with the assumption that you will stick to a single, officially sanctioned version of the language runtime and core libraries.
The problem here is that it's impossible to build a coherent distro if you allow arbitrary mixing of versions, without parallel installability. I don't think it can really be solved in the general case.
The first step is to separate system tools from userland tools. For example if system tools depend on python then they should be using their own isolate python and not the one that gets run when a user types 'python' on the command line. This way you can upgrade /usr/bin/python to 3.12 while /sbin/python (or whatever) can be locked at 3.8 to make sure important OS tools don't break. /sbin/python is not in the path of any user and only gets upgraded as part os a major OS upgrade.
Or you go the functional package management route and strip out these global paths entirely. nix and guix do this. Every package that needs python is built against a specific version of python (where a "version" is defined by its source code, compiler flags, toolchain versions, bootstrap toolchain versions, etc.; basically anything that can make the build result behave differently) and will only ever use that. Different packages can use different python versions. A system configuration can also choose to make a specific version available in the PATH, but it doesn't have to. A user environment can do the same, as can a project specific environment. All in one package management system.
> Which is a reasonable assumption to make, for example, if the language in question is also used by system utilities in a traditional Linux distribution. You really don't want yum invoking the wrong version of python.
Yep, this is the core reason: there needs to be a "system" distribution of Python, Ruby, etc. for packages written in those languages, and that distribution fundamentally needs to prioritize the interests of the package manager over "external" uses. This is particularly painful in the Python world, since using a "system" Python distribution for local development can break the system itself in all kinds of fun and exciting ways (usually beginning with a well-intentioned `sudo pip install ...`).
Homebrew suffers from this like other system package managers do, with the added pain that many macOS users expect Homebrew to do everything (whereas Linux users typically understand that they need `pyenv` or similar). PEP 668[1] helps a bit here by clarifying the responsibilities/expected behavior of system Python distributions, but at the cost of heartburn for some user scenarios (i.e. ones that depend on `pip install -U` and similar).
Sure, Nix is one of the few package management systems that makes a credible attempt at being good for all three major package management roles (the system; user applications; project-specific packages)
pip and venv are officially sanctioned. (venv and "ensure-pip" are included in python-3.x) That's all you really need (after you have the desired version of python).
If you use pip and requirements.txt, the dependency resolution is not great, it just installs things in order.
There isn't a lock file. How do you deal with dev dependencies? How do you work on a lib and its calling code without pushing it to pypa? Working with c deps?
All this stuff feels harder to do then it should be.
I have never used anything other than pip and venv. Maybe I'm missing something, but I never felt like I'm missing anything. And Python package management is so easy and simple if you stick to that (well, except version conflicts, but that's always a pain somehow).
It absolutely would. But there are many many things that would make the Python world better that the Python developers seem generally oblivious to so I wouldn't hold your breath.
Probably a little early to officially bless Rye though anyway.
> This entire article has dog in burning house vibes, but I can’t tell if it was intentional or not. I’ll give them credit for being a good sport about it at any rate.
Yeah, I appreciate the honesty. I think it's just a straightforward "this is good enough for me, but I'm not going to hide the glaring red flags that will stop anybody from using this at work" approach, which I think is cool. Some people feel compelled to pretend that the choices they make for a weekend hobby project are exactly the same choices they would make with other people's livelihood on the line, and it really doesn't need to be the same standard.
It’s worth calling out that it’s still early days for Rye. The ownership recently transitioned from Armin Ronacher to the team that develops ruff (https://astral.sh). No doubt limitations exist today, but it’s going to look a lot more like cargo as they put out more releases.
I'm curious what other people's experiences are like with Rye? I've been using it for data projects (a little bit of web stuff with fast api) and haven't hit a single hiccup at any point - it's been an absolute dream and I usually forget it's even there.
There's a lot of compiled c code in things like numpy etc but it's mostly precompiled in wheel files so I guess I've dodged the authors issue (unless I've misunderstood something?)
Am I just lucky that the data landscape is different - would the author's issue happen with more or less any Django project?
I've had some issues with compiling extension that didn't have a wheel(I was using an older python version needed to support older library versions) and being required to install clang and then some hard to debug header missing issues. But after reinstalling clang versions several times and removing rye caches manually it worked. Changing CC=gcc did try to use gcc which has other issues when compiling. I think a lot of this has to do with fundamental issues with compiled extensions in Python that happens on any package manager when you don't have wheels.
This is why I believe the package management tool should be separate from the interpreter management tool. Downloading a random third-party build can be valuable for some, but most people have a good enough interpreter on their machines.
This would also largely invalidate the need to write the package manager in Rust. A Rust program managing Python packages adds an unreasonable barrier to entry and makes Python seem like a toy language if it can't manage its own packages.
Rye is hamstrung by Python’s historic lack of desire to create portable Python distributions on platforms other than Windows. If it wouldn’t be for that I think the experience would already be much better.
Rust, Go and node (particularly in combination with Volta) show that combining package management and interpreter management into a close coupling can create a great developer experience.
For me (Rye author here) I wanted to see how close to that I can go. Unfortunately alone it’s quite a lot of work.
Thank you for your hard work! Don't be let down by the negativity this author displays. Probably the result of his youth. Your work is still useful and even though imperfect, very much appreciated.
Combining package and interpreter management can be a good developer experience for some, but it also means some people can’t or won’t use the tool, because they prefer their OS’s Python, or are forced to use a specific Python.
Decoupling the two while designing the interpreter manager CLI to forward some actions or parts of them to the package manager would be the best of both worlds. Users who can/want your interpreters use the merged CLI, others can still use the package manager separately with their favourite Python.
> Combining package and interpreter management can be a good developer experience for some, but it also means some people can’t or won’t use the tool, because they prefer their OS’s Python, or are forced to use a specific Python.
This doesn't give any reason to separate the package management and toolchain management. They very obviously should be managed by the same tool, and virtually all modern languages do it like that.
The only reason it is awkward in Python is because of Python nonsense, not because it's a bad idea.
> makes Python seem like a toy language if it can't manage its own packages.
Well, quite. But plenty of people have to use Python so I don't see why we shouldn't make good tooling for it in better languages. The same thing is happening for JavaScript.
I recently switched to Rye from Hatch. The switch was pretty painless for a project.toml-based workflow.
I really liked Hatch, but Rye - uniquely among standards-compliant new-gen package managers, afaict - has reasonable support for Python monorepos through its “workspaces”, letting you specify multiple local packages as editable installs.
I do believe Hatch is considering adding this at some point, but for now Rye is a good fit for monorepos.
Very fast, too. Armin and Astral just do great work.
Yes that feature will be coming right after PyCon! I just have too much to do right now between work, preparing for my talk and getting the next minor release out.
I look forward to checking it out! I’m subbed to the GH issue, so following it closely. We had a fantastic experience with Hatch over the past year or so. Thank you for your excellent work.
Rye is the only game in town for monorepos atm, but the approach is far from definitive - I believe Armin has acknowledged as much. In particular we’re not completely convinced that the benefits of a single repo-wide interpreter are worth the trade-offs. Very interested to see where you land with it.
I've replaced every bit of poetry and pdm with uv, which is clean, really fast and uses pyproject.toml and requirements.txt files. I don't get why one would want to use rye.
I think this article could have been more compelling with a more recent Django project. Django 2.0 officially supported Python ends at 3.7. Python 3.7 is out of support. Still annoying it does not work seamlessly, but I have foolish hope that recent builds would have pre-compiled artifacts on which you might be able to rely.
I am 100% bullish on some combination of PyBi, uv, or rye taking over the terror that is the ecosystem.
I agree with the author that using standard pip and virtualenv etc. is fine. I feel like people want software with massive guard rails that they can just prod at until it does the right thing with no danger of it doing the wrong thing. That's not been my experience with working with computers or any other machines and it won't get you very far as a software engineer. The Python tools are really powerful and flexible but you need to learn how it works. Just a little bit.
Using pip is not fine. It's broken it way too many ways.
Using virtualenv is just a really, really weird choice. The only reason anyone today would use virtualenv is because they want to support Python 2.X. Not that I think there's anything wrong with Python 2.X, in fact, I'd personally take it over 3.X any time... but I doubt this is what you had in mind.
I was trying out a flask and tailwind based course on udemy and one of the first things they wanted me to do was download rye and do a bunch of other boilerplate stuff. I didn’t fancy downloading some random executable from the internet and running it just for a udemy course so I tried to grab the GitHub repo and build it. No dice. I finally gave up on rye and other python tooling and did the udemy course using sinatra and tailwind. Which was vastly easier.
> I didn’t fancy downloading some random executable from the internet
Flask and Sinatra contain code. They may not be "executables" by default but you will probably not read the source and they could obfuscate sending your local data to some malicious server or something. Also lots of python dependencies run code at install time (mostly to complie native dependencies but it is running python code that could be doing anything) so even before you run the test server for the first time just installing the dependencies may run code.
Open source is built on a web of trust. You could argue there's too much of it but I don't see a reason to declare rye one step too far.
One must assume cargo comes pre-installed on your system. Right?
On similar note: it's funny how you believe that installing from a Git repository is going to provide reproducible experience for everyone. Like, you haven't even considered the possibility that some time after you've done that another commit was added to that repository and things stopped working in the way they did before, did you?
Like... I mean, just give it some... a tiny bit, really, it doesn't require that much of critical thinking... and you'll see how your "advise" is ill-advised.
Astral have said they eventually want uv to replace rye, but are going to build off of what rye has already done. Rye is currently using uv under the hood for the venv basics and the dependency resolver.
Rye is kind of being used as the test bed for what uv will eventually be.
I tried out rye + uv on a recent greenfield project. They are awesome tools and I'm really excited about their potential.
For me, rye (+ uv underneath) has perhaps the perfect workflow for an open source Python project. So I'm definitely using rye for that from now on -- instead of, say, poetry -- or hatchling directly, following the PyPA boilerplate[1].
You have a way of doing local development against any Python interpreter version. You have a way of tweaking dependencies and a straightforward sync/lock workflow. It all works atop "standard" PyPA infrastructure like pyproject.toml. You have a single command to build[1] project artifacts, like wheels. And you have a single command to publish new artifact versions to PyPI[2].
I think if you're doing local development on a project that is not meant to be published to PyPI, like a private Django project, then whether to use rye becomes more of a debate. For example, for a Django project I'm working on, I decided to just use uv directly, along with a Makefile. This is because during development of a Django project, I preferred to just use a plain requirements.txt (really, requirements.in) file, avoid the sync/lock workflow that rye imposes, and avoid the need to use something like "rye run". And rye's ability to build and package didn't solve a problem this project had, since the Django project wasn't being deployed via a PyPA packaging mechanism.
But this is probably also because the Python interpreter management problem, for me, is already handled by pyenv. I think if you're not already a pyenv user, rye is even more appealing because it handles "all" of the Python issues -- interpreters, requirements/dependencies, and packaging/publishing. (As well as a number of other standard dev-time issues besides, like testing, linting, and formatting.) But, in my case, I could hand venv management to uv, and then make dependency management part of a larger Makefile for my Django project, including custom linting, testing, and deployment steps. I wrote a little bit about my high level thoughts on Python packaging and dependency management, though this post was written before rye and uv were out.[4] If I were to update this post now, I'd say that you could swap uv for pyenv-virtualenv + pip-tools and be happier for it, and you could swap rye for the PyPA boilerplate. That is, you could upgrade to rye, or use it from the start, if you need packaging support, or if you like its dev-time workflow.
I'll also say, I found a little bug in how rye (+ hatch) interacted with my local git setup, and reported it to the rye team, and they helped me get to the bottom of it rather quickly.[5]
Yes, I saw the issue was already reported in hatch and being worked on so I assumed it'd make its way into rye eventually! Thanks for closing the loop!
Thanks to all 3 :) ... Ryelang in sort of "not a concern at this stage" and "go with the flow" way currently has this idea of per project static binary with all dependencies compiled in.
It sounds a little odd at first, but besides being the simplest to implement in Go, this also solves a problem for which Python needs tools like venv, pyvenv, virtualenv, ... that also this blog post mentions. The sheer number of these tools shows that this is a problem and that maybe that the solutions aren't optimal.
It also offloads most of package management (ie. pip like stuff) to just "go mod".
A specific binary per project also has other benefits in terms of securing the running environment, I think ... I'm still exploring this.
This is not yet in focus, but maybe starting from a totally different position will make some interesting or simpler tooling possible.
> I don’t actually consider the Python ecosystem and its tooling to be "hot garbage".
And in the same breath proceeds to describe Python ecosystem as hot garbage, no quotation marks needed.
Comeon, why do all these mental gymnastics trying to convince yourself your tools don't suck, when you already have all the evidence that they do?
As for Rye itself: yeah, no. pyproject.toml was supposed to do that... Of course it didn't work. And so did plenty of other "solutions" (Pipenv? Poetry?) that try to work around the defective basic components instead of building from scratch.
To fix this problem for real, the first stage needs to be the one where the people working on the fix understand the problem. As long as these people aren't willing to be honest with themselves and won't admit how bad their work has been so far, it's not going to get better. It will be more of the same band-aid.
I find the Python community Stockholm syndrome over dependency management kind of charming, it feels like one of the old internet holy wars.
It's absolutely nuts that you'd repeatedly meet the same criticism, independently arrived at all over the place, and just conclude it's some statistical anomaly of bigotry and fools.
"Well, I don't have these problems, I just {{500 word article}}"
Homebrew is an interesting beast. I was debugging something architecture-related recently during one of my first forays into rust land and quickly learned that my brew-installed rust wasn’t going to cut it because I needed rustup. Meanwhile, I don’t think I’ve used brew-installed node since at least v12 because I use nvm to support and debug multiple projects with different node versions. Is this kind of thing just becoming par for the course?
For the pain with the Django project, I guess I’m also willing to cut Rye slack because the underlying goal seems admiral and sadly the problem the author encountered seems common in Python land in general.