Setting up a development environment five years ago: here's a Word document that tells you which tools to install. Forget about updating the tools - good luck trying to get everyone to update their systems.

Setting up a development environment two years ago: here's a README that tells you which commands to run to build Docker images locally which will build and test your code. OS X users have fun burning RAM to the unnecessary VM gods when they have little to spare to begin with because MacBook Pros. No reproduceability between Dev and CI despite using container images because Dev will have debugging tools made available in the image that will be removed from the final image. Be limited while debugging locally because all debugging must happen over a network connection.

Today: Install Nix. Run build.sh. Run test.sh. Run nix-shell if you want more freedom.

I'm missing something here. (Disclaimer: I've just tried to understand the OP, I don't know all the details about what is going on there).

The text starts with "In my last post about Nix, I didn’t see the light yet." ... Then it continues with "A popular way to model this is with a Dockerfile." So I have expected that the post will demonstrate how Nix can be used without docker... But then later:

"so let’s make docker.nix:"

and continues in that way. So there is some dockering involved? To this casual reader, it doesn't seem that using Nix allows one to avoid docker, as a technology, but just that some additional goal is achieved by using Nix over that.

I just missed what was actually achieved, other than that the text mentions a few megabytes less here or there, of 100 MB, and that I also miss the information if the megabytes were traded in the build time or, if I understand correctly, in some way even more dependencies (more places from which something has to download something). Can anybody explain?

I'm sure that these tradeoffs were obvious to the author, but I as a reader hoped to somehow get the idea of those, and I missed that (yes, it's hard to "see the light" especially indirectly).

Docker is a tool that lets you package a Linux distribution and roll it out on any random server cleanly and safely. But the stuff inside that distribution is still very much bespoke and unmaintainable.

Nix is a tool that lets you create a custom Linux distribution with absolutely minimal effort. (Basically, just list your packages in a file and hit 'go'.) But the packaging story for Nix is pretty bad.

To bridge that gap, Nix has code that puts a Nix package with all dependencies into a Docker container. It works, but of course kind of icky; something more integrated and smart would be preferred.

This, a thousand times. If you are deploying the same container 10,000 times, with no modifications - well it makes sense to spend time maintaining that distribution. But if you're using Docker for dev environments (for example) and each one is different... hiiiya you haven't really moved on in terms of maintainability than using Vagrant or any VM setup.

- development

- production

For development, using Nix or Guix is in my opinion extremely nice. Using Docker in development would mean mounting your dev directory as a volume into a dev-container and depending on your application, this might end up being a pain in the ass. Editors often depend on the same dependencies to compile your code on the go and check for errors - if you don't have these on the host, you will then have to start solving new problems.

With Nix, you can have a package definition and either install all the necessary dependencies globally on your machine or spawn a shell with all of the needed binares in your PATH. Recently an application needed a different Node version than the one that was globally installed on my machine. Instead of having to build a Dockerfile or whatever, I just spawned a shell with the newer Node version, ran the command that I needed and was done.

For production, you might still want to use Docker, as there's a lot of great software built on top of it (Kubernetes and other platform-specific managed services). You can turn a Nix/Guix package definition into a Docker image quite easily, if you already have one. As extra benefit, you remove the small chance of still ending up with incompatible dependencies that you get when using a traditional package manager in a Dockerfile.

But sometimes your ops team has standardized on Docker, maybe because you're using Kubernetes. In that case, Nix will happily build those images for you.

Now I miss the point of that one too: if just a Go executable alone can be enough anyway, as it is statically linked, why not just copying it, instead of complicating around?

You can build the binary then have a Dockerfile copy it in from the scratch base image. However if you are using nix for deterministic builds you might as well add the few lines of code to have nix build the docker image to vs a Docker file with a single copy command.

If you are not building a static binary you get the advantage nix will copy in only the dependencies needed. You also get the advantage when you are building images you not randomly downloading urls from the internet that may be dead. Artifacts can come from a nix build cache which is cryptographically signed based on the build inputs so you know building the same image every time produces the same output.

With typical Dockerfiles that is not true. Docker images are not immutable so fetching the same Docker image may result in a different image being fetched. Likewise a lot of Docker files just wget / yum install random packages from places that may not exist anymore. If you maintain your own nix build cache you will always be able to build, get speedup from hitting the build cache vs compiling and know the build is deterministic. Running the same build multiple times will result in the exact same output.

If this is how you were setting up a development environment 5 years ago, anything would be better.

Also nix makes it pretty trivial to package and patch propriety binary vendor software. It would take at least one person on the development team feeling comfortable with nix, but they could craft a nix file that does everything.

Seriously, I dread every time I have to compile C++ in NixOS and there's no existing derivation already.

Oh, and when the Omnisharp guys start requiring a new version of mono that won't build the same way for whatever reason, so you're stuck on old versions of your IDE plugins if you want C# support until someone else figure out what broke.

GNU autotools assists in making source code packages portable accross Unix-like systems, while make is for build automation and has fine-level dependencies (inter-files).

Nix is a package manager, and though is can build your software for you transparently, it is not tied to and specific software configurationand building toolset, can manage buildtime and runtime dependencies (including downloading the sources), caching build results transparently, etc.

I'd argue the opposite—it's beautiful in it's simplicity. It's a tool that lets you express how to create something in a deterministic way.

A large part of what creates the learning curve that's best described as a wall is that this simplicity is due to building on a number of concepts that themselves are likely initially foreign: a pure functional language, content addressable storage etc. These are all good concepts to learn about in isolation, but as with any domain, learning a large number of new things at the same time (particularly those that may directly conflict with your current mental models) is hard.

What's truly nice about the nix approach is that it is not bound to any stack. You can use it for any language, for building any software (Nix - the package manager). You can also use it to define a entire machine config (NixOS), or even an entire collection of machines (NixOps). It is not something that you need to re-learn every few months, or learn then not use because some of your other tooling changed. Yes, the initial mental load is high, but the return on it is continuous and you'll likely learn about some other useful things along the way.

>I'd argue the opposite—it's beautiful in it's simplicity. It's a tool that lets you express how to create something in a deterministic way.

I'd take a different angle and argue that Nix is Version 1 of what to me looks like a very cool idea, and it's both beautiful and not straightforward to use.

If that matters, wait for version 5. Look at the progression before Docker came on the market - it does what other tools could do, but packaged in a nice, developer-friendly toolkit. I expect the same for Nix in a few years time.

But we're not going to get there without people doing this sort of stuff that can't possibly be worth the effort today.

In the end, excessively clever people make me nervous, because they leave complex problems in their wake, which I'm not clever enough by half to solve.

You can keep your wget, build script, db config, and nginx config all in the same file!

I think if you weighed the complexity of Nix, and NixOS in particular, against the tools it replaces (or anyway, could potentially replace) it's a no-brainer. I'm a convert.

Nix, unlike say docker, actually solves a problem correctly in a principled way: reliably and reproducibly going from source artefacts and their dependencies to installed binaries/libraries/etc and without conflicts or undesired interactions between other installed binaries/libraries/etc, even different versions of the same thing.

It represents a major innovation and step forward in the depressingly dysfunctional mainstream OS landscape, that still is stuck with all the abstractive power of 80's BASIC but with concurrency and several orders of magnitude more state to PEEK and POKE.

Also, even if you come up with a forward facing solution and demonstrate it only on a particular domain you still, IMO, deserve most of the credit for solving the problem even if it takes others to translate it straightforwardly to other domains.

I'm less sure about windows, can you explain a bit more how you use docker containers for providing "native" windows stuff (as opposed to as a more lightweight linux VM replacement when developing something that you really want to deploy on linux)?

Usually the traditional option would be to snapshot the VM right after the first successful install.

However stuff like Windows Nano now makes it interesting to start playing with containers on Windows.

I know some people write Windows software from Nix, but this is a way for Linux developers to make desktop apps for Windows users, not a solution to any problem Windows developers have.

1. They want to create a Game or Windows end user UI App (e.g. Overwatch or Photoshop)

2. They write some internal corp tooling or B2B software for windows shops (e.g. gluing some SAP or Oracle garbage together)

3. They want to write a server application (which means deploying on linux, which is basically the only server OS left). But they do not want to run linux as their desktop OS for corp or private reasons.

The 1st category probably has limited use for either docker or nix (in the absence of first class windows support). Might still be useful for tooling though.

The second category probably has use for docker mostly as a shitty linker, maybe also for isolation/security (I don't know the windows docker ecosystem).

The 3rd category can and totally should be using nix and I'd guess is at least double digit market share (so not insignificant; e.g. before Google banned them, a large fraction of Googlers had windows notebooks).

I don't use Windows as pretty UNIX, rather for its own capabilities.

So sound very alienated and have a realistically cynical that accurately reflects of most of the industry. But please don't assume everything has to be that way. Tools like Nix and community efforts like Nixpkgs really are the exception.

I absolutely love this quote and very much feel the same. Often times I feel like the industry at large has a bit of a tricky problem, and then in fixing that problem they forget about a whole set of issues the thing they are trying to fix dealt with, so often times they've just traded one set of problems for another, just now with more complexity. Monolith vs. microservices is probably the best example of this: monoliths have a bunch of problems that make developing with large teams hard, but microservices add a whole host (e.g. transactional consistency, performance) of issues that are much more difficult to solve for most dev teams.

Nix real power is being a language that allows to describe all of those things in a declarative way.

They certainly do! And quite often, the reason they leave those problems is either a) they're not clever enough either, b) there's enough actual work that solving them doesn't make them feel clever, and/or c) there are other things they could do that make them look more clever.

I have made some really good money ripping out somebody's too-clever-by-half solution and replacing it with something simple, well supported, and dreadfully unfashionable.

Since your runtime requirements depend on how you compile a system, you usually have to be quite careful with keeping your Dockerfile in sync with what you're building. This busywork just goes away.

Nix involves quite a large upfront time commitment to understand it, but it solves problems that I haven't seen solved elsewhere (well, I guess guix is similar), and does it across all the languages you write for. That it can work across toolchains and languages is a unifying force, and so I think it's one of the better systems for reducing the "exponential fragmentation" referred to above.

"There are other solutions"?

Well, look, well defined, easily reproducible, immutable environment configurable by text files is something that pretty much everybody wants, even if he never even thought about concepts like this. I mean, literally, your grandma would want it on her iPad, because it would mean you could buy her a new iPad and she couldn't see the difference aside from the fact the glass is not broken anymore. Every programmer wants it, because it would free them from that awful system administrator hegemony. I want it, because getting a new laptop that I plan to use for slightly different purposes than my current hardware, instead of building up my environment from scratch gradually installing more and more shit I forgot years ago even exists, I could review a couple of 200-line config files (which I possibly didn't even write myself, the system just managed to express in a well-readable format what I got by experimenting in the shell years ago), remove some lines, add a couple of others, and get exactly what I want on that laptop.

And we don't really want any tradeoffs here. Yeah, as a real rock'n'rollers we want a fucking lot, but that's the point, we want to find some way to make all problems caused by the complexity of making lots of software and hardware work together disappear.

Yet, I don't see this ultimate dream come true on every — be it real or virtual — device, for every purpose. So be it the problem of marketing, or the "other solutions" being a bit more of a tradeoff than we are ready to accept, or something else, but these "other solutions" don't seem to do as much of a good job as you imply. So, my understanding is that we are just "not there yet". Hence "all these technologies popping up left and right, switching names, dying abruptly, getting super-seeded or hacked with vulnerabilities". There is a dream, maybe even a vision, but there is no solution. Not that I'm aware of. (And, once more, the real solution is not just something that is "possibly possible", but also includes solving the problem of user-adoption.)

I share the parent's frustration in trying to keep my head wrapped around what all the different players are in the space. I tend to agree that the more I look into Nix, the less it really seems like it will actually take off. But it's utterly confused to segue from that to "Therefore, they must not be solving any problems we really have."

I'm always wondering when I read one of these posts what setup they must have to get work done. Seriously, what do most people do? How is it apparently so easy to share development environments between your co-workers in a manageable way? And if the answer is "each employee starts out with the same laptop baselined in the same way, and then they work on one project which is in a monorepo, whose system dependencies are already installed on the laptop, and those dependencies never change" then... hello? You don't see that this is not a solution? I fundamentally _do not_ understand how anybody can live in the world and not be annoyed by this. What do they do? WHAT DO THEY DO?!

Nix besides the build, can provide a shell that contains all the tooling necessary to build the application. The definition lives together in the same repo as application any everyone who uses it will get the exact same tools with the same versions.

Once you do this it saves all the “it runs on my machine”. No more days following a readme setting up a vagrant/virtualbox machine, which is then slow as a vm with limited memory on a laptop.

Instead you type one command, “nix-shell” and get everything installed locally as required in a deterministic way. Agree, this has been a revelation to teams I’ve worked on.

I've had 40 years to observe and think about why this is all happening.

What I think happens: new blood enters the industry, ignores the accrued wisdom of the elder old blood, re-invents things, regurgitates, recycle (every time someone makes money with it), and then .. becomes the old blood.

Like, there seriously is some sort of industry-bound amnesia affect at 3 and 4 orders of scale, in my opinion.

There are those who read the docs, and write new ones/edit old ones. There are those who don't read the docs, but write new ones only. There are also those who read the docs, and don't write anything new, too. Then, there are those who don't read the docs and don't write new docs, either.

Alas, all these are tied up in a universal struggle to gain dominance over each other, and its called the technology startup battle.

Docs don't get old. People do.

[1] Eelco is the creator of Nix: https://nixos.org/~eelco/pubs/phd-thesis.pdf

No, we don't.

What else can let me install two different versions of gcc and glibc and choose one with a one-line config change?

Seriously, if you have any suggestions, I'm all ears. I agree Nix is too complex.

Not sure how Nix(OS) handles isolation in terms of cgroups/namespaces though, seems that it relies on systemd for that.

Well said. And not only your sanity, but by using this stuff in production you’re also dragging everyone else’s sanity with you. The benefits need to be compelling and aligned with the business needs, and saving a few MBs here and there at the cost of time is definitely not in alignment except for in the most exceptional cases.

Fun to read about and toy around with, but let’s leave it at that. The first example using multi-stage docker build and alpine is fantastic.

It will be corrected the next time we see a major bubble burst/recession. Right now we have literally millions of people around the world working in tech, and everyone has ideas.