The advantage (or disadvantage) of getting older: Seeing the younger generations come up with brilliant "new" ideas. New to them, whereas I think "yep, that was done 40 years ago, and 20 years ago, and now it's new all over again."
Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.
> Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
The saddest thing about the way this changed is that the barrier to learning is so much lower today, but what do you actually do once you've learned?
When I was a kid, there were no programming classes in school and I had to go buy books and magazines to learn how to code in BASIC or assembler or Pascal. But once I've learned a new trick, I could immediately do something with it and feel good about it. It didn't matter it wasn't something spectacular, it didn't matter it wasn't very useful, because it was mine and there weren't 1000 of other things just like it, available to me at a touch of a button.
Compare that to what my kid is going through. He wanted to try learning to code, so we signed him up for some classes where he learned some Python. Later, in high school, he took AP CS Principles and they taught him some rudimentary JavaScript. Now, in his junior year, he's taking AP CS A, where they're teaching him Java.
Thing is, what can he do with all that? He's surrounded by devices running code. Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding. Writing some actually useful code is harder, and it feels pointless, because there's almost always an existing app you can have at a click of a button and it does what you thought of and probably better. Writing useful code that doesn't already exist -- like e.g. your own mod for such-and-such game -- is even harder than that.
> Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
To top off the irony, generalists are not being appreciated or sought much.
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding.
But there's joy to be found in writing pointless and abstract programs. It's more about the journey and less about the utility of the finished product. Writing, for example, an ASCII-art drawing system that works on a big rectangular 2D character array can teach you so many concepts – from OO and encapsulation, through linear algebra, to file IO.
> But there's joy to be found in writing pointless and abstract programs. It's more about the journey and less about the utility of the finished product.
Sure, and that's how you know you're really into programming. That's how I could tell my kid wasn't really into it, but rather wanted to try out this thing his dad loves to do so much ;)
I guess that hasn't really changed: if programming is something that "calls to you", it will still "call to you" despite all these changes.
I guess what I'm wondering is whether there's a "middle ground" we're missing. There are kids who will choose software development just because "it's a good career". I don't want my kid to take that path and risk ending up with a job that is absolutely devoid of joy, rather than trying to find something that would be more fulfilling. On the other extreme are the kids who are the way I was when I was a kid: programming is something that fascinates them regardless of whether it's useful or a good career, because they find joy in the act of programming itself.
Does the middle ground exist? Are there kids who would actually be happy writing software for reasons not so banal as "it's a good career", but aren't in the "holy shit, this totally abstract and pointless thing I did was so cool" category either?
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding.
No, but in the 80s, when the guy at Radio Shack didn't know how to stop it, it was still worth it somehow.
What can your kid do to assert some goofy kind of power in today's adult world?
In retrospect, I realize that the part you quoted might sound like I felt that toy programs like that weren't magical, when I was a kid. On the contrary, I'm trying to say that they don't feel like magic in today's world.
> Writing some actually useful code is harder, and it feels pointless, because there's almost always an existing app you can have at a click of a button and it does what you thought of and probably better.
Well, there are a lot of people who enjoy cooking, or gardening, or carpentry, or making their own clothing, or playing in a local softball league, even though there are many professionals with professional-level skills and equipment who can do those jobs better than they can.
I think it's like you suggested in your later comment: you either have the bug for programming, or you don't. There are a lot of programmers nowadays who don't -- they just program to make a buck.
Not that there's anything wrong with making a buck, but it's like the guy who flips burgers at McDonalds. He does it to make a buck, not because it's speaking to his inner need to be a creative chef.
> Writing a toy program that fills the screen with "HELLO, I AM COMPUTER" does not feel like you just learned a magical incantation that makes this machine do your bidding
Find out what his favourite hobby or favourite game is, then demonstrate a little script... Suddenly the conditional branches and loops don't seem so abstract anymore !
Isn't there is a similar spread for older technologies and crafts like horticulture/agriculture, textiles, ceramics, metalwork, printing, music, cooking... is this just evidence that software has arrived?
>That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms.
I find a lot of these tools and frameworks and languages and platforms are attempts to correct perceived problems in the original tools and methods. But these new things have their own issues which someone else will attempt to correct. Then you get your cliques and reddit subs that make fun of you for using X because it's old.
My point is that there is no back to basics--no grounding in the fundamentals we grew up with--or should have at least.
I think you do not understand just how much stuff it out there. Just in the web world, can you do React and Angular and NodeJS and Spring? Have you really looked at the CSS specification? That's a career in itself.
Moving slightly to mobile apps, what about Jetpack Compose for Android and the equivalent for iPhones? Electron? Or classic programming: How is your SQL, including stored procedures? Note that Oracle is different from Microsoft is different from MySQL. Want to learn ABAP? How about writing a back-end system in Java, with a front-end in JavaFX?
I'm just getting started...
No, you cannot learn all of this sufficiently to be even remotely productive. You may, like me, be aware of a lot of it. I'm sure you could pick any single new platform or technology, and become productive in it in a few weeks or months. But to be a real generalist, able to do it all? That is simply no longer possible.
I disagree with the Jack of all trades, master of none cliché. You can effectively be a master of multiple fields because the true applicable depth of so many of them are completely shallow.
There are entire niches in software development that you can dismiss wholesale.
Virtually everything in the web that isn't the web standards themselves or from the most popular libraries isn't worth learning because it's just regurgitation.
If you know CSS as an applicable professional level, any CSS framework you see is irrelevant. You can pick it up. Etc.
I generally agree with this. And so much of modern dev… especially in the app/web dev space is just pattern recognition and hooking up different systems to each other. That’s not to discount its difficulty… but you can be good at several disciplines. I actually think being a generalist (to some degree) can actually be helpful, because it allows you to think in multiple modalities, where as someone who specializes in something is less likely to be able to break out of their thought patterns.
> I actually think being a generalist (to some degree) can actually be helpful, because it allows you to think in multiple modalities, where as someone who specializes in something is less likely to be able to break out of their thought patterns.
GP mentioned the two-part "jack of all trades", but there's a relatively modern third part to it that expresses this sentiment: "A jack of all trades is a master of none, but oftentimes better than a master of one"
Yep I also disagree. I would say I have a mastery of a lot of backend programming, architecture/infra deployment, microcontroller programming, assembly, React + Redux, and at least 3 or 4 other languages I’ve worked in through my career. I’ve had jobs doing data engineering - building and deploying machine learning models at scale, advanced computer vision stuff (before ML became the rage for it), building, designing, and deploying tiny sensors on wind tunnel models, etc etc. I have a couple of open source multiplayer projects with a few thousand MAU each. Now, the one thing I will never master or care to is CSS - the devil himself must have come up with this abomination.
Maybe true, but I run into very few experts to go head-to-head with.
I started programming in 1974, professionally since 1979, and I'm still working, mostly web sites and lots of system admin. I don't know most things at expert level, but I can learn and adapt quickly and start producing right away, because I have decades of experience with analogous or ancestor technologies (like C and C++, SQL, Unix) at least at a very proficient level, and I have lots of business domain expertise.
I have worked with true experts. No need to go head-to-head, there's no competition, just a great opportunity to learn. Mostly I have worked with people who have mediocre skills, and narrow skills, who can't adapt to change or come up to speed on new things. Most programmers live in the "expert beginner" zone Erik Dietrich described so well, not actual experts, and I wouldn't worry about going head-to-head with them if that was part of the job. But programming isn't competitive gymnastics, we don't have to perform better than the guy next to us, we only have to satisfy management or customers that usually have no idea what programmers do.
> because I have decades of experience with analogous or ancestor technologies (like C and C++, SQL, Unix)
I think a lot of people underestimate just how much knowledge transfers between systems within the same domain. I have nowhere near your length of experience, but even in just my time I've been able to pick up plenty of languages and frameworks and get going right away because I already learned the underlying concepts when working with other similar systems.
No one should be starting from scratch every single time they start something new, and the larger your variety of experience the faster it's going to get each time you start with a new language/framework/system in that domain.
The opposite of the jack of all trades is the one trick pony and the one trick pony always falls onto hard times when their single trick is no longer in fashion.
The thing is to produce an app these days you need to have mastery of 3 languages + some frameworks. You need HTML, JavaScript, CSS, and then the frameworks.
If you work on a team you can specialize in one of those but when you are learning you really need to use them all yourself.
When everything was a command-line application you didn't need to know more than C or Perl or Basic etc. But then you couldn't really do cool user-interfaces, with mouse and events fifty shades of color-gradients.
> Part of the problem is the sheer breadth of IT and software development today
No, the problem is wholly human limitation. All humans start as an ignorant babe and have to both mentally develop and learn everything from scratch. This includes the rebellious teenage years and not listening to the wisdom of their elders (and for the adults around them, how to properly raise and mentor kids).
We have been doing this exact dance every dozen years for thousands of years for every single field and every single little task, from how to cook the best bread, the best way to saddle an ox, how to fix a toilet, etc. Learning and discovering, and relearning and rediscovering, ad infinitum.
This is a common thing I've read in books and heard from my more experience co workers. My favorite current example is OpenAPI specs for building a client or generating docs. SOAP was another big thing. So was CORBA. We're really good at rebuilding RPCs. Not to say that they don't have their differences and benefits.
NextJS always gives me a laugh too. I remember when it was newer in 2019/2020 and people raving about its file based routing and server-side rendering.
These new waves sometimes have new benefits to them, and sometimes they don't. I have to imagine that we won't ever seen an end to it, but at least it keeps some exciting new stuff on the table.
Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
40 years ago: your physicist knows nothing of how to write RPG, your mainframe developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin wouldn't touch FORTRAN with a ten-foot pole.
... Which is to say, I don't really buy it. Or rather, I don't think that specialization is a recent phenomenon. It is true that the personal computing market consisted of comparatively simpler machines, which allowed for greater breadth of understanding.
We call this reinventing the flat tire. In the old days, you would chastise people for reinventing the wheel. Now we beg, “Oh, please, please reinvent the wheel.” -- Alan Kay
On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things. In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament
That approach works so well that a program written in '70s is so well abstracted away from the intricacies of the system that it can input data from a network socket, a pipe, a file, or output to whatnot, and doesn't need to be touched for all that to work. A plain program reading from stdin and writing to stdout can be turned into a network service via inetd or similar by just redirecting file descriptors.
But for the contemporary software coders the ingenious idea is of course rubbish. They would replace everything by node_modules.
It's great that it is easy to do simple things with ancient tools, and it's completely true that you can inetd and CGI your way to covering numerous use cases on small systems.
The problem is the lack of innovation holds us back from solving complex use cases. The sockets API in particular is a gigantic mess, because the protocols are more complicated than people imagined forty years ago. For example it is simple to send a datagram. What if you want to get notified of the timestamp when the datagram hits the wire? Now you will 1) read a lot of man pages, 2) write 100s of lines of gross C code, 3) read the source code of the Linux kernel to figure out which parts of the manual were lies, then 4) fix your program with some different code and a long, angry comment. None of they would be necessary if the last quarter-century of network features hadn't been jammed into the ioctl and setsockopt hack interfaces.
In other words there's plenty of room for operating systems to reconsider interfaces in light of the systems and protocols we have now, instead of Space Age imaginary use cases.
> The problem is the lack of innovation holds us back from solving complex use cases.
Innovation is awesome, when it is in the pursuit of achieving something better. (Better, not just newer for the sake of rewriting everything Yet Again.)
What burns me out about this industry is that a huge amount of "innovation" is just a pursuit to create complexity for the sake of complexity and resume-driven development.
Thus we get to classics like the "Command-line Tools can be 235x Faster than your Hadoop Cluster" (a well-known one, but countless similar scenarios play out everywhere continuously).
To all of you setting up your 20 node cassandra cluster behind your enormous kubernetes setup, all just to serve your 100rps app: you're not innovating, you're doing it wrong.
(Not "you" you. The whole software engineering industry.)
Sure, but there are hundreds of thousands of people in this industry who are dealing with data problems on a scale that certainly will not be handled on your laptop with sed and awk. The needs of industry shouldn't be sacrificed to the artisanal aesthetic of small solutions.
> Sure, but there are hundreds of thousands of people in this industry who are dealing with data problems on a scale that certainly will not be handled on your laptop with sed and awk.
Yes there are, but those 100k's of people doing that are working in a small handful of megacorporations.
For 99+% of the companies out there, they don't need the complexity. But it gets shoehorned in because resume.
Reading timestamps is indeed a clusterfuck, but are you sure that whoever came up with that mess would have done better with a deprogramming model?
Edit: I won't elevate the Unix model to a pedestal, but in his favour I will say that the relative simplicity of the model and it's inherent difficulties to extensibility helped limit scope creep and overengineering.
Most recent horror I discovered: `mincore` is intentionally broken, always returns positive results when the calling task is running in a mount namespace. Because the hallowed unix file abstraction is fundamentally bad.
The devil is in the details. You can't turn really a command from the 70s into a webservice because its input are command line options/arguments and not query params or RPC arglists, and its output is probably intended for humans to read or parse easily with a regex/sed/awk. Web services speak in structured data for their input and output over protocols that can't be abstracted as file i/o.
Because they're not just files. There are additional semantics and concerns that you cannot abstract away with read/write calls. You also need ioctls, you need send/recv, you need parsers on top to do something useful with the data, and so on.
The reason that the simple model dies out is because it's not as useful as it seems in practice.
It was originally developed as a multi-user time-sharing system. Multiple users would access the system simultaneously via the attached teletype terminals. The "shell" program handled such an interactive user session. All IO was blocking, and programs were written in a sequential manner. Piping enabled them to feed the standard output of one process to the standard input of another.
It's a good framework for text-based computing, but nowadays we have so much more problems to cope with. This original idea never addressed networking, binary versioning and control, configuration management, async IO, multithreading, GUI, etc.
You're not inventing new things, take a look at CORBA and compare that to COM objects. It's not "no new things", it's "build things that are actually principled, don't just cram a bunch of garbage together because it's the path of least resistance".
>Every increase in expressiveness brings an increased burden on all who care to understand the message.
There are plenty of new things that are largely compatible with unix philosophy. MQTT is one of my favorite examples, it's a message bus that follows a lot of unix philosophy and I find it a joy to work with compared to stuff like DBUS. Obviously it doesn't fill the same role as DBUS, but still.
You can't blame Unix for getting popular, or for almost no research and progress in operating systems since Unix got popular. Rob Pike wrote a good paper about that.
At this point it's network effect, familiarity, and the free price tag, not a quasi-religious sacrament. Anyone can try to take that different approach, but that doesn't mean it will go anywhere. Some very smart people have worked on alternatives to Unix, or different directions for operating systems -- including the original Unix team with Plan 9, Wirth with Oberon, the Lisp machines. No Unix Inquisition shut those down, they just didn't offer a 10x improvement along enough axes.
What are some alternatives that you'd recommend people look at?
I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline.
Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
Generally I think people are happy piping structured data around, the problem is that ties you into a particular data structure. JSON seems to be winning there, and I think people would be very happy if more commands had a JSON environment variable, or even if it was possible to output data on a /dev/stdjson pipe.
Anyone know how to add a new stdout-like interface to unix-like OSes?
> The functions defined in libxo are used to generate a choice of TEXT, XML, JSON, or HTML output. A common set of functions are used, with command line switches passed to the library to control the details of the output.
JSON is a good start, but Powershell is great in that a date is actually a date object, which means that I can do operations on that without faffing around with parsing and worrying about whether what comes in the JSON might depend on the locale of some system, or that somebody didn't take into account that I might want microseconds and truncated the timestamp.
Right, but at that point you're tying it to a universal type system.
Personally I like YAML, since you can add type-hints to data which you can use to turn simple text types into more complex types, but I can see why that standard wouldn't take off.
Most of the unix-philosophy people I know are interested in stuff like that, it just has to be implemented in a thoughtful way.
What value would a new standard pipe for JSON bring? It's already serialized text that can be sent to stdout. The real rub is getting programs to speak JSON natively.
>> The real rub is getting programs to speak JSON natively.
You mean getting programs to ingest arbitrary data structures? OK, JSON is not arbitrary - it is limited to a certain overall format. But it can be used to serialize arbitrarily complex data structures.
Meaning most Unix command line tools are line-oriented meaning input and output are expected to have a series of text records, each on its own line. Stray too far from this lingua franca and you'll need to get creative to get piped IO to work.
JSON doesn't care so much about lines and doesn't necessarily represent an array of records, so it doesn't fit into this box.
I think it would be cool if stdout worked something like the clipboard, where you have different representations, and when an application copies something from the clipboard it selects the representation it wants. I'm not sure how to avoid making it terribly wasteful though, so that the producing application doesn't have to write in every format possible.
Imagine a magical `ls` that can emit text/plain, application/json, what have you:
ls -f text
ls -f json
ls -f csv
ls -f msgpack
...
Now instead of specifying formats on both ends:
ls | jq # jq accepts application/json => ls outputs as json
ls | fq # fq accepts a ton of stuff => "best fit" would be picked
ls | fq -d msgpack # fq accepts only msgpack here => msgpack
ls # stdout is tty, on the other end is the terminal who says what it accepts => human readable output
Essentially upon opening a pipe a program would be able to say what they can accept on the read end and what they can produce on the write end. If they can agree on a common in-memory binary format they can literally throw structs over the fence - even across languages, FFI style - no serialisation required, possibly zero-copy.
And I mean, we really know: the last one we already do! Tons of programs check for stdin and/or stdout being tty via an ioctl and change their processing based on that.
It'd allow a bunch of interesting stuff, like:
- `cat` would write application/octet-stream and the terminal would be aware that raw binary is being cat'd to its tty and thus ignore escape codes, while a program aiming to control the tty would declare writing application/tty or something.
- progressive enhancement: negotiation-unaware programs (our status quo) would default to text/plain (which isn't any more broken that the current thing) or some application/unix-pipe or something.
- when both ends fail to negotiate it would SIGPIPE and yell at you. same for encoding: no more oops utf8 one end, latin1 on the other.
Interesting post! when prototyping what would end up being fq i did quite a lot of tinkering with how it could work by splitting it up into multiple tools, and use it like: <decode tool> | <query tool> | <display tool>. Never got it to feel very neat and problem is what would be piped? i tried JSON so that <query tool> could jq but whatever is would be would have to be quite verbose to be able for <display tool> to be able to show hexdump dumps, value mappings, descriptions etc. So in the end i ended up doing more or less the same design but in jq where the values piped between filters is kind of a "polymorphic" JSON values. Those behave behave like JSON value but can be queried for which bit range and buffer they originate from or if the value is symbolic representation, description etc.
Maybe an interesting note about fq is that for format like msgpack it kinds of can decode in two modes, by default it decodes msgpack into like a "meta"-mode where things like integer encoding, sizes etc can be seen and then there is another "representation"-mode that is JSON value of what it represents.
One alternative is Apache Arrow. It's an efficient in-memory format used to represent flat or hierarchal data. Multiple databases and programs like pandas support this, so they become compatible with each other.
> What are some alternatives that you'd recommend people look at?
For me, this would mean an OS that supports both static and runtime reflection, and therefore code-generation and type generation. Strong typing should become a fundamental part of OS and system design. This probably means having an OS-wide thin runtime (something like .NET?) that all programs are written against.
UNIX's composability was a good idea, but is an outdated implementation, which is built on string-typing everything, including command-line options, environment variables, program input and output, and even memory and data itself ('a file is a bag of bytes').
The same thing has happened to C (and therefore C++) itself. The way to use a library in C (and C++) is to literally textually include function and type declarations, and then compile disparate translation units together.
If we want runtime library loading, we have to faff with dlopen/dlsym/LoadLibrary/GetProcAddress, and we have to know the names of symbols beforehand. A better implementation would use full-fledged reflection to import new types, methods, and free functions into the namespace of the currently-executing program.
James Mickens in The Night Watch[1] says 'you can’t just place
a LISP book on top of an x86 chip and hope that the hardware
learns about lambda calculus by osmosis'. Fair point. But the next-best thing would have been to define useful types around pointers as close to the hardware as possible, rather than just saying 'here, this is the address for the memory you want from HeapAlloc; go do what you want with it and I will only complain at runtime when you break it'. Pointers are a badly leaky abstraction that don't really map to the real thing anyway, given MMUs, memory mapping, etc.
There are so many ways we could make things a bit more disciplined in system and OS design, and drastically improve life for everyone using it.
Of the major vendors, I'd say only Windows has taken merely half-hearted steps towards an OS-wide 'type system'. I say half-hearted, because PowerShell is fantastic and handles are significantly more user-friendly than UNIX's file descriptors, thread pointers, and pipes. But this is still a small oasis in a desert of native programs that have to mess with string-typing.
> In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament
Long time ago I asked my karate sensei how come I see some of the black belts doing katas in a different way. His answer was that to improve on something one must first have deep mastery of how it is done.
(i.e. "Shut up and do the katas correctly. If you ever get to black belt then you can improvise.")
Always found that answer to apply to everything. In particular it applies to a lot of the poor quality libraries and code we see today as a direct consequence of not understanding the past and what has been tried and failed and why.
The fact that we're talking in 2024 about using the file abstractions of Unix created in the 70s is proof of how extremely powerful they are. Nothing is perfect, but that's quite an achievement.
A mistake we often make I believe is to focus on code-quality, rather than code-content-quality. Code is basically mathematics. Is it good math or bad that is the question. Does it do something useful, in an economical way.
To write great code, I believe, means you have to be a domain expert of the domain for which your code solves problems.
"Everything is a file" is not a lesson, it is an imposition that with some some things that are very different to files it is a very wrong thing.
The lesson is: Minimise as much as possible the different interfaces necessary and reuse them. Interlisp or Lisp machines used Expressions so everything could be read and written by everything.
Unix is a mess of binary formats, a mess of different console emulator formats, a mess of different text formats.
And the article is just about one guy who does not like Wayland not using files for everything, which is fine for me.
The main lesson from unix that is getting lost is composability of systems.
So many silos and proprietary "cloud" systems, so much glue that needs to be constantly rewritten because there isn't any stability in an ecosystem composed of competing rather than cooperating actors.
There are great things that have come along, I would say k8s has been very successful at introducing a highly interoperable, composable API for abstracting over a bunch of machines.
But the trend towards hosted services that are incentivized to create lock-in and data-gravity isn't great.
I don't think this lesson is getting lost accidentally, but intentionally. There was a brief moment in the late 00's/early 10's where companies were creating these rich web API's that would let you (and other services acting on your behalf) interact with their system in arbitrary and novel ways. But that turned out to not be very profitable (or at the very least, wouldn't let you create a walled garden that would get 100x exit for your VC's) so we've steadily moved to more locked down, less interactive, less open, less useful, but ultimately more profitable services.
It's not just unix, but programming languages themselves and general patterns for how software is built and how projects are run. It's what I'm most frustrated by in this industry. So much waste. I once thought surely this industry would mature, but now I don't see that happening in my lifetime. Reminds me of human civilizations before people developed language and wrote things down and almost all knowledge was lost with each new generation, keeping progress limited to a snails pace.
It’s a ramble that ends up with a complaint that Wayland doesn’t fit into the stream file model that’s supposedly critical to “unix”. Block files have been in Unix forever and they don’t either.
Maybe read “Worse is Better” before writing part 3.
And “Unix” is orthogonal with FLOSS.
Sorry to be such a downer — been reading el reg since it started.
I read the article, and I didn't see this point made. Mostly it was a very long description of the history of Unix, with a tangent about microkernels.
Your headings are
What is 'Unix' anyway?
Generations and the gaps between
Turn it up to 11
I believe your ultimate point is about Wayland and perhaps that it should follow everything-is-a-file. But that comes so late in the article that you barely give it a passing mention, and it isn't consistent with the premise of the article.
Nah. :-) I am not terribly interested in Wayland, myself, and I suspect that in the fullness of time it will turn out to be a not-very-interesting blind alley.
The existence of more than one article based on a talk does not by itself suggest that they are dependencies of one another; nor does it excuse rambling.
The title of the OP is "Forgetting the history of Unix is coding us into a corner." At no point do you explain who is forgetting what lessons, or why such forgetting is coding us into a corner.
In any case, the history is interesting and clearly well-researched. I offer as constructive criticism that your structure and clarity could benefit from continued editing.
> The existence of more than one article based on a talk does not by itself suggest that they are dependencies of one another
I beg to differ. This is part 2 of an estimated 4. They are heavily interdependent; a talk is, of necessity, linear.
> nor does it excuse rambling.
Guilty as charged.
My plan was 3 parts, but this section would have been circa 3½ thousand words, which is way too long, so I cut it in approximately half. Currently I think there'll be 4 parts, probably published over the next 2 weeks.
And, as I said, if you want to skip to the end, the script is right there, but it's not as coherent as something intended to be read would be.
P.S. Thanks for the comments!
Sadly deadlines in online publishing come twice a day, and thus, on average, I only have 4 hours to bash an article into shape.
As Blaise Pascal wrote in 1657: “I have only made this letter longer because I have not had the time to make it shorter."
Or, as Churchill allegedly said:
"If you want me to speak for two minutes, it will take me three weeks of preparation. If you want me to speak for thirty minutes, it will take me a week to prepare. If you want me to speak for an hour, I am ready now."
You might have some grand vision of all this ties together in part 4 or whatever, but the problem is that it doesn't exist yet so we are left on discussing now this part 2, and it really doesn't stand on its own at all. Nor does the article itself state this fact that there are still two parts coming, so at least I was quite surprised when I reached the end; it truly felt like half of the article was missing.
So maybe the biggest problem here was just posting this on HN prematurely, maybe it would have worked better if you'd waited until the whole thing had been published.
I linked to it in the first paragraph. You can read the script if you want.
Redox, OTOH, no: it does not even enter into it. Although I do not address it, if pressed, I might concede that if there is a Rust involvement, it's R9, the successor to Harvey OS.
I'm really not a fan of UNIX in the slightest, other than as a basic kernel OS to build something nicer on, like what macOS did. I think it's much better to make it easier to write programs and have a rich system library than to glue together text streams. And in fact that's basically what has happened instead: the "UNIX philosophy" has largely been replaced by Python and the like. People now reach for Python for many tasks that were previously shell one-liners, and imo that's a positive development.
When I was a kid learning how to code reading online, everyone was religiously defending the "UNIX philosophy" showing off their fancy and indecipherable one-liners (that don't work on my system bc I have GNU awk instead of whatever awk they have or whatever). This has basically died out year by year as so many people switch over to scripting languages with great standard libraries and super-powered easy to install libraries.
Now, I'm not really a fan of the super overkill object-oriented approach that systems like mac and smalltalk take, but I think our modern approach is a good compromise
On the topic of "UNIX shell one-liners versus Python", I enjoy letting my students solve the "make a list of the words in a file with their counts" exercise in Python, then I show them a sh one-liner.
The Python scripts takes them from 10 minutes to two hours, they're buggy, and they cannot process big files - everything is in RAM (with a monster dict) - without a crash/system freeze.
The one thing holding Python back from completely eating my workflow of utils, glue, and hacks is environment isolation. Nothing sucks more than when activating some venv breaks a background tool I wrote 3 years ago and forgot about. I recognize that this is a surmountable problem with at most moderate effort, but getting to a point of complete frictionlessness would really move the whole ecosystem forward a lot, IMO.
Sure, I've basically just accepted that I'll need to trash and rebuild my various Linux daily drivers every 1-2 years because of environment decay. All solutons I've seen (Qubes, Nix, etc) come off as a massive PITA given that I just want a FOSS laptop upon which I can tool around a little and maybe watch Netflix. For now I just use docker for Unix-y isolation, but still cumbersome. If Python could finally figure this out I think it would be a reason for massive celebration. Conda got close, but never quite nailed it
The random Wayland bashing seems both completely unrelated to whatever the topic of the article is, and uncalled for. Wayland literally is a protocol that is communicated over a file (WAYLAND_DISPLAY). Sure, the file is a (UDS) socket, but it is still a file! Also Wayland is purely an userland thing, so it is very much constrained on what primitives kernel has to offer.
The way the Wayland section is included there, it seems like author is heavily implying that it is prime example of "spiralling software complexity and bloat", which to me sounds mostly like they do not know what they are talking about. Core Wayland is distinctly smaller and simpler than X11 ever was. Sure, modern graphics (etc) are intrinsically complex and Wayland exposes some that, but it does it mostly by keeping Wayland itself as simple and transparent as possible, instead of making Wayland (or rather its implementations) hugely complex and bloated.
But besides this Wayland thing, I'm really confused what the article is actually trying to say and how does the title relate in any way to the content? The intro implies that there are some lessons to be learned from UNIX history, but the article sure as hell doesn't cast any light on what those lessons might be. There is also this implication that we are somehow coding ourselves into a corner, but that is never actually expanded on what they are meaning with that?
In terms of lessons learned, the key observation maybe is that UNIX peaked somewhere around SysIII before BSD etc hippies set their grubby hands on it and messed everything up.
“Those that fail to learn from history are doomed to repeat it.”
Winston Churchill, 1948.
As can be seen, it equally well applies to history of computing. We keep reinventing the same things only because young coders don't learn from computing history and outright dismiss anything that's more than 1 year old.
Churchill was paraphrasing George Santayana, and leaving off most of the original. The author of the article we're commenting on included the original quote. The context seems relevant here:
Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it. -- George Santayana
> So any Wayland evangelists out there, tell us: where in the file system can I find the files describing a window on the screen under the Wayland protocol? What file holds the coordinates of the window, its place in the Z-order, its colour depth, its contents?
I haven't used X11 in several years, and it's been well over a decade since I've had to put any serious thought into it, but I honestly don't remember being able to do this there.
> For instance, macOS is a Unix™. It passes the tests, and Apple pays for the certification. But it hides most of the real Unix directory tree, its /etc is relatively empty, it doesn't have an X server – it's an optional extra. And most of all, it's not case sensitive.
What does it mean for the operating system to be “case sensitive”? Certainly APFS is case sensitive, so this must refer to something else?
OS X / macOS has always defaulted to using filesystems that are case-preserving but not case-sensitive. You can cd to ~/library and see the contents of ~/Library and trying to mkdir ~/library will fail because ~/Library exists.
I started working on the embedded industry in the early 90s when other OS'es with the likes of Greenhills and VxWorks.
My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks.
What call me attention the most compared with other embedded OS was the transparency of the system. We all had our door-stop manuals 10 lbs weight manuals for our embedded OS documentation where a collection of tools like Tornado (In VxWorks) gave us a way to interact with the OS. But Unix, in my understanding, needed no added tool. A kernel, a filesystem a shell... go.
Many years a go a developer tried to introduce a patch in Linux to give graphics a leg up because Linux PC Desktop experience lagged behind Windows and Macs. The gate-keepers said no and the guy forked the kernel and went his own merry way.
Going back to embedded real-time systems and Unix/Linux efforts like MonteVista and HardHat created smaller leaner footprints that started competing in the embedded field. The OS evolves wells. My lasts embedded projects with Yocto make the point.
> My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks.
You know what, although I experimented with Lasermoon Linux/FT on CD in ~1996, and tried and failed to install Slackware 3.0 around then, the first Linux I ran as my daily desktop OS for a while was Caldera OpenLinux 1.
Your memory deceives you: it came on a CD. It needed 3 or 4 boot diskettes to get to the point where it could find your CD drive and start the installation process.
It had the first graphical installation in the Linux world, and while it wasn't a live environment and had no desktop, it did multitask: you could play Tetris while it installed.
When I think of the Unix approach that is based on text stream and the pipes that carry them, I think of telephony. (What a coinky-dink.)
The Unix way is "in-band", where metadata (for telephony: signaling) is in the same stream (channel) as the data. It makes reinvention (upgrading) of the shell super difficult. Nobody can even agree precisely how to (for example) include column headers in a CSV stream.
The other way is "out-of-band", where metadata/signaling is done in a parallel channel. Newer shells can try to approximate this by using JSON when the pipeline is appropriate.
I kinda wish the latter (OOB) had taken hold back in the day.
I'm a little worried this encourages ignoring things that have been learned recently. But it is very hard to not be annoyed when you realize how much ground was covered early on.
If one distro can pass the test, then in principle, any distro can.
If two distros passed, then it is case closed.
The entire point of the argument is not "can Linux pass?" but "will anyone pay to put Linux through the test?" And the answer is yes, repeatedly, although not any more, and who can blame them.
Unix has meant "passes the compatibility test" for ⅓ of a century. Linux passed. Once a Unix, always a Unix. It's Unix.
FWIW the time to protest was a year ago when I wrote the "Unix is dead" story. Here was the brief discussion:
>The specification encompasses the base operating system environment, networking services, windowing system services, and internationalization aspects and programming languages.
All distros MIGHT pass if they meet the qualifications but just because these two distros passed does NOT mean ALL Linux distros can pass and it doesn't mean any other distros can pass.
I disagree, and it seems to me that you are using words in senses other than their strict meanings, but I can see this is a big deal to you, and life is too short to argue over something which to me is unimportant.
Haha, yes. I was thinking more about things like containers, and the concepts around cgroups etc. that I think is currently a mess (from the user's point of view).
Yet another article fetishizing Plan 9 accompanied with a history lesson that doesn't support the conclusion.
No, not everything is a file. Having a somewhat standardised way of accessing certain features in a stream-like way is cool, but it's not the fundamental feature of any unix, it's just a cute piece of user-space. And there's better ways to evaluate a design than aesthetic appreciation.
Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge.
That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants.
Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.