While text editing interfaces could be more polished and modern--the underlying problem that causes your dissatisfaction is the text. Storing code as text is an evolutionary dead end. Storing code graphs in text requires the use of plaintext names for graph references, which binds logic and presentation together. The gap between the goal-completion logic a user comes up with and how they turn those mental instructions into plaintext is an unnecessary jump--an encoding--that produces no end of trivial but infuriating miscommunications.
Storing code as pure structure (not XML or anything silly, the serialization is trivial) avoids a huge class of artificial problems we've had to deal with since the dawn of compilers, due to the disconnect between human meaning encoded in plaintext and machine parsing.
Visualizing code as text--text as the primary view in the MVC, complemented with colours and annotations and hyperlinks and hints--is extremely useful. What we really need is a structural editor with the familiarity and ease of use of a text editor. But that is a hard problem, and more polished text manipulators are a nice stop-gap in the meanwhile.
Here's a problem: anyone who's ever used Microsoft Word has experienced random invisible bits of formatting stuck in the text somewhere, difficult to extricate, impossible to replicate, impossible to script. Anyone who's used GUI tools to edit data for long periods of time has experienced actions that are hard to perform in bulk-- that might be scriptable with some complicated API, true, but that would have been trivial to perform if the underlying data was in a text file, probably ending up as some search-and-replace.
If what I'm editing is not the original data, it had best be a perfect, 1:1 representation of that data; I should be able to edit any part of it with a regex without the fear of losing data or missing some annotation I can't see. Certainly in the case of an editor designed for programmers, that will mostly be true-- but there's a good chance that it won't be completely true, and then suddenly the editor is getting in my way and wasting my time.
I don't want to fight against technology. With current editors, I can get pretty much the same functionality, but all the metadata required to colorize text, tab complete, jump to definition, refactor, navigate by s-expr, etc. is just a cache, not something I have to think about.
"Microsoft Word has experienced random invisible bits of formatting stuck in the text somewhere"
In following the links to the paper on Femke, I found in that paper this expansion of the acronym Femke: "emphfunctional micro kernel experiment". I had to look at it a while before I realized that the markup had leaked into the result.
Then I though that "emphfunctional" might just be a very useful neologism, perhaps one that well describes the primary article's idealistic shell, but I'm not sure what it should mean yet.
These are the questions that plague the mind that is having trouble focusing on its work.
When I'm working with Lisp in Emacs using Paredit, I am not editing text. Instead it is an interface to the expressions directly. I add, remove, transpose, cut, paste, move around, up and down, all with whole expressions at a time.
If someone were to build a new structural editor for programs, I could only hope it is as good as Paredit.
It's not as hip as Lips/Emacs, but Java/Eclipse works in much the same way. IDEs with deep introspection capabilities are great. I hate all those mini refactorings when I'm programming Javascript or Coffeescript.
Exactly, a usable structure editor is extremely difficult to write.
The problems with Lisp are that your source code ends up flattened into plaintext in the end, and that identifiers and symbols must be resolved by string lookup; there is no "first-class" reference, just a collection of resolution rules and environments and symbol manglers. You could fix these, but you wouldn't really have Lisp anymore.
We code with text because it leverages our innate language skills. Words and grammer are how we communicate naturally. Typing in words and numbers transmits an enormous amount of information very quickly, compared to e.g. manipulating graphical controls. We use diagrams for very narrow domains of information, not for general concepts.
If you've got something better in mind, let's see it. I mean, let's see exactly how it is supposed to work. The general idea has been around a long time, but nobody has produced even a design that is compelling.
Typing in words and numbers transmits an enormous amount of information very quickly,
This reminded me of NASA 'Mission Control' used for Apollo.
To eyes used to the movies or video games the consoles look bizarre: there are no graphs, no icons, no _pictures_. All the data came in, and went out, as numbers, or text.
They did it like that because to the engineers on the console the numbers could be grasped more quickly and meant more than a picture.
Communication using numbers instead of pictures is how engineers failed to stop mission control from allowing the Challenger to launch and crash with extremely out-of-spec frozen O-rings.
Good post. Put my money where my mouth is. My replies in this thread all relate to the insights I've gained from a side project I've been picking at for four years now. But the editor is only one part of it which I've barely touched. I'll work on a blog post with illustrations.
I do completely agree with you that we ought to transition toward structural code editors (and therefore, I suppose, structural terminals), but as you say, that's a very hard problem -- I have yet to see one which isn't terribly clumsy to use.
VOPs in SideFX:Houdini are the best use of code-with-nodes I've ever used.
You 'write' your code as a network of nodes, then on the fly it is converted to vex (a houdini internal scripting language) and compiled, the result runs very quickly.
If you would like to check it out, get Houdini Apprentice and have a play with the 'VOPsop".
I think this is a cool idea, but it seems to necessitate a language with a pure tree structure, i.e. lisp without conventional line-comments. Perhaps code merges would be more enjoyable too if the SCM software operated on trees instead of text files with lines.
It doesn't necessitate pure tree structure. The visuals are like conventional programming languages even if the underlying structure is tree-like. (Which is essentially the case anyway for any AST-based language.) Comments and other metadata are annotated onto the AST. (For example, English identifiers. In this way, a module of code can have an overlay of names in English, a French overlay, etc. And renaming a function or variable doesn't cause a rebuild or break any references because nothing is referenced by human-language names internally, even though it appears as so in the editor.)
Yes, the idea is to unify the structure editor with version control. Having the entire history of how nodes were moved around in the code gives you perfect knowledge of what was moved where or changed into something else, unlike line-based editing, where the problem is AI-complete.
Visual programming for dataflows - as in a series of mapreduce jobs on Hadoop - is an excellent case for this kind of programming. If you haven't read it, check out this paper on PigPen from Yahoo Research: http://bit.ly/v6eqwq
s-expressions, then. High-end Lisp editors can do a good bit of structural editing based on the fact s-expressions are pure structure.
> The gap between the goal-completion logic a user comes up with and how they turn those mental instructions into plaintext is an unnecessary jump--an encoding--that produces no end of trivial but infuriating miscommunications.
I think this is the wrong idea. Coming up with the logic in the first place is the hard part; encoding it is trivial once the encoding is learned. The fact is, thinking logically long enough to come up with a nontrivial piece of logic for a computer to execute is hard for humans, as is turning that logic, in whatever form it's stored, into an intuitive mental structure.
S-expressions are still stored as text. Editing as structure is what IDEs do, but they serialize back to text in the end. This is the mistake. Smalltalk is closer to what I'm talking about. Code stored like git stores data, with hashes and version control.
> I think this is the wrong idea [...]
You're arguing about something else. I worded it badly. The logic is hard and the transformation into a series of instructions is hard. Once you're about to type those in, though, you should be having a conversation with the computer, as opposed to sending a series of tokens into a black box and wondering if it understood what you meant.
And in Smalltalk, applications lived in the image and were difficult to extract from it. They didn't work well with code outside the image.
That was a bigger deal when the Desktop was King. It might actually be an advantage for web applications: Everything in the image is trusted in your security model and nothing much from the outside can get in. Run the image in a virtual machine (think Xen) to enhance the iron box effect.
> Code stored like git stores data, with hashes and version control.
I don't... we already have git. What could be more 'like git' than git?
> Once you're about to type those in, though, you should be having a conversation with the computer, as opposed to sending a series of tokens into a black box and wondering if it understood what you meant.
There has been some slight work done towards visual development, where you create 'circuit-diagram software': wire up logical components with visual control flow (or visual data flow, perhaps). It never seems to catch on.
> There has been some slight work done towards visual development, where you create 'circuit-diagram software': wire up logical components with visual control flow (or visual data flow, perhaps). It never seems to catch on.
We've done this in the past. You can blame implementation, perhaps, but we've found that in our use case (3D graphics processing workflows), it starts to get too hard to understand what is occuring for any non-trivial "program", since you're bound by the constraints of how much information we decided to display visually, whereas with a general purpose domain-specific programming language you can choose how much information to surface, by being able to choose the level of abstraction you want (or not, as the case may be).
The reason we chose to go visually in the first place, is because someone has the bright idea that "anyone" should be able to create these workflows, but it ends up that a technically minded 3D graphics engineer does it anyway, and they just get frustrated by not being able to write a proper program/script :)
It strikes me that IC design systems started out oriented towards a 2d visualization scheme but have evolved towards a text representation. Chip design was once done by laying out components in a 2d plan view of the chip, but is now done in VHDL/RTL.
An opaque image is the wrong idea, yes. A structure editor necessarily makes the structure easy to extract and manipulate, like git's tooling makes it easy to work with the git DAG.
Think of code where every blob in the git DAG is one AST node. (Obviously this is horrendously inefficient, this is just an example.) No plaintext.
I'm not talking about that particular subset of visual systems like circuit diagrams or other awkward things like LEGO mindstorms. A good representation would still look like code as you know it, since that's proved to be such an information-dense and useful form. The "view" is the same, but the model has been separated from it.
Storing code as pure structure (not XML or anything silly, the serialization is trivial) avoids a huge class of artificial problems we've had to deal with since the dawn of compilers, due to the disconnect between human meaning encoded in plaintext and machine parsing.
Visualizing code as text--text as the primary view in the MVC, complemented with colours and annotations and hyperlinks and hints--is extremely useful. What we really need is a structural editor with the familiarity and ease of use of a text editor. But that is a hard problem, and more polished text manipulators are a nice stop-gap in the meanwhile.