Ideas for free
Starting from scratch

Friday 20 December 2024 at 16:00 CET

Sometimes, I have ideas, and while it’s unlikely I’ll ever pursue them, I can’t stop thinking about them. So I write them down. I hope to get them out of my head, and ideally, into the head of someone else’s, who might benefit from them. If you want to make this, or even just take one small idea from the pile, please do. It’s yours.

Code is text. It’s (usually) UTF-8-encoded characters, structured in a jagged grid, and then serialised as a set of files which relate to each other.

It doesn’t have to be.

Language

Code is a directed, cyclic graph. It’s a set of nodes and edges, following a known structure, in which nodes encompass and/or refer to other nodes.

We could serialise it that way, using a simple, standard, general format of typed nodes and edges.

The “structure” (the programming language’s syntax tree) would then be imposed on top as a kind of validation step, similar to type-checking. This structure would again be described in a standard format, i.e. a formalisation of the language grammar. (Tree-sitter’s grammar format might be a good place to start.) It would probably be a representation of the language in higher-order abstract syntax or an abstract semantic graph.

A node might not be small; it could contain a lot of binary data, e.g. an image. This should not be that scary, and the tools should allow for it.

I have no idea how the C preprocessor, or other text-based pre-processing such as M4, would work here. My assumption is that they simply wouldn’t. Advanced manipulation of the syntax tree such as Agda’s mixfix operators, might also prove to be very difficult.

Persistence

If we are to go down this road, nodes and edges would have unique, random identifiers so that they can be referred to and tracked, and a way of specifying a unique path to a particular node or edge. Edges might not be “real” in the persisted sense; perhaps nodes will just be able to reference other nodes. These relationships might be cyclic; for example, two functions might call each other in a mutually recursive manner.

Perhaps a single file would encapsulate the entire program or library; there’s no reason to save multiple files in this situation. (One could imagine an opaque block in which nodes and edges are allocated space, becoming fragmented over time, similar to a classic file system.) Or, for simplicity, every single node and edge would become a file in a kind of incomprehensible file tree (similar to an addressable store such as Nix, but probably based on identifiers, so not content-addressable).

There’s plenty of good reasons to err on the side of simplicity, with a text-based format and a file per node, but: I prefer to push for a new binary format, in which nodes and edges are represented compactly, in a single file. This would allow an editor or compiler to feasibly memory-map an entire program and allow for very fast access to any part of it.

I could, of course, be totally wrong, and there may be good reasons to support both, similar to how Protocol Buffers supports both text and binary formats.

Why would we give up text?

One of the best things about programming is that free, open-source technology won. It’s not even a debate any more. Even Microsoft products are open-source, kind of.

And let’s be clear: to be open-source, you have to be able to read the source. It must be text. You must be able to independently produce the product from the source code, or it does not count.

So I want to be absolutely clear: I am not proposing a new, closed, binary format, any more than UTF-8 is not a closed format that maps characters on top of bits and bytes. I consider this graph format to be about the same level of complexity as a new character set.

Editor

If the code is a graph, then the editor must edit a graph.

This doesn’t mean the programmer has to type a graph; they might still write code as we currently do, but it’d be parsed upon saving it to disk. The first level of parsing would become part of the serialisation, not the compiler.

Syntax highlighting should be basically free.

It’s OK for people not to have the code in a valid state at all times. I expect there’ll be a special kind of node, “unfinished”, marking some work-in-progress that’s just text and not parseable. Comments too.

The rendering would be configurable; the editor should indent, wrap, etc. however the programmer prefers. Tabs vs. spaces becomes a preference, not a debate. You might also imagine the programmer customising how the code looks to e.g. show or hide parentheses or comment boundaries, use indentation or braces… it doesn’t matter, because it’s all local and parsed away before persisting. (Perhaps there’d be an override to specify specific layout, if necessary.)

With total freedom on presentation and layout, the editor could present the code in many different ways; Peter Hilton puts forward a few ideas in his article, Beautiful Code.

You could imagine much more advanced ways of manipulating code, similar to IntelliJ IDEA’s refactorings, to be written on demand in a small DSL (such as Comby’s simple DSL, though one could imagine more complex transformations).

Prototyping

Marce Coll’s wonderful essay on Exploratory Programming speaks about “rigidness”:

The problem is that most development environments are not designed to help with the exploratory nature of this way of mapping a problem space. They seem to be designed around what the end-program needs to look like. Well-structured files in clean hierarchies and packaged within cleanly delimited interfaces and modules. Who can write those from the start? My guess is no-one.

We write a function. And then we write another one. We want to write a third, but a little angel on our shoulder whispers, “Shouldn’t we do that in a new file?”

Segmenting code is often a good idea, but not when we’re prototyping; we need the freedom to experiment without the constraints. While I will not claim that this approach to structure and editing will solve all these problems, it will make it easier, because we get rid of the file.

Instead, we could potentially consider zoom levels: perhaps I want to look at a single class in isolation, but maybe it’s more useful to look at an entire module and scroll through it. (You could consider this similar to Code Bubbles.) I should be able to do both: this is now a local rendering problem, not an achitectural one. In addition, more advanced refactoring tools would, in theory, make it much easier to start simple and “do the right thing” once we’ve figured out what we’re doing.

(We’re just looking at code-as-written here, not running code. Lisp and Smalltalk programmers are, as always, way ahead of the game.)

Compilers and Tools

Tools such as compilers, linters, etc. would be written to work on the syntax graph, not text, which mostly just removes a step.

Autoformatters become unnecessary.

Tools often need to refer back to the source. As there are no line numbers, we’d use node and edge paths.

It should be easier, in this format, to track changes and how they impact entrypoints such as tests. We could therefore imagine a test runner that watches for changes and runs tests for the changed behaviour constantly.

Feedback to the programmer in the editor could, hopefully, be a lot faster, as the editor is doing part of the work (and highlighting the relevant areas instantly when something doesn’t parse). Further feedback would require tool integration using a protocol similar in nature to the Language Server Protocol (LSP), but oriented around graphs, not text.

If we’re imagining a complete redesign of compiler front-ends, perhaps this is a good opportunity to demand that every compiler needs to be a compile server with a CLI attached. Many language compilers do not support this, and as such, the LSP facades are slow and brittle.

Version control

This graph structure does not lend itself well to text-based version control systems such as Git. We need something else.

I don’t know what it’d look like, only that you’d need to start again. Maybe you’d build it on top of an existing VCS, but it seems like that’d end in tears.

Departing from the existing VCSes means we could look into real-time version control, where changes are rapidly synced as they’re typed, allowing programmers to decouple version-control-as-checkpointing (commits) vs. version-control-as-backup, as demonstrated by Jujutsu, or version-control-as-collaboration.

Diffs

Structural/tree diffing is an area of active research. There are a number of known very good algorithms, e.g. Difftastic, the Fantastic Diff.

Because nodes have IDs, you’d be able to see in the diff that e.g. a function moved from one module to another, and one line was changed inside it.

Renaming a module or class should no longer result in a gigantic diff; it’s just one change, and the edges stay the same.

Collaboration

We all know that pull requests are miserable. We can do better.

We could follow changes as they’re made by our colleagues and synced with the version control system, and perhaps leave comments for them or work alongside them on the same branch (like the Visual Studio Live Share extension for VS Code).

If we’re reviewing after the fact, why not view the diff in the same editor we use for editing, and even allow the reviewer to modify the code or run examples or tests?

And if we’re redesigning everything, why not start with collaboration in mind?

What are we trading off?

I’m imagining a different world here. As with any kind of science fiction (no matter how dry), there are good reasons why this might not work. We already have an incredible number of free tools that let us work with text in many, many different ways, and many translate across languages. I don’t need to write a new plugin to sed or grep to get them to work with a new language; they just do.

I think that a language-agnostic graph format would allow for the same kind of tools to be written, but they don’t exist yet. And we’d be able to write new tools that operate at a higher level across code. As an example, every LSP server needs to reimplement refactoring operations such as renaming a variable. In this format, we’d only need to write that once, and we’d get the capability across all languages (as long as they make good use of referencing variables through graph edges, rather than by name).

Past that, I’m really not sure whether this would work or not. I don’t think anyone can be sure about it. Consider this a proposal for an experiment, with a hypothesis that the advantages would outweigh the disadvantages; we can’t know the results until we try it.

Down with the Integrated Development Environment!

All of the above might give you the impression that I am campaigning for an entirely new ecosystem. While I hope that’d be the end state, I don’t really expect anything of the sort to emerge overnight. I am not expecting a company to invest a gazillion dollars into a new compiler, IDE, and the associated trimmings.

Instead, I’d prefer if we moved away from IDEs altogether, and towards what I like to think of as CDEs, or “composable development environments”.

There is, broadly speaking, an axis on which we can measure code editors, which we might title integratedness: IDEs on one end, and simple editors on the other. Those editors might be something very nerdy (and wonderful) such as emacs or vim, or they might be very barebones, such as Microsoft’s Notepad (though I hear that’s had a revamp recently; do not tell me, I don’t want to know). But there’s another axis: the composability of the editor.

Typically, IDEs are extensible, but it’s difficult: you have to write a complex plugin using the relevant framework. And of course, Notepad is not extensible at all. But emacs and vim (or neovim)… they’re special. Sure, there are complex plugins there too, but the simplest integrations take a few lines of code. And because these editors are programmable, they are magic. Hard to learn, but incredibly rewarding once you do. Once you’ve learned a bit, you can trivially compose them with other powerful tools to make them more than the sum of their parts.

In the realm of the composable, we can swap out parts as we please. So we might start with persistence, and how we store a graph, as opposed to text, without changing the editor at all. Perhaps we’d write an emacs or neovim plugin to read and write from the graph structure. Once that’s done, we could look into using it for syntax highlighting too. For a long time, we’d probably persist both text and graph formats, relying on the editor to keep them in sync, so all other tools could keep working.

Next, we might look into version control, and then start treating the graph format as the canonical one. In the spirit of composability, this would be a separate tool, with optional editor integration.

And slowly, carefully, we could extend, modifying other tools to work with either format, until the text format becomes vestigial.

Perhaps, after all that, the IDEs would catch up.

This idea is free as in birds

If you like this idea, it’s yours. While I’d be happy to discuss it with you (and please get in touch!), and there’s even a non-zero chance I might get on board, it’s more likely I’ll wish you the best of luck, help a little when I can, and tell everyone I know how wonderful you are.

You can read more ridiculous ideas by browsing the series:

1. Starting from scratch
2. Structured archival, and the web as it once was
3. Search is broken

Thank you to Gary Verhaegen, Nat Pryce, Nick Chapman, and sleepyfox for reviewing this and offering lots of useful feedback.