A Lisp REPL as my main shell

What a computer user (and in particular a technically minded user such as a developer) does on a computer can be understood as one of three actions:

• Data collection and filtering.
• Data visualization (which I also call “inspection” in this article).
• Data transformation (which I also call “processing” in this article).

Opening a file or multiple files with a program, say, to play a music album, is in effect “collection + visualization”. Shrinking a bunch of photos is “collection + processing”.

I believe that most, if not all our interactions with a computer can be summed up in those three tasks.

More interestingly, there is a feedback loop between collection and visualization, which is very common when manipulating large bunches of files from a shell: the user first displays the file list to process, then filters it, displays the filtered list, re-filters if needed, re-displays it, and so on. When the file list is finalized, the user can process it and finally visualizes the result.

This feedback loop that’s best represented by this simple diagram:

It turns out that traditional shells are particularly poor interactive tools to deal with such a feedback loop.

To paraphrase the example in Howard’s presentation on piper, typical “collection + processing” in the shell happens with a pipeline and some control structures:

for S in $(systemctl --all | grep openstack | sed 's/\.service.*//' | cut -c3-)
do
    systemctl restart $S
done

In the above example, as is customary in the shell, the whole “collection + processing” step remains a blackbox to the user who can visualize the data only before and after, but never for the intermediate steps.

While we are at it, shell languages from the sh family have poor control structures, which make simple things like a for loop too cumbersome to write and riddled with pitfalls.

There is a common misconception that terminals and shells are inherently bound to each other, to the point that there is sometimes a confusion between the two.

• A terminal, actually a terminal emulator, is a program that visually emulates the hardware of the 1970s and 1980s, such as the VT100. These tools are by definition stuck in the past.

  (A common misconception is that they are fast. Ironically, they are not: emulators emulate the physical properties of the terminal such as the baud rate, and this limits the speed at which text can be printed.)

• A shell is a programming language interpreter. A REPL of a sort. It often embeds interactive features such as a prompt with history support.

Terminals have no reason to continue to be used in my opinion. Note that this does not mean we shouldn’t use “textual” interfaces, quite the opposite: textual data is a bliss to manipulate. But we can very well manipulate text, along with other types of data, in something other than a terminal, that is faster, prettier, more powerful. (Graphical Emacs is one such example.)

In the past I’ve discussed the drawbacks of using a terminal as an interface (see my article Eshell as a main shell). I won’t go into the details again, but allow me to summarize terminal-induced limitations of traditional shells:

• Can’t search the outputs.
• Can’t interact with output (like opening a path under the cursor).
• Can’t copy/paste without a mouse (barring a few hacks).
• Can’t navigate the prompts.
• Limited colors (barring a few hacks), formatting and rendering is convoluted and not portable.
• Can’t render anything beside text.
• Interface toolkits like ncurses can’t render structured widgets (to see what I mean, try selecting text in an ncurses frame: the selection will grab the whole line, beyond the frame).
• Slow text output.
• Poor font support (single size).
• No per-pixel elements (e.g. cannot draw a line separator).
• Prompts and their output cannot be folded, moved arround, etc.

I was an Eshell user for a while, then I switched to M-x shell. Both these non-terminal-based shells rid me of most of the aforementioned terminal limitations, since graphical Emacs is a full-blown graphical application.

But I still wasn’t satisfied, in particular with the programming language used by the shell (I’ve used Bash, Zsh, Fish and Eshell).

I fancy the Lisp programming languages, so why not use the language I’m most comfortable with, say Common Lisp?

For a long time I tried really hard to stick to the paradigm of what I call the “readline shells”. Almost all shells are developed with the idea of running in a terminal with a readline-based prompt (if not readline, then a similar interface). Even shells as exotic as Xonsh, Ammonite Shell, Oil shell or SHCL are developed with this paradigm in mind.

This can only perpetuate the problem: these shells can’t really escape the poor feedback loop problem I mentioned above because of the lack of interactivity options in the user interface. And it is still limited in terms of rendering, prompt navigation and manipulation, etc.

At this point, it became clear to me that I was still trying too hard to follow the status quo. What if I started thinking the other way around? Instead of trying to bring Lisp (or your favourite programming language) to the shell, why not bring the shell to the Lisp REPL?!

Turning the problem around has many immediate benefits: it gets us (for free) top-notch programming tools and features such as a debugger, a stepper, interactive stacktraces and, very importantly as we will see below, an inspector. From there, we can implement what traditional shells are specialized with: process management, convenient input-output redirection, pipelines… (It’s surprisingly not that much!)

It seems much easier this way than the other way around!

Like many shell aficionados, your favourite sport might be to customize your prompt (also know as PS1) :)

The following screenshot showcases my prompt customization in SLY:

It looks like a rather regular shell, but let’s not jump to the conclusion too fast, as there are many subtle and yet important features at display.

• It’s a multi-line prompt. The first line indicate the path. The second line has two noteworthy elements:
  • The 0 is the back-reference of this prompt result (if any). We will come back to it in a while.
  • The $ is the current Common Lisp package (called “namespace” in many other programming languages), here my own “shell” package. This is great because this means that we have namespacing support right at the prompt!

Here SLY prompt customization frees itself from the shell cryptic syntax by using regular Lisp (for instance colors are referred their actual name like “green”). Instead of being just a string, my prompt can be generated dynamically with Lisp code!

Since SLY runs in Emacs, you get all the features of a powerful text editor right at your prompt!

• Search the whole REPL with incremental highlighting.
• With helm-occur or similar, you can list the search matches in a new window, narrow them down, navigate them (press C-c C-f), etc.
• Use you favourite keybindings, either CUA, Emacs-style or VI-style (with the Emacs Evil and Evil Collection packages).
• Smart S-exp manipulation with Paredit, Lispy or similar, which is very handy to write and manipulate Lisp code smoothly and rapidly.
• Keyboard macros.
• Multiple cursors.

Considering how extensible and mature Emacs is, the list goes endless…

Typically, multiple shells don’t share the same underlying system process, which means that if you define a function or variable in one shell, it one be seen in others.

While it can sometimes be useful to isolate shells from each other, other times I wish I could share code and data between my shell instances!

SLY supports “multi-REPLs” out of the box. When you open a new REPL, you can decide whether to start a new process or reuse an existing one.

Another benefit is that REPLs sharing an instance use the memory of only one process. More on that in the section on Size and memory usage.

Maximum flexibility, maximum power.

If you use the shell a lot, and if you use it both for regular shell usage and your programming projects like I do in Common Lisp, it’s only too easy to get lost between the many shell windows.

So I worked on a Helm extension called Helm Selector.

It allows you to fuzzy-search among all SLY REPLs. You can group even them by inferior Lisp process.

More importantly, you can select multiple REPLs and run an action on all of them in one go, such as deleting them or restarting them.

Notice that this “selector” also gives you the list of Lisp files and all related windows, such as the debugger and compilation results windows.

Tip: You can configure the appearance of the shell listing with various information, such as the Lisp compiler being used, the name of the window (“buffer” in Emacs parlance), etc. Example:

(defun ambrevar/helm-sly-format-connection (connection buffer)
  (let ((fstring "%s%2s  %s"))
    (format fstring
	    (if (eq sly-default-connection connection)
		"*"
	      " ")
	    (helm-sly-connection-number connection)
	    (replace-regexp-in-string
	     "*$" ""
	     (replace-regexp-in-string
	      "*sly-mrepl for " ""
	      (replace-regexp-in-string "*sly-inferior-lisp for " ""
					(buffer-name buffer)))))))
(setq helm-sly-connection-formatter #'ambrevar/helm-sly-format-connection)

An essential feature, in my opinion, is the ability to “go to a given prompt”. When the output of a command is long, it can be cumbersome to seek back to the previous prompt (it’s only worse if the output is heavily colored, since then the prompt does not stand out so much then).

In SLY, you can move the cursor to the various prompt in your REPL with sly-mrepl-previous-prompt and sly-mrepl-next-prompt.

It only gets better: with helm-comint-prompts-all, you can list all prompts of all REPLs, fuzzy-search them, narrow down live and finally confirm to go to the desired prompt.

With this weapon in your hands, you won’t ever lose a prompt input nor its output again!

Tip: Since it’s not bound by default, I like to bind it to M-s f:

(define-key sly-mrepl-mode-map (kbd "M-s f") 'helm-comint-prompts-all)

In Searching and editing we talked about searching the whole REPL. Sometimes it can be useful to restrict the search to a single or select outputs.

One way to do this is to “narrow” the REPL to the desired prompt. If I place my cursor on a prompt or its output and press C-x n d (narrow-to-defun), all other prompts and outputs disappear (only virtually), thus restricting searches and other commands to just what I see.

To display the whole buffer again, press C-x n w (widen).

A back-reference is like an automatic variable that is assigned to every single result of all prompt commands.

(GIF from the SLY GitHub page.)

Without back-reference, you’d have to systematically store the results in well-defined names. In Bash, you could do this:

$ v1=$(command1 ...)
some output...
$ v2=$(command2 ...)
some other output

This quickly gets cumbersome. If you get the number wrong, you may skip a number (which would be confusing) or accidentally overwrite a previous result.

Automatic back-references solve all these issues.

Back-references play a crucial role in rethinking how a shell can be used. See the following section Better than pipelines: graphs! for an introduction.

Some jokingly call cd the poor man file manager! :) It’s probably only justice: It’s slow, cumbersome, inefficient. We can do much better.

I believe everyone should be able to use their favourite file manager. Good news: with SLY, it’s possible to “change directory” to the one corresponding to what your file manager points to!

As a big fan of Helm, I use helm-find-files as a file manager. I find navigating directories with it really nice since you can fuzzy-search the directory name, no need to type the name precisely.

Together with the Helm Switch to REPL extension, pressing M-e from anywhere in helm-find-files will switch the desired REPL to the corresponding directory.

Examples:

• To go 3 parents up, I press s-f C-l C-l C-l RET.
• To go to a previously visited directory, I can press M-p to prompt the fuzzy-searchable history.
• With helm-locate, I can fuzzy-search any file and directory, anywhere on all my hard drives, within a finger snap. Then I press M-e to switch to the directory of the selected file.

I use Helm to add live, narrowing-down fuzzy completion to the history search.

Notice how matches are found regardless of the search term order.

Since SLY does not use Helm by default, I simply replace the history binding with the corresponding Helm command in my Emacs config:

(define-key sly-mrepl-mode-map (kbd "M-p") 'helm-comint-input-ring)

In the traditional shell world, completion power is all the rage. Zsh and Fish boast amazing completion features.

With SLY, the topic is turned upside down since now you write Common Lisp, and you are given Common Lisp completion!

So you can complete Common Lisp functions, symbols, etc. You can also fuzzy-search any symbol, from a given package or any package, with sly-apropos (or helm-sly-apropos).

When calling a function, the signature is automatically displayed (thanks to eldoc) so that you know which arguments the function takes.

Common Lisp natively supports

• positional arguments
• optional arguments
• key arguments

which makes it advantageously nicer to use than the variety of inconsistent command-line arguments programs expose in the shell.

SLY also gives you completion against filenames. Take the following:

> (sha1 "/path/to/dir/<|>")

In the above <|> represent my cursor position. If I press Tab here, it will prompt a list of possible completion which I can fuzzy-search!

You can also trigger file completion via other means, like after a #p (which is the syntax for a pathname in Common Lisp). See this discussion.

Moreover, you don’t even have to use completion to insert a path. Instead, you can use your file manager to insert the selected file path in the REPL.

Helm supports these key bindings by default:

• C-c i: Insert full path at point.
• C-u C-c i: Insert short (possibly relative) path at point.
• C-u C-u C-c i: Insert base name.

I’d recommend remapping these keyboard shortcuts considering how useful they are.

What most good text editors provide is interactive documentation: Point at a function and display its documentation!

SLY also has sly-apropos which searches all known symbols (functions, variables, classes, etc.). And helm-sly-apropos which does the same but with live fuzzy completion.

Bash and friends don’t have such approachable features. Man pages don’t really compete here in my opinion, especially if they prevent you from accessing the prompt! You could use two terminals side-by-side, one reserved to man calls… Or some other niftier trick.

A killer feature of SLY is its inspector.

As a simple example, let’s list a bunch of files:

<7:$> (finder ".")
(#<FILE 02. And the Day Turned to Fright (Eat Static Remix).mp3 {10052FF613}>
 #<FILE 04. A New Way to Say Hooray (Prometheus Remix).mp3 {1005301933}>
 #<FILE 04. Without Thought (Youth Remix).ogg {1005303B93}>
 #<FILE 06. Dorset Perception (Total Eclipse Remix).mp3 {1005305BA3}>
 #<FILE 06. Timeless E.S.P.ogg {1005307D53}>
 #<FILE 08. Aranyanyara (Abakus Mix).ogg {1005309AC3}>
 #<FILE 08. Once Upon the Sea of Blissful Awareness (Esionjim Remix).mp3 {100530B973}>
 #<FILE Dialogue of the Speakers - Back.jpg {100530E0E3}>)

Here we have a list of FILE objects. Pressing Enter on the result opens up the inspector:

#<CONS {1005301927}>
--------------------
A proper list:
0: #<FILE 02. And the Day Turned to Fright (Eat Static Remix).mp3 {10052FF613}>
1: #<FILE 04. A New Way to Say Hooray (Prometheus Remix).mp3 {1005301933}>
2: #<FILE 04. Without Thought (Youth Remix).ogg {1005303B93}>
3: #<FILE 06. Dorset Perception (Total Eclipse Remix).mp3 {1005305BA3}>
4: #<FILE 06. Timeless E.S.P.ogg {1005307D53}>
5: #<FILE 08. Aranyanyara (Abakus Mix).ogg {1005309AC3}>
6: #<FILE 08. Once Upon the Sea of Blissful Awareness (Esionjim Remix).mp3 {100530B973}>
7: #<FILE Dialogue of the Speakers - Back.jpg {100530E0E3}>

Each element is recursively inspectable, which in effect allows me to navigate the whole structure of the result that was printed at the prompt!

So if I press Enter on an element, I now get:

#<FILE {100530B973}>
--------------------
Class: #<STANDARD-CLASS AMBREVAR/FILE:FILE>
--------------------
 Group slots by inheritance [ ]
 Sort slots alphabetically  [X]

All Slots:
[X]  ACCESS-DATE       = @2021-01-24T18:19:50.000000+01:00
[X]  CREATION-DATE     = @2021-01-24T18:19:14.000000+01:00
[ ]  GROUP-ID          = 998
[ ]  INODE             = 19738333
[ ]  KIND              = :REGULAR-FILE
[ ]  LINK-COUNT        = 1
[X]  MODIFICATION-DATE = @2020-09-07T12:44:11.000000+02:00
[ ]  PATH              = "/home/ambrevar/projects/fosdem2021/music/collection/08. Once Upon the Sea of Blissful Awareness (Esionjim Remix).mp3"
[ ]  PERMISSIONS       = (:USER-READ :USER-WRITE :GROUP-READ :OTHER-READ)
[ ]  SIZE              = 6635729
[ ]  USER-ID           = 1000

[set value]  [make unbound]

A FILE is a Common Lisp object, so here the inspector lists the object “slots” (sometimes known as “attributes” in other programming languages). The slot values are further inspectable.

Notice that I’ve selected some slots (the dates, marked with [X]). If I click on [set value], I can change the selected slot values

The first and foremost feature of a shell is its ability to run commands.

Interestingly, not all programming languages can do process management properly. Thankfully, Common Lisp is rather complete in this area:

• Execute processes directly, without relying on a shell.
• Execute processes asynchronously.
• Print process output live.
• Set the standard input, standard output and standard error. Common Lisp streams can be used here.
• Connect the output of a process to the input of another one (efficiently).
• Collect the exit code.

The de-facto “process manager” in Common lisp is launch-program (from the UIOP library), but its name alone is too long to type to quickly run commands.

Syntax really matters here: the raison d’être of Bash and friends, after all, is that they have the shortest syntax possible for executing a program with arguments.

The good news is that we can get just 1-2 character close to this conciseness in Common Lisp, due to the language meta-programming capabilities and the ability to customize its reader. This means that running ls -l can be written as

> #! ls -l

or even

> !ls -l

depending on your taste.

(In Common Lisp it’s good practice to customize the reader only for strings starting with #, so #! is preferred here.)

I wrote an example implementation of this syntax in my dotfiles.

SHCL has a more advanced syntax table with the possibility to interweave Bash and Common Lisp expressions.

This syntactic sugar is key here to ease adoption and transit from a traditional shell to a REPL.

Indeed, many of us still use Bash, many code snippets on the Internet rely on a sh-like shell, so it’s important to be able to run these code snippets seamlessly in our REPL as well.

This can only smooth the transition and ease the adoption.

In the line of the previous section, while we can preserve back-compatibility with Bash commands, it would be nice if we could do the same with “visual programs” (to use Eshell terminology), such as ncurses programs like htop.

Again, this would make the transition smoother for those who would like to venture out of the traditional shell territory.

Talking about Eshell, the latter has a nice workaround: it knows a list of “visual program names”, and every time the user tries to input a command starting with one of those names (e.g. htop), Eshell will automatically forward the execution to a preferred terminal like Xterm, or Vterm if you’d like to stay in Emacs.

I’ve implemented a similar workaround in Common Lisp:

https://github.com/ruricolist/cmd/issues/10

All that said, my opinion is that we should ultimately not have to rely on decade-old hardware emulators for user interfaces, and all your favourite ncurses programs should have a fancier, programmable alternative (in Lisp or any modern language).

If we can run Bash commands directly in our Lisp REPL, (for instance by prefixing them with #!), then we can run Bash pipelines.

But what about pipelines using a Lispy syntax? Still with little typing?

With the cmd- helper I wrote (which should be published at some point), it’s just as easy. To remain Common-Lispy, I decided not to use the reserved | for pipes. We could have use a single character like ! but I’ve opted for :- for various reasons. (This may change if this gets published as a library.)

As above, I’ve leveraged the editor to help me insert the extraneous characters:

(defun ambrevar/sly-insert-double-quotes ()
  "Convenient to write list of strings, e.g. when writing a shell command line."
  (interactive)
  (while (sly-inside-string-p)
    (forward-char))
  (cycle-spacing)
  (insert "\"\"")
  (backward-char)
  (when (and (boundp 'evil-state)
	     (not (eq evil-state 'insert)))
    (call-interactively #'evil-insert)))

(defun ambrevar/sly-insert-pipe ()
  "Convenient to write a `:-' pipe."
  (interactive)
  (while (sly-inside-string-p)
    (forward-char))
  (newline-and-indent)
  (insert ":-")
  (ambrevar/sly-insert-double-quotes))

(define-key sly-mrepl-mode-map (kbd "S-SPC") 'ambrevar/sly-insert-double-quotes)
(define-key sly-mrepl-mode-map (kbd "C-S-SPC") 'ambrevar/sly-insert-pipe)
(define-key sly-mrepl-mode-map (kbd "<C-M-return>") 'ambrevar/sly-insert-cmd-) ; Defined as for `ambrevar/sly-insert-cmd'.

Now if I want to write

(cmd- "sort" "./dict"
      :- "uniq" "-c"
      :- "sort" "-nr"
      :- "head" "-3")

all I’ve got to press is

C-M-return
sort
S-SPC
./dict
C-S-SPC
uniq
S-SPC
-c
C-S-SPC
sort
S-SPC
-nr
C-S-SPC
head
S-SPC
-3

which is, again, just one key press longer than with Bash.

(And “./dict” could be completed since we have file completion.)

The above still feels very “sh-ish” as a pipeline: you need to write the whole thing in one go. It’s still cumbersome to write and even harder to get right from the first shot. What we lack here is a more iterative approach to sequencing the various steps of the pipeline.

Remember the diagram at the beginning? What a pipeline essentially does is “collect → process”, possibly multiple times. Crucially, it lacks any “visualization” step. While you can hack it in with a bunch of tee commands duplicating the various outputs, it’s so cumbersome to write that you’ll rarely bother unless you are writing a proper script.

This is were Back-references come to play their crucial role: we can call each step separately and pass the output of one command to the input of another by using the appropriate back-reference.

This is very handy since it allows us:

• to inspect the intermediate results at any point;
• to correct the intermediate result before passing it on, without having to re-run the previous commands.

What’s also better than pipelines here is that the various steps don’t have to form a linear pipeline: since various sources of data can be combined at any point, it forms a graph!

Or, preferably, a direct acyclic graphs (DAG). Allowing cycles in a process graph may result in hard-to-debug issues (like dead locks or non-reproducible results).

In SLY, back-references already allow us to compose the execution of processes as a graph. For an example, see User stories.

But we could have hoped for something more declarative.

As we saw in Searching and editing and Inspecting and editing, we can search and edit with all the power of Emacs.

This makes most filtering tools obsolete, like grep, cut, head, etc.

Maybe you’d think that more advanced text processing programs like AWK still have their use. But even there, the Common Lisp alternative CLAWK provides both support for the original syntax and a more Lispy syntax:

(for-file-lines (filename)
  (with-fields ((name payrate hrsworked))
    (when ($> hrsworked 0)
      ($print name payrate hrsworked))))

If you are not a nostalgic of AWK, maybe a dumb-simple string tokenizer will do:

(defun tokenize (string)
  "Return list of STRING lines, where each line is a list of each word."
  (mapcar (lambda (line)
	    (sera:tokens line))
	  (str:split (string #\newline) string)))

(defun token (line column lines)
  "Return token at line LINE and column COLUMN in the list of strings LINES."
  (nth column (nth line lines)))

then selecting the third column of the second line of the output of ls -l is as easy as:

(token 1 2 (tokenize ($cmd "ls -l")))

Here again, this shows how the Common Lisp language is vastly more powerful at text processing than any shell tool.

We just saw how Common Lisp makes many Unix tools obsolete. Let’s push it further: what about the filtering Unix tools like sort, uniq, grep (for filtering and not searching this time), etc.?

Here the Common Lisp language, even without any library, has many nice functions on offer:

• sort which accepts a key and a predicate parameter.
• remove to remove elements matching the given item.
• remove-if to remove elements matching a predicate.
• remove-duplicates which, unlike uniq also works on non-adjacent data.

And more sophisticated list/set manipulation helpers:

• set-difference
• union
• intersection

We will see how these helpers prove useful (and in my opinion superior to Unix tools) in the User stories section.

After Executing external commands, maybe the most common thing to do in a shell is operate on files.

Interestingly, this is something Bash is very bad at. In particular, it falls on its face as soon as it hits files with whitespace (or worse, line breaks) or if we must deal with files nested in different sub-directories.

The deep reason behind this weakness is because Bash pipelines can only pass text from process to process, and not structured data. This means that if we want to filter data, the file collection process (e.g. find) must return the file list as a string buffer: we just lost the structure of the file list. An alternative which is supported by some find versions is to use \0 as a separator, but then either the receiving program must support this, or you must use a version of xargs which also understands \0. In any case, it’s cumbersome and limiting.

In Common Lisp, we don’t need to start a new process to do any filtering, so it doesn’t suffer from this limitation. We can use any data structure we like: lists, hash tables, sets, etc.

One library that seems to be missing from the Common Lisp ecosystem is a find alternative, coupled with a file class which would embed properties like the Unix attributes and more, in an extensible way.

So I wrote the ficle library which implements exactly this.

I’ve shown an example use in Inspecting and editing.

There are many great benefits to this finder helper compared to Unix find:

• Predicates can be arbitrary: so I can write my own predicate, say match-date>, that matches files newer than the given period, and thus

  (finder "/path/to/dir" (match-date> 60) (match-extension "txt"))
  

  finds text files newer than 1 minute ago.

  By default, the predicates are joined by a logical and (meaning they must all be satisfied. If you want a logical or instead, you can make use of the disjoin higher-order function (from the Alexandria library):

  (finder "/path/to/dir" (disjoin (match-date> 60) (match-extension "txt")))
  

  This time we find all the text files as well as all the files newer than 1 minute.

• It’s more readable and flexible than find! :)
• The find syntax is hard to remember. With finder, I can complete against any function starting with match- or from the ficle namespace.
• finder returns a list of file objects. It’s structured and inspectable.
• If the resulting list is not exactly what I want, I can further filter this result (using a back-reference and without recalling finder) using remove and friends, or simply by editing the list by hand if that’s enough. Indeed, hand-editing of an S-expression remains “structured” thanks to the editor support (e.g. Paredit, Lispy).

The file class can be specialized against specific types of files. For instance, the mediafile class contains more information, like the MIME type and all the information returned by ffprobe (the information tool from the FFmpeg suite).

Now we can inspect the audio tags, the video codecs, the picture resolution, etc., by just using the SLY inspector. No need to even learn an API here, it’s explorable from the interface itself.

With the introduction diagram, we discussed the feedback loop between data filtering and visualization. Good tools are key in order for this loop to roll smoothly.

The traditional shell practice of groping around with piped Unix tools until we get the results right is certainly as frustrating as it is unproductive.

Previously, we’ve shown some more efficient ways to filter data iteratively. What we haven’t talked much about yet is how to visualize this data.

We’ve shown how the SLY inspector can inspect data recursively. While extremely useful, it does not allow for other representations. For instance, interactive tables could give us an overarching view of sequences of data, such as a file list.

To work around this, I’ve implemented a small Emacs widget. Now I can call it on my file list and it displays the following tabulated list view:

Notice that I can sort each column (and reverse the sort). I can apply interactive filters to this list, for instance a filter which prompts the file type to keep, or the date range to exclude.

In the future I’d like to be able to send the filtered and sorted list back to SLY. This is very much a work-in-progress, but it shows how we can communicate back and forth with Emacs and leverage this communication to use Emacs and its widgets as an interface to display the data.

That said, Emacs itself is also somewhat limited in terms of widgets: no real support for interactive graphs (beside maybe the chart library), limited support for pictures, no support for videos or 3D rendering.

But Emacs is not the end of it.

I wanted to play music files and videos found recursively in sub-directories, at random.

This is surprisingly hard to do in Bash. (Exercise left to the reader :p)

With SLY, cmd, my aforementioned finder helper and Alexandria’s shuffle, it’s a one-liner:

(apply #'cmd "mpv" (shuffle (finder "/path/to/media/library/")))

In Better than pipelines: graphs!, we talked about an important paradigm shift in how to use a shell. Rather than writing pipelines, we can write the various steps separately and combine them as we go.

Let’s see what it gives us with a practical example. The following session is a real life task I had to do some weeks ago.

I had two versions of a game files, let’s call them superdiffer-v1 and superdiffer-v2.

They have many files in common, some files are new in v2, some files have been modified. I wanted to remove all identical files from superdiffer-v2, in order to make a v1-to-v2 patch.

Should be easy, right? Turns out it’s pretty tough to get right with conventional Unix tools. I tried with rsync but couldn’t figure out the right incantation.

I gave up with the shell and did it all in Common Lisp. Here follows a recording of the session, including the mistakes. I’ve interspersed it for comments to clarify what I was trying to do:

(finder "/path/to/superdiffer-v1")
;;    (#P"/path/to/superdiffer-v1/Manual.html"
;;     #P"/path/to/superdiffer-v1/Readme.txt"
;;     ; ...
;;  )

(finder "/path/to/superdiffer-v2")
;; (#P"/path/to/superdiffer-v2/ChangeLog"
;;  #P"/path/to/superdiffer-v2/Manual.html"
;;  #P"/path/to/superdiffer-v2/Readme.linux"
;; ...)

;; Optional: Here I decided to name the results with easier-to-remember names,
;; but of course I could have just kept using the back refernences.
(defvar source #v1)

(defvar target #v2)

;; Collect the files relative paths paired with their checkum.
(mapcar (lambda (p) (list p (checksum p))) (mapcar #'relative-path source))
;; ((#P"superdiffer-v1/Manual.html" "83483fc4155ec32038fc5f2f0c5c56a205bf03d6")
;;  (#P"superdiffer-v1/Readme.txt" "13b3cc1f97c5f871b550ae0ee722256ef12a8ef4")
;; ...)

(mapcar (lambda (p) (list p (checksum p))) (mapcar #'relative-path target))
;; ((#P"superdiffer-v2/ChangeLog"
;;   "094db450c8352b3a5687e63984eb59ccb89f2533")
;;  (#P"superdiffer-v2/Manual.html"
;;   "83483fc4155ec32038fc5f2f0c5c56a205bf03d6")
;;  (#P"superdiffer-v2/Readme.linux"
;;   "d2481dc94eedb19f56277124889c7130ee73d9e7")

(defvar source+checksum #v5)
(defvar target+checksum #v6)

;; Oops!  The relative path is wrong, we need to store the path relative to
;; the game directory itself.
;; No problem, since we proceed iteratively, we can fix the previous result
;; without having to start over, and without having to recompute the checksums.
(mapcar (lambda-match
	  ((list path sum) (list (relative-path path "./superdiffer-v1/") sum)) )
	source+checksum)
;; ((#P"Manual.html" "83483fc4155ec32038fc5f2f0c5c56a205bf03d6")
;;  (#P"Readme.txt" "13b3cc1f97c5f871b550ae0ee722256ef12a8ef4")

(mapcar (lambda-match
	  ((list path sum) (list (relative-path path "./superdiffer-v2") sum)) )
	target+checksum)
;; ((#P"ChangeLog" "094db450c8352b3a5687e63984eb59ccb89f2533")
;;  (#P"Manual.html" "83483fc4155ec32038fc5f2f0c5c56a205bf03d6")
;;  (#P"Readme.linux" "d2481dc94eedb19f56277124889c7130ee73d9e7")

;; Since we've _visually_ verified the result is correct, let's update our
;; references:
(setf source+checksum #v9)
(setf target+checksum #v10)

(set-difference target+checksum source+checksum :test #'equalp)
;; ((#P"data/libssl.so.1.0.0" "00df752e95496f3f68f3938e0478942b6d2c124f")
;;  (#P"data/libdraw.so" "1b4847cf117190e8c8de6cddabdf36c61797c2e9")
;;  (#P"data/libcrypto.so.1.0.0" "9ef2f7749ebd3d42b7c6044e2aa3d1b4732cfae3")

;; Now let's take the complement:
(set-difference target+checksum #v14 :test #'equalp)
;; ((#P"palettes/enemies/zamza7.pal" "72716fa0f509f8b1c8bd199ad22c2f30f70357fa")
;;  (#P"palettes/enemies/zamza6.pal" "3a9d2afa2d5ee1b90bdf8ff2650e13e25e3953dc")
;;  (#P"palettes/enemies/zamza5.pal" "051c1230d5beea757b9acfe70fe2867a6b9661ee")

;; Once we've visualized which data is going to be removed, we can proceed with
;; confidence!
(mapc (alex:compose #'delete-file #'first) #v15)

Hopefully the above walkthrough highlights the usefulness of back-references and list manipulation helpers (here set-difference).

This took me less than five minutes and, more importantly:

• The logic of the process flowed naturally, I knew there would be no blockers. The cognitive effort is much lower I believe.

  In Bash, I would have to resort to the expressiveness of tools like rsync or find, but if those are not able to do what I want, I could very well get stuck.

• Writing the Lisp was faster for me than just reading the manual of find or rsync.
• Errors that happened any time before the actual processing (the file deletion at the very end) can be corrected without having to start over.

The startup time of the REPL matters if you are going to fire up dozens of them.

An Xterm window running Bash starts in a fraction of a second and this is what we should be aiming for.

Personally, I found that a REPL starting in more than 1s would become frustrating over time.

The case of Common Lisp is interesting. Most implementation start up really fast by default:

$ time sbcl --no-userinit --quit

real	0m0.004s
user	0m0.000s
sys	0m0.004s

Sadly, this only gets us the language standard, which is very limited as a shell: no library management, no modern string manipulation, no regular expressions, no high-level concurrency library, etc.

If we want to add these features to the language, we must load third-party libraries, which means we must load ASDF first:

time sbcl --no-userinit --eval '(require :asdf)' --quit

real	0m0.094s
user	0m0.086s
sys	0m0.008s

Ouch! That alone added a significant overhead. Add to this some more libraries and you quickly end up with a REPL that’s much too slow to start.

So I worked on lisp-repl-core-dumper, a little tool that caches the initialization (technically, it “dumps a Lisp image”). Now if we use it to start an SBCL preloaded with Alexandria and Bordeaux Threads:

$ time lisp-repl-core-dumper -p 'alexandria bordeaux-threads' sbcl --quit
Running '/home/ambrevar/.cache/lisp-repl-core-directory/sbcl-2.1.0-alexandria+bordeaux-threads.image'.

real	0m0.032s
user	0m0.024s
sys	0m0.012s

Much better!

Pro-tip: SLY has an option which you can tweak to dramatically reduce the startup of an “mrepl”:

(setq sly-connection-update-interval 0.1)

See the variable documentation for the details.

If you are going to use many shells, memory usage might be a concern. This is were “mrepls” (multiple REPLs sharing the same instance) are a life saver.

But for the times you’d like to use different instance, or if you are going to run a REPL on a system with very low memory, you might want to save in both compiler size and memory usage. For this, different Lisp compilers can be good options.

The transitive size (package with all dependencies) of the compilers (except for ABCL) is generally very low:

$ guix size ccl
store item                                                       total    self
/gnu/store/kxiilibc62zxp0dj4ywg4gqw8nvvhp40-ccl-1.12               109.1    37.0  34.0%
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31              38.4    36.7  33.7%
/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib           71.0    32.6  29.9%
/gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16       1.6     1.6   1.5%
/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16     39.4     1.0   1.0%
total: 109.1 MiB

Compared to Bash:

$ guix size bash
store item                                                       total    self
/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31              38.4    36.7  43.4%
/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib           71.0    32.6  38.6%
/gnu/store/87kif0bpf0anwbsaw0jvg8fyciw4sz67-bash-5.0.16             84.6     6.3   7.4%
/gnu/store/zzkly5rbfvahwqgcs7crz0ilpi7x5g5p-ncurses-6.2             76.9     5.9   7.0%
/gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16       1.6     1.6   1.9%
/gnu/store/knp4rkdm39ph4brkbzsp07q248nfffi1-readline-8.0.4          78.3     1.4   1.7%
total: 84.6 MiB

But then Bash can be compiled statically:

$ guix size bash-static
store item                                                       total    self
/gnu/store/bcjcd97xvh0qkvq1maqj6qab88xb30dv-bash-static-5.0.16       1.6     1.6 100.0%
total: 1.6 MiB

Some compilers like ECL can compile down to binaries, which can save on disk-usage.

There is even an attempt to generate static executable with SBCL.

In terms of memory usage, I’ve measured the memory taken at startup (doing nothing), once without loading anything, and once just loading the implementation initialization file which was set to load ASDF. All tests are done with Guix System commit 51418c32d95d8188d8877616829f26479f1135c6. The results are in Kbytes as per GNU time.

$ command time -f '%M' bash -c 'exit'
3580

$ command time -f '%M' /gnu/store/kxiilibc62zxp0dj4ywg4gqw8nvvhp40-ccl-1.12/bin/ccl -n -e '(quit)'
18648

$ command time -f '%M' /gnu/store/kxiilibc62zxp0dj4ywg4gqw8nvvhp40-ccl-1.12/bin/ccl -e '(quit)'
43508

$ command time -f '%M' /gnu/store/lzfxjn036h3kis13lcc222rpwcnqazkr-ecl-20.4.24/bin/ecl --norc --eval '(quit)'
13336

$ command time -f '%M' /gnu/store/lzfxjn036h3kis13lcc222rpwcnqazkr-ecl-20.4.24/bin/ecl --eval '(quit)'
27656

$ command time -f '%M' sbcl --no-userinit --quit
23072

$ command time -f '%M' sbcl --quit
184160

$ command time -f '%M' /gnu/store/0r1x1pp2da4cilbj3y5bklr9b8y8z272-clisp-2.49-92/bin/clisp -norc -x '(quit)'
8544

$ command time -f '%M' /gnu/store/0r1x1pp2da4cilbj3y5bklr9b8y8z272-clisp-2.49-92/bin/clisp -x '(quit)'
13200

$ command time -f '%M' /gnu/store/16ajvwh94mrz8amghxvd9l2bz8n96pzr-abcl-1.8.0/bin/abcl --noinit --eval '(quit)'
260304

$ command time -f '%M' /gnu/store/16ajvwh94mrz8amghxvd9l2bz8n96pzr-abcl-1.8.0/bin/abcl --eval '(quit)'
597360

Conclusion: CLISP performs the best, ABCL the worst, ECL comes close to CLISP.

Benchmarks are always to take with a grain of salt. The following exponential-cost Fibonacci implementation is useful to measure just a few things: function calls, memory usage and simple arithmetic (here the addition).

All tests are done with Guix System commit 51418c32d95d8188d8877616829f26479f1135c6. on an AMD Ryzen 5 2600 processor.

POSIX sh Fibonacci implementation:

fibonacci () {
  if [ $1 -lt 2 ] ; then
    echo $1
  else
    echo $(($(fibonacci $(($1 - 1)))
	    +
	    $(fibonacci $(($1 - 2)))))
  fi
}

$ time fibonacci 30
832040

real	27m37.401s
user	23m54.411s
sys	4m47.767s

Notice it’s just 30 here! 40 (as in the following tests) would have taken much to long to be practical. Even though Bash can do slightly better, they are obviously not made for calculus. It’s an embarrassing limitation because this means that even simple computations can block a script forever while it would have taken a few seconds with a higher performance programming language.

Now to the Lisp measurements.

(defun fibonacci (n)
  (if (< n 2)
      n
      (+ (fibonacci (- n 1)) (fibonacci (- n 2)))))

;; SBCL
(time (fibonacci 40))
Evaluation took:
1.915 seconds of real time

;; CCL
(time (fibonacci 40))

(FIBONACCI 40)
took 753,161 microseconds (0.753161 seconds) to run.

;; ABCL
(time (fibonacci 40))

75.654 seconds real time

;; ECL
(time (fibonacci 40))

real time : 61.357 secs
run time  : 77.137 secs

;; CLISP
(time (fibonacci 40))

Real time: 117.89895 sec.
Run time: 116.89732 sec.

Both SBCL and CCL are performing really well here, so it’s unlikely that performance is ever going to be a problem when used as a shell.

CCL is both the fastest (here) and a good middle ground in terms of memory usage.

If memory is the bottleneck, CLISP or ECL are good alternatives.

In Performance, memory usage and startup time, I talked multiple Lisp implementations.

SLY lets you use any of these implementations and you can quickly jump between them. Here is my setup which leverages Guix to create independent environments:

(setq sly-lisp-implementations
      (let ((maybe-core-dumper (when-let ((exec (executable-find "lisp-repl-core-dumper")))
				 (list exec))))
	`((sbcl-ambrevar ("lisp-repl-core-dumper" "-p" "ambrevar" "sbcl"
			  "--eval" "(in-package :ambrevar/all)"
			  "--eval" "(named-readtables:in-readtable ambrevar/syntax:readtable)"))

	  (sbcl (,@maybe-core-dumper "sbcl"))

	  (sbcl-failsafe ("sbcl"))

	  (sbcl-nyxt (lambda () (ambrevar/sbcl-for-nyxt :no-grafts? t)))

	  (sbcl-nyxt-pre-inst-env
	   (,(expand-file-name "~/projects/guix/pre-inst-env")
	    "guix" "environment" "-l"
	    ,(expand-file-name "~/common-lisp/nyxt/build-scripts/guix.scm")
	    "--ad-hoc" "glib" "glib-networking" "gsettings-desktop-schemas"
	    "--" "sbcl"))

	  (sbcl-nyxt-site ("guix" "environment" "--pure"
			   "-m" ,(expand-file-name "~/common-lisp/nyxt-site/guix-manifest.scm")
			   "--" "sbcl"))

	  (ccl (,@maybe-core-dumper "ccl"))

	  (clisp (,@maybe-core-dumper "clisp"))

	  (ecl ("ecl")))))

This allows me to easily jump between different Lisp implementations, with different startup options.

When an implementation is started with guix environment, the environment is includes the specified packages, such as extra (Common Lisp or not) libraries and tools, but these packages are not seen by other REPLs! I can pass it the --pure option to further isolate the environment, i.e. not inherit from any existing environment variable.

In other words: these various environments don’t spill on each other, they remain clearly separated. This gives many guarantees in terms of setup and reproducibility.

While not a Common Lisp exclusivity, it’s important to note that SLY can connect to a remote Common Lisp process, including a process running in a container.

I made this little wrapper script to quickly start SBCL in a Guix container:

#!/bin/sh

port=4006
[ -n "$1" ] && port=$1

guix environment --network --container --manifest=$HOME/guix-manifests/common-lisp-manifest.scm -- \
     sbcl --eval "(require :asdf)" \
     --eval '(dolist (p (list "" "sly/contrib/" "sly/slynk/")) (push (pathname (format nil "~a/share/emacs/site-lisp/~a" (uiop:getenv "GUIX_ENVIRONMENT") p)) asdf:*central-registry*))' \
     --eval "(asdf:load-system :slynk)" \
     --eval "(slynk:create-server :port $port)" \
     --eval "(asdf:load-system :cmd)"

Then I can connect to this instance with M-x sly-connect. The result is a usual shell, but with limited file system access and all the limitations that I can configure with a Linux container.