Rich Unicode

There are many well-known tricks for fancy formatting using Unicode. Below is an incomplete tour of how a powerful DSL or compiler could implement various ‘fancy’ formatting in Unicode, starting with inline elements:

• Italics: Mathematical Symbols Range’s sans italics

• Bold: Mathematical Symbols Range’s bolds

• Italic Bold: ditto (as well as ‘script’ variants)

• Superscripts/subscripts: Unicode superscripts/subscripts

  • Fractions

  • Footnotes

  • Hyperlink numbering: while ‘hyperlinks’, in the sense of interactive elements, could not be supported, they must still be supported—people are not about to stop writing URLs!—and can be usefully listed at the end of a utext.

    Note that several alternative sets of superscript numbers are provided in the Dingbats block, so there need not be any visual confusion between various uses of superscripts. URLs could get the ‘normal’ superscripts, while footnotes (being much less used) get weirder ones like ‘➀’/‘❶’/‘➊’.

• Underlining: s͟i͟n͟g͟l͟e (COMBINING DOUBLE MACRON BELOW or COMBINING LOW LINE), d̳o̳u̳b̳l̳e̳ (COMBINING DOUBLE LOW LINE)

• Strikethrough: s̵t̵r̵i̵k̵e̵t̵h̵r̵o̵u̵g̵h̵ (COMBINING SHORT STROKE OVERLAY)

• Unordered list: Unicode supports countless icons which could be used for unordered lists of almost arbitrary depth

• Ordered list: Standard approaches like 1. are fine, but Unicode also supports much weirder numbers—in addition to the ‘circle’ ones shown above, there are also PARENTHESIZED DIGIT n points like PARENTHESIZED DIGIT ONE: ‘⑴’, ‘⑵’, ‘⑶’, ‘⑷’ etc.

• Math: Unicode can encode much simple math as-is, as even unusual fonts like ‘double-struck’ letters are in Unicode. For example, the quadratic equation in Unicode:

                            −𝘣 ± √̅(𝘣² − 4𝘢𝘤)
                        𝘹 = ————————————————                            (1)
                                  2𝘢

  For more advanced math, the user can write in the familiar LaTeX notation and it can be compiled to UnicodeMath (demo), as is already supported by some tools like Pandoc.⁴

• Code syntax highlighting: while color is unavailable, italics/underline/bold are already adequate for syntax highlighting, as demonstrated by monochrome syntax highlighting themes like used in the influential ALGOL 60 report (eg. Pygments’s algol_nu—which inspired the Gwern.net syntax-highlighting theme)

• Automatic rewrites with fancier versions: why display a boring (1) to the reader when you could replace it with something cool like ‘➀’? Or add ligatures—yes, it was a bad idea for Unicode to include them, but if they’re there, might as well use them and replace any ‘ff’ with ‘ff’ to look cooler.

• Stylistic variations: there are many Unicode alphabet variations, like fraktur (or letters in white circles & black circles or black squares, or even upside-down!), which do not map onto standard markup because they would usually just be a different font choice; a Utext could support these transformations as attributes set on a specified span of text.

  Attributes could be more complex, and allow defining additions like adding diacritics to each character, avoiding typing. In addition to the weird stylistic variations, some useful shortcuts would be: all-uppercase, all-lowercase, alternating-case each letter, adding n spaces between each letter… (Even more transformations, on a more semantic level, are possible using AI.) Transformations could be chained: [She said what!​?]{transform="uppercase,bold,italics"} → 𝙎𝙃𝙀 𝙎𝘼𝙄𝘿 𝙒𝙃𝘼𝙏‽.

  Utext is not quite homoiconic, but one can transform text in-place (eg. with a keybinding in one’s IDE to ‘execute’ a selected range of text): a snippet like [I am talking calmly!]{transform="uppercase"} can be rewritten in-place to I AM TALKING CALMLY!; this is potentially lossy, so it would be convenient to keep the original around as a backup, and one could then swap back and forth as necessary ([I AM TALKING CALMLY!]{original="[I am talking calmly!]{transform=\"uppercase\"}"}).

Block elements:

• Lists: handled by indentation + rarer characters for list marker visual flair (even just for stars, there are countless Unicode characters to choose from)

• Sidenotes/margin notes: handled by the text layout reflowing the body text around them, as seen in many books—tedious to do by hand, but easy for a text layout algorithm.

• Semigraphics, for drawing images/art:

  • Fleuron & Ornamental Dingbats decoration: these can be used to provide horizontal rulers, bolded quotes, and outline/box-drawing characters like checkerboard patterns (extended substantially in Unicode 13+ in “Symbols for Legacy Computing”, for examples see the Unscii font homepage)

  • Sparklines: Block Elements can draw Tufte-style sparklines like ‘▁▂▃▅▇’ (eg. on the CLI).

  • Data visualization & statistical graphs: gnuplot (matplotlib/asciiplotlib/), termeter, plotlib/bashplotlib, txtplot

  • Shading: Block Elements include shaded blocks like ‘░’ (LIGHT SHADE), ‘▒’ (MEDIUM SHADE), & ‘▓’ (DARK SHADE); as well as

  • Dots: Braille Patterns could draw various binary patterns, for a dithered look (eg. drawille)

• Header outlines, tables⁵, diagrams, rectangles/squares/wrappers of any kind: box-drawing characters

           ┏━━━┳━━━┓
           ┃ | ┃ – ┃
           ┣━━━╋━━━┫
           ┃ H ┃ ‡ ┃
           ┗━━━┻━━━┛
  
    “Is this loss”—in Unicode?
    Depends how generous you feel.
    (And what fonts you installed.)

• Headers: header level priority can be expressed nicely with the foregoing.

  Working our way up from the smallest headers to the largest, a header hierarchy might go: eg. italics, underlined, bold, bold underlined, underscored (add a second line like ======), capitalized underscored, outlined header…

  At the higher levels, we can draw increasingly large squares or rectangles; the text cannot be upscaled, of course, but the header text can be redrawn as ASCII art (eg. 1, 2, 2, 3)—they are fonts-within-fonts, one might say, and providing these sorts of ‘fonts’ was a common feature of ANSI/ASCII art text editors like TheDraw. The transition point would be governed by whatever looks good. (Maybe a 4×4 square is too small, and the lower limit is 5×5?)

  • Dropcaps: if we have ‘fonts’, we can also have dropcaps to decorate our documents. They are simply a box with an upscaled letter and perhaps ‘ornamentation’ inside it around the upscaled letter. These can be packaged, or generated on the fly procedurally.

  • Progress bars: one nice feature of more complex document readers is providing a progress bar. We can’t provide a GUI/TUI which computes this, of course, but we can encode progress bars into headers if we like.

    For example, in any underscored header (which is most of the large headers), we could transition characters, corresponding to document position. At the beginning of the document, the underscore line might be ---------- (0%), and at the halfway point, the next header is =====----- (50%), ========-- (75%), and finally, ========== (100%), providing a visual progress bar for the reader. (This sort of ‘scroll spy’ approach is increasingly common on web pages & mobile apps.)

Utext Markup

For ease of user adoption, the markup should probably be based on Markdown. Markdown already has constructs corresponding to most of the above features, and the compiler can take care of things like hyperlink numbering & appending.

A Utext user probably needs more fine-grained control over 2D layout than a Markdown user usually needs (or wants). This could be encoded as tables, like Pandoc’s pipe tables. Tables can be given arbitrary attributes to control use of advanced layout features, like auto-generating text borders through a neural net.

Text + Source Storage

Because the output of Utext is the same as the input as far as everything else is concerned (ie. UTF-8 text), one can provide both the ‘source code’ and ‘compiled document’ of a Utext in a single file. While (less than) doubling the space consumption, this would have a few benefits:

• Distributing source by default helps with community-building & licensing (similar to how many people learned HTML by reading the HTML of cool web pages they visited, and built on copies)

• Utexts can be re-compiled:

  • Greater quality: particularly with the more advanced features I speculate about above, Utext compilers may improve over time. If Utexts didn’t come with source, improved renderings would be difficult as now the Utext has to be decompiled/reverse-engineered, and it may be hard to know what the original prompts or styles were.

  • Responsive design: any default size width will ill-suit someone; perhaps they prefer to provide the compiler a much wider canvas to paint on, or alternately, want the best possible rendition for their particularly narrow smartphone screen-width. (See also HTML Utext.)

  • Personal preference: providing source lets readers customizes the compilation. They can disable or upgrade the advanced features, or change the esthetics: perhaps they don’t like the ‘default retro’ appearance, and will instead target Commodore fonts with their characteristic slanting ‘italic’ style

• Accessibility: source-availability means that a screenreader can skip to the source, or the user could program their browser to skip to the source, or the Utext could simply be provided by a third party in a HTML wrapper which marks up the source as what the screenreader should

Utext Format

The primary purpose of Utext is to generate the coolest output, so the main challenge is just achieving any of the suggestions above. But as an optional feature, mostly independent of the foregoing, I’ll discuss a special-purpose Utext format.

Fancy Unicode tricks do not, in principle, require a new document format beyond the classic .txt file. One can simply store the source for a Utext in an ordinary Markdown text file with a .md extension, and ‘compile’ to a .txt file.

But it is convenient to define a new format (especially if Utext winds up adding extensions or syntax which make no sense to a Markdown compiler), and since we are using strictly UTF-8, we can do something interesting and combine a fancy Unicode text file with a text file into a single file format.⁸

The Utext format (.utxt / text/plain-utext;charset=UTF-8) is a UTF-8 document which has two kinds of lines: compiled and source. A utext could contain a single document, or concatenate many documents. There is no explicit document-level division or header, and they are distinguished on a line-by-line basis.

Each line of a compiled utext is prefixed with an invisible special character (eg. ASCII’s Start of Text control character) which indicates it is a compiled line.⁹ A compiled line is human-readable UTF-8 text line. (120 characters would be a good maximum width for all devices; but a Utext could be rendered at multiple widths so as to support other devices, like smartphones which usually a comfortable width of ~40 characters.) But a compiled line is not meant to be human-editable, because it uses all available Unicode tricks for rich formatting, like ASCII art. For example, a source line may use italics or bolds, which normally cannot be displayed in pure UTF-8 text without using terminal control codes or relying on markup conventions like HTML/Markdown, but the compiled version abuses Unicode to map each italic letter onto the math symbols—thereby providing ‘italics’ and ‘bolds’ in pure UTF-8 text rendering in any text viewer.

Any line without the prefixed special character is a source line. A source line would be whatever document format the user prefers, such as a Markdown dialect with custom extensions for Utext-specific features.

Since both compiled & source lines are UTF-8 text by definition, they can be stored in the same file. Utexts can be compiled ‘in-place’ and a text editor could easily provide a ‘WYSIWYG’ view of a utext being edited, and this interoperates with version-control systems.¹⁰ A standard utext provides the full compiled version first, followed by the source. This is most convenient for the reader, who can read a document in its pretty compiled form, and then skip the source. (It also helps solve accessibility issues: if a 40-column smartphone rendition is unacceptable, then one can provide the source with some HTML like a <pre style="white-space: pre-wrap"></pre> wrapper; ASCII/ANSI art is inherently screenreader hostile, but a screenreader could simply skip compiled lines for the human-readable source text, and so a Utext would be far more accessible than a hand-written equivalent, which lacks any formal structure or documentation of intent.)

But they can also be arbitrarily ordered: one could provide a Literate Utext, which is a range of compiled lines, followed by the corresponding source lines. (The standard utext is easily recreated from the literate utext by simply sorting on the first character of each line, and ‘bubbling’ up lines with the prefix and bubbling down lines without.) In this approach, as long as compilation is kept to one-pass compilation, Literate Utext can be generated incrementally and streamed.¹¹

And because compiled vs source lines can be matched by a per-line regexp, a literate utext can be viewed easily as an incremental streamed—simply grep for the compiled lines. This means one can do command line stunts like create an IRC client/server with nice formatting simply by telnet: a client opens a telnet connection and pipes in the source text they type, sending it to the server; they open a second telnet connection, which downloads an (incrementally streaming) literate utext, filtering out the source lines with grep; and the server simply pipes each client line of text through the Utext compiler and appends to the file that each client is indefinitely downloading. (This can be done to some degree with regular IRC, but not with formatting.)

<html>
    <body>
<pre>Normal Utext content here, but then one wants to link to
some URL like</pre> <a href="https://example.com/"><pre>Example.com</pre></a> <pre>to demonstrate how
linking works in HTML Utext.</pre>
    </body>
</html>

Normal Utext content here, but then one wants to link to
some URL like
Example.com
to demonstrate how
linking works in HTML Utext.

<style>pre { display: inline; }</style>

<!DOCTYPE html>
<html lang="en-us">
  <head>
    <meta charset="utf-8">
    ...

    <style>pre { display: inline; }
    @media all and (max-width: 649px) {
        .wide { display: none; }
        }
    @media all and (min-width: 650px) {
        .narrow { display: none; }
        }
    </style>
  </head>

  <body>
<pre class="narrow">Normal Utext content here, but then one wants to link to
some URL like </pre><a class="narrow" href="https://example.com/"><pre>Example.com</pre></a><pre class="narrow"> to demonstrate how
linking works in HTML Utext.</pre>

<pre class="wide">Normal Utext content here, but then one wants to link to some URL like</pre> <a class="wide" href="https://example.com/"><pre>Example.com</pre></a>
<pre class="wide">to demonstrate how linking works in HTML Utext.</pre>
  </body>
</html>