Shortlog - a log of everyday things

2012-02-17

On plaintext markup formats.

I can't say I'm satisfied with any of the current plaintext markup formats. All of them have warts. The most popular on the web at this point is probably Markdown. Github uses it. So do Reddit and StackOverflow. So do a variety of blog engines, including Posterous, Tumblr, and WordPress. And for good reason: it's probably the best one out there.

What do I think about it? I love some things, and I really hate some others. First, the good:

• The usual text markup features, like paragraph reflowing, horizontal rules, lists, etc.
• Reasonable escaping behavior for HTML entities
• A decent compromise on line break behavior
• Intelligent auto-linking
• Classy reference-style links probably inspired by proper email use of the same
• Easy to embed fancier HTML constructs as needed

My dislikes tend to stem from a few things where I think the syntax either looks atrocious or strikes me as poorly designed:

• Support for two kinds of headers, both of which are ugly. Also, whose idea was it to underline things on a separate line to make them headers? It looks uglier, and it's harder to parse.
• Somewhat icky list behavior (why does adding an empty line between two items make a new <p> tag inside both?) along with poor indentation policy (just as in publishing, your bullets should not align with the left edge of your text). To be fair, lists in general are hard to make featureful, attractive in plaintext, and still uniquely parseable.
• Numbered lists requiring digits, but ignoring the digits provided. (Some people consider this a feature; I do not.)
• Single underscore on either side of a word makes it italic, making for disasters like this if you aren't religious with your escaping, where you meant to have words_with_underscores but got wordswithitalics instead. (If I can tell what tool you use by the defects in your final product, that's a good sign you should either learn to use the tool better, or get a better tool.)
• Wasting the obvious use of underscores to indicate underlining by making them...do the exact same thing as asterisks.
• Punting to HTML for tables, which are uniformly awful to try to read in plaintext (the alternatives aren't great either, but they're less bad).
• I have no idea what's going on with the image embedding syntax; a more general asset embedding syntax might be useful here, with possible auto-detection of, say, video or image formats based on suffix (though I guess it's perfectly reasonable to punt to HTML for most such things)
• John Gruber apparently has no interest in maintaining the darn thing, to the chagrin of Markdown users

Even with all these complaints, Markdown is the best we have. There are projects that have extended it to improve (among other things) the table situation. The alternatives fairly uniformly suck: reStructuredText suffers from about two-thirds of the same criticisms, lacks nearly any redeeming features, and requires way too many empty lines. AsciiDoc has the same atrocious header behavior, has horribly difficult to read plaintext, also wastes underscores on italics, and has horrid list nesting if you have list items longer than six characters. BBCode is practically HTML replacing < and > with [ and ]. Mediawiki is intended to have a richness of features that goes far beyond what most text-to-HTML engines require, but does some silly things with using nothing but 's for italics and bold, and the lists are ugly. Textile has a somewhat redeeming table structure, but the lists have the same issues as most other markup formats, code blocking and headings are so close to HTML proper as to be not worth using, and they use underscores for italics again.

Now, I happen to like a lot of things about dokuwiki's syntax. I find the list syntax of dokuwiki to be attractive. It's got its own heap of problems - lists can't contain a whole lot of other possibly interesting fields, nested lists behave inconsistently depending on how you indent them, and list items can't span multiple lines. That last criticism I see as a feature; if you're writing such long bullet points that you need to split them into multiple paragraphs, you're using bullet points incorrectly. Alone of the text markup formats I've seen, dokuwiki's lists require the indentation of the standin for the bulletpoint itself at the zeroth level.

Dokuwiki's table format is also terse, yet effective. It doesn't appear to support super-fancy use, but is simple and readable enough that I like it. Using the whitespace inside cells to determine alignment is clever and well-done. Dokuwiki also is rare in that it gets the use of underscores and asterisks correct as well.

Dokuwiki's linking system is subpar - it's clearly meant for a wiki, which makes it slightly harder to repurpose, and it's not clear how embedding of assets works if you're not using them in the context of a wiki. I'm not a fan of using \\ as a line break either.

The biggest problem with Dokuwiki's syntax, though, is that it appears to have exactly one implementation, and that's the (extensible!) codependent lexer and parser that sit in the dokuwiki PHP source. I cannot find any renderers for its markup in any other language. And that is a shame.

I'm in the process of making my own derivative plaintext markup language that takes the behaviors I like from markdown and dokuwiki. I will, when it's ready, use it internally on my blog when I write entries. I hate that it'll be Yet Another Plaintext Markup Format, but at least I'll be happy with it.

And that's just one of the ~six technical things I've been working on this week. If you're curious about the others, drop me a line, and I'll try to write up my progress on each of them a little more.

Comments: