Welcome to my github world

Well, I did it. Against all reason and my better judgment, I went and made rstweaver depend on my fork of docutils.

This is not how we're supposed to do things. We're supposed to submit patches when we can and find workarounds while we have to, which is usually. This keeps everyone on the same libraries, and makes dependencies navigable by mere humans.

But go look at the most recently updated python-language repos on github. Right now the top ones are:

Repo	Lines of code
dqwiki/lisabot	1369
benjschiller/checkmyclones	124
nolar/shortener	1458
eteq/astropysics	43,195
codebrainz/geany-sphinx	216

With one exception these are little projects. And what's the point of having such tiny projects if you can't bend them to your needs? And who's going to notice the extra space consumed by 3 slightly incompatible versions of lisabot?

Go look at the Python Package Index and see how small most of those projects are. Software is tiny these days. I think a large part this is due to Python being sufficiently easy that a lot more people can throw together one thing in Python than could in C++. And it interfaces better with itself so you have no incentive to make these massive sprawling projects.

So why?

Well cause there were some bugs I wanted to fix and a feature I wanted to add. Not compelling reasons but that's the experiment: how low can you set the threshold?

Avery Pennarun makes an interesting argument on a not-quite-the-same topic. (Linking to that might make it sound like I have some problems with docutils. I don't. Docutils is awesome).

With Python (or Ruby or Perl or ...) + Git we can do anything

I hope. No one really knows what the programming future holds. Wasn't Prolog supposed to be the future?

Why are all the "easy" languages dynamically typed?

Actually, that one's not hard to answer; a harder one is "Why do I think I still prefer static types?". The evidence is right there both in my own experience and in the numbers around me that dynamically-typed programming is just easier. You get more done. Why do I resist in admitting this?

"Languages of the future" always make me feel uneasy. Well I guess some day we'll all use Prolog and I can stop worrying.

(I want to like Scala. I really do. It's got everything I think I want in a language. But why do I have this uneasy feeling about it?).

Sweeping changes to rstweaver

Improvements:

• New languages: C++, Python
• Caching
• Pure docutils (so conversion to LaTeX works -- conversion to ODT which was one of my primary motivations for removing HTML-specific code still fails, which I think may be due to a bug in docutils but I'm still figuring that out)

And it's sitting on my github site with some examples.

Implementing new features doubled as a chance to encounter newer and yet more frustrating problems. It is an eternal fact of programming that reality is smarter than you are: the ideas that you design from your own creativity will simply never measure up to the horrors nature throws at you haphazardly. She makes it look so easy.

An example would be the case of Python decorators when the decorator function also happens to have class scope.

     decor.py 

  1  class A:
  2  
  3      def fivify(func):
  4          def handler():
  5              return func(5)
  6          return handler
  7  
  8      @fivify
  9      def a(x):
 10          return x
 11  

We are now to wonder what transformations may have been applied on fivify (such as making it a bound method) before it is used as a decorator. These thoughts give us an error message with the approximate clarity of

     decor-thoughts.py 

  1  class A:
  2  
  3      @classmethod
  4      def fivify(func):
  5          def handler():
  6              return func(5)
  7          return handler
  8  
  9      @fivify
 10      def a(x):
 11          return x
 12  
 13  a = A()
 14  

Traceback (most recent call last):
  File "decor-thoughts.py", line 1, in <module>
    class A:
  File "decor-thoughts.py", line 9, in A
    @fivify
TypeError: 'classmethod' object is not callable

So we are lead to see that decoration must happen while the methods are still sitting in the uninstantiated class (this can cause other surprises).

A fair amount of frustration comes from the docutils library itself. Docutils is actually quite wonderful, but a conspicuous need for polishing renders it a capable adversary. Most of the snags can be traced to rather boring implementation bugs, but one of its more archetictural flaws helps remind us why in programming python we stick to the pythonic.

At the heart of docutils' extensibility lie directives, which serve much the same role as tex commands except that they are written in python and manipulate docutils rather than tex. So I set out to registor a directive:

     directive.py 

  1  from docutils.parsers.rst import Directive, directives
  2  from docutils.core import publish_parts
  3  

  4  class MyDirective(Directive):
  5  
  6      def __init__(self, *a, **b):
  7          Directive.__init__(self, *a, **b)
  8  
  9      def run(self):
 10          return []
 11  

 12  directives.register_directive('mydir', MyDirective)
 13  publish_parts('\n.. mydir::\n\n')
 14  

From this we learn that register_directive() appears to take a factory, which it is going to instantiate, and pass some arguments which I appear not to care about.

I was almost content with this arrangement -- I just wanted to pass some context to MyDirective whenever it was created. Seeing as register_directive() appeared to take just a callable factory, I added some context:

     directive.py (cont)

  4  class MyDirective(Directive):
  5  
  6      def __init__(self, ctx, *a, **b):
  7          Directive.__init__(self, *a, **b)
  8          self.ctx = ctx
  9  
 10      def run(self):
 11          return []
 12  

and some factory:

     directive.py (cont)

 13  def create(*a, **b):
 14      return MyDirective(None, *a, **b)
 15  
 16  directives.register_directive('mydir', create)
 17  publish_parts('\n.. mydir::\n\n')
 18  

Traceback (most recent call last):
  File "directive.py", line 17, in <module>
    publish_parts('\n.. mydir::\n\n')
  File "/usr/lib/pymodules/python2.7/docutils/core.py", line 427, in publish_parts
    enable_exit_status=enable_exit_status)
  File "/usr/lib/pymodules/python2.7/docutils/core.py", line 641, in publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "/usr/lib/pymodules/python2.7/docutils/core.py", line 203, in publish
    self.settings)
  File "/usr/lib/pymodules/python2.7/docutils/readers/__init__.py", line 69, in read
    self.parse()
  File "/usr/lib/pymodules/python2.7/docutils/readers/__init__.py", line 75, in parse
    self.parser.parse(self.input, document)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/__init__.py", line 157, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 233, in run
    context, state, transitions)
  File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 454, in check_line
    return method(match, context, next_state)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2281, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2293, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2035, in directive
    directive_class, match, type_name, option_presets)
  File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2093, in run_directive
    'Directive "%s" must return a list of nodes.' % type_name
AssertionError: Directive "mydir" must return a list of nodes.

Oh docutils. This error message had me for... I'd say near twenty minutes, because darn it I am returning a list of nodes.

So I was a bit suprised to find out that register_directive doesn't actually take a callable object -- and it's not thinking of it as a "factory" either -- it's expecting to get either a class, which it will instantiate, or a function, which it will leave as is until it gets called to handle a directive. And there's the problem.

Now nothing against docutils because it's actually a very nice library and for the most part well designed but checking the type of an object and then branching based on the result is decidedly unpythonic.

And that's how I learned that "pythonic" is something that actually matters and isn't just something people say.

(It's not pythonic because it doesn't respect duck typing -- it's not supposed to matter what the type of the object is so long as it has the right properties.)

But then it was later in the same project that I... I found myself wanting to commit the same error! You see I had an "interface" like

     weaver.py 

  1  class WeaverLanguage:
  2  
  3      def run(self, code, args):
  4          '''
  5          Returns content to be added to the document.
  6          '''
  7          raise NotImplementedError
  8  

I could always make run() return a docutils node -- because that would cover all cases. You want just plain text? Stick the text in a node. HTML? make a raw HTML node. So that would solve all my problems.

But I just didn't want to do that. I wanted to make the most common case of returning raw text easy, and not require looking at docutils (sorry again docutils, there's nothing wrong with you, really). I could break it up into two stages:

     weaver-stages.py 

  1  class WeaverLanguage:
  2  
  3      def run_text(self, code, args):
  4          '''
  5          Returns content to be added to the document.
  6          '''
  7          raise NotImplementedError
  8  
  9      def run_node(self, code, args):
 10          text = self.run_text(code, args)
 11          return nodes.inline(text, text)
 12  

Except that I don't want to do that either, because it puts the output type in the name of the function, making it look way more important than it actually is (solution: type inference and typeclasses -- also not pythonic).

So I went with the unpythonic hack. And maybe some day someone'll hate me for it.

Expression

I was at first so disheartened by Luke Plant's little essay:

I feel like a human compiler, translating the Haskell or Python in my head
into a language which is a whole level lower.

Haskell, Python and R are pretty much all I use these days. Will I ever be able to go back to one of those rigid, Industrial Grade tongues?

I think there is hope.

Remember, programming is poetry. When you have written 50 lines of C, don't think to yourself "gee, that would have been 10 lines of Haskell"; think, ah, here are 50 lines in which to express myself.

The program can describe more than the algorithm. You can put your thoughts into it. Make every line count.

Also, be grateful that you have verbs. Haskell doesn't have verbs; it has many nouns and prepositions. Nouns and prepositions can be combined every which way, just like pieces of Haskell code.

Verbs, on the other hand, are rigid. They can occur only so often, and they propagate their structure to the space around them, requiring a subject and inviting an object or complement.

The presence of verbs of course limits C, but it makes it more natural. You spend most of your life speaking with verbs, and you should welcome the chance to use them in code.

I can't tell you that Haskell or Python wouldn't be better. But I can tell you you can use what you have to reflect your thoughts. Maybe just not the same thoughts you would have otherwise.

It works this time

When I started learning Haskell I heard that the type system could be used to check all kinds of constraints at compile time -- far beyond just the type of the data. For the most part I have found this not to be the case -- not because the type system can't do these things but because most people who program Haskell are either 1) not leet enough to use it that way (this includes me) or 2) too leet for me to understand their code, so I have no idea what kind of constraints are being checked.

But I did run into a beautiful example using Maps. A Map stores data with ordered keys, so to do that you need an ordering function, and they used the most obvious possible type signature for that ordering:

1 compare :: a -> a -> Ordering

And here's the great thing: the Map consistency cannot possibly be screwed up by using mutable keys.

Ah, but that's easy, you say: just use immutable keys -- make them const. But this is even better: you don't have to use immutable keys. I could use a key like this:

1 data Key = Key { 2 x :: Int 3 , y :: IORef Int 4 }

That's mutable. I can change it. Even while it's being used as a key. But there is no way (and I really hope someone doesn't come and tell me that there is a way because that would make me sad) for compare to see those changes -- because of its type.

In fact you can pack as much mutable state as you want into the Key, and it is partitioned off cleanly from ever affecting the ordering.

But here's what I really love about this: it works because the guys who wrote Map picked the simplest most obvious type for compare. It's the type you'd come up with if someone pointed a gun to your head and said "you have 5 seconds give me the type for compare". They could have written it

1 compare :: (Monad m) => a -> a -> m Ordering

except that that's substantially less obvious and in this case turns out to be undesirable.

It's not always like this. A lot of the standard functions for operating on lists could have been made monadic because they don't need consistency. This is one of those rare beautiful cases of the language working exactly the way it's supposed to.

(In other news, untracked side-effects may be "dangerous" but nothing beats the havoc of using a backtracking monad transformer on the ST monad. Nothing I tell you.)

Visual Editors

While I sit here typing away at an especially depressing bit of code there's about 85 lines of Vim script trying to make the indentation look pretty by searching the nearby text for symbols it likes and trying to shift things around appropriately, using lots of regexes. At first this seems like a terribly inefficient combination -- the compiler already has a parser that knows where the structural elements in the source code start and stop -- why does Vim have to duplicate (poorly) this logic using regexes?

Of course it doesn't work like that: the compiler is designed to accept a complete, fully written program, and as a service to you doesn't tolerate deviations from the syntax, but the editor has to accept gross syntactic problems because you're just about to change it.

Which is one reason why linear flowing text is such a great way to represent what is really a hierarchical and rigidly structured document -- it's more flexible than the document it's ultimately going to be, so it can represent incomplete syntax. I can write this:

if ( ) {
}

And fill in the details later.

I was thinking about this because I was wondering why visual editors -- like LabView's flow thingy or EasyC or Scratch aren't more commonly used. I don't use them and all the ones I've tried annoy the hell out of me but in theory they seem like a better idea than pure text editors because they could do so much more. (Even super fancy IDEs that do a ton ultimately start from just text, which is why they will always screw stuf up (and get confused by incompletely written code)).

So what's missing? There are a few things I can see rigt away. In my opinion, to be useful visual editors must:

1. Avoid visual distractions

This is where EasyC is very good and Scratch could be better. Think about what a text editor gives you: small characters freely placed on a large, blank canvass. Blocks that fill the screen, excessive background colors (notice most editors let you use background colors in syntax highlighting and no one uses them) or lines and arrows cluttering up the space will just make you sad when looking at your code.

2. Represent incomplete syntax flexibly.

This includes incomplete syntax in all directions -- maybe you write a loop and omit the body, or maybe you write the body but omit the loop -- any conceivable way to omit parts of a program must be possible.

3. Give the programmer flexibility in laying out the code.

This might not be so much of an issue for structured imperative languages like C (though I think it's still necessary), but it would be absolutely inconceivable to write Haskell (or Lisp dear god) in a language that did the layout for you -- functions can be combined in too many different ways to derive the feel of a piece of code from the syntax.

This gets at a deeper point which is that editors must be able to communicate more than the code; they also must communicate what the code means in your head, by more than just identifier names and comments. Python tries to wrest this ability from you by using indentation to indicate blocks, and to some extent does succeed in frustrating me like that but they left enough flexibility in the language to make the code expressive.

4. Not be marketed as a kids' toy.

Because while many adults might be ok spending their days playing with a kids' toy, there is a certain age at which this is very unfashionable, and it's an age where a lot of programming gets learned. And a lot of habits are frozen.

5. Be as polished as a typical text editor.

I would never use EasyC for serious programming because it jus takes too much pointing and clicking, and it's slow. The thing is, even if text editors are starting out from a disadvantage, they've just been around so much longer, so many more people use them, and have acquired so many more hours of development and testing that most of the sharp edges have been worn off. And with a tool that you interface with constantly it matters less that it have a clean design than that the sharp edges be removed.

This last point is also why I keep using Vim, despite the many horrific stories I have accumulated while using it. Like how I was wondering why paragraph formatting in comments works strangely -- it formats them OK and keeps the leading '#' but it seems to hit everything in the comment even when I use 'gqip' to get just one paragraph. The reason (of course) was that Vim over the years has acquired two entirely different ways of delimiting paragraphs: the one used to format paragraphs by gq, and an entirely different one used to capture a paragraph with ip -- the former being a series of hacks for getting around things like comments and lists and the latter being a huge list of NROFF macros describing spaces and such. But I keep using Vim because most of the sharp edges have been hacked away and it's scriptable so I can get rid of most of the ones that still bug me (except paragraph formatting which I don't have the energy to tackle).

This last point is also the reason I think it's unlikely we'll get away from programming in text any time soon -- there's just too much effort put into text editing to make it worth starting from scratch. NI is trying with LabView, and a number of researchers are trying with tools for kids to program. (Notice both of these are about getting people who otherwise wouldn't be programming (scientists who have other stuff to do, or kids who have other stuff to do) to do so in a very limited boxed-in way -- but importantly the target is people who don't have preconceived notions about what programming is. The scientists aren't going to turn into the next generation of software developers but the kids might -- maybe they'll change stuff. And maybe I'll hate it and end up complaining about how kids these days do it all wrong. But oh well.)

Literate literate literate programming

In an effort to better document my Haskell experiments I wrote a set of reST directives for Sweave-like literate programming.

What makes rstweaver different from Sweave is that a single reST document can write and execute multiple source files, in stages, and parts can be inserted or modified, then reexecuted. This gives the illusion of an interactive session for a language that is largely not interactive.

If you look at the tutorial you will notice that it too includes literate reST... which in turn includes literate Haskell. It is literate literate programming!

And the society for putting things inside of other things demands that we go one level further: an excerpt from the tutorial: I present you literate literate literate programming:

-- so-nested.rst --

You can name the stages: .. weaver:: test4.rst exec block .. haskell:: Named.hs :name: imports import Control.Monad And then go back and edit them: .. weaver:: test4.rst exec block .. haskell:: Named.hs redo :name: imports import Control.Monad import Control.Applicative Or insert a stage after a named stage: .. weaver:: test4.rst exec block .. haskell:: Named.hs :after: imports x = 5

You can name the stages:

-- test4.rst --

.. haskell:: Named.hs :name: imports import Control.Monad

-- Named.hs --

1 import Control.Monad

And then go back and edit them:

-- test4.rst (cont) --

.. haskell:: Named.hs redo :name: imports import Control.Monad import Control.Applicative

-- Named.hs (cont) --

1 import Control.Monad 2 import Control.Applicative

Or insert a stage after a named stage:

-- test4.rst (cont) --

.. haskell:: Named.hs :after: imports x = 5

-- Named.hs (cont) --

4 x = 5

Haskell is that guy

I've been programming a lot of Haskell recently, and I've noticed something.

Haskell is that guy. You know the one who's like "pshhh I don't need side-effects" and then obsesses over side-effects. And I mean obsesses.

I've been using Template Haskell, which involves traversing and creating syntax trees, so, a very tree-like sort of task, you know that kind that functional languages are just perfectly suited for. Except for one little thing.

The Quotation Monad.

It's really a terribly minor detail. It does one tiny tiny hardly significant thing: it keeps a counter to make unique names. It's really not at all what Template Haskell or my project are about. It would be better not to think about it.

The problem is that quotation, being a monad, changes the type of expressions, and you then have to pull that type up the syntax tree using <*>so<$>many=<<annotations>>=, or use a do block and make everything flat and linear, losing all the benefit of a functional language.

Here's an example:

-- Main.hs --

1 matchLambda kw (Match pat body whrs) = 2 let 3 names = patNames pat 4 parts = bodyParts body 5 qLams = [buildLambda (names ++ bNames) <$> (expToSimpExp kw x) 6 | (bNames, x) <- parts 7 ] 8 in do 9 lams <- sequence qLams 10 whrs' <- fmap (id =<<) $ sequence $ (decToSimpDecs kw) <$> whrs 11 tup <- buildTuple lams 12 return $ letify whrs' tup

Granted it's still shorter than it would have been in Java, but it could be so much better if not for that one little counter. If you look at my code it looks like I wrote 500 lines for manipulating the Q monad and maybe a little bit here and there that deals with syntax trees -- especially since the parts that don't use the Q monad are short and clean.

Worse, the monad operators are the most visually familiar parts of the code, so someone who had no idea what the code does but knows Haskell would look at it and say "well, I don't know what this does, but I can see you're using a monad".

The problem is that the inconvenience of using a monad grows with the depth with which it is nested in the syntax tree. So

-- Main.hs --

1 f <$> a

is no trouble but

-- Main.hs --

1 f <$> (g <$> (h <$> a) <*> (pure 2)) <*> (pure 3)

is a pain, and has none of the beauty of

-- Main.hs --

1 f (g a 2) 3

Is there any design reason (other than syntactic) why monads should be harder to use when they are deeper? I do not know -- it's possible there is and I haven't seen it yet.

It turns out it's actually fairly simple to insert the <$>applicative<*> operators in the right place automatically (simple in terms of the idea -- it took a lot of code to get right). I don't have the energy to describe it right now but it's kind of obvious if you think about it -- it's probably been done before but I don't know how to search for it. Just trust that the following code works:

-- Demo.hs --

1 {-# LANGUAGE TemplateHaskell #-} 2 3 import Language.Haskell.TH.MetaLang 4 import Language.Haskell.TH.MetaLang.ApplicativeLang 5 6 main = do 7 print $ $(apl [| 8 let a = (nd [0,1]) 9 in 10 (a, a, a) 11 |])

[(0,0,0),(0,0,1),(0,1,0),(0,1,1),(1,0,0),(1,0,1),(1,1,0),(1,1,1)]

I mean no disrespect to Haskell -- it's a great language, this one problem has just been irritating me. But I may or may not have solved it now.

(It's also interesting that the Q monad shouldn't, I think, have the usual problem of the interaction between lazy evaluation and side-effects, because you don't care what order the counter gets incremented in, just that it does).

Other random things:

• pozorvlak's tutorial on Template Haskell is very goood.