This is based on the talk I gave at ErlangDC.

The common perception of Erlang is that it’s a good language for big projects where you need massive scalability, distribution, fault-tolerance and so on. The language itself is weird and ugly, and it’s got a lot of annoying restrictions, but that’s what you have to put up with to get the good stuff. I want to challenge both sides of that, and say that Erlang is also a great language for small projects – even little scripts – and that it’s really beautiful once you understand it. It just has its own way of doing things. And a key here is that it does have A Way of Doing Things; There are design patterns and programming idioms that you can follow. It’s not rocket science; you can learn it. So what I’m going to do here is show you some of those patterns and idioms, walk through a couple of simple programs, and hopefully give you what you need to know (and a bit of a nudge) to get started having fun with Erlang.

Synergistic Weirdness

Erlang has what I’ve come to think of as “synergistic weirdness”. When I was first starting out with Erlang, there were all these things that struck me as weird. Like, I want to write a for loop that increments a counter, and Erlang’s like, “Nope. No can do.” You can’t increment a counter because variables are immutable, and there’s no for loop. There are no loop controls, period. And there are no classes or objects, while we’re at it. How do I do anything in this language?

Then there’s all this weird extra stuff. There’s efficient tail recursion. Ok, fine, but how often do I write recursive functions? About never. There’s pattern matching and guard clauses. That’s kinda cool, but I’m still not sure how much I’d use it. All of the inter-process communication (IPC) stuff – process spawning and message passing – is definitely cool if you’re writing a big concurrent app, but otherwise? All this stuff is fine, but how often would you actually use any of it? The answer is, “All the time.” All these isolated bits of weirdness combine to something very elegant. For example, let’s take a look at…

Recursion

You don’t use recursion much in OO languages because (a) you rarely need to, and (b) it’s scary – you have to be careful about how you update your data structures. Recursive methods tend to have big warning comments, and nobody dares touch them. And this is self-reinforcing: Since it’s not used much, it remains this scary, poorly-understood concept.

In Erlang, recursion takes the place of all of the looping constructs and iterators that you would use in an object-oriented (OO) language. Because it’s used for everything, there are well-established patterns for writing recursive functions. Since you use it all the time, you get used to it. Erlang’s immutable variables actually simplify recursion, because they force you to be clear about how you’re changing your data at each step of the recursion. Pattern matching and guard expressions make recursion really powerful and expressive, because they let you break out the stages of a recursion in a very declarative way. Let’s look at the basics of recursion in Erlang with a very simple example: munging a list of data.

Like a story, a recursive function has a beginning, a middle, and an end. The beginning and end are usually the easiest parts, so let’s tackle those first. The beginning of a recursion is just a function that takes the input, sets up any initial state, ouput accumulators, etc., and recurses. In this case, we take an Input list and set up an empty output list.

    %% beginning
    func(Input) ->
        Output = [],
        func(Input, Output).

The end stage is also easy to define. We pattern-match on an empty input list, and return our output list. (I’ll get to why we reverse the output list in a minute.)

    %% end
    func([], Output) -> lists:reverse(Output).

The middle stage defines what we do with any single element in the list, and how we move on to the next one. Here, we just pop the first element off the input list, munge it to create a new element, push that onto the output list, and recurse with the newly-diminished input and newly-extended output. (And note that we add our new element at the beginning of the list, rather than the end.)

    %% middle
    func([First | Rest], Output) ->
        NewFirst = munge(First),
        func(Rest, [NewFirst | Output]);

That’s all there is to the basics of recursion. You may have multiple inputs and outputs, and there could be multiple middle and end functions to handle different cases (and we’ll see a more interesting example in a minute), but the basic pattern is the same.

Digression: Backwards Lists

Why do build our output list backwards? Why don’t we just add new elements to the end, and not have to reverse it when we’re done? This was one of those little Erlang weirdnesses that really bugged me until I understood it. The key is that lists in Erlang are not just arrays; they’re linked lists, and critically, singly-linked lists. So when you create a list like

    Foo = [cat, dog].

You get a logical list structure that looks like

    Foo
    |
    cat - dog

If you create a new list by prepending an element to Foo,

    Bar = [monkey | Foo].

The logical structure now looks like

    Bar      Foo
    |        |
    monkey - cat - dog

And if you create another new list by prepending more elements to Foo,

    Baz = [elephant, tiger | Foo].

The logical structure will now look like

    Baz       Bar      Foo
    |         |        |
    |         monkey - cat - dog
    |                 /
    elephant - tiger /

So the new lists are efficiently re-using Foo’s elements, but this only works because Foo itself is immutable. If you could add elements onto the end of Foo, or modify its elements in place, you’d see that change in every list built off of Foo.

Back to recursion: Bowling Game

A more interesting example of recursion is the bowling game. This is a standard programming exercise – write a program to calculate the score for bowling. It’s fairly simple, but not trivial. Your input is a list of rolls (number of pins knocked down), and your output is just a number, a final score. There’s also this concept of frames you have to keep track of; the game has a fixed number of frames, but the number of rolls may vary. The score for a frame may depend on rolls you made in other frames. Writing this in an OO language, and trying to break this functionality cleanly out into classes, can be tricky. In Erlang, we’re just going to define a recursive function that takes a list of rolls and returns a number.

Again, we start by defining the beginning of the recursion. We get a list of rolls, set our initial frame to 1 and score to 0, and recurse.

    %% Beginning: score/1 -> score/3
    score(Rolls) ->
        Frame = 1,
        Score = 0,
        score(Rolls, Frame, Score).

The end is even simpler. There are ten frames in a game, so when our frame count gets to 11, we’re done. Just return our score.

    %% End
    score(_Rolls, 11, Score) -> Score.

For the middle, we’re going to start with the normal case for scoring a frame, ignoring strikes and spares. Here, we just pop the next two rolls off the input, add them to our total score, and recurse with an incremented frame.

    %% Middle
    score([Roll1, Roll2 | Rest], Frame, Score) ->
        NewScore = Score + Roll1 + Roll2,
        score(Rest, Frame + 1, NewScore).

Now we need to deal with the strike and spare cases. Erlang’s pattern matching lets us do this very cleanly. For strikes, define a score function (in proper terms, a clause of the score function) that matches when the first roll is a 10. For spares, we use a guard expression to match only when the next two rolls add up to 10. In both cases, we need to look at rolls in following frames (2 for a strike, 1 for a spare) and add those to our score.

    %% Strike
    score([10 | Rest], Frame, Score) ->
        [Bonus1, Bonus2 | _] = Rest,
        NewScore = Score + 10 + Bonus1 + Bonus2,
        score(Rest, Frame + 1, NewScore);
 
    %% Spare
    score([Roll1, Roll2 | Rest], Frame, Score) when Roll1 + Roll2 == 10 ->
        [Bonus1 | _] = Rest,
        NewScore = Score + 10 + Bonus1,
        score(Rest, Frame + 1, NewScore);

That’s pretty much it for the scoring rules. We still need to handle incomplete frames, so by the time we’re done with that, the whole thing looks like this.

    score(Rolls) -> score(Rolls, 1, 0).
 
    score(_Rolls, 11, Score) -> Score;
 
    score([10 | Rest], Frame, Score) ->
        score(Rest, Frame + 1, Score + 10 + strike_bonus(Rest));
 
    score([Roll1, Roll2 | Rest], Frame, Score) when (Roll1 + Roll2 == 10) ->
        score(Rest, Frame + 1, Score + 10 + spare_bonus(Rest));
 
    score([Roll1, Roll2 | Rest], Frame, Score) ->
        score(Rest, Frame + 1, Score + Roll1 + Roll2);
 
    score([Roll1], _Frame, Score) -> Score + Roll1;
    score([], _Frame, Score) -> Score.
 
 
    spare_bonus([]) -> 0;
    spare_bonus([Bonus1 | _Rest]) -> Bonus1.
 
    strike_bonus([]) -> 0;
    strike_bonus([Only]) -> Only;
    strike_bonus([Bonus1, Bonus2 | _Rest]) -> Bonus1 + Bonus2.

Ok, that’s an algorithm. To turn it into a usable application, we need to put some sort of interface in front of it, and we need some way to store our data as the game progresses. The simplest interface is the command line, so let’s start with that.

Sketching the CLI

We can start sketching out the command-line interface in the Erlang shell. io:get_line/1 prompts the user and reads a line from standard input. We can enter a player name and a roll. string:tokens/2 will split that into separate strings. string:to_integer/1 will convert the roll to a number we can work with.

    Eshell V5.8.4  (abort with ^G)
    1> Line = io:get_line("Next> ").
    Next> colin 4
    "colin 4\n"
    2> [Player, RollText] = string:tokens(Line, " \t\n").
    ["colin","4"]
    3> {Roll, _} = string:to_integer(RollText).
    {4,[]}

Now we need somewhere to store it. A dictionary will let us keep track of multiple players. dict:new/0 creates it. dict:append/3 takes a key-value pair and adds the value to the list of values for that key. Note that it does not replace the key’s value (dict:store/3 does that). dict:find/2 returns the value for a key, which in this case is a list of rolls.

    4> GameData = dict:new().
    {dict,0,...
    5> NewGameData = dict:append(Player, Roll, GameData).
    {dict,1,...
    6> {ok, Rolls} = dict:find(Player, NewGameData).
    {ok,[4]}

Finally, we pass the list of rolls to our scoring function (not very interesting yet).

    7> Score = bowling_game:score(Rolls).
    4

Normally now, you’d throw a while loop arounds this stuff, but this is Erlang, so we wrap it in a recursive function.

    loop(GameData) ->
        Line = io:get_line("Next> "),
        [Player, RollText] = string:tokens(Line, " \t\n"),
        {Roll, _} = string:to_integer(RollText),
        NewGameData = dict:append(Player, Roll, GameData),
        {ok, Rolls} = dict:find(Player, NewGameData).
        Score = bowling_game:score(Rolls).
        io:format("New score for ~s: ~p~n", [Player, Score]),
        loop(NewGameData).

This is the middle of a recursion, so we need to add a beginning. That’s simple enough – invoke loop with a new dictionary. We could put this in a module and call it from the Erlang shell, but it’s easier if we wrap it in an Escript. (If you’re not familiar with Escript, it lets you run Erlang code as a script, the way you would with Perl, Python, Ruby, or whatever. You don’t even need to define a module, just a main/1 function that takes the command-line parameters as a list of strings.)

    #!/usr/local/bin/escript
    #
    # scorekeeper.erl 
 
    -import(bowling_game).
 
    main(_) -> loop(dict:new()).
 
    loop(GameData) ->
        ...

That’s the beginning and middle of the recursion. We’re not going to bother defining an end – you can just ctrl-c out of the loop. Here’s a sample of what we get when we run it:

    $ ./scorekeeper.erl
    Next> colin 3
    New score for colin: 3
    Next> colin 4
    New score for colin: 7
    Next> colin 10
    New score for colin: 17
    Next> colin 3
    New score for colin: 23
    Next>

(Note that it correctly handles the strike!)

Webify!

So that’s a command-line interface. Now let’s turn it into a simple web app. We’re going to have a single-page rich web client that will send Ajax requests to a REST service, and use Javascript to update its display. The REST service will have pretty much the same API: It’ll take a player and a roll, and return the player’s new score. We’ll use jQuery on the front end and Spooky on the back end. Spooky is a very simple web application framework, much Ruby’s Sinatra, if you’re familiar with that.

The other change here is that we’ll have to deal with concurrency. The command line is inherently sequential, but web services are inherently concurrent. We’ll need to create a mini-service which will control access to our bowling data.

Bowling Service

A bit of a digression: I’ve seen it argued that Erlang is one of the most truly object-oriented languages. The original theory behind object-oriented design was that objects would be like living things that talk to each other. Rather than having a dumb data structure that you manipulate, you would send messages to an object, requesting that it give you information, update its state, perform a calculation, or whatever. You would have an interface to talk to it, but you couldn’t know or manipulate its state directly. Most OO languages implement this by wrapping a dumb data structure in a bunch of smart (or not so smart) accessor methods – usually optional. What Erlang does is create processes to manage access to data. Rather than data having associated methods, Erlang has processes that own data. The only way to get to the data is to talk to the process, and that’s where IPC comes in.

So what does that look like? Well, remember the command-line loop? Let’s break that up into sections.

    loop(GameData) ->
        %% receive input
        Line = io:get_line("Next> "),
        [Player, RollText] = string:tokens(Line, " \t\n"),
 
        %% process input
        {Roll, _} = string:to_integer(RollText),
        NewGameData = dict:append(Player, Roll, GameData),
        {ok, Rolls} = dict:find(Player, NewGameData).
        Score = bowling_game:score(Rolls).
 
        %% print new score
        io:format("New score for ~s: ~p~n", [Player, Score]),
 
        %% recurse with new state
        loop(NewGameData).

Now here’s the message-handling loop.

    loop(GameData) ->
        %% receive input
        receive {From, {append, Player, RollText}} ->
 
            %% process input - this is the same
            {Roll, _} = string:to_integer(RollText),
            NewGameData = dict:append(Player, Roll, GameData),
            {ok, Rolls} = dict:find(Player, NewGameData),
            Score = bowling_game:score(Rolls),
 
            %% respond with new score
            From ! Score,
 
            %% recurse with new state
            loop(NewGameData)
        end.

That’s it. Instead of waiting for command-line input, we wait for a message. Instead of printing our response, we send a message back. GameData is a local variable to loop/1. Nobody else can see it; there’s only one process that can change it.

Again, that’s the middle; how do we start this recursion? As with the CLI, we need a beginning function that creates the dictionary. The difference here is that instead of calling loop/1 directly, we wrap it in a closure and spawn it off as a new process.

    init() ->
        Data = dict:new(),
        Start = fun() -> loop(Data) end,
        spawn(Start).  % returns process id

For convenience, we’ll add an append/3 function for our clients. It hides the message format, and makes the asynchronous request synchronous. This makes it look a lot like we’re creating an object and updating it.

    append(Player, RollText, Pid) ->
        Pid ! {self(), {append, Player, RollText}},
        receive Resp -> Resp end.

REST API

Now that our back-end service is done, we move to the REST interface. Let’s keep this simple. Let’s just take a GET request – a straight URL – something like this:

http://localhost:8000/add/Player/Roll

So for example:

http://localhost:8000/add/colin/4

Yes, I know this isn’t entirely kosher for a REST API – we shouldn’t be modifying state with a GET – but we’re just doing the simplest thing that works here.

Spooky App

To create a Spooky web application, we just need to create a module that has the “spooky” behavior and exports Spooky’s callback functions. To use our bowling module, we’ll need to import that.

    -module(bowling_web).
    -behaviour(spooky).
    -export([init/1, get/2]).  % Spooky API
    -import(bowling_service).

init/1 is called once, when the server starts up. It starts up the bowling service we just defined, and registers its process id as bowl_svc. That lets us refer to it by name, so we don’t have to pass the PID around somehow. This would be especially useful in a situation where the service might be restarted and get a new process id. Other processes could continue to use it without needing to know the new PID. The return value configures Spooky to start up on port 8000.

    init([])->
        register(bowl_svc, bowling_service:init()),
        [{port, 8000}].

get/2 is called to handle each HTTP GET request. Spooky splits up the URL’s resource path into a list of strings. That makes it easy to match on patterns, like so.

     %% REST handler
    get(_Req, ["add", Player, RollText])->
        Score = bowling_service:append(Player, RollText, bowl_svc),
        {200, io_lib:format("~p", [Score])};

We’ll also need to define handlers for the base web page and any associated resources, such as Javascript or CSS files, or images. If there is no resource path – the URL is just the host and port – we’ll return our base page. Otherwise, we treat the resource path as a relative path to a file, and try to return that.

     %% static page handlers
    get(Req, [])-> get(Req, ["form.html"]);  % main page
 
    get(_Req, Path)->  % other static resources
        Filename = filename:join(Path),
        case file:read_file(Filename) of
            {ok, PageBytes} -> {200, binary_to_list(PageBytes)};
            {error, Reason} -> {404, Reason}
        end.

Run it!

We can start up the Spooky server from the Erlang shell as long as we add Spooky and its dependencies to the code path. Note that what we’re doing here is starting the Spooky server, and telling it to use our bowling_web module as its plug-in request handler.

    $ erl -pa $SPOOKY/ebin -pa $SPOOKY/deps/*/ebin
    ...
    Eshell V5.8.4  (abort with ^G)
    1> spooky:start_link(bowling_web).
    {ok,<0.35.0>}
    2>

Of course, I got tired of typing that every time, so I wrote an Escript to do it. This uses a SPOOKY_DIR environment variable to find all the dependencies. As an extra bonus, it compiles all our modules for us. Note that the same process is compiling them and then loading them. This is Erlang’s hot code reloading in action, in a low-key way.

    #!/usr/local/bin/escript
 
    main([]) ->
        SpookyDir = os:getenv("SPOOKY_DIR"),
        %% Add spooky and its dependencies to the code path.
        true = code:add_path(SpookyDir ++ "/ebin"),
        Deps = filelib:wildcard(SpookyDir ++ "/deps/*/ebin"),
        ok = code:add_paths(Deps),
 
        %% Compile our modules, just to be safe.
        c:c(bowling_game),
        c:c(bowling_service),
        c:c(bowling_web),
 
        spooky:start_link(bowling_web),
        io:format("Started spooky~n"),
 
        io:get_line("Return to exit...  "),
        spooky:stop().

This is a script, and when it gets to its end it shuts down any processes it started, including Spooky. So while we started up Spooky as before, we then needed a way to keep the script from exiting. So we called io:get_line/1, which will hang until the user enters something. At that point it returns and goes on to the spooky:stop/0 line, which shuts down gracefully.

REST interaction

Now that the server is up and running, we can test it by hitting our REST service directly from a browser. We can see that it gets the same results as our command-line run did.

Webapp interaction

Now we get to the web client itself. Hitting our base URL brings up the main page.

We start off by adding a player. This is entirely client-side. Our REST service has no separate way of creating or registering players other than adding scores for them.

Now that we have a player, we can start entering scores for them. Again, we get the same results as with our command-line and REST interfaces.

Now, if you poke at this a little bit, you’ll find it’s far from perfect. As is, it won’t handle invalid data (rolls greater than 10, for example). The rolls are stored independently in the client and the server (try sending a direct REST request from another browser in the middle of a game). It might be nice if the server returned the full list of rolls along with the score, so the client didn’t have to keep any state in its display. It would be extra nice if it grouped the rolls by frame. If you want a good little learning project, try fixing any of these. You could also try implementing this in a different framework, like mochiweb or webmachine.

Ta-dah!

So, we’ve created an elegant little algorithm, and built both a command-line and web interface to it. You’ve gotten a little taste of what it’s like to work with Escript and Spooky. Hopefully, you’ve started learning to think in Erlang, and are getting the hang of the recursion and IPC patterns.

There’s a bunch of extra stuff that I didn’t have time for in this talk, including a command-line testing tool for the REST service. You can find that, along with the full source for these examples, unit tests and so on in my GitHub project for this talk. That also has the S9 markup source for my slides, which you can see on GitHub Pages. You can follow my continuing adventures, and catch up on previous experiments, ponderings, and rants here on my blog.

Think small, have fun

By coincidence, I’d done it about a year earlier in Python for another hack night. In an OO language, it’s pretty obvious to model it as a Game object which contains a list of Frame objects. It’s less clear how to handle the fact that the score for a frame may depend on rolls in other frames. Either frames have to know about other frames, or rolls have to be added to multiple frames, or the Game class has to handle the spare and strike calculations, or something like that. All of the options feel a little awkward, so you can get wrapped around the axle there. But in the end, I had a solution I was pretty happy with. It weighed in at 53 lines of code.

The final version* in Erlang was 15. Wow. I think of Python as a pretty compact language, and the Erlang code is less than a third as long. And it’s not some high-density, Perl-style line noise; it’s clearer – hardly more than a definition of the rules of the game.

(* Thanks to Rusty for pointing us in the right direction here.)

score(Rolls) -> score(Rolls, 1, 0).
 
score(_BonusRolls, 11, Score) -> Score;
 
score([10 | Rest], Frame, Score) ->
    score(Rest, Frame + 1, Score + 10 + strike_bonus(Rest));
 
score([Roll1, Roll2 | Rest], Frame, Score) when (Roll1 + Roll2 == 10) ->
    score(Rest, Frame + 1, Score + 10 + spare_bonus(Rest));
 
score([Roll1, Roll2 | Rest], Frame, Score) ->
    score(Rest, Frame + 1, Score + Roll1 + Roll2);
 
score([Roll1], _Frame, Score) -> Score + Roll1;
score([], _Frame, Score) -> Score.
 
 
spare_bonus([]) -> 0;
spare_bonus([Bonus1 | _Rest]) -> Bonus1.
 
strike_bonus([]) -> 0;
strike_bonus([Only]) -> Only;
strike_bonus([Bonus1, Bonus2 | _Rest]) -> Bonus1 + Bonus2.

(If you’re completely new to Erlang, the multiple function definitions are essentially implicit if-elseif conditions matched against the values of the parameters.)

So what makes the Python code so much longer? A lot of it is spent querying and updating the state of the objects. That’s also where a lot of the design complexity came from, figuring out how each object interacts with the others. In the Erlang solution, we do an end run around all that by just using simple lists and variables. The input is a list of numbers; the output is a single number; why use anything more complex in between?

The point of this is not which language is better; it’s that using Erlang showed me a different way to solve this problem. I went back and translated the Erlang code into Python. It’s straightforward: one function with a big if-elif block. You can do it as either a recursive function or a while loop. Either way, it ends up being pretty much the same length as in Erlang.

(As a side note: A nice thing about using recursion is that it simplifies variable scoping. At each step, you’re calling a function with an explicit set of parameters. This makes it clear what state is being passed along. If you’re just modifying variables and looping, it would be easy to forget to update a value.)

So why didn’t I come up with the simpler solution when I wrote this in Python? It’s mostly a matter of culture. As with natural languages, programming languages come with a layer of culture; what the normal way of writing programs is. Because Python is an OO language, the first question I asked was “What are the objects?” What are the concepts in the problem domain, and how do I map them to classes? While that may turn out to be a necessary intermediate step, it’s not actually the end goal. It’s easy to lose sight of that. You can jump right in and start coding up classes: constructors, accessors, unit tests and so on. You can write a fair amount of code without thinking too much about what you’re trying to accomplish. That feels like progress, but it may just be a diversion. In this case, it’s not necessary and only adds complexity.

In a functional language like Erlang, the first question is “What are my inputs and outputs?” What is the end result I’m trying to get to, and where am I starting from? It keeps you focused on the data you have and what you’re trying to accomplish. Erlang’s pattern matching brings a lot of power to working with simple data structures. If you want to create real mutable objects, you can, but Erlang makes you deal with concurrency up front, so it’s less trivial. It encourages you to think of the simplest thing that works.

It seems like this should be a transferable skill, but I suspect there’s a catch. I could certainly use recursion and simple data structures in my Python code (or Ruby and maybe to a lesser extent Java). If it cuts out a lot of extraneous object munging and dramatically reduces the line count, that should make it more maintainable. The trouble is that “maintainable” means “maintainable by other programmers”. My code may be more elegant and concise, but if it’s using programming idioms they aren’t familiar with, they’re going to have trouble with it. I can trust that Erlang programmers are familiar with recursion because Erlang uses it for everything. That’s not true of Python. This is the flip side of culture: There are things that you could say – that are grammatically correct – but you wouldn’t.

So for reference, here’s the Python version:

#!/usr/bin/python
 
max = 100
for x in range(1,max+1):
    if ((x % 5) == 0) and ((x % 3) == 0):
        print "fizzbuzz"
    elif (x % 3) == 0:
        print "fizz"
    elif ((x % 5) == 0):
        print "buzz"
    else:
        print x

Here’s my first draft of the Erlang version

#!/usr/local/bin/escript
-mode(compile).
 
main([]) -> fizzbuzz(1).
 
fizzbuzz(X) when X > 100 ->
    ok;
fizzbuzz(X) when ((X rem 5) == 0) and ((X rem 3) == 0) ->
    io:format("fizzbuzz~n"),
    fizzbuzz(X + 1);
fizzbuzz(X) when ((X rem 3) == 0) ->
    io:format("fizz~n"),
    fizzbuzz(X + 1);
fizzbuzz(X) when ((X rem 5) == 0) ->
    io:format("buzz~n"),
    fizzbuzz(X + 1);
fizzbuzz(X) ->
    io:format("~p~n", [X]),
    fizzbuzz(X + 1).

So it works, and it uses pattern matching and recursion, but it feels kinda messy. That version is all about the side effects. Let’s split out the printing, and see what we get.

#!/usr/local/bin/escript
-mode(compile).
 
main([]) ->
    Out = fizzbuzz(1, []),
    print(Out).
 
fizzbuzz(X, Out) when X > 100 ->
    lists:reverse(Out);
fizzbuzz(X, Out) when ((X rem 3) == 0) and ((X rem 5) == 0) ->
    fizzbuzz(X + 1, [fizzbuzz|Out]);
fizzbuzz(X, Out) when (X rem 3) == 0 ->
    fizzbuzz(X + 1, [fizz|Out]);
fizzbuzz(X, Out) when (X rem 5) == 0 ->
    fizzbuzz(X + 1, [buzz|Out]);
fizzbuzz(X, Out) ->
    fizzbuzz(X + 1, [X|Out]).
 
print([]) -> ok;
print([First|Rest]) ->
    io:format("~p~n", [First]),
    print(Rest).

Ok, that’s conceptually a bit cleaner, but the code itself actually feels more cluttered. Thinking about it some more, I realize the fizzbuzz function itself isn’t recursive. Each value can be calculated independently. fizzbuzz(6) is fizz, whether it appears in a sequence or not. We have the fizzbuzz calculation mashed together with iterating over a range. Let’s rework that so that the list printing function handles the iteration instead.

#!/usr/local/bin/escript
-mode(compile).
 
main([]) ->
    print_loop(1).
 
print_loop(I) when I > 100 ->
    ok;
print_loop(I) ->
    io:format("~p~n", [fizzbuzz(I)]),
    print_loop(I + 1).
 
fizzbuzz(X) when ((X rem 5) == 0) and ((X rem 3) == 0) -> fizzbuzz;
fizzbuzz(X) when ((X rem 3) == 0) -> fizz;
fizzbuzz(X) when ((X rem 5) == 0) -> buzz;
fizzbuzz(X) -> X.

Now the fizzbuzz function itself is very declarative, and the looping is cleaner and simpler. This separation lets us parameterize the function and max value in the print_loop without modifying the fizzbuzz rules. That’s kinda nice.

#!/usr/local/bin/escript
-mode(compile).
 
 
main([]) ->
    print_loop(fun fizzbuzz/1, 100, 1).
 
print_loop(_Func, Max, I) when I > Max ->
    ok;
print_loop(Func, Max, I) ->
    io:format("~p~n", [Func(I)]),
    print_loop(Func, Max, I + 1).
 
fizzbuzz(X) when ((X rem 5) == 0) and ((X rem 3) == 0) -> fizzbuzz;
fizzbuzz(X) when ((X rem 3) == 0) -> fizz;
fizzbuzz(X) when ((X rem 5) == 0) -> buzz;
fizzbuzz(X) -> X.

The end result is about the same number of lines of code as the Python version, so I can’t say that it’s more efficient. But it’s surprisingly different. You can go back and re-write the Python version to have that functional separation, but it doesn’t feel the same.

#!/usr/bin/python
 
def fizzbuzz(x):
    if ((x % 5) == 0) and ((x % 3) == 0):
        return "fizzbuzz"
    elif (x % 3) == 0:
        return "fizz"
    elif ((x % 5) == 0):
        return "buzz"
    else:
        return x
 
max = 100
for x in range(1,max+1):
        print fizzbuzz(x)

And would you really? The original Python version looks fine; the second one is actually uglier, with its multiple returns. It’s not just a matter of what’s possible in a language, or what’s more conceptually elegant; it’s what the language can express cleanly and easily, and what’s familiar to its developer community.

What’s cool about Erlang?

Normally, if you ask “What’s cool about Erlang?” people will talk about scalability. They’ll talk about performance, about servers that handle thousands of requests a second without breaking a sweat. They’ll say crazy things like “Nine 9s uptime.” They’ll talk about how concurrency is baked in to the language, how spawning processes, monitoring them, and sending messages between them is trivial. They’ll talk about how you can upgrade your app while it’s running, with each process loading its new code when it’s convenient. They’ll admit it has some quirks – weird syntax and awkward constraints – but that’s what you have to put up with to get the good stuff.

That is in fact amazingly cool stuff, but what do I do with it? I don’t run into a lot of problems that require that massive scale of solution. I run into a lot of problems that require some quick little scripty thing. Maybe bigger ones might run to a few hundred lines of code. If I did have to build something massively scalable, I’d be leary of doing it in a language I didn’t know. There’s a chicken-and-egg problem here.

I’d like to take a different look at Erlang, one that might help break that deadlock. I’m going to talk about how Erlang is actually a good language for small programs, and how its “weird” syntax and language features actually help you write clearer, more concise, and more elegant code. They take a bit of getting used to, but you’ll adapt faster than you expect, and you won’t look at other languages quite the same way again. Part of what’s cool about Erlang is what’s cool about functional programming, but it goes beyond that with its own features that make it even more powerful and expressive.

Procedural, OO, Functional

A little background in case you’re not familiar with functional programming. There are three basic styles of programming: Procedural, object-oriented, and functional. (Wikipedia lists about 60, but in my mind, and for the purpose of this discussion, these are the big ones.) Like other human languages, programming languages are not just rules for expressing thoughts; they also embody a culture, a way of looking at and making sense of the world. Most are not strictly one style, but creoles, mutts to varying degrees, blending influences from the others.

Procedural programming is very narrative. It explains the world as a sequence of actions. Do this, do that, then do this other thing; in that case, do this instead. Functions are like anecdotes, little incidents that the narrative is built up from. There are things – data structures – but they’re fairly simple. They’re passive – things are done to them. The narrative may be meandering and incoherent, but there’s a continuity to it. C is a procedural language. Perl is a procedural language with objects bolted on as an afterthought.

This fits pretty well with how people see the world. We make sense of it through stories. Even the barest facts need narrative context to be meaningful.

Object-oriented programming focuses on things; things that do things. This is another way that people explain the world, though in a narrower context. It looks more at actors and elements in isolation. We talk about how people behave, or animals; what their capabilities are. We talk about the utility of things, their features and limits.

OO programming focuses on these actors and elements, what you can do with them and how they interconnect. It can be powerful and flexible, but it also runs the risk of losing the narrative. Instead of jumbled strands of procedural spaghetti code, you can end up with ravioli code – code where it’s clear what each individual part does, but it’s hard to piece together the whole. By the time you’ve traced your way down a chain of a half-dozen object linkages, you have no idea of what they’re actually accomplishing. Java is a mostly OO language. Python and Ruby are strongly OO, with some functional influence (stronger in Ruby).

Functional languages define a mapping from inputs to outputs. This is not a normal human way of looking at the world; it’s very abstract and mathematical.

But all programming is ultimately about taking in some data and producing some sort of output, whether it’s to a screen, database, file, network, or whatever. It’s inherently abstract and mathematical. A language that forces you to focus on that can be very powerful and expressive. This is all hard to explain without examples, so we’ll be getting into those soon.

Doing this requires that you kinda rewire your brain. The good news is that that’s much easier than you’d expect, and a lot more fun. Especially at first, you’ll get a lot of “a-ha!” moments, with that little burst of neuro-chemical reward for figuring something out. As with most games, it can be bewildering at first, but there are patterns and strategies you learn that help you make sense of it.

Working with procedural languages, you can do this stream of consciousness thing. Just start at the first thing you do, and step through it from there. I think of OO programming as kinda like working with Play-Doh. I figure out what the main objects are, the big chunks of functionality. Then I mush those around, occasionally ripping out some bit of functionality from one class and glomming it onto another. As I work on them, they gradually get smoother and more coherent. Both of these are very incremental processes. I can just start writing code and see what comes out of it.

The experience of writing functional code is kinda like doing math. It requires a lot more up-front thought. There’s a lot more staring at a blank screen or scribbling tentative ideas on a whiteboard. But when you figure it out, you know it; it really clicks. It’s an almost physical sensation.

So why would you bother? Why would you code in an unituitive language? Why would you take longer to figure out how to do something functionally, when you could just slap it together quickly in an OO or procedural language? Because what you end up with may be much simpler and clearer. Sometimes, the procedural solution looks like this:

1 + 2 + 3 + 4 + ... + n

and the functional solution looks like this:

n(n+1)/2

Reading Erlang

In case you’re not familiar with Erlang code, I’ll take a couple minutes to give you the basics – just what you’ll need to be able to read the examples I’m going to throw at you; not enough to really write Erlang from scratch.

Even people who like Erlang will talk about its “weird” syntax. Really, “weird” just means “doesn’t look like C.” Almost every mainstream language since C has adopted similar syntax. Java, the current 800-pound gorilla, deliberately tried to look as familar as possible to the legions of existing C developers. C# went as far as keeping the name. Javascript, Perl, Php, Python, and even Ruby are all basically C-like.

So, to start with the basics of Erlang:

%% This is a comment.
Variable  % starts with a capital letter.
atom  % this is a value, not a variable - like :atom in ruby
[1,2,3]  % list
{1,2,3}  % tuple

You can do multi-assignments with lists like so:

[X, Y, Z] = [1, 2, 3]  % X=1, Y=2, Z=3
[First, Second | Rest] = [1, 2, 3, 4]  % First=1, Second=2, Rest=[3,4]
[First, Second | Rest] = [1, 2]  % First=1, Second=2, Rest=[]

Note that this will cause an error, because 3 & 4 have nowhere to go:

[First, Second] = [1, 2, 3, 4]

If you want to ignore them, use an underscore, either by itself or in front of a variable name:

[First, Second | _ ] = [1, 2, 3, 4]  % 
[First, Second | _Rest ] = [1, 2, 3, 4]

Function definitions are simple, since neither the return value nor the parameters are typed:

my_func(Param1, Param2) ->
    %% do stuff
    X = some_func(Param1),
    Y = some_other_func(Param2),
    Status = ok,  % assign value to variable
    {Status, X, Y}.  % return a tuple

That may look a bit quirky to most programmers, but mathematicians would be happy that this is a valid function definition in Erlang.

f(X) -> X * 2.

You can overload functions, defining different versions of a function with the same name but different numbers of parameters:

my_func(Param1) ->
    my_func(Param1, "default").
 
my_func(Param1, Param2) ->
    %% actually do stuff...

Erlang takes overriding a step further, and lets you do it by value.

my_func([]) -> ...
 
my_func([Only]) -> ...
 
my_func([First, Second | Rest]) -> ...

It also lets you use “guards” – boolean expressions – to further break up your function definitions:

my_func(0) -> ...
 
my_func(X) where X > 0 -> ...
 
my_func(X) where X < 0 -> ...

It may not make much practical difference, but this style of code feels more declarative. The function definitions feel more distinct and self-contained than a sequence of if-else checks.

Guards also show up in ‘if’ expressions. In Erlang, these are implicitly if-elseif checks. Here, we’ll go down the left side conditionals until we find one that matches; then we’ll return the corresponding value on the right.

if
    X >= 100 -> 100;
    X >= 0   -> X;
    true     -> 0
end

Case statements work similarly. Here we get the return value of my_func(X) and go down the left side until we find a match; then return the value on the right. If the left side is an unassigned variable, it counts as a match, and the case value is assigned to it. This example says, “if my_func(X) returns ‘error’, return an empty list; otherwise assign the value to List and return that.”

case my_func(X) of
    error -> [];
    List -> List
end

That may seem a little bizarre, but it makes for very elegant error handling:

Output = case file:open(Filename, [read]) of
    {ok, Handle} ->
        process_file(Handle);
    {error, Reason} ->
        log_error(Reason, Filename),
        []
end.

Essentially, this says that if the return status of file:open() is ‘ok’, then
Output = process_file(Handle)
If it’s ‘error’, then log it and set
Output = []

Now, turning your Erlang code into a command-line script is as simple as this:

#!/usr/bin/escript
 
main(_Args) ->
    io:format("Hello World!~n").

If you want to handle command-line arguments, you can do this:

#!/usr/bin/escript
 
main(Args) -> main("World", Args).  % invoked first, sets Name="World"
 
main(_, ["-h" | _Args]) ->
    io:format("Help! Help!~n");  % print message, exit.
 
main(_, ["-n", Name | Args]) ->  % ignore old name, pop new one off Args.
    main(Name, Args);            % recurse to next arg with new Name.
 
main(Name, []) ->
    io:format("Hello, ~s!~n", [Name]).  % print message, exit.

Note that this will handle any number of “-n name” options, and only keep the last one. It will exit when it gets a “-h” arg, or when it runs out of args. It will exit with an error if it gets any other kind of arg, or if “-n” is the last arg.

Indent Parser

I first looked at Erlang about three years ago. It was cool, and I was excited to build an app in it, but what to do? I didn’t have any personal projects that made sense as massively distributed, concurrent apps. So it sat on the shelf.

I came back to Erlang in a roundabout way. I was going through one of my periodic “get my act together” phases. This time I decided that what I really needed was a project database that I could keep in a flat file in some sort of markup language. That would allow me to review the whole thing easily, and print it all out in a compact form if need be. I’d group projects, sub-projects, and actions just by indents.

project
    action
    sub-project
        action
    action
project
    sub-project
        sub-project
            action

And the actions would have markup to indicate context, due date, priority, and so on.

work on presentation [computer, quiet] (2011-08-24) !!

Then I could write a script to parse it and list my next actions by context in the order I should tackle them.

I never got that far. I got as far as writing a Ruby script to parse the indented text into a logical tree structure, that would turn this

into this

It had the OO structure you’d expect: Parser, Lines, Trees, Nodes, Leaves. It took each input line, and tacked it on to the tree in the next appropriate spot, building it out like so:

That much went well, and I felt pretty good about it. I thought I had a nice clean separation of reponsibilities, and some pretty concise code. But then I read Hackers and Painters. It’s a collection of essays by Paul Graham, of HackerNews and Y Combinator fame. Very smart guy. You probably won’t agree with everything he says, but he’ll make you think. Throughout the book, he plugs Lisp as the most powerful language in the world and the choice of all True Programmers. Fine. I’ll pick up that gauntlet and give it a shot.

I figured I’d take my little Ruby script and re-write it in Lisp, just sort of transliterate it. Pull all the functions out, write them in Lisp; convert the objects to simpler data structures and pass them in as parameters. That didn’t go so well. It’s hard to explain quite why, but the end result was that the code was messy, convoluted, and incoherent, and still didn’t build certain structures correctly. I’d lose track of where I was in tree and where I was in the input stream. Trying to fix one case tended to break another. Most of all, it felt wrong. It felt like I was trying to jam a square peg into a round hole. So how could I do this in a more “Lisp-y” way?

Recursion

Lisp is all about recursion. You’d think that anything involving a tree structure would have to be recursive, but not really. The way I was going about it was actually very linear: taking the input one line at a time and walking up and down the tree until I found where to attach it. Recursion is really about breaking a problem into self-similar pieces, so you can define the whole in terms of its parts. Ok, but what does that look like in the real world?

Well, for contrast, here’s the procedural way of transforming a list of things. You take an input array, create an output array, and iterate through, mapping each input element to its corresponding output element.

String[] process(String[] input) {
    String[] output = new String[input.length];
    for (int i=0; i < input.length; i++) {
        output[i] = do_stuff(input[i]);
    }
    return output;
}

The recursive approach is to focus on defining (1) how you transform each part, and (2) how each part relates to the whole. The idiom here is to take the first thing off the input list, transform it, add it to the output list, and then invoke yourself with the new lists.

    process([First|Rest], Output) ->
        NewFirst = do_stuff(First),
        process(Rest, [NewFirst|Output]);

This defines what you do for each element N, and how you get to N+1. To round it out, you need to define your begin and end points.

    Output = process(Input, []).
 
    process([], Output) ->
        lists:reverse(Output).

(It’s convention to add each element to the beginning of the output list, so you need to reverse it when you’re all done.
It’s apparently more efficient.)

The key thing here is that this is a common idiom in Erlang. Procedural and OO languages allow recursion, but you don’t often have a good reason to use it, so it remains this strange and intimidating concept. If you do use it, you usually put comments all around it saying “Recursion happening here! Stand back! Do not touch!” In Erlang, it’s how you do everything. Anything that you’d use any sort of loop structure for in another language, you do with recursion. You get used to it pretty fast.

Lisp

So, back to the problem at hand: How do I do this parsing thing in a more recursive way? It ended up being the classic math problem experience. I beat my head against it for a couple hours, sketching out ideas in code, all of them abject failures. Eventually I gave up and went to bed. Lay there staring at the ceiling. About twenty minutes later, I was just starting to drift off, and it hit me. It was one of those solutions where you immediately know you’re right because it’s so simple and obvious in retrospect, and you feel like an idiot for not having thought of it sooner.

The first line in the input is the first node in the tree. Every other line either belongs to one of its children, or one of its siblings. So you can run through the lines until you hit one with the same indent as yours. That’ll be your first sibling; everything before it belongs to your children.

All you have to do now is recurse and apply the same logic to each block. Your first child (if any) splits its list into its children and siblings. Your first sibling (if any) does the same with its list.

And so on…

And so on…

And so on…

…Until you’re all the way down at the leaf nodes. Here, the logic is trivial: No lines, no children, no siblings. And you return back up the tree.

So what does the code look like? Well, I don’t have time to teach you how to read Lisp. Fortunately, at this point it occurred to me, “Hey, Erlang is a functional language, too.” Maybe porting from Lisp to Erlang would be fairly straightforward. As it turned out, not only was the straight conversion easy,
but I was also able to go back and rework bits to take advantage of Erlang’s pattern matching features to make it even cleaner and simpler.

Erlang

Here’s the core of the Erlang functionality. We define records – data structures – for our input lines and output nodes. We pop the first line off the list, and split the rest of it into child and sibling lines. We then recurse on the child lines to get our child nodes, and on the sibling lines to get our sibling nodes. We create a node with its list of children, add it to the front of its list of siblings, and return that.

-record(line, {content, indent}).
-record(node, {content, children}).
 
build_nodes([First|Rest]) ->
    {ChildLines, SiblingLines} =
        split_list(First#line.indent, [], Rest),
    Children = build_nodes(ChildLines),
    Siblings = build_nodes(SiblingLines),
    Node = #node{content=First#line.content, children=Children},
    [Node|Siblings];
 
build_nodes([]) -> [].

The split_list function follows much the same pattern as our earlier recursion example. All the lines start out in the sibling list; the child list is empty. Instead of transforming each sibling line, it pops it off and adds it to the child list. When it hits a line that’s not indented greater than the minimum, it pushes that back onto its siblings, and returns the two lists. (Again, reversing the children because we’ve been adding them to the beginning of the list.) If that doesn’t happen before its sibling list is empty, it’s all children and no siblings.

split_list(MinIndent, Children, [First|Rest]) ->
    if 
        First#line.indent > MinIndent ->
            split_list(MinIndent, [First|Children], Rest);
        true ->
            {lists:reverse(Children), [First|Rest]}
    end;
 
split_list(_Indent, Children, []) ->
    {lists:reverse(Children), []}.

I’m not sure if that’s actually fewer lines of code than the equivalent procedural code, but it feels clearer. The stop condition is unambiguous, as opposed to the usual “i >= list.length”.

Bowling

I stumbled across another good example in the Bowling Game. The challenge is just to write a program to score a bowling game. This is a classic beginner coding exercise. It’s very self-contained, but it has just enough complexity and ambiguity to be interesting.

When you’re programming in OO languages, there’s a pretty obvious class structure. You have Games and Frames and Rolls, and you calculate a score. What makes this interesting is that spares and strikes cause rolls from future frames to be added to the score for that frame. This gives you a bit of metaphor shear. Rolls belong to one frame, but may be counted double or even triple. Do you put that logic in the Frame class, requiring frames to know about their following frames? Or do you put it in the Game, as the keeper of inter-frame knowledge?

I had written this in Python as part of a pair programming exercise about a year ago. I was pretty happy with the results. The code itself was fifty-some lines. There was a clean separation of responsibilities. The Frame class had all sorts of useful methods for querying its state. The Game class hid all that and provided a clean interface for adding rolls and getting the score.

Then for an Erlang meetup, we did the same exercise. This was the result:

score(Rolls) -> frame(Rolls, 1, 0).
 
frame(_BonusRolls, 11, Score) -> Score;
 
frame([10|Rest], Frame, Score) ->
    [Bonus1, Bonus2|_] = Rest,
    frame(Rest, Frame + 1, Score + 10 + Bonus1 + Bonus2);
 
frame([First,Second|Rest], Frame, Score) when (First+Second==10)->
    [Bonus1|_] = Rest,
    frame(Rest, Frame + 1, Score + 10 + Bonus1);
 
frame([First,Second|Rest], Frame, Score) ->
    frame(Rest, Frame + 1, Score + First + Second).

That’s ten lines of code, a 5x improvement over Python. And (again, once you get used to Erlang’s syntax) it’s amazingly clear. frame() takes a list of remaining rolls, a frame number, and a total score. The different definitions of frame() handle the end-of-game, strike, spare, and normal conditions. Each consumes rolls, boosts the score, and recurses to the next frame.

So what about unit tests? Here you go:

test() ->
    test(0,   [0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 0,0]),
    test(20,  [1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1]),
    test(150, [5,5, 5,5, 5,5, 5,5, 5,5, 5,5, 5,5, 5,5, 5,5, 5,5, 5]),
    test(47,  [1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 10, 10 ,9]),
    test(173, [7,3, 7,3, 7,3, 7,3, 7,3, 7,3, 7,3, 7,3, 7,3, 7,3, 10]),
 
test(Expected, Rolls) ->
    case score(Rolls) of
        Expected -> io:fwrite("Pass~n");
        Scored -> io:fwrite("Fail: expected=~p, scored=~p~n",
            [Expected, Scored])
    end.

That’s it: a list of test cases with expected result and input rolls; and a simple method for evaluating them. There are unit test frameworks for Erlang, but why bother when it’s as simple as this?

The Funhouse Mirror

Looking back at the Python code, I realize that it spends lot of time on peripheral issues: What is the nature of a Frame? How does it relate to the Frames around it? Is it responsible for validating its inputs, or does the Game do that? (Usually both, which makes for some redundancy.) Does it bear responsibility for calculating its own value, or does that also fall to the Game class? What’s the “right” way to divide up functionality that involves two or more classes? It gets very metaphysical.

And in the end, these questions have no direct bearing on the task of calculating the score. They’re all about the model, which is just an intermediate stage between input and output. You also tend to end up with a lot of “what if?” code: functions that could be useful, but aren’t actually used. You get boilerplate accessor methods. After working with Erlang for a bit, OO code seems to spend a lot of time talking about the problem without actually solving it. You look at all these interlocking parts and think, “Well! It certainly is very busy, but what is it actually doing?” Erlang takes more up-front thought, but is it really a good sign that you can write a lot of OO model code without really understanding what you’re trying to accomplish?

So I say, Embrace the weirdness. Give Erlang a shot. Spend a few hours – it shouldn’t take long. The syntax and idioms start to feel natural in a surprisingly short time. Even if you never get to use it for work, it’ll change the way you think about programming.

Remember that you can start small, that you can write Erlang programs that aren’t distributed or concurrent. You can write little shell scripts or file parsers or cgi scripts. And you can be secure in the knowledge that you could scale if you had to, without having to significantly re-work your code.

I’m dealing with data that gets passed through layers and layers of components, shuttling to various web services and back. There’s all this dynamic XML configuration and auto-generated Java code. When something gets garbled in translation, it’s great to have a four-line script to send a bare-bones HTTP POST to a web service and see what comes back.

Deploying a new version of our code is this tedious and unforgiving process of ftping the code down from the staging server, unpacking it to a deployment directory, shutting down the web server, backing up the old version, copying the new version into place, and restarting the web server. It’s actually more complicated than that, but I forget all the details – because I don’t have to remember them. It’s all scripted. I just run my script and hit return a bunch.

Computers have great memory and terrible judgement. My scripts remember all the nitpicky details of what I have to do, but they stop and ask me whenever a decision has to be made. We work well together.

I’ve never really thought of myself as a “Bash programmer”. Scripting is just an adjunct to “doing stuff on Linux machines.” But I’ve been doing enough of it lately that I decided to actually treat Bash as a programming language: Learn more about what you can do with it, and try to create re-usable code instead of single-use scripts. It turns out it has a ton of features that I’ve never used. Most surprisingly, it even allows some limited metaprogramming. For example:

$ name="Colin MacDonald"
$ city="Arlington"
$ state="VA"
$ echo $name
Colin MacDonald
$ vars="name city state"
$ for x in $vars ; do echo "$x=${!x}" ; done
name=Colin MacDonald
city=Arlington
state=VA

In the for loop, when x is “city”, ${!x} is $city. Ok, clever, but what do you use it for? Well, it lets me save configuration settings to a file with a generic function that goes something like:

function save_config () {
    for x in $vars ; do
        echo "$x=\"${!x}\"" >> $config_file
    done
}

And gets called like so:

save_config "name city state"

I’ve been able to extend this sort of meta-variable handling to parsing command-line parameters and prompting the user for input, so I now have a reusable library for managing script configuration in a fairly clean way. Now my script can just have:

load_config
process_options "[d]=debug" "[n]=name [c]=city [s]=state" "$@"
confirm_config "[name]=\"Name:\" [city]=\"City:\" [state]=\"State:\""
save_config "name city state"

When you run it, it sets the variables based on the command-line params, then prompts you to confirm the settings and add any missing ones.

$ ./my_script.sh  -n "John Smith"
Name: [John Smith] 
City: [] Arlington
city changed to 'Arlington'
State: [] VA
state changed to 'VA'

Those settings will be saved in the config file

$ cat .my_script.cfg 
name="John Smith"
city="Arlington"
state="VA"

And the next time you run your script, you get

$ ./my_script.sh
Name: [John Smith] 
City: [Arlington] 
State: [VA]

In my world, these will be things like URLs and SOAP message definitions – things that will be different on each machine, and which I really only want to type in once. So, handy.

If you want to grab this library and a template of how to use it, they’re up on GitHub. If you want to learn more about metaprogramming tricks in Bash, pop the hood and dig around in there. The syntax is a bit strange and can be very finicky, so there’s some hard-won knowledge baked into the library.

Here’s the meat of the Erlang code:

-record(line, {content, indent}).
-record(node, {content, children}).

parse_text_line(Text) ->
    %% converts "    text" to (:content "text" :indent 4)
    Content = string:strip(string:strip(Text, left), left, $\t),
    Indent = length(Text) - length(Content),
    #line{content=string:strip(Content, right, $\n), indent=Indent}.

%% Split a list of lines in two, breaking on the first line whose indent is =< the minimum.
split_list(_MinIndent, List1, []) -> [lists:reverse(List1), []];
split_list(MinIndent, List1, [First|Rest]) when First#line.indent =< MinIndent ->
    [lists:reverse(List1), [First|Rest]];
split_list(MinIndent, List1, [First|Rest]) ->
    split_list(MinIndent, [First|List1], Rest).

%% Parse a list of 'line' record into a tree structure, based on indent
build_nodes([]) -> [];
build_nodes([First|Rest]) ->
    [ChildLines, SiblingLines] = split_list(First#line.indent, [], Rest),
    Children = build_nodes(ChildLines),
    Siblings = build_nodes(SiblingLines),
    Node = #node{content=First#line.content, children=Children},
    [Node|Siblings].

And the equivalent Lisp:

(defun parse-text-line (line)
  ;; converts "    text" to (:content "text" :indent 4)
  (let ((content (string-left-trim '(#\Space #\Tab) line)))
    (list :content content :indent (- (length line) (length content)))))

(defun split-list (criterion data-list)
  ;; break a list into two lists; the second begins with the first element that matches the criterion.
  (if data-list
      (if (funcall criterion (first data-list))
          (list () data-list)
          (let ((me (pop data-list)))
            (let ((chunks (split-list criterion data-list)))
              (push me (first chunks))
              chunks)))
      (list () ())))

(defun build-nodes (lines)
  (if lines
      (let* ((me (pop lines))
             ;; split off our children from the rest of the lines.
             ;; the first line with an indent <= ours is a sibling, not a child
             (chunks (split-list (lambda (x) (<= (getf x :indent) (getf me :indent))) lines)))
        (let ((children (build-nodes (pop chunks)))
              (siblings (build-nodes (pop chunks))))
          (let ((node (list :content (getf me :content))))
            (if children (setf (getf node :children) children))
            ;; (format t "~a~%" (getf node :content))
            (push node siblings))))
      nil))

Erlang has obvious Lisp roots, so most of the concepts translated 1-for-1. (I didn’t have any macros in my Lisp code, which might have made that hairier.) Most of it was changing dashes to underscores and moving parentheses around, but Erlang has some new features of interest.

Pattern Matching

Most of the Erlang functions take advantage of its pattern matching. This is a more extreme form of the method overloading that Java developers are familiar with. Erlang does allow varying numbers of parameters, as in

show_indented(Nodes) -> show_indented(Nodes, 0).
show_indented(Nodes, Indent) ->
    ...

where the one-parameter version just calls the two-parameter version with a default indent. But more interestingly, it lets you define functions by the parameter values, as we see here:

build_nodes([]) -> [];
build_nodes(Lines) ->
    ...

The first build_nodes function matches when it’s passed an empty array. If any other value is passed, it fails to match the first version and tries the second. The second matches any parameter, and assigns its value to the Lines variable. The really wacky case of this is in the definition of

main([[$\:|FuncName]|Filenames]) ->
    ...
main(Filenames) ->
    ...

The first one only matches a list whose first element is a string beginning with a ‘:’ (“:my_function”), and handily assigns the rest of the string to the FuncName variable (FuncName = “my_function”) and all the other arguments to Filenames. No match, and it just assigns the whole argument list to Filenames. You could implement that functionality in any other language, but this expression is very concise and clear, very declarative.

The case clause works in a similar way:

case Node#node.children of
        [] -> io:fwrite("~s~n", [NodePath]);
        Children -> show_path(Children, NodePath)
    end

This says, “If node’s children are an empty list, write out the node path; otherwise, assign them to Children, and pass that to show_path() along with NodePath.

On the whole, as odd as Erlang’s syntax is, I find it friendlier than Lisp’s. The pattern matching is weird at first, but very elegant and powerful once you wrap your head around it. I’d be interested to know the reaction of someone new to programming. I suspect they’d be less thrown by the syntactic quirkiness – the arrows and the comma vs. semicolon vs. period for line endings.

Spawn!

But all this is not why I’m learning Erlang. If it were just a stylistic improvement over Lisp, that wouldn’t justify its existence. We’re here for crazy parallel processing hijinks. So ok, here, let me re-write this so that it spawns a new process for every node. When the node is done building its children and siblings, it sends a message with all its data to its parent node, and so on up and down the tree. Ready? Here we go…

build_nodes(Parent, Group, []) -> Parent ! {Group, []};
build_nodes(Parent, Group, Lines) ->
    [First|Rest] = Lines,
    %% split off our children from the rest of the Lines.
    %% the first line with an indent =< ours is a sibling, not a child
    Criterion = fun(X) -> X#line.indent =< First#line.indent end,
    [ChildLines, SiblingLines] = split_list(Criterion, Rest),
    spawn(?MODULE, build_nodes, [self(), children, ChildLines]),
    spawn(?MODULE, build_nodes, [self(), siblings, SiblingLines]),
    receive
        {children, Children} ->
            Node = #node{content=First#line.content, children=Children}
    end,
    receive
        {siblings, Siblings} ->
            io:fwrite( "[~s] ~p~n", [Node#node.content, self()]),
            Parent ! {Group, [Node|Siblings]}
    end.

Here’s the full thing, but that’s about it. Instead of calling build_nodes() directly, we spawn them off and wait for the results. Instead of just returning our list of nodes, we send it to our Parent. (“Group” is just to keep children and siblings separate.) If you look at the diff, you’ll see a couple more minor things, but that’s all there is to the functional change.

So Erlang is surprisingly useful as a general-purpose scripting language. Probably not the first tool I’d reach for, but it doesn’t suck. And all of its limitations and eccentricities pay off when you start spawing processes. They straightjacket you into designing something that’s easily parallelized. I’m imagining trying to do a similar conversion in Java, where I’d be building a shared data structure and having to manage synchronization, thread completion, ordering results, and all that. Bleah.

(I have to point out that this is actually a bad example – this is not the kind of processing you want to parallelize. It schleps around a fair amount of data, and does very little processing. You really want that the other way around.)

What?

Erlang was developed back in the 80s for programming telephone switching equipment. That sounds pretty bizarrely specialized, right? The problem they had was pretty much unique back then. They had all these machines that had to talk to each other. The individual messages, the requests they had to process, were relatively simple, but the system as a whole had to deal with an absolutely ludicrous volume of them and be perfectly reliable. You probably make hundreds, if not thousands, of phone calls a year. If even two or three of them got dropped, you’d say your phone system is pretty crappy. “It keeps dropping calls.” And if the whole system went down for like an hour, they’d never hear the end of it.

These days, a lot of people have this problem. Not just phone companies, but pretty much anyone who runs a popular web site or instant messaging service. A lot of enterprise software now relies on a “message bus” to handle communications between various chunks of code on different servers. Anything having to do with credit card processing or financial markets also needs that cocktail of high volume and high availability.

So what’s so special about Erlang?

Erlang was built from the ground up with this sort of thing in mind, not just as a concern but as its raison d’etre. There are really basic things that Erlang doesn’t let you do because they cause problems at scale in a massively multi-processing environment. Like, it doesn’t let you modify data structures. Once you’ve created them, you can copy them over and change them on the way, but you can’t change the original. Why? Because that forces you to communicate the change explicitly, and nobody has to worry about data getting yanked out from under them.

That’s the programming equivalent of rugby players taping their ears to their heads. The fact that people feel the need to do that tells you something about the seriousness of the situation.

On the other hand, Erlang has features that make it easier to write these sorts of massively concurrent and robust programs. Spinning off a new process is trivial, a single command, as is sending messages to it and even registering it as a service so that other processes can talk to it. Doing that on a remote machine isn’t much more than adding its name or IP to the command. I’ve done that sort of thing in Java, but holy crap it’s a lot of hassle.

The fact that this is a simple and routine part of Erlang programming also tells you something about the nature of the situation.

The crazy thing that Erlang lets you do (again, easily) that I haven’t seen in any other language is that it lets you update code while it’s running. This is not to be confused with the half-assed thing Java application servers do: If they’re running in “development mode” (so not recommended for production use), they can recompile updated JSPs; or they can shut down your whole app and restart it, leaking memory like a sieve the whole while. In Erlang, all these separate processes can individually reload their code whenever it suits them. You don’t have to shut down the server. You don’t have to coordinate all the processes in some way. You just set it up so that the next time a process does its thing, it does it with the new version of the code. Here’s what it looks like:

-module(countdown).
-export([init/0, reload/0]).
-export([tick/1]).  % so we can spawn this properly.

%% spawn a countdown process with a default start time of 10 seconds.
init() -> init(10).
init(Time) ->
    register(ticker, spawn(?MODULE, tick, [Time])).

tick(Time) when Time >= 0 ->
    io:format("Tick ~p~n", [Time]),
    timer:sleep(999),
    receive reload ->
        ?MODULE:tick(Time - 1)
    after 1 -> tick(Time - 1)
    end;
tick(_StartTime) ->
    io:format("Boom.~n", []),
    done.

reload() ->
    ticker ! reload.

(Calling “?MODULE:tick()” causes the code to reload; plain “tick()” doesn’t. Full example with alternate version and commentary.)

The syntax probably looks kinda foreign, but simple enough that you can puzzle it out. It just creates a countdown timer called “ticker”, but it comes with a reload command that tells it to update itself. In the middle of counting down, you could swap in a new version that prints something different, or counts slower, or starts counting up. It’s a trivial example, but that’s the point: it’s trivial. Doing this in any other language would be somewhere between mind-bendingly complicated and flat-out impossible. It certainly wouldn’t be something you’d do to make your program more reliable. But Erlang has this whole infrastructure for not only updating your application while it’s running, but rolling back to the old version if you find you’ve screwed something up.

So Erlang is technologically cool, and useful in an increasingly broad range of applications. But part of the appeal for me is also that it’s kinda weird. As with Lisp, the programming toolkit is not like most other languages, and it forces me to think about problems in a very different way. That makes it more interesting as a hobby, and it may also make it more valuable as a professional skill. There’s a lot of demand for Java, but there’s also a lot of competition. Erlang is more of a gamble. It may not catch on, but if it does, all that weirdness is going to thin the competition.

Until then, it also means that the Erlang community is mostly made up of people who actually love programming. They’re not just trying to get a job. They’re smart and passionate about what they do. They’re fun to hang out with and they seem to like beer. So worst case, I’ll have fun learning it.

(Most of Paul Graham’s older essays about why you should use Lisp also apply to Erlang.)

Lisp is the Latin of programming languages. It’s been around forever (since 1958, which is the paleolithic era for programming). It never really caught on, outside of niche applications and user groups, but it’s never quite died out, and it seems to be enjoying a resurgence. It’s also inspired other languages, like Ruby and Erlang, that I’ve gotten interested in, and people who know it say that even those just have watered-down, training-wheel versions of features found in Lisp.

I had a good little sample problem to try it out on. I’d been sketching out a to-do list tool, something that would let me keep track of projects, nested sub-projects, and tasks. The key was that I wanted to keep the data in plain text, in some readable format, but also be able to munge it – filter and sort it and such; some sort of simple wiki-style markup language. The first part of this was the structure: I’d have a project, with its sub-projects indented below it, and their sub-projects and tasks indented further. The parser would build up a tree data structure. So you’d have plain text input that looks like this:

and you’d want to end up with a data structure like this:

Now, I’d already done this in Ruby. I’d designed a set of classes to represent the stages of the parsed data. I start out with a list of strings, then I break each string into a simple object that just knows “indent” and “content”. It then adds that to the tree structure as a node that knows what its parent node is. As more nodes get added, they become its children. Eventually, I could extend those classes to understand particular bits of project-related markup, like priorities or dependencies. But for now, just the tree structure.

Already, there’s a bit of recursion here: I have a “current” node, the most recently added one. If the next node is indented further, I add it to the current node’s children. Otherwise, I tell the current node’s parent to try to add it. This way, you could pop up several levels before you find a parent to add the new node to. In this example, I try to add node “e” to “d”, then “c”, before finally adding it to “a”. Despite that, it’s still a pretty linear process. Each line is added in order, and you’re just moving up and down the tree, stitching each into place.

So for my first stab at doing this in Lisp, I just sort of transliterated it. I took the logic and structure of my Ruby code, and tried to just turn it into Lisp syntax. That didn’t go so well. It’s a bit hard to explain how it went wrong, other than to say that it came out like the programming version of Engrish. It didn’t work right, it was hard to follow what was going on in the code, and trying to fix it just made it uglier.

Normally, when I’m writing code it’s kinda like sculpting with clay or Play-Doh. I start out with a rough idea of what I want and gradually refine it, mushing chunks of logic around, whittling away bits I don’t need. I can see it taking shape. My understanding of what’s going on, the scope and nature of the problem I’m solving, gets clearer and clearer. This code was just getting messier. It felt more and more like a duct tape and bailing wire job. Even more than other programmers, Lisp people go on about the elegance and clarity of their code. So I gave up for the evening and went to bed pondering how to make it “Lispier”, maybe really work that recursion thing.

In the classification of programming languages, Lisp is a “functional” language, as opposed to an object-oriented or procedural language. The distinctions are a bit blurry, but they’re about how the language encourages you to organize your code and suggests the terms you use to describe the problem you’re dealing with. Like I said, vague, but programming is fundamentally an act of communication, so the mental constructs you bring to it are important.

Programming languages all have nouns and verbs: Data and things you do with the data. Procedural languages focus on the verbs, the procedures. You’re stepping through a process; it’s linear. Procedures are broken down into sub-procedures, but it’s still all about what you’re doing as you step through the process. You have data structures, but you’re kinda just trundling them along through the procedures like a shopping cart.

Object-oriented languages focus on the data, how it’s organized. The verbs are “methods” tied to the data, what you can do with it. Methods may do things with other objects, but they’re defined in terms of the thing doing it.

Functional languages are a little harder to explain. Rather than the steps of a process, they focus on the start and end points, what you put in and what comes out; results, not actions. That isn’t really how people think. We tell stories; we make sense of the world through narratives. But while it’s not entirely natural, the functional way of thinking can be useful because it forces you to focus on what you’re trying to accomplish.

The other part of “Lispy” code is its focus on handling lists of data recursively. You define a function that does something with the first element of the list, then applies itself to the rest of the list. Rather than

new_list = []
for thing in list:
    new_thing = do_stuff(thing)
    new_list.add(new thing)

You do something more like

(defun process-list (list)
  (list (do-stuff (first list))
        (process-list (rest list))))

process-list is a function that calls do-stuff on the first thing in a list, then calls itself on the rest of the list, and gloms the results together to make a new list. Weird way to do things, huh?

Anyway, in my case, I’ve got a node and a list of lines that are all either children or siblings of that node. Once you break the list into those two chunks, you just create a child and a sibling node, and hand them their lines to process. Your output is a list of the siblings, with you in the front. Your child hands you back a list of its siblings, your children, with itself in the front. Your sibling gives you it siblings (which are also your siblings) with itself in the front. By the time they get back to you, all of your siblings and children already have their siblings and children.

That’s a very clean, mathematical way of thinking about it, but it’s seriously non-linear. The Ruby code built its structure in order, “a” through “l”. The Lisp code built it in the following order: d e c b i h k j g l f a. It processes the children from the deepest up, and the siblings from the last to the first. Not intuitive.

The experience of programming Lisp ended up being a lot more like doing math than sculpting Play-Doh. I had to step away from the keyboard and just think about it for a while. But when I figured it out, everything just snapped into place. It wasn’t the gradual refining and polishing of an idea; it was like solving a puzzle. When I hit on the solution, I knew I had it. I felt it in my bones.

So it was a good learning experience. My hope was that Lisp would force me to think differently, and it did. The Lisp program I came up with is fundamentally unlike the Ruby version. (You can see the Ruby code here and the Lisp code here.) I think I’ve kinda wrapped my head around this functional programming approach. That, more than Lisp syntax, is the potentially useful bit.

The thing about functional programs is that they’re much easier to break into seprate processing chunks, whether you’re running them on some Google-grade cluster or compute cloud, or just on a multi-core chip. There’s a lot more demand for that kinda thing these days. The Ruby version of my code has this central tree structure that it adds onto line by line. There isn’t any way to farm that work out to a bunch of independent processes. In the Lisp version, at each node you’re splitting the work into two batches, children and siblings. You could distribute it naturally across as many processors as you have nodes. Each just has to get its set of lines to process, and the tree comes together in this non-linear way as they return their results.

I’ve always been all about code as a communication between programmers, rather than just a way of telling the machine what to do. Lisp doesn’t do so well on this count. It’s certainly not as friendly to the novice programmer. One of the things I love about Python is that pretty much anyone who knows how to program can read it – it looks like pseudocode. That’s so not true of Lisp. The syntax, with all those parentheses, isn’t like any other languages. It makes no sense at first, and takes a good bit of getting used to. (To be fair, it was there first.) But the harder thing is shifting to a new set of ideas and customs. Early on (and occasionally still) I would try to do something that should be simple and obvious, but in Lisp they just don’t do things that way. Like any foreign language, it’s much harder to write than to read, and the idioms are harder to learn than the grammar.

But again, like any foreign culture, once you spend some time there, you get used to it and it starts to feel comfortable, even normal. Recursion goes from being alien and academic to highly practical. All the crazy parentheses actually feel like a more logical and coherent way of grouping functions. And when you come home to Ruby or Java or whatever, you bring a little bit of all that with you.

That doesn’t mean replacing people with machines. It’s about letting the computer handle the tedious, repetitive crap, and freeing people up to work on the more interesting stuff. In almost any business process, there will be some amount of straightforward bookkeeping that a computer can handle; but there will also be a need for judgement and common sense, and the ability to deal with ambiguous or unforeseen situations, which computers are fundamentally incapable of.

The simplest, most common example of this is spreadsheet software. A computer can’t tell you what numbers and formulas to put in, or what the results mean—what’s important, what action to take—but it can save you the hassle of re-doing the math every time something changes. It’s a power tool, like a band saw or a nail gun.

The first time this was really driven home for me was years ago, when I was working at a little web development shop. I was chatting with a friend of mine who was a designer there, and she was griping that she’d gotten stuck with this crappy assignment. One of our clients had these tables of financial data on their site that had to be updated every month. They sent the new data to us in a PDF document. Her task was to copy the data from the PDF into HTML. So she opened up the PDF in one window, and the HTML source in another, and copied it one entry at a time: click, drag, ctrl-c, click, ctrl-v; for row after row, table after table. She’d just finished the first batch of this, and it had taken her nearly two full days. Absolutely mind-numbing, and she was going to have to do this every month.

She told me all this, and I said, “You did what? No, that’s crazy; don’t do that again. Give me a little while.” I went off and found a utility to dump out the PDF as text, then wrote a Perl script to parse the data out and stuff it into an HTML template. It took me maybe a day and a half. Now, each month, she just had to run the script and spend maybe fifteen minutes proof-reading the pages it spat out.

This is a tiny example, but it demonstrates the alchemy you can work with software. I got to do this fun little side-project; I freed up nearly 10% of her time, so she could do more valuable work for the company; and I made her job way less sucky. Everybody wins.

Let me give you a definition of Business Software that’ll shed a bit of light on the situation: Business Software is software that isn’t used by programmers. It’s used by non-technical people, suits. It’s not useful to us. We don’t care about it. Programmers use programming tools and play video games; end of story. In fact, if the suits think it’s great, that’s a sign that it sucks.

Since we don’t use business software, we don’t know what it should do, whether it’s working right, or how to make it better. There isn’t a good, objective definition of what it’s supposed to do; it’s “whatever the user wants.” Sometimes, it’s based on bizarre business, organizational, historical, or legal requirements. There’s no way to reason it out. If you try, you’ll often find out that your perfectly logical conclusion is in fact disastrously wrong. So we have to ask the users how the software needs to work. That means sitting in meetings. With suits.

Since the users aren’t technical, they can’t just tell you what they want. They probably don’t know, themselves. They don’t know what the limits of the technology are. They have no idea what’s feasible. They ask all sorts of really dumb questions, and expect the impossible. They’re not good at visualizing how they’ll actually use the software. They’re usually wrong about what they think they want. And they will change their minds. So you have to bridge that gap, and you don’t get to just meet them halfway. They might be able to learn a few basic technical things, but you need to learn a lot about their business. So now you have to be the one asking a lot of dumb questions.

So just figuring out what your program needs to do is the really hard bit. You’d think that after suffering through all of that, you’d at least get to do something really cool, but no – the technical solution often turns out to be fairly straightforward, usually something involving a web server and a database. I think that’s why there’s such frantic innovation in tools and frameworks for building web applications: There are all these programmers out there trying to make the technical work interesting.

In fact, to write successful business software, you have to make it deliberately uncool. You don’t use development builds of cutting edge tools and elaborate frameworks; you use simple, stable tools that lots of other people know how to use. In business software, you’re really trying to program yourself out of a job, to make yourself replaceable. You’re trying to take all of this business knowledge, and embody it in code. I’ve heard programmers brag that the guys who came after them weren’t smart enough to maintain their code. That’s completely backwards. If you’re really good, anyone should be able to pick up your code and understand what it’s doing and why. Code is communication, the expression of an idea. Your code should make all of the complex business logic clear and precise. If you’ve done really well, you can even walk non-technical people through it and give them a better understanding of their own business. If other programmers can’t work with it, you either didn’t really understand it, or you didn’t communicate it clearly.

“Legacy code” is a term of distain. It’s shorthand for “crusty old software written in some boring language on an obsolete platform.” In practical terms, it’s anything more than about 6 months old. Real legacy code can even be decades old, written back in the days of Big Iron. It’s been tweaked and stretched and hacked up ever since. It has years of business logic layered on like shellac. To work with it, you have to waste valuable brain space learning about how people did things before we Figured It Out. If you’re really skilled and really lucky, your reward will be that your business software becomes legacy code. It’ll become critical to the business and they’ll use it for years to come. The fullness of time will expose all of the mistakes and misjudgements that you made. You will learn humility, and it will make you a better programmer. It’s a tough love kinda thing.

So why do I write business software? Because I like people. I like trying to figure them out. I like trying to get my head inside their world. All of that stupid, illogical business logic? Most of it really does make sense once you learn the context and history of it. I like that it’s unintuitive; I like that little jolt of surprise I get when I learn that the world is more complex than I’d thought. I also like being useful. Just on the level of pure self-preservation, I figure I’m better off being useful rather than just clever. But I also like the feedback. I get all kinds of warm fuzzies from writing a bit of code that makes someone else’s life a little easier. I’ve come to the conclusion that I was put on this earth to program the Suck out of people’s jobs.