I don’t disagree with that, but it’s just not the point of this series of posts. This is about understanding the computational structure of event-driven code, and see how it’s possible to transform it into a less awkward form without introducing concurrency (or at least not in the traditional sense of the term).

Using threads to solve what is essentially a control flow problem is cheating. And you pay in terms of increased complexity, and code which is harder to reason about, since you introduced a whole lot of interleaving opportunities and possible race conditions. Using a non-preemptive concurrency abstraction with manual yield directives (like my Python gist does) will solve that, but then you’d have to think of how to schedule your coroutines, so that is also not a complete solution.

Programmable semicolons

To find an alternative to the multitask-based approach, let’s focus on two particular lines of the last example:

    reply = start_request();
    get_data(reply)

where I added an explicit semicolon at the end of the first line. A semicolon is an important component of an imperative program, even though, syntactically, it is often omitted in languages like Python. It corresponds to the sequencing operator: execute the instruction on the left side, then pass the result to the right side and execute that.

If the instruction on the left side corresponds to an asynchronous operation, we want to alter the meaning of sequencing. Given a sequence of statements of the form

    x = A(); B(x)

we want to interpret that as: call A, then return control back to the main loop; when A is finished, bind its result to x, then run B.

So what we want is to be able to override the sequencing operator: we want programmable semicolons.

The continuation monad

Since it is often really useful to look at the types of functions to understand how exactly they fit together, we’ll leave Python and start focusing on Haskell for our running example.

We can make a very important observation immediately by looking at the type of the callback registration function that our framework offers, and try to interpret it in the context of controlled side effects (i.e. the IO monad). For Qt, it could look something like:

1

    connect :: Object -> String -> (a -> IO ()) -> IO ()

to be used, for example, like this:

1
2

    connect httpReply "finished()" $ \_ -> do
        putStrLn "request finished"

so the first argument is the object, the second is the C++ signature of the signal, and the third is a callback that will be invoked by the framework whenever the specified signal is emitted. Now, we can get rid of all the noise of actually connecting to a signal, and define a type representing just the act of registering a callback.

1

    newtype Event a = Event { on :: (a -> IO ()) -> IO () }

Doesn’t that look familiar? It is exactly the continuation monad transformer applied to the IO monad! The usual monad instance for ContT perfectly captures the semantics we are looking for:

1
2
3
4
5

    instance Monad Event where
      return x = Event $ \k -> k x
      e >>= f = Event $ \k ->
        on e $ \x ->
          on (f x) k

The return function simply calls the callback immediately with the provided value, no actual connection is performed. The bind operator represents our custom semicolon: we connect to the first event, and when that fires, we take the value it yielded, apply it to f, and connect to the resulting event.

Now we can actually translate the Python code of the previous example to Haskell:

1
2
3
4
5
6
7
8
9
10

    ex :: Event ()
    ex = forever $ do
      result <- untilRight . replicate 2 $ do
        reply <- startRequest
        either (return . Left) (liftM Right . getData) reply
      either handleError displayData result

    untilRight :: Monad m => [m (Either a b)] -> m (Either a b)
    untilRight [m] = m
    untilRight (m : ms) = m >>= either (const (untilRight ms)) (return . Right)

Again, this could be cleaned up by adding some error reporting functionality into the monad stack.

Implementing the missing functions in terms of connect is straightforward. For example, startRequest will look something like this:

1
2
3
4
5

    startRequest :: Event (Either String Reply)
    startRequest = Event $ \k -> do
      reply <- AccessManager.get "http://example.net"
      connect reply "finished()" $ \_ -> k (Right reply)
      connect reply "error(QString)" $ \e -> k (Left e)

where I took the liberty of glossing over some irrelevant API details.

How do we run such a monad? Well, the standard runContT does the job:

1
2

    runEvent :: Event () -> IO ()
    runEvent e = on $ \k -> return ()

so

1

    runEvent ex

will run until the first connection, return control to the main loop, resume when an event occurs, and so on.

Conclusion

I love the simplicity and elegance of this approach, but unfortunately, it is far from a complete solution. So far we have only dealt with "one-shot" events, but what happens when an event fires multiple times? Also, as this is still very imperative in nature, can we do better? Is it possible to employ a more functional style, with emphasis on composability?

I’ll leave the (necessarily partial) answers to those questions for a future post.

Most of modern programming is based on events. Event-driven frameworks are the proven and true abstraction to express any kind of asynchronous and interactive behavior, like in GUIs or client-server architectures.

The core idea is inversion of control: the main loop is run by the framework, users only have to register some form of “callbacks”, and the framework will take care of calling them at the appropriate times.

This solves many issues that a straightforward imperative/procedural approach would present, eliminates the need for any kind of polling, and creates all sorts of opportunities for general-purpose optimizations inside the framework, with no impact on the complexity of user code. All of this without introducing any concurrency.

There are drawbacks, however. Event-driven code is hideous to write in most languages, especially those lacking support for first class closures. More importantly, event-driven code is extremely hard to reason about. The very nature of this callback-based approach makes it impossible to use a functional style, and even the simplest of interactions requires some form of mutable state which has to be maintained across callback calls.

As a very simple example, suppose we want to perform a GET request and retrieve some data, handling any HTTP error that might occur. In a generic event-driven frameworkm, we would need to implement a simple state machine whose graph will look somewhat like this:

Each state (except the initial one) corresponds to a callback. The transitions are determined by the framework. To avoid starting more than one request at a time, we will need to explicitly keep track of the current state.

Now let’s try to make a simple change to our program: suppose we want to retry requests when they fail, but not more than once. Now the state machine becomes more complicated, since we need to add extra nodes for the non-fatal error condition.

In our hypotetical event-driven code, we need to keep track of whether we already encountered an error, check this flag at each callback to perform the right action, and update it appropriately. Moreover, this time the code isn’t even shaped exactly like the state machine, because we reuse the same callback for multiple nodes. To test our code exhaustively, we need to trace every possible path through the graph and reproduce it.

Now assume we want to allow simultaneous requests… you get the idea. The code gets unwieldy pretty fast. Small changes in requirements have devastating consequences in terms of the state graph. In practice, what happens most of the times is that the state graph is kept implicit, which makes the code impossible to test reliably, and consequently impossible to modify.

Towards a solution

A very simple but effective solution can be found by observing that state graphs like those of the previous examples have a very clear operational interpretation in the context of the equivalent synchronous code.

A single forward transition from A to B can be simply modelled as the sequence A;B, i.e. execute A, then execute B. Extra outward transitions from a single node can be mapped to exceptions, while backward arrows can be thought of as looping constructs.

Our second state machine can then be translated to the following pseudopython:

while True:
    for i in xrange(2):
        error = None
        try:
            reply = start_request()
            data = get_data(reply)
            break
        except Exception as e:
            error = get_error(e)
    if error:
        handle_error(error)
    else:
        display_data(data)

This code is straightforward. It could be made cleaner by splitting it up in a couple of extra functions and removing local state, but that’s beside the point. Note how easy it is now to generalize to an arbitrary number of retries.

So the key observation is that we can transform asynchronous code into synchronous-looking code, provided that we attach the correct semantics to sequencing of operations, exceptions and loops.

Now the question becomes: is it possible to do so?

We could turn functions like start_request and get_data into blocking operations that can throw. This will work locally, but it will break asynchronicity, so it’s not an option.

One way to salvage this transformation is to run the code in its own thread. Asynchronous operations will block, but won’t hang the main loop, and the rest of the program will continue execution.

However, we need to be careful with the kind of threads that we use. Since we don’t need (and don’t want!) to run multiple threads simultaneously, but we need to spawn a thread for each asynchronous operation, we have to make sure that the overhead is minimal, context switching is fast, and we’re not paying the cost of scheduling and synchronization.

Here you can find a sketched solution along these lines that I wrote in python. It’s based on the greenlet library, which provides cooperative multithreading.

In the next post I will talk about alternative solutions, as well as how to extend the idea further, and make event-driven code more declarative and less procedural.

So, suppose you need to expose a C++ function like:

void applyEffect(QImage* img, float arg);

that takes a QImage, an argument, and applies a graphic effect, mutating the image in place.

Directly exposing this function to ruby using the extension API is not easy, because you need to extract a QImage pointer from the ruby object corresponding to the QImage, and that would require you to make assumptions on exactly how QObjects are wrapped by the ruby binding code, which is not ideal for a number of reasons.

Fortunately, there exists an elegant solution to this problem. First, you need to define your C++ function as a slot of some QObject. For example:

class Extensions : public QObject
{
Q_OBJECT
public slots:
  void applyEffect(QImage* img, float arg) const;
};

Then in your extension initialization function you can instantiate it with something like:

Extensions* ext = new Extensions(QCoreApplication::instance());
ext->setObjectName("__extensions__");

And finally access it from ruby code and wrap it in a nicer package:

$ext = $qApp.findChild(Qt::Object, "__extensions__")
class Qt::Image
  def apply_effect(arg)
    $ext.applyEffect(self, arg)
  end
end

This works because Qt allows you to call slots dynamically using runtime introspection of QObjects. It’s not as fast as a native function call, but in the context of a ruby method call, the additional cost should be pretty much negligible.

Of course, unless your extension is particularly large and complicated, you don’t need to create an Extension object for each of the functions you want to expose: you can add all of them as slots in a single Extension object, which is loaded at startup, and create a ruby-esque API for them directly in ruby code.

One of the first problems that pop up when starting a new Qt/KDE project in ruby is how to use it in such a way that your code doesn’t end up being completely unidiomatic. This can happen very easily if one tries to stick to the usual conventions that apply when writing Qt code in C++.

If you take any piece of C++ code using Qt, you can very trivially translate it into ruby. That works, and sometimes is useful, but writing code in this way completely misses the point of using a dynamic language. You might as well write directly in C++, and enjoy the improved performance.

So I believe it’s important to identify the baggage that Qt brings from its C++ roots, and eliminate it when using it from ruby. Here are some ideas to achieve that.

button.on(:clicked) { puts "hello world" }

def on(sig, types = nil, &blk)
  sig = Signal.create(sig, types)
  candidates = if is_a? Qt::Object
    signal_map[sig.symbol]
  end
  if candidates
    if types
      # find candidate with the correct argument types
      candidates = candidates.find_all{|s| s[1] == types }
    end
    if candidates.size > 1
      # find candidate with the correct arity
      arity = blk.arity
      if blk.arity == -1
        # take first
        candidates = [candidates.first]
      else
        candidates = candidates.find_all{|s| s[1].size == arity }
      end
    end
    if candidates.size > 1
      raise "Ambiguous overload for #{sig} with arity #{arity}"
    elsif candidates.empty?
      msg = if types
        "with types #{types.join(' ')}"
      else
        "with arity #{blk.arity}"
      end
      raise "No overload for #{sig} #{msg}"
    end
    sign = SIGNAL(candidates.first[0])
    connect(sign, &blk)
    SignalDisconnecter.new(self, sign)
  else
    observer = observe(sig.symbol, &blk)
    ObserverDisconnecter.new(self, observer)
  end
end

combobox.on(:current_index_changed, ["int"]) {|i| self.index = i }

model.on(:rows_about_to_be_inserted) do |p, i, j|
  # ...
end

rowsAboutToBeInserted(const QModelIndex& parent, int start, int end)

Kaya is written in ruby, and I have learned quite a few things about writing non-trivial GUI applications in ruby while working on it, so I decided to share some of my experience and code in the hope that it might be useful to others, and possibly inspire other Qt/KDE developers to try out ruby for their next project.

Advantages

Here is a list of what I think are the most important points that make programming GUIs in ruby so much more productive than in C++. I’ll leave out subjective arguments like “it’s more fun” or “it has a nicer syntax” because I think that the actual facts are more than compelling already.

Fast Prototyping

This is not specific to GUI programming. It is greatly aknowledged that ruby is orders of magnitude more convenient than C++ for throwing quick scripts together and in general for trying new ideas out. This turns out to be very important in a GUI context as well.

For example, it is very easy to write setup code for a single component of your application so that you can run it standalone and test it more efficiently. I have found myself resorting to this kind of trick very often, and it sometimes makes debugging a lot less painful. In C++, it would be unreasonable to write a new application skeleton (and alter your build scripts) just to test a new dialog you are developing.

Another example is “fancy logging“. If something breaks in the middle of complicated or highly interactive code, sometimes printing to the console or setting breakpoints doesn’t quite cut it. In these cases, it is a very good idea to hack together a simple but powerful visualization tool to show the internal state of the application in real time, and that is usually very helpful to identify the issue. Again, since this is basically throwaway code, ease of prototyping is very important.

Declarative GUI Definition

Ruby has very powerful metaprogramming facilities that make creating an embedded DSL extremely straightforward. I use a simple mechanism in Kaya that allows me to write things like:

    @gui = KDE::autogui(:engine_prefs,
                        :caption => KDE.i18n("Configure Engines")) do |g|
      g.layout(:type => :horizontal) do |l|
        l.list(:list)
        l.layout(:type => :vertical) do |buttons|
          buttons.button(:add_engine,
                         :text => KDE.i18nc("engine", "&New..."),
                         :icon => 'list-add')
          buttons.button(:edit_engine,
                         :text => KDE.i18nc("engine", "&Edit..."),
                         :icon => 'configure')
          buttons.button(:delete_engine,
                         :text => KDE.i18nc("engine", "&Delete"),
                         :icon => 'list-remove')
          buttons.stretch
        end
      end
    end
    setGUI(@gui)

I find it very convenient to define GUIs at this slightly higher level of abstraction, plus you have none of the boilerplate code typical of GUI construction in C++.

Defining GUIs in this way is so quick (and it’s easy to run them to immediately see the results), that it makes tools like Qt Designer completely redundant, in my opinion, except possibly when GUI design and coding are done by different people.

Using Closures for Slots

Take a look at this simple example, and try to imagine how many lines of code would be needed to implement it in C++:

win.display.text = "0"
inc = 1
Qt::Timer.every(1000) do
  win.display.text = (win.display.text.to_i + inc).to_s
end
win.button.on(:clicked) { inc = -inc }

Here win.display is a QTextEdit and win.button is a QPushButton. What the example does is count up seconds in the QTextEdit, and toggle between that and counting down when the button is pressed.

Pretty trivial, granted, but in C++ you would need to define two slots to do that, plus keep track of the direction in which you are counting by using a member variable. In ruby you can just use local variables without littering the parent scope.

Toolkit Independence

If you write your application in C++, once you’ve decided that you are going to use KDElibs (or Qt), that decision is pretty much set in stone. You can’t even switch from KDE to pure Qt very easily, and it’s so hard to support both environments in a single code base, to make it almost completely not worth the effort.

Of course, this might not seem that big of an issue, given that KDE is now a lot more portable, but not many people are running KDE SC on Windows or Mac at the moment, and if you want to reach as many users as possible, having a Qt-only version of your application is going to help a lot.

So Kaya can currently run on Qt-only as well as KDE. When in KDE mode, all the usual KDE goodies are running under the hood: KPushButtons, KDialog, K-everything, and of course KXMLGUI and all the good stuff that comes with it, but it can switch to plain Qt classes and a simplistic replacement for the XML GUI if you don’t have KDElibs installed.

Easier Automatic Testing

Dynamicity, mock objects, plus the whole culture of test-driven development that characterizes the ruby ecosystem make it a lot easier to devise automated tests for your application. Effective GUI testing is still basically an open problem in software engineering, so don’t expect it to be a piece of cake, but in my experience, you can definitely reach a considerably higher test coverage with ruby than with C++.

Trivial Extensibility Through Plugins

KDE really shines when it comes to making applications easily extensible via scripts or plugins, but it can’t compare with ruby. The distinction between plugins and user scripts is now nonexistent, and loading a plugin is just a matter of calling the ruby load function.

Creating a sensibly extensible application still requires careful planning, of course, but it’s a lot easier when you can directly expose application functionality to plugins, without creating tons and tons of interfaces for even the most trivial uses.

In Kaya, the use of a dynamically typed language turned out to be key: the flexibility required to make its plugins easy enough to write without knowing much about the application internals is pretty much impossible to achieve in a statically typed context.

Disadvantages

Of course, like everything in software engineering, choosing ruby over C++ for GUI development involves a number of tradeoffs.

Performance

Everyone knows that ruby is slow. Painfully slow sometimes. That seems to be improving with 1.9, but ruby isn’t going to get faster than C anytime soon.

Now, the good thing is that its poor performance is almost always completely irrelevant. A typical GUI application is nowhere near being CPU bound, and the few computationally intensive parts are usually in the GUI library anyway. That said, if you have performance critical sections in your application, you are better off writing them in C/C++ and accessing them from your ruby code using the ruby extension API.

For example, Kaya includes a small C++ extension to supplement the missing blur functionality in Qt < 4.6. Writing it and integrating it was trivial.

Testing is Essential

Automated testing is very important for software written in any language, but for dynamic languages you simply can’t do without it. Untested code will inevitably contain silly typos and type errors which will cause it to crash in your user’s faces, or in the best case will make your testing sessions painful and slow.

So you don’t have any choice but to add lots and lots of unit tests, integration tests and the like. Units tests for pure GUI code are not really effective, useful or easy to write, but you can settle on smoke tests that will cover the most common mistakes and be pretty confident that no silly errors will pop up that a compiler would catch.

Conclusion

I hope this gives a nice overview of the benefits you can get by switching from C++ to a dynamic language for your KDE (or general GUI) development. I used ruby as the main example, because that’s what I have experience with, but most of the points probably apply to dynamic languages in general (python, perl, clojure, etc).

In the following posts, I will dig a little bit more into Kaya’s code to show the kinds of tricks that I employed to better exploit the advantages that I discussed, and offer even more ways to get the most out of ruby for GUI programming. Stay tuned!

Suppose you need to model a finite Markov chain in code. There are essentially two ways of doing that: one is to simply run a simulation of the Markov chain using a random number generator to obtain dice rolls and random cards from the decks, the other is to create a stochastic matrix containing the transition probabilities for each pair of states.

In this post I will show how a single monadic description of the Markov chain dynamics can be used to obtain both a simulator and the transition matrix.

> {-# LANGUAGE MultiParamTypeClasses, 
>     FlexibleInstances, 
>     GeneralizedNewtypeDeriving #-}
>
> import Control.Arrow
> import Control.Monad
> import Control.Monad.State.Strict
> import Data.Array
> import Random

Let’s start with an example of Markov chain and how we would like to be able to implement in Haskell. Consider a simplified version of the familiar Monopoly game: there are 40 squares (numbered 0 to 39), you throw two 6-sided dice each turn, some special squares have particular effects (see below), if you get a double roll three times in a row, you go to jail. The special squares are:

30: go to jail
2, 17, 33: Community Chest
7, 22, 36: Chance

Community Chest (CC) and Chance (CH) make you take a card from a deck and move to some other place depending on what’s written on the card. You will find the details on the code, so I won’t explain them here.

This is of course a Markov chain, where the states can be represented by:

> type Square = Int
> data GameState = GS {
>       position :: Square,
>       doubles :: Int } deriving (Eq, Ord, Show)

and a description of the game can be given in a monadic style like this:

> sGO :: Square
> sGO = 0
> 
> sJAIL :: Square
> sJAIL = 10
>
> finalize :: Square -> Game Square
> finalize n
>     | n == 2 || n == 17 || n == 33 = cc n
>     | n == 7 || n == 22 || n == 36 = ch n
>     | n == 30 = return sJAIL
>     | otherwise = return n
> 
> cc :: Square -> Game Square
> cc n = do i <- choose (1 :: Int, 16)
>           return $ case i of
>                      1 -> sGO
>                      2 -> sJAIL
>                      _ -> n
> 
> ch :: Square -> Game Square
> ch n = do i <- choose (1 :: Int, 16)
>           return $ case i of
>                      1 -> sGO
>                      2 -> sJAIL
>                      3 -> 11
>                      4 -> 24
>                      5 -> 39
>                      6 -> 5
>                      7 -> nextR n
>                      8 -> nextR n
>                      9 -> nextU n
>                      10 -> n - 3
>                      _ -> n
>     where
>       nextR n = let n' = n + 5
>                 in n' - (n' `mod` 5)
>       nextU n
>           | n >= 12 && n < 28 = 28
>           | otherwise = 12
> 
> roll :: Game (Int, Int)
> roll = let r1 = choose (1, 6)
>        in liftM2 (,) r1 r1
> 
> markDouble :: Bool -> Game ()
> markDouble True = modify $ \s -> s {
>                     doubles = doubles s + 1 }
> markDouble False = modify $ \s -> s {
>                      doubles = 0
>                    }
>
> goTo :: Square -> Game ()
> goTo n = let n' = n `mod` 40
>          in modify $ \s -> s { position = n' }
> 
> game :: Game ()
> game = do n <- liftM position get
>           (a, b) <- roll
>           markDouble (a == b)
>           d <- liftM doubles get
>           if d == 3
>            then do markDouble False
>                    goTo sJAIL
>            else do let n' = n + a + b
>                    n'' <- finalize n'
>                    goTo n''
> 

As you can see, Game is a state monad, with an additional function choose that gives us a random element of a range:

> class MonadState s m => MonadMC s m where
>     choose :: (Enum a) => (a, a) -> m a

This can be implemented very easily using the (strict) state monad and a random generator:

> newtype MCSim s a = MCSim (State ([s], StdGen) a)
>     deriving Monad
> 
> instance MonadState s (MCSim s) where
>     get = MCSim $ liftM (head . fst) get
>     put x = MCSim . modify $ \(xs, g) -> (x : xs, g)
> 
> instance MonadMC s (MCSim s) where
>     choose (a, b) = MCSim $
>                     do (xs, g) <- get
>                        let bnds = (fromEnum a, fromEnum b)
>                        let (y, g') = randomR bnds g
>                        put (xs, g')
>                        return . toEnum $ y
> 
> -- type Game a = MCSim GameState a
>
> runSim :: StdGen -> Int -> s -> MCSim s () -> [s]
> runSim g n start m = fst $ execState m' ([start], g)
>     where
>       (MCSim m') = foldr (>>) (return ()) $ replicate n m

The runSim function runs the simulation and returns the list of visited states. This is already quite nice, but the best thing is that the same code can be used to create the transition matrix, just swapping in a new implementation of the Game type alias:

> newtype MC s a = MC (s -> [(s, Double, a)])
> 
> instance Monad (MC s) where
>     return x = MC $ \s -> return (s, 1.0, x)
>     (MC m) >>= f = MC $ \s ->
>                    do (s, p, x) <- m s
>                       let (MC m') = f x
>                       (s', q, y) <- m' s
>                       return (s', p * q, y)
> 
> instance MonadState s (MC s) where
>     get = MC $ \s -> return (s, 1.0, s)
>     put x = MC $ \s -> return (x, 1.0, ())
> 
> instance MonadMC s (MC s) where
>     choose (a, b) = let r = [a..b]
>                         p = recip . fromIntegral . length $ r
>                     in MC $ \s -> map (\x -> (s, p, x)) r
> 
> type Game a = MC GameState a

The idea is that we keep track of all possible destination states for a given state, with associated conditional probabilities. For those familiar with Eric Kidd’s series on probability monads, this is basically:

type MC s a = StateT s (PerhapsT [] a)

Now, how to get a transition matrix from such a monad? Of course, we have to require that the states are indexable:

> markov :: Ix s =>
>           MC s () -> (s, s) -> Array (s, s) Double
> markov (MC m) r = accumArray (+) 0.0 (double r) $
>  	     	    range r >>= transitions
>     where
>       mkAssoc s (s', p, _) = ((s, s'), p)
>       transitions s = map (mkAssoc s) $ m s
> 	double (a, b) = ((a, a), (b, b))

So we iterate over all states and use the probability values contained in the monad to fill in the array cells corresponding to the selected state pair.

To actually apply this to our Monopoly example, we need to make GameState indexable:

> nextState :: GameState -> GameState
> nextState (GS p d) = if d == 2
>                      then GS (p + 1) 0
>                      else GS p (d + 1)
> 
> instance Ix GameState where
>     range (s1, s2) = takeWhile (<= s2) .
>                      iterate nextState $ s1
>     index (s1, s2) s =
>         let poss = (position s1, position s2)
>         in index poss (position s) * 3 +
>            doubles s - doubles s1
>     inRange (s1, s2) s = s1 <= s && s <= s2
>     rangeSize (s1, s2) = index (s1, s2) s2 + 1

then finally we can try:

> monopoly :: (GameState, GameState)
> monopoly = (GS 0 0, GS 39 2)
> 
> initialState :: Array GameState Double
> initialState = let n = rangeSize monopoly
>                    p = recip $ fromIntegral n
>                in listArray monopoly $ replicate n p
> 
> statDistr :: Int -> [(GameState, Double)]
> statDistr n = let mat = markov game monopoly
>                   distributions = iterate (.* mat)
>                         initialState
>                   st = distributions !! n
>               in assocs st

where .* is a simple vector-matrix multiplication function:

> infixl 5 .*
> (.*) :: (Ix i, Num a) =>
>            Array i a -> Array (i, i) a -> Array i a
> (.*) x y = array resultBounds
>               [(i, sum [x!k * y!(k,i) | k <- range (l,u)])
>                | i <- range (l'',u'') ]
>         where (l, u) = bounds x
>               ((l', l''), (u', u'')) = bounds y
>               resultBounds
>                 | (l,u)==(l',u') = (l'', u'')
>                 | otherwise = error ".*: incompatible bounds"

Calling statDistr 100 will return an association list of states with corresponding probability in an approximation of the stationary distribution, computed by applying the power method to the transition matrix. The number 100 is a pure guess, I don’t know how to estimate the number of iterations necessary for convergence, but that is out of the scope of this post, anyway.

While I tend to think that rewriting from scratch is usually a bad idea, if we are talking about open source projects, there’s another element you should take into account: how pleasurable it is to work on it. If you end up with a big and hairy code base you cannot touch without suddenly feeling all sad and pessimistic about the fate of the world, you should realize that something has gone deeply wrong: you are supposed to work on your project for fun (yeah, to scratch an itch, maybe, too…), and when the fun stops, most of the reasons to continue disappear. You have no (paying) customers, you have no deadlines, you will be able (or unable, depending on what you do in the rest of your time) to pay your bills anyway.

At this point, either you throw everything into the garbage bin – which may be sad, disappointing and frustrating, both for you and your users – or you take whatever lessons you may have learned in the process, and use your extra wisdom to start from scratch a hopefully better and more mature design and implementation that won’t have those shortcomings that brought you to this miserable situation.

It will probably have other defects and issues, though, more subtle and harder to spot in an upfront design. You have to accept it. And you have to accept the fact that you may need to start from scratch once again in the future.

Here is a simple form you can use to announce your decision to start from scratch. Just copy-paste it into your blog, filling in the required fields. It may save you precious time that you will be able to spend on your n-th rewrite

I'll rewrite my pet project from scratch because:

[ ] It's not modular enough
[ ] The original design didn't take __________________ into consideration
[ ] It's too slow
[ ] There is too much legacy code which I can't seem to get rid of
[ ] I want to write it in ____________________

----

[ ] And this time I'll use TDD/BDD

[ ] And this time it will be done right!

Signed (the developer)

_____________________________

What I did manage to get working is the following: just spawn the process in a temporary child process. This way the grandchild will be orphaned and won’t require a wait from its grandfather.

There are however some problems with this approach. Namely, the child needs to send the PID of its own child to the parent. I’ve accomplished that through a pipe. Here is some example code to better illustrate this technique:

from subprocess import Popen, PIPE, STDOUT
import sys

def spawn(cmd):
    cmdline = ('python', __file__) + cmd
    return int(Popen(cmdline, stdout = PIPE).stdout.read())

if __name__ == '__main__':
    print Popen(sys.argv[1:], stdout = open('/dev/null', 'w'), stderr = STDOUT).pid
    sys.exit(0)

Why git rules

• It’s decentralized. I fully agree with Linus on this point: the decentralized model is superior, for various reasons I won’t talk about in detail here.
• It’s simple. I don’t understand why people keep complaining about git being overly complicated. The object model is fairly straightforward, as soon as you stop thinking about revisions the cvs/svn way, and branching/merging is trivial.
• The object model is very well designed, flexible and useful.

Why git sucks

Well, git has a number of minor (?) defects, like portability issues, impossibility of doing partial checkouts (actually I think this is a problem with all the decentralized VCS’s) and stuff like that, but let me concentrate on those issues which are fundamental to git as an open source project, and are unlikely to be addressed in the future:

• It is written in (unreadable) C. I don’t want to start a flame war (really!), but browsing through git’s code is a painful experience. Huge .c files with tons of big functions with no apparent structure and almost no comment. Ubiquitous micro-management stuff for handling manually allocated buffers. In other words… a mess.
• It (ab)uses the UNIX philosophy. Do one thing, and do it well. Yeah, of course it’s a great idea, but maybe they shouldn’t have taken it too literally. I mean, if we talk about modularity and reusable components, I’m all for it, but it doesn’t mean you have to make a different executable for each task (even low level ones that should be invisible to the end user) and chain them together via text pipes and bash scripts!
• It’s not reusable by third party applications, at least not in any efficient/convenient way. If you want to build a higher-level abstraction over git (a so-called porcelain), you need to fork out every now and then to get input from git, and then parse its textual output. Yes, I know about libgit.a, but that’s not really usable as a library (half of git functions call exit on failure) and has no API documentation at all.

That said, I have to admit that most of its defects aren’t that important if you are just going to use it, and not develop it. However, using something you have problems inspecting and modifying feels a bit like using a closed source product…