Correct me if I'm wrong, but isn't this a bit like reinventing the wheel when PLY is around? http://www.dabeaz.com/ply/

There's another popular library that escapes me, too.

I wrote one of these a while back: http://knot.appspot.com/e/simple-python-scanner

For his example, it needs one more line:

  simple = Scan(
    IDENTIFIER = r'[\w_]\w*',
    OPERATOR = r'[-+\*=]',
    DIGIT = r'\d+(\.\d+)?',
    END_STATEMENT = r';',
    SPACE = r'\s+',
  )
  simple.ignore('SPACE')
  simple('foo = 5 * 30; bar = bar - 60;')
  # generates tokens.