Comments on: The Magic Behind Parser Combinators http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators (permanently in beta) Sun, 29 Aug 2010 20:01:44 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: Søren B. Vrist http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-5113 Søren B. Vrist Mon, 26 Jul 2010 09:28:43 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-5113 Any chance for a follow up expanding on scala 2.8 angle on parser combinators? (packrat?) Any chance for a follow up expanding on scala 2.8 angle on parser combinators? (packrat?)

]]> By: Seth Tisue http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4955 Seth Tisue Mon, 29 Mar 2010 19:33:33 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4955 You write: "hand-written parsers have a tendency toward poor performance (think: the Scala compiler)". Yes, the Scala compiler is slow, but the parsing phase isn't the bottleneck. It's later phases such as the type checker. You write: “hand-written parsers have a tendency toward poor performance (think: the Scala compiler)”. Yes, the Scala compiler is slow, but the parsing phase isn’t the bottleneck. It’s later phases such as the type checker.

]]> By: Alexander Lehmann http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4954 Alexander Lehmann Sat, 27 Mar 2010 12:11:54 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4954 Wonderful post. Exactly what I was looking for (although, admittedly, it seems like I'm one year late ;-) Wonderful post. Exactly what I was looking for (although, admittedly, it seems like I’m one year late ;-)

]]> By: Axel Sammet http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4953 Axel Sammet Fri, 26 Mar 2010 12:02:45 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4953 I just stumbled from a google search across your blog. Thanks for the great insight. This helped me a lot to understand the behavior of my experimental parsers written in scala. I also noted the lack of activity in your blog ;-) Any chance, you find the time for more elaborations in the future? I just stumbled from a google search across your blog. Thanks for the great insight. This helped me a lot to understand the behavior of my experimental parsers written in scala. I also noted the lack of activity in your blog ;-) Any chance, you find the time for more elaborations in the future?

]]> By: Daniel Spiewak http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4872 Daniel Spiewak Sat, 01 Aug 2009 05:24:31 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4872 @gatoatigrado GLR is interesting, but it has some fairly serious flaws. For example, despite Tomita's claims, the algorithm is not O(n^3) in the worst case. This grammar in particular is O(n^4) for all valid input: <pre> S ::= S S | S S S | b </pre> Also, GLR is not fully generalized. There are grammars for which GLR (as with any shift-reduce parsing technique) is undefined: <pre> A ::= B A a | a B ::= \epsilon </pre> Granted, most GLR parser generators will transform the above grammar into something which is actually usable within the constraints of the algorithm, but it doesn't change the fact that GLR is not entirely what it claims to be. Fortunately, there are algorithms which are fully-general and which attain cubic asymptotic bounds without absurd memory usage. For example, GLL (Generalized LL) is one such technique presented at LDTA 2009. It's basically just recursive-descent, except that it can handle multiple simultaneous parse trails. And, as with recursive-descent, it is extremely amenable to representation within a functional, composable framework (like parser combinators). In other words, you get all of the benefits of a true, generalized parsing algorithm, cubic performance in the worst case with linear performance on most grammars, all within the easy-to-use framework of parser combinators. Share and enjoy: http://github.com/djspiewak/gll-combinators @gatoatigrado

GLR is interesting, but it has some fairly serious flaws. For example, despite Tomita’s claims, the algorithm is not O(n^3) in the worst case. This grammar in particular is O(n^4) for all valid input:

S ::= S S
    | S S S
    | b

Also, GLR is not fully generalized. There are grammars for which GLR (as with any shift-reduce parsing technique) is undefined:

A ::= B A a
    | a

B ::= \epsilon

Granted, most GLR parser generators will transform the above grammar into something which is actually usable within the constraints of the algorithm, but it doesn’t change the fact that GLR is not entirely what it claims to be.

Fortunately, there are algorithms which are fully-general and which attain cubic asymptotic bounds without absurd memory usage. For example, GLL (Generalized LL) is one such technique presented at LDTA 2009. It’s basically just recursive-descent, except that it can handle multiple simultaneous parse trails. And, as with recursive-descent, it is extremely amenable to representation within a functional, composable framework (like parser combinators). In other words, you get all of the benefits of a true, generalized parsing algorithm, cubic performance in the worst case with linear performance on most grammars, all within the easy-to-use framework of parser combinators. Share and enjoy: http://github.com/djspiewak/gll-combinators

]]> By: gatoatigrado http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4871 gatoatigrado Fri, 31 Jul 2009 08:00:52 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4871 unfortunately, most of the regex libraries used are the unintuitive, potentially exponential time (or worse) backtracking Perl libraries. It's a nice idea in theory. I think the research in Elkhound is the nicest technology -- a GLR parser that runs as fast as LALR on most inputs. Unfortunately the implementation is c++ and rather geared towards parsing c++ (which demonstrates that it can tackle challenging grammars, but doesn't so much make it accessible for small languages). unfortunately, most of the regex libraries used are the unintuitive, potentially exponential time (or worse) backtracking Perl libraries. It’s a nice idea in theory. I think the research in Elkhound is the nicest technology — a GLR parser that runs as fast as LALR on most inputs. Unfortunately the implementation is c++ and rather geared towards parsing c++ (which demonstrates that it can tackle challenging grammars, but doesn’t so much make it accessible for small languages).

]]> By: Nathan Matthews http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4869 Nathan Matthews Wed, 29 Jul 2009 21:13:47 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4869 Hi Daniel, I've stumbled across your blog. I just want to say thank you, you have a great writing style and the content is hugely informative. I'm trying to learn Scala better and because the language is so grand in scope and capability it can be hard getting started. I know how much time it takes to write quality material. Anyways, thanks again. Hi Daniel,

I’ve stumbled across your blog. I just want to say thank you, you have a great writing style and the content is hugely informative. I’m trying to learn Scala better and because the language is so grand in scope and capability it can be hard getting started. I know how much time it takes to write quality material. Anyways, thanks again.

]]> By: Daniel Spiewak http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4826 Daniel Spiewak Mon, 13 Apr 2009 14:49:14 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4826 @Walter Whitespace and comments are actually a trickier issue. My personal preference on handling them is to first feed the input source through a proper scanner, which then produces tokens and removes all unnecessary whitespace (and comments). Parser combinators are certainly compatible with this technique, but for some reason it is rarely employed. In order to implement whitespace-stripping in my parsers, I would have to do something similar to what RegexParsers does in the Scala combinator framework, which is basically pre-process the stream before every parser, stripping any leading whitespace. Comments can be handled in the same pass. @Walter

Whitespace and comments are actually a trickier issue. My personal preference on handling them is to first feed the input source through a proper scanner, which then produces tokens and removes all unnecessary whitespace (and comments). Parser combinators are certainly compatible with this technique, but for some reason it is rarely employed.

In order to implement whitespace-stripping in my parsers, I would have to do something similar to what RegexParsers does in the Scala combinator framework, which is basically pre-process the stream before every parser, stripping any leading whitespace. Comments can be handled in the same pass.

]]> By: Walter Chang http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4825 Walter Chang Mon, 13 Apr 2009 09:34:26 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4825 thanks for the great article! just curious, how are whitespaces handled in your parsers? what would be best strategy to handle whitespaces and comments? thanks for the great article! just curious, how are whitespaces handled in your parsers? what would be best strategy to handle whitespaces and comments?

]]> By: Fabien Todescato http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators/comment-page-1#comment-4814 Fabien Todescato Wed, 01 Apr 2009 09:43:21 +0000 http://www.codecommit.com/blog/scala/the-magic-behind-parser-combinators#comment-4814 Hi Daniel, thanks for that great post. Parser combinators are indeed an impressive example of functional programming techniques. A while ago, I had the opportunity to use in a few professional projects the Clean functional programming language (http://clean.cs.ru.nl/), which is a close cousin to Haskell. I used monadic parser combinators a lot, and found the following paper to be a nice introductory text, as well as an illustration of the use of more advanced techniques focused on efficency. The paper demonstrates the use of continuations to improve the execution efficiency of parser combinators. P. Koopman and M.J. Plasmeijer. Efficient Combinator Parsers, In Proc. of Implementation of Functional Languages (IFL '98), London, UK, K. Hammond, A.J.T. Davie and C. Clack (Eds.), Springer Verlag, LNCS 1595, pp. 120-136. http://www.st.cs.ru.nl/papers/1999/koop98-EffCombParsIFL98.ps.gz As Clean isa lazy functional programming language, which supports tail recursion, I am not sure these techniques do apply to Scala, but you may find the paper interesting nevertheless. Hi Daniel, thanks for that great post. Parser combinators are indeed an impressive example of functional programming techniques.

A while ago, I had the opportunity to use in a few professional projects the Clean functional programming language (http://clean.cs.ru.nl/), which is a close cousin to Haskell. I used monadic parser combinators a lot, and found the following paper to be a nice introductory text, as well as an illustration of the use of more advanced techniques focused on efficency. The paper demonstrates the use of continuations to improve the execution efficiency of parser combinators.

P. Koopman and M.J. Plasmeijer. Efficient Combinator Parsers, In Proc. of Implementation of Functional Languages (IFL ‘98), London, UK, K. Hammond, A.J.T. Davie and C. Clack (Eds.), Springer Verlag, LNCS 1595, pp. 120-136.

http://www.st.cs.ru.nl/papers/1999/koop98-EffCombParsIFL98.ps.gz

As Clean isa lazy functional programming language, which supports tail recursion, I am not sure these techniques do apply to Scala, but you may find the paper interesting nevertheless.

]]>