summaryrefslogtreecommitdiff
path: root/src/core/utils/Tokenizer.cpp
AgeCommit message (Collapse)Author
2014-11-19fixed a bug preventing the Tokenizer from finding the right token if a parse ↵Benjamin Paassen
was incomplete beforehand. Also cleared the buffers if a subclass returns false from doPrepare. Failing to clear the buffers lead to subsequent problems.
2014-11-14implemented tokenizer test and started implementing CodeTokenizer under ↵Benjamin Paassen
supervision of Maester Stoeckel.
2014-10-31finished first draft of tokenizer code.Benjamin Paassen
git-svn-id: file:///var/local/svn/basicwriter@90 daaaf23c-2e50-4459-9457-1e69db5a47bf
2014-10-31finished work on TokenTree and tested it.Benjamin Paassen
git-svn-id: file:///var/local/svn/basicwriter@88 daaaf23c-2e50-4459-9457-1e69db5a47bf
2014-10-31added tokenizer class code.Benjamin Paassen
git-svn-id: file:///var/local/svn/basicwriter@87 daaaf23c-2e50-4459-9457-1e69db5a47bf