diff options
| author | Benjamin Paassen <bpaassen@techfak.uni-bielefeld.de> | 2014-11-19 19:44:29 +0100 |
|---|---|---|
| committer | Benjamin Paassen <bpaassen@techfak.uni-bielefeld.de> | 2014-11-19 19:44:29 +0100 |
| commit | 45c6acf226f7867f847a9085d84960d337b1a2ae (patch) | |
| tree | d248b1155a87b89af0f877284ac69254b9024962 /clean.sh | |
| parent | ec31aae293f88e36190aa32169a97a776873567a (diff) | |
1.) fixed the word tokenization mechanism in the CodeTokenizer which returned rubbish previously. 2.) Allowed multiline tokens if the mode is LINEBREAK (obsiously). 3.) returned LINEBREAK tokens in normal mode. 4.) added a CodeTokenizer test.
Diffstat (limited to 'clean.sh')
0 files changed, 0 insertions, 0 deletions