Logos

Create ridiculously fast Lexers.

Logos has two goals:

• To make it easy to create a Lexer, so you can focus on more complex problems.
• To make the generated Lexer faster than anything you'd write by hand.

• Combines all token definitions into a single deterministic state machine.
• Optimizes branches into lookup tables or jump tables.
• Prevents backtracking inside token definitions.
• Unwinds loops, and batches reads to minimize bounds checking.
• Does all of that heavy lifting at compile time.

Example

use logos::Logos;
#[derive(Logos, Debug, PartialEq)] #[logos(skip r"[ \t\n\f]+")] // Ignore this regex pattern between tokens enum Token {     // Tokens can be literal strings, of any length.     #[token("fast")]     Fast,
#[token(".")]     Period,
// Or regular expressions.     #[regex("[a-zA-Z]+")]     Text, }
fn main() {     let mut lex = Token::lexer("Create ridiculously fast Lexers.");
assert_eq!(lex.next(), Some(Ok(Token::Text)));     assert_eq!(lex.span(), 0..6);     assert_eq!(lex.slice(), "Create");
assert_eq!(lex.next(), Some(Ok(Token::Text)));     assert_eq!(lex.span(), 7..19);     assert_eq!(lex.slice(), "ridiculously");
assert_eq!(lex.next(), Some(Ok(Token::Fast)));     assert_eq!(lex.span(), 20..24);     assert_eq!(lex.slice(), "fast");
assert_eq!(lex.next(), Some(Ok(Token::Text)));     assert_eq!(lex.slice(), "Lexers");     assert_eq!(lex.span(), 25..31);
assert_eq!(lex.next(), Some(Ok(Token::Period)));     assert_eq!(lex.span(), 31..32);     assert_eq!(lex.slice(), ".");
assert_eq!(lex.next(), None); }

For more examples and documentation, please refer to the Logos handbook or the crate documentation.

How fast?

Ridiculously fast!

test identifiers                       ... bench:         647 ns/iter (+/- 27) = 1204 MB/s
test keywordsoperatorsand_punctators ... bench:       2,054 ns/iter (+/- 78) = 1037 MB/s
test strings                           ... bench:         553 ns/iter (+/- 34) = 1575 MB/s

Acknowledgements

• Pedrors for the Logos logo.

Thank you

Logos is very much a labor of love. If you find it useful, consider getting me some coffee. ☕

If you'd like to contribute to Logos, then consider reading the Contributing guide.

Contributing

Logos welcome any kind of contribution: bug reports, suggestions, or new features!

Please use the issues or pull requests tabs, when appropriate.

To release a new version, follow the RELEASE-PROCESS

License

This code is distributed under the terms of both the MIT license and the Apache License (Version 2.0), choose whatever works for you.

See LICENSE-APACHE and LICENSE-MIT for details.