Skip to main content

Module lexer

Module lexer 

Source
Available on crate feature lexer only.
Expand description

First pass lexer that gets run on the source code and returns a series of tokens with their associated Fragments.

Every byte of the source ends up in exactly one token – whitespace and comments are produced as tokens too (see TokenTy::is_trivia), which lets the parser build a lossless syntax tree.

Modules§

comments
Implementation of comment token lexing.
identifier
Implementation related to parsing keywords and identifiers.
integer_literal
Implementation for lexing integer and float literals.
quoted
Lexing implementation for quoted literals.
token
Token models.
trivial
Trivial tokens and their implementation.

Structs§

Lexer
The lexical analyser for wright. This produces a series of tokens that make up the larger elements of the language.