Struct Lexer

pub struct Lexer {
    pub remaining: Fragment,
}

Available on crate feature lexer only.

The lexical analyser for wright. This produces a series of tokens that make up the larger elements of the language.

Fields

remaining: Fragment

The remaining source code that has not been processed and returned as a token from the iterator yet.

Implementations

impl Lexer

pub const fn bytes_remaining(&self) -> usize

Get the number of bytes remaining that we need to transform into tokens.

pub fn new(source: SourceRef) -> Self

Construct a new Lexer over a given source reference.

pub fn fork(&self) -> Self

“Fork” this lexer, creating a new Lexer at the same position as this one that can be used for failable parsing. This can be compared to the original lexer it was forked from using Lexer::offset_from.

pub fn offset_from(&self, origin: &Self) -> usize

Get the number of bytes between the origin’s remaining and this Lexer’s remaining using Fragment::offset_from.

Panics

• This function panics under the same conditions as Fragment::offset_from.
• Generally the best way to avoid panics is to only call this function on Lexers created using Lexer::fork on the origin lexer.

pub fn matches(&self, pattern: &str) -> bool

Check if a pattern matches at the start of the Lexer::remaining Fragment.

pub fn next_token(&mut self) -> Option<Token>

Get the next token from the lexer.

Trait Implementations

impl Clone for Lexer

fn clone(&self) -> Lexer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Lexer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.