Struct tantivy::tokenizer::Token

pub struct Token {
    pub offset_from: usize,
    pub offset_to: usize,
    pub position: usize,
    pub text: String,
}

Token

Fields

offset_from: usize

Offset (byte index) of the first character of the token. Offsets shall not be modified by token filters.

offset_to: usize

Offset (byte index) of the last character of the token + 1. The text that generated the token should be obtained by &text[token.offset_from..token.offset_to]

position: usize

Position, expressed in number of tokens.

text: String

Actual text content of the token.

Trait Implementations

impl Default for Token

fn default() -> Token

Returns the "default value" for a type.