Struct tantivy::tokenizer::SimpleTokenizer

pub struct SimpleTokenizer;

Tokenize the text by splitting on whitespaces and punctuation.

Trait Implementations

impl Clone for SimpleTokenizer

fn clone(&self) -> SimpleTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Tokenizer<'a> for SimpleTokenizer

type TokenStreamImpl = SimpleTokenStream<'a>

Type associated to the resulting tokenstream tokenstream.

fn token_stream(&self, text: &'a str) -> Self::TokenStreamImpl

Creates a token stream for a given str.

fn filter<NewFilter>(
    self,
    new_filter: NewFilter
) -> ChainTokenizer<NewFilter, Self> where
    NewFilter: TokenFilter<Self::TokenStreamImpl>, 

Appends a token filter to the current tokenizer.