Struct tantivy::tokenizer::RawTokenizer

pub struct RawTokenizer;

For each value of the field, emit a single unprocessed token.

Trait Implementations

impl Clone for RawTokenizer

fn clone(&self) -> RawTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Tokenizer<'a> for RawTokenizer

type TokenStreamImpl = RawTokenStream

Type associated to the resulting tokenstream tokenstream.

fn token_stream(&self, text: &'a str) -> Self::TokenStreamImpl

Creates a token stream for a given str.

fn filter<NewFilter>(
    self,
    new_filter: NewFilter
) -> ChainTokenizer<NewFilter, Self> where
    NewFilter: TokenFilter<Self::TokenStreamImpl>, 

Appends a token filter to the current tokenizer.