Struct tantivy::tokenizer::FacetTokenizer

pub struct FacetTokenizer;

The FacetTokenizer process a Facet binary representation and emits a token for all of its parent.

For instance, /america/north_america/canada will emit the three following tokens - /america/north_america/canada - /america/north_america - /america

Trait Implementations

impl Clone for FacetTokenizer

fn clone(&self) -> FacetTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Tokenizer<'a> for FacetTokenizer

type TokenStreamImpl = FacetTokenStream<'a>

Type associated to the resulting tokenstream tokenstream.

fn token_stream(&self, text: &'a str) -> Self::TokenStreamImpl

Creates a token stream for a given str.

fn filter<NewFilter>(
    self,
    new_filter: NewFilter
) -> ChainTokenizer<NewFilter, Self> where
    NewFilter: TokenFilter<Self::TokenStreamImpl>, 

Appends a token filter to the current tokenizer.