Struct tantivy::tokenizer::JapaneseTokenizer

pub struct JapaneseTokenizer;

Simple japanese tokenizer based on the tinysegmenter crate.

Trait Implementations

impl Clone for JapaneseTokenizer

fn clone(&self) -> JapaneseTokenizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Tokenizer<'a> for JapaneseTokenizer

type TokenStreamImpl = JapaneseTokenizerStream

Type associated to the resulting tokenstream tokenstream.

fn token_stream(&self, text: &'a str) -> Self::TokenStreamImpl

Creates a token stream for a given str.

fn filter<NewFilter>(
    self,
    new_filter: NewFilter
) -> ChainTokenizer<NewFilter, Self> where
    NewFilter: TokenFilter<Self::TokenStreamImpl>, 

Appends a token filter to the current tokenizer.