Struct tantivy::tokenizer::RemoveLongFilter

pub struct RemoveLongFilter { /* fields omitted */ }

RemoveLongFilter removes tokens that are longer than a given number of bytes (in UTF-8 representation).

It is especially useful when indexing unconstrained content. e.g. Mail containing base-64 encoded pictures etc.

Methods

impl RemoveLongFilter

pub fn limit(length_limit: usize) -> RemoveLongFilter

Creates a RemoveLongFilter given a limit in bytes of the UTF-8 representation.

Trait Implementations

impl Clone for RemoveLongFilter

fn clone(&self) -> RemoveLongFilter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<TailTokenStream> TokenFilter<TailTokenStream> for RemoveLongFilter where
    TailTokenStream: TokenStream, 

type ResultTokenStream = RemoveLongFilterStream<TailTokenStream>

The resulting TokenStream type.

fn transform(&self, token_stream: TailTokenStream) -> Self::ResultTokenStream

Wraps a token stream and returns the modified one.