tokenize

To reduce to a token or set of tokens by lexical analysis.

Verb

  1. To reduce to a token or set of tokens by lexical analysis.

    Coordinate Terms: codify encode

  2. To substitute sensitive data with meaningless placeholders.
  3. To treat as a token minority.

Origin

Etymology tree English token Proto-Indo-European *-id- Proto-Indo-European *-yéti Proto-Indo-European *-idyéti Proto-Hellenic *-íďďō Ancient Greek -ῐ́ζω (-ĭ́zō)bor. Late Latin -izōder. Middle French -iserbor. Middle English -isen English -ize English tokenize From token + -ize.

Forms

tokenizes tokenizing tokenized tokenise

Related

tokenism

Derived

detokenize mistokenize retokenize tokenizable tokenization tokenizer untokenized