Options  

Tokenizer

• The GPT family of models process text using tokens, which are common sequences of characters found in text. The models understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens.

• Text: Text to process and return token quantity.
• Tokens: Contains tokens quantity in text, and list with tokens ids.
  • Tokens: Token quantity.
  • TokensIds: Token id list.