TTokens

[Properties (by Name)] [Methods (by Name)] [Events (by Name)]

This is the string tokenizer

Declaration

Source position: tiTokenLibrary.pas line 33

`type TTokens = class(TObject) end;`
`public`
`constructor Create(); overload;`		Tokenizers constructor
`destructor Destroy; override;`		A standard class destructor.
`function Token();`		Returns a token
`function TokenCount;`		Number of tokens found in the original string.

Inheritance

`TTokens`		This is the string tokenizer
\|
`TObject`

Description

This is the string tokenizer. It is based on a Finite State Machine design pattern. The following image shows how the State Machineworks.

Example

    { Sample Usage }
    SomeTokenLine := '1,2,"3,4"';
    tokenizer := TTokens.Create(SomeTokenLine, ', ', '"', '"', '\', tsMultipleSeparatorsBetweenTokens);
    try
      i := tokenizer.TokenCount;   // i will now contain the value 3
      s := tokenizer.Token(1);     // s contains the string value '1'
      s := tokenizer.Token(3);     // s contains the string value '3,4'
    finally
      tokenizer.Free;
    end;