[Overview][Types][Classes][Index] Reference for unit 'tiTokenLibrary' (#tiopf)

Reference for unit 'tiTokenLibrary'

String tokenizer library


The TToken tokenizer is extremely flexible, being able to tokenize quite complex strings. See the unit tests for examples of these. It is also a lot faster than the previous string tokenizer function in tiUtils.

The tiToken/tiNumToken functions are wrapper functions for the TToken class. Unfortunately they will not be able to take full advantage of the new tokenizer, because tokenized strings will not stay in memory between calls.

Feature List:

The following diagram explains how the states change.

Documentation generated on: 2017-08-16