struct C0::Tokenizer

• C0::Tokenizer
• Struct
• Value
• Object

Overview

High-performance zero-copy tokenizer for C0DATA.

Scans a byte buffer for control codes (< 0x20) and emits tokens as offsets into the original buffer. No allocation for data values.

The hot loop is a single comparison: byte < 0x20.

Defined in:

c0/tokenizer.cr

Constructors

• .new(buf : Bytes)

Instance Method Summary

• #each(& : Token -> ) : Nil

  Yields each token to the block.

• #to_a : Array(Token)

  Collects all tokens into an array.

Constructor Detail

Instance Method Detail