18. Compiler Internals: Lexical Analysis

Lexical analysis—commonly referred to as tokenization or lexing—is the initial stage of the Quorlin compiler pipeline. Positioned at the very front of the compilation lifecycle in quorlin/compiler.cpp, the Lexer transforms a raw sequence of UTF-8 characters (the source text) into a structured linear stream of strongly typed tokens.

This chapter dives deep into the inner workings of the lexical analysis phase within the kortana::quorlin C++ codebase, detailing character classification routines, identifier/keyword resolution, bounds checking, and the compiler's diagnostic error-handling strategy.

---

1. Overview of the Lexical Phase

Before the parser can construct an Abstract Syntax Tree (AST) or semantic analysis can verify type compatibility, the source code must be converted into discrete atomic units called tokens.

In Quorlin, compilation progresses through four strict phases defined in compile(std::string_view source) inside quorlin/compiler.cpp:

   Raw Source Code (.ql)
             │
             ▼
   ┌──────────────────┐
   │ 1. Lexical Phase │ ◄── (Lexer parses raw text into Token vector)
   └─────────┬────────┘
             │
             ▼
   ┌──────────────────┐
   │ 2. Parsing Phase │ ◄── (Parser constructs SourceUnit / AST)
   └─────────┬────────┘
             │
             ▼
   ┌──────────────────┐
   │ 3. Semantic Phase│ ◄── (Analyzer performs type checks & symbols)
   └─────────┬────────┘
             │
             ▼
   ┌──────────────────┐
   │ 4. Codegen Phase │ ◄── (CodeGenerator emits KVM Instructions)
   └──────────────────┘

The primary duty of the lexical analyzer is to classify every character sequence into a token category (TokenKind), track line/column positions for error reporting, and reject unparseable or out-of-bounds input before down-stream compiler stages run.

---

2. Lexer Architecture and Pipeline Integration

The lexer is instantiated in quorlin/compiler.cpp by passing the entire source buffer as a string view, alongside a reference to a DiagnosticBag:

// Excerpt from quorlin/compiler.cpp
CompilationResult compile(std::string_view source) {
    CompilationResult result;

    // --- 1. Lex ---------------------------------------------------------------------------------//
    Lexer lexer{source, result.diagnostics};
    std::vector<Token> tokens = lexer.tokenize();

    // A lexical error means the token stream no longer describes the source. Parsing it would report
    // grammar errors about tokens the user never wrote.
    if (result.diagnostics.has_errors()) return result;
    
    // ... Parsing, Analysis, and Codegen follow ...
}

Strategic Design Choice: Lexical Guarding

Notice the diagnostic check immediately following lexer.tokenize(). The Quorlin compiler enforces a strict early-return policy upon encountering lexical errors. If a character cannot be mapped to a valid token (e.g., an unclosed string or illegal symbol), compilation halts immediately.

Attempting to parse a corrupted token stream causes cascading grammar errors—reporting syntactic failures for tokens that were synthesized improperly or missed completely. By cutting execution off at stage 1, the compiler presents clean, actionable error diagnostics to the developer.

---

3. Character Classification Primitives

Low-level tokenization relies on fast, inline character predicate functions defined in the anonymous namespace of quorlin/lexer.cpp. These primitives operate directly on ASCII character values to validate identifiers, numbers, and hexadecimals.

namespace kortana::quorlin {
namespace {

[[nodiscard]] bool is_identifier_start(char c) noexcept {
    return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_';
}

[[nodiscard]] bool is_identifier_part(char c) noexcept {
    return is_identifier_start(c) || (c >= '0' && c <= '9');
}

[[nodiscard]] bool is_decimal_digit(char c) noexcept { 
    return c >= '0' && c <= '9'; 
}

[[nodiscard]] bool is_hex_digit(char c) noexcept {
    return is_decimal_digit(c) || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F');
}

[[nodiscard]] uint8_t hex_value(char c) noexcept {
    if (c <= '9') return static_cast<uint8_t>(c - '0');
    if (c <= 'F') return static_cast<uint8_t>(c - 'A' + 10);
    return static_cast<uint8_t>(c - 'a' + 10);
}

}  // namespace
}

Classification Breakdown

1. Identifier Boundaries:

   • An identifier must start with an ASCII letter (a-z, A-Z) or an underscore (_).
   • Subsequent characters in an identifier can include decimal digits (0-9).
   • Security Note: By restricting identifier start characters to [a-zA-L_], the lexer eliminates ambiguity between numeric literals and user-defined variable names.

2. Hexadecimal Decoding:

   • is_hex_digit(c) checks if a character falls in the range 0-9, a-f, or A-F.
   • hex_value(c) performs branchless byte translation to extract raw 4-bit nibbles from hex characters during numerical parsing.

---

4. Identifiers, Keywords, and Syntax Mapping

Quorlin's syntax is explicitly designed to read like plain English while enforcing strict type boundaries. The lexer maps identifier strings to reserved keywords using describe(TokenKind) in quorlin/lexer.cpp.

Reserved Keywords Table

Quorlin Keyword	Internal TokenKind	Description
contract	TokenKind::Contract	Declares a contract scope
record	TokenKind::Record	Declares a structured record type
interface	TokenKind::Interface	Declares an external contract interface
constructor	TokenKind::Constructor	Execution block on deployment
reads	TokenKind::Reads	Non-mutating function modifier (read-only/view)
writes	TokenKind::Writes	Mutating function modifier (state changes permitted)
event	TokenKind::Event	Contract event declaration
emit	TokenKind::Emit	Emits a log/event topic
indexed	TokenKind::Indexed	Marks event parameter for log indexing
if	TokenKind::If	Conditional control flow statement
else	TokenKind::Else	Alternative branch control flow

Primitive Type Keywords

Unlike standard languages using C-like type names in source code, Quorlin uses explicit English words for primitive types. The parser and semantic analyzer translate these human-friendly types into machine types:

// Excerpt from quorlin/parser.cpp
std::string_view type_name(Type type) noexcept {
    switch (type) {
        case Type::U256:    return "number";
        case Type::Bool:    return "truth";
        case Type::Address: return "address";
        case Type::Text:    return "text";
        case Type::Void:    return "nothing";
        case Type::Invalid: return "<invalid>";
    }
    return "<unknown>";
}

• number: Represents a 256-bit unsigned integer (Type::U256).
• truth: Represents a boolean condition (Type::Bool).
• address: Represents a 20-byte Kortana execution account or contract (Type::Address).
• text: Represents a string bound by length limitations (Type::Text).
• nothing: Represents the return type of a procedure that yields no value (Type::Void).

---

5. Literal Parsing and Constraint Enforcement

The lexer enforces memory limits and bounds directly at token generation time.

String/Text Literals and kMaxTextBytes

To prevent memory-exhaustion attacks against the Kortana node during compilation, text literals processed by the lexer share a strict boundary constant with the semantic analyzer: kMaxTextBytes (imported from quorlin/sema.hpp).

If a literal string exceeds this byte limit, the Lexer logs an error directly into the DiagnosticBag and marks the compiler pass as failed.

Hex and Address Literals

Address literals in Quorlin are lexed as hexadecimal values prefixed with 0x and validated to be 20 bytes (40 hex characters) in length.

// Valid Quorlin Code Context
address recipient = 0x1122334455667788990011223344556677889900;

When the lexer encounters 0x, it delegates scanning to a dedicated hexadecimal reader routine that uses is_hex_digit and converts character streams into raw byte arrays.

---

6. Diagnostic Handling and Lexical Errors

Diagnostics are managed centrally through the DiagnosticBag. When the lexer encounters an unexpected character or illegal format, it emits a structured error containing:

1. The source code location (Line number, Column offset).
2. A human-readable description of the lexical failure.

Diagnostic Example Scenario

Consider an invalid character like # appearing outside a comment in a Quorlin source file:

contract Token {
    number balance = #100;
}

The lexer processing line 2 hits #. Because is_identifier_start('#') and is_decimal_digit('#') both return false, the lexer emits a diagnostic error:

[Error] Line 2, Column 22: unexpected character '#' in source file.

Because diagnostics.has_errors() returns true, the pipeline aborts immediately before the parser attempts to construct the AST, preventing confusing downstream error cascade messages.

---

7. Step-by-Step Execution Walkthrough

To understand how the lexer breaks down code into tokens, let's trace the tokenization of a Quorlin transfer function:

writes truth transfer(address recipient, number amount) {
    require amount > 0, "invalid amount";
    return yes;
}

Execution Flow:

1. Keyword Analysis:

   • The lexer encounters w-r-i-t-e-s. is_identifier_start('w') is true. It collects characters until whitespace.
   • String matches reserved keyword writes.
   • Emits: Token{ Kind: TokenKind::Writes, Text: "writes" }.

2. Type Mapping:

   • Character sequence t-r-u-t-h matches reserved keyword truth (Type::Bool).
   • Emits: Token{ Kind: TokenKind::TruthType, Text: "truth" }.

3. Identifier & Punctuation:

   • Scans transfer -> TokenKind::Identifier.
   • Scans ( -> TokenKind::LeftParen.
   • Scans address -> TokenKind::AddressType.
   • Scans recipient -> TokenKind::Identifier.
   • Scans , -> TokenKind::Comma.
   • Scans number -> TokenKind::NumberType.
   • Scans amount -> TokenKind::Identifier.
   • Scans ) -> TokenKind::RightParen.
   • Scans { -> TokenKind::LeftBrace.

4. Statement Body & Literal Constraints:

   • Scans require -> TokenKind::Require.
   • Scans amount -> TokenKind::Identifier.
   • Scans > -> TokenKind::GreaterThan.
   • Scans 0 -> TokenKind::NumberLiteral.
   • Scans , -> TokenKind::Comma.
   • Scans "invalid amount" -> Lexer verifies string length against kMaxTextBytes. Emits TokenKind::TextLiteral.
   • Scans ; -> TokenKind::Semicolon.
   • Scans return -> TokenKind::Return.
   • Scans yes -> TokenKind::BooleanLiteral (mapped to true).
   • Scans ; -> TokenKind::Semicolon.
   • Scans } -> TokenKind::RightBrace.

The resulting std::vector<Token> is cleanly formatted and verified, ready to be safely consumed by the Parser in Phase 2.