CRATER: Complex Reflective Angular Transformer with External Retrieval

Notation

1   Introduction

The Transformer architecture (Vaswani et al., 2017) has become the dominant paradigm in deep learning, powering state-of-the-art models across natural language processing, computer vision, and multimodal learning. At its core, the standard transformer relies on real-valued representations and dot-product attention to compute token relationships. While remarkably effective, this framework has inherent limitations: the dot-product attention mechanism captures only magnitude-based similarity, and retrieval-augmented generation (RAG) systems remain architecturally disconnected from the core transformer computation.

Complex-valued neural networks have a rich theoretical history (Hirose, 2012; Trabelsi et al., 2018), offering advantages including richer representational capacity through amplitude-phase decomposition, natural handling of periodic and oscillatory patterns, and built-in rotational equivariance. However, their application to transformer architectures has remained largely unexplored.

In this paper, we present the Complex Reflective Angular Transformer with External Retrieval (CRATER), a complete architecture that integrates complex-valued computation, angular geometry, and retrieval augmentation into a unified framework. CRATER introduces five tightly coupled innovations. First, multi-frequency positional encoding captures dependencies at multiple temporal scales through learnable frequency bands. Second, complex state representation embeds tokens into the complex plane with phase-normalized rotational operators, complex residual connections, and dedicated normalization (Complex Layer Normalization) and activation functions (complex GELU). Third, angular attention replaces dot-product similarity with trigonometric phase differences, enhanced through specialized head partitioning and low-rank complex projections. Fourth, Imaginary Reflection Windows (IRW)—a legacy name; the final design queries the full complex state, not the imaginary part alone (§4.4)—perform multi-query retrieval with weighted fusion and angular similarity, integrating RAG directly into the transformer's core computation. Fifth, Complex Mixture of Experts (CMoE) increases model capacity through sparse complex-valued expert routing without proportionally increasing computation.

Training stability is maintained through a dedicated suite of techniques: component-wise gradient clipping for complex parameters, phase regularization to prevent oscillatory instability, and magnitude regularization to control representation norms.

Our contributions can be summarized as follows:

• Core contributions. (i) a complex block representation with phase-preserving Complex Layer Normalization and complex GELU (with multi-frequency positional encoding and phase-normalized rotational operators); (ii) Angular Attention, a trigonometric attention mechanism with specialized head partitioning and an optional non-causal linear-time (O(n)) angular-kernel prototype (softplus-approximate; the causal path is not yet implemented); (iii) Imaginary Reflection Windows with multi-query retrieval, weighted fusion, and complex-domain angular similarity; (iv) a Complex Mixture of Experts for sparse complex-valued capacity; and (v) a complete training-stability framework (component-wise gradient clipping, phase regularization, and magnitude regularization).
• Engineering optimizations. Low-rank complex projections, a magnitude-reliability gate over phases, causal masking, and embedding/output weight tying — the choices that keep the per-token cost near 1.4× (Table 5b) rather than the naive 2×.
• Optional modules (implemented but not yet evaluated). A lattice phase mode (§4.10), RoPE-style relative phase encoding (§4.11), a calibrated retrieval-confidence gate (§4.12), conjugate (phase-mirror) attention heads (§4.13), and phase-coherence-guided decoding (§4.14).

We stress at the outset that these gains come with a genuine trade-off. Complex-valued computation roughly doubles the arithmetic per element, introduces additional hyperparameters (frequency bands, regularization weights, gate temperatures), and complicates optimization relative to a real-valued transformer. Our claims are further tempered by the fact that all results below come from single-seed runs; we therefore present CRATER as an exploratory formulation whose expressivity benefits must be weighed against its added complexity, and whose empirical advantages await multi-seed and larger-scale confirmation.

2   Related Work

2.1   Complex-Valued Neural Networks

Complex-valued neural networks have been studied since the early work of Georgiou and Koutsougeras (1992). Deep complex networks (Trabelsi et al., 2018) demonstrated the viability of complex arithmetic in modern deep learning. More recently, complex-valued representations have found applications in signal processing (Choi et al., 2019), speech enhancement, and physics-informed neural networks. Our work extends this line of research to the transformer architecture, using complex representations not merely as a computational tool but as the foundation for a new attention mechanism.

2.2   Attention Mechanisms

Since the introduction of scaled dot-product attention (Vaswani et al., 2017), numerous alternatives have been proposed, including linear attention (Katharopoulos et al., 2020), sparse attention (Child et al., 2019), and kernel-based attention (Choromanski et al., 2021). Rotary Position Embeddings (RoPE) (Su et al., 2021) introduced complex rotations for positional encoding but retained real-valued dot-product attention. CRATER goes further by computing attention scores entirely in the angular domain; via an optional linearized angular kernel (Section 4.3), this can in principle reduce asymptotic complexity from \(O(n^2)\) to \(O(n)\) in a non-causal setting; the causal autoregressive path is not yet implemented.

2.3   Retrieval-Augmented Generation

RAG approaches (Lewis et al., 2020; Borgeaud et al., 2022; Guu et al., 2020) augment language models with external knowledge retrieval. Existing methods typically treat retrieval as a pre-processing step or an auxiliary module. RETRO (Borgeaud et al., 2022) introduced chunked cross-attention for retrieved passages, while FiD (Izacard and Grave, 2021) processes retrieved documents in the encoder. CRATER's Imaginary Reflection Window offers a structurally distinct approach: it uses the mathematical structure of complex representations to generate multi-query retrieval signals intrinsically, with angular similarity in the complex domain providing a shared geometric basis for internal attention and external retrieval (a unifying design principle rather than a claim of mathematical equivalence).

3   Architecture Overview

The CRATER architecture processes input tokens through a unified pipeline of seven integrated stages, as illustrated in Figure 1. The computation within each layer is complex-valued end-to-end—attention, normalization, and expert mixing all act on \(\mathbb{C}^{d/2}\) states—but the residual stream between layers is carried as the stacked real pair \([\Re z;\, \Im z] \in \mathbb{R}^{d}\) and re-lifted to \(\mathbb{C}^{d/2}\) at the entry of the next layer (Stage 2). CRATER is therefore block-complex: each block is internally complex end-to-end, rather than a single persistent complex tensor threaded unchanged through the full depth of the network. In other words, CRATER should not be interpreted as maintaining a single complex hidden state across all layers; each block lifts the real residual stream into a fresh complex representation.

Stage 1 — Multi-Frequency Positional Encoding (§4.1). Input tokens are embedded and augmented with positional information through \(K\) learnable frequency bands, capturing dependencies from local syntactic patterns to long-range discourse structure.

Stage 2 — Complex State Construction (§4.2). Real-valued embeddings\(h_t^{(0)} \in \mathbb{R}^{d}\) are projected into \(\mathbb{C}^{d/2}\) via two learned linear maps\(W_{\text{re}}, W_{\text{im}} \in \mathbb{R}^{(d/2)\times d}\)—an explicit \(\mathbb{R}^{d} \to \mathbb{R}^{d/2}\times\mathbb{R}^{d/2}\) split that creates no width. A content-dependent complex rotation\(\tilde{z}_t = z_t \cdot e^{i\theta_t}\) is applied, followed by phase-preservingComplex Layer Normalization (CLN)—the complex analogue of RMSNorm with a positive-real gain \(\gamma \in \mathbb{R}_{>0}\) (via softplus) and a complex bias\(\beta \in \mathbb{C}\). Split-complex GELU (cGELU) activations, a non-holomorphic choice justified by the Wirtinger calculus framework (Kreutz-Delgado, 2009), introduce smooth non-linearity in both the real and imaginary pathways.

Stage 3 — Angular Attention (§4.3). Attention scores replace standard dot-product similarity with a branch-cut-free phasor formulation: unit phasors\(u(x) = x / (|x| + \varepsilon)\) are formed from complex queries and keys, and the angular score is the \(1/\sqrt{d_h}\)-normalized sum over channels of\(\operatorname{Re}(u(Q)\,\overline{u(K)})\)—equal to the cosine of the per-channel phase difference without ever computing or differencing phases explicitly. A learnable per-head inverse-temperature \(\gamma_h\) rescales the logits, and an optional per-head magnitude confidence bias \(\eta_h\) adds bounded additive salience from key magnitudes. Attention heads are partitioned into real, imaginary, and mixed groups for specialized processing. An approximate positive angular-kernel variant (softplus on phasor coordinates) optionally reduces complexity from \(O(n^2)\) to \(O(n)\), low-rank complex projections reduce parameter counts by 30–60%, and a causal mask is applied for autoregressive language modeling.

Stage 4 — Imaginary Reflection Window (§4.4). The full complex state \(\hat{z}_t\) (not the imaginary part alone, despite the historical name) generates\(M\) parallel retrieval queries via complex projectors, each independently searching the external knowledge base using angular similarity in \(\mathbb{C}\). An non-differentiable, out-of-graph ANN index (FAISS/Contriever) pre-selects \(c \ll N\) candidates per query; CRATER then differentiably re-ranks and fuses these candidates, so the \(O(N)\) corpus scan is never materialized inside the computational graph. An orthogonality penalty on the query Gram matrix decorrelates the \(M\) queries, and a dynamic retrieval temperature\(\tau_{\text{eff}} = \tau / (1 + \bar{|q|})\) sharpens scoring for high-energy queries. Results are integrated via a temperature-controlled reflection gate. Two additional retrieval refinements are integral to the IRW design: a dual-index “harmonic search” that re-ranks ANN candidates by complex phase coherence, and phase-coherent key consolidation that fuses near-duplicate memory entries agreeing in both real magnitude and Lattice phase code (§4.10), shrinking the external index without removing the ANN dependency.

Stage 5 — Complex Residual Connections (§4.5). Skip connections employ complex residual coefficients \(\alpha + i\beta\), providing directional residuals that adjust both magnitude and phase across layers.

Stage 6 — Complex Mixture of Experts (§4.6). A sparse CMoE layer routes tokens to specialized complex-valued expert networks via a complex representational routerthat computes routing logits as the angular projection of the complex state onto learned complex gate centroids \(\{G_e\}\), maintaining the Hermitian geometry of the representation. Noisy gating (Gaussian perturbation of routing logits) and a router-entropy bonus stabilize expert assignment. Only the top-\(k\) (typically \(k=2\)) experts are activated per token, increasing capacity without proportionally increasing computation.

Stage 7 — Output Projection (§4.8). Complex-valued hidden states are concatenated (real and imaginary parts) and projected back to the real-valued output space through Layer Normalization and a weight-tied linear classifier.

Training stability (§4.7) is ensured throughout by component-wise gradient clipping, phase regularization (\(\mathcal{L}_{\text{phase}}\)), and magnitude regularization (\(\mathcal{L}_{\text{mag}}\)), which together keep complex-valued representations in a well-conditioned region of \(\mathbb{C}\). A Lattice Phase Mode (§4.10) further enables structured phase quantization for discrete reasoning tasks.

Optional extensions (§4.11–4.14). The core seven-stage pipeline is complemented by four optional modules, each implemented in the reference code but not yet evaluated: RoPE-style relative phase encoding (§4.11), which makes angular scores translation-equivariant; a calibrated confidence gate (§4.12), which adds an ECE penalty so the retrieval gate tracks true predictive confidence; conjugate (phase-mirror) attention heads (§4.13), which detect semantic contradiction via phase conjugation (\(\phi_Q \approx -\phi_K\)); and phase-coherence guided decoding (§4.14), which uses circular variance of hidden-state phases as a free uncertainty signal at inference time. Section 7 discusses a broader family of speculative future extensions.

4   Mathematical Framework

4.1   Multi-Frequency Positional Encoding

Given an input sequence of tokens, we first embed each token into a dense vector and augment it with positional information through multiple learnable frequency bands. Let the token at position \(t\) be represented by its one-hot vector \(x_t \in \{0,1\}^{|V|}\). The token embedding is:

where \(W_E \in \mathbb{R}^{d \times |V|}\) is the embedding matrix, \(b_E \in \mathbb{R}^d\) the bias, and \(|V|\) is the vocabulary size. Rather than a single-frequency sinusoidal scheme, CRATER employs a multi-frequency positional encoding that sums over \(K\) learnable frequency bands:

where \(\omega_k^{(i)}\) and \(\delta_k^{(i)}\) are learnable frequency and phase parameters for each dimension \(i\) and frequency band \(k\). The \(1/\sqrt{K}\) factor keeps the encoding variance independent of the number of bands \(K\), preventing the amplitude from growing with\(K\). Because a plain sine vanishes at \(t=0\) for every band, the sines-only form would be degenerate at the origin; the learnable per-band phase offsets \(\delta_k^{(i)}\) break this degeneracy and give position 0 a distinct, learnable signature. This captures dependencies at multiple temporal scales simultaneously—high-frequency bands encode local syntactic patterns while low-frequency bands capture long-range discourse structure. The combined representation is:

4.2   Complex State Representation

The core of CRATER maps real-valued embeddings into the complex plane:

where the real and imaginary components are obtained through learned linear projections:

Concretely, \(W_{\text{re}}, W_{\text{im}} \in \mathbb{R}^{(d/2)\times d}\) each map the real embedding \(h_t^{(0)} \in \mathbb{R}^{d}\) into \(\mathbb{R}^{d/2}\), so the pair\((h_t^{\text{re}}, h_t^{\text{im}})\) realizes an explicit split \(\mathbb{R}^{d} \to \mathbb{R}^{d/2}\times\mathbb{R}^{d/2}\), read as the single complex vector \(z_t \in \mathbb{C}^{d/2}\). Whenever a real-valued interface is required—residual adds, the retrieval gate, or the output head—we reverse the identification with the concatenation\([\Re(z_t);\Im(z_t)] \in \mathbb{R}^{d}\), so no width is created or lost: \(d\) real coordinates in, \(d/2\) complex coordinates internally, \(d\) real coordinates out.

We then apply a complex rotation parameterized by a learned angle \(\theta_t\):

Expanding this rotation:

where \(\theta_t = W_\theta \, h_t^{(0)} + b_\theta\). This rotation allows the model to learn position- and content-dependent transformations in the complex plane.

Complex Layer Normalization (CLN)

Standard LayerNorm is designed for real-valued tensors; applied naively to a complex state (e.g., to the concatenated real and imaginary parts, or per-component with independent scales and shifts) it rescales and re-centers \(\Re(z)\) and \(\Im(z)\) differently and therefore rotates and distorts \(\arg(z)\), destroying the angular information the rest of the model relies on. CRATER instead employs Complex Layer Normalization (CLN) that stabilizes the magnitude of the complex state while preserving its phase when the bias is zero (the normalization core—division by a positive real scalar—never rotates phase; the optional complex bias β can then shift it):

where \(\text{RMS}(z_t)\) is a single positive-real scalar for token \(t\), the root-mean-square magnitude over that token's \(d/2\) complex channels (not across the batch or across positions), the complex analogue of RMSNorm (Zhang & Sennrich, 2019; cf. Ba et al., 2016); \(\gamma \in \mathbb{R}_{>0}\) is a learnable positive real gain (parameterized as \(\gamma = \text{softplus}(g)\) so it can never rotate the phase), \(\beta \in \mathbb{C}\) is a complex bias, and \(\varepsilon\) is a small constant for numerical stability. The role of the bias is deliberately different from that of the gain: because \(\beta\) is complex, it applies a learnable translation in the complex plane that can re-center both the real and imaginary parts of the normalized state and hence shift its aggregate phase, whereas the positive-real \(\gamma\) can only rescale magnitude. Setting \(\beta = 0\) recovers a purely magnitude-normalizing, phase-preserving map; a nonzero \(\beta\) lets each channel learn a fixed complex offset without disturbing the magnitude normalization applied before it. The normalizing factor is a single positive real scalar, so dividing by it leaves \(\arg(z)\) unchanged: CLN rescales magnitude without rotating phase. This is deliberate. An earlier revision of CRATER applied a full\(2\times 2\) covariance whitening \(V^{-1/2}(z-\mu_z)\) that decorrelates the real and imaginary parts and maps each feature to an isotropic unit distribution. Geometrically, forcing isotropy spreads the phase uniformly around the circle—it maximizes phase entropy—which works directly against the phase regularizer of Section 4.7, whose whole purpose is to concentrate phases (minimize their circular variance). The two objectives fight at every layer. Replacing whitening with the phase-preserving RMS form above removes that conflict by construction: magnitude is stabilized by a phase-commuting rescale, and all angular shaping is left to the phase regularizer and the angular-attention geometry. In our single-seed runs, this phase-preserving normalization reduced the frequency of gradient-norm spikes (training steps whose global gradient norm exceeded the clipping threshold) relative to applying standard LayerNorm independently to the real and imaginary parts, while keeping the angular information the model relies on intact; we report this as a qualitative stabilizing effect rather than a precise percentage.

Complex GELU Activation (cGELU)

CRATER extends GELU (Hendrycks & Gimpel, 2016) to the complex domain by applying it independently to each component:

This preserves the non-linear gating property of GELU while maintaining the complex structure. The component-wise application ensures that both the real (content) and imaginary (relational) pathways benefit from smooth non-linearity. Formally, cGELU is a split-complex (component-wise) activation, following standard practice in complex-valued networks (Trabelsi et al., 2018). It is deliberately non-holomorphic, but this is unproblematic: because the training loss is real-valued, all complex parameters are optimized via Wirtinger calculus (Kreutz-Delgado, 2009), which accumulates the conjugate Wirtinger derivative \(\partial \mathcal{L}/\partial \bar{z}\)—the correct steepest-descent direction for a real function of a complex variable—regardless of holomorphicity. Every complex operation in CRATER (cGELU, CLN, the phasor maps, the complex projections) is therefore trained with well-defined gradients.

4.3   Angular Attention Mechanism

CRATER replaces standard dot-product attention with Angular Attention, which operates on the phases of complex-valued query and key vectors. For each head \(h\):

where \(W_Q^{(h)}, W_K^{(h)}, W_V^{(h)} \in \mathbb{C}^{d_h \times d/2}\).

Phase Normalization

Computing angular differences directly via \(\arg(q)\) can suffer from oscillatory instability during early training. CRATER extracts phases component-wise using the numerically stable two-argument arctangent, which already returns a principal value in \((-\pi, \pi]\):

Using atan2 rather than arctan resolves the quadrant ambiguity and avoids the discontinuity at \(\Re(q)=0\). The principal phase nonetheless retains the usual branch cut at\(\pm\pi\), and differencing two principal phases—as an explicit\(\phi_Q - \phi_K\) would—reintroduces a \(2\pi\) jump whose gradient is ill-behaved whenever a query–key pair straddles the cut. CRATER therefore never differences phases explicitly. It forms unit phasors\(u(x) = x / (|x| + \varepsilon)\) and reads the phase agreement off their Hermitian product, using the identity \(\cos(\phi_Q - \phi_K) = \Re\!\big(u(Q)\,\overline{u(K)}\big)\). The angular attention score between positions \(t\) and \(s\) is thus the \(1/\sqrt{d_h}\)-normalized sum (not a plain average) over the \(d_h\) channels of head\(h\), of this branch-cut-free per-channel agreement:

The two forms are equal in value; the left-hand phasor form is what CRATER actually evaluates. Because it depends on\(Q\) and \(K\) only through the smooth map \(u(\cdot)\), the score and its gradient are continuous everywhere—including at \(\pm\pi\)—so the branch-cut discontinuity an explicit phase difference would inject is removed by construction. This continuity refers to the phase wrap-around at \(\pm\pi\); the phases of near-zero-magnitude coordinates remain uninformative and are down-weighted by the magnitude masking of Section 4.4. In our single-seed runs this phasor formulation produced smoother, more stable training without a measurable convergence penalty.

Attention weights are obtained via softmax:

Each cosine term is bounded in \([-1, 1]\), so the raw sum lies in \([-d_h, d_h]\); dividing by\(\sqrt{d_h}\) keeps the score variance \(O(1)\) as \(d_h\) grows—the same role the\(1/\sqrt{d_k}\) factor plays in scaled dot-product attention—so the softmax neither saturates nor collapses to uniform. Crucially, the score depends only on the phases of the complex representations and is therefore invariant to their magnitudes, focusing purely on directional relationships.

On the \(1/\sqrt{d_h}\) scaling and a learnable temperature. The angular case differs from dot-product attention in a subtle way. Each summand here is a cosine confined to \([-1,1]\), so theinformative part of the score—its expected per-channel phase agreement—does not itself grow with\(d_h\); the \(1/\sqrt{d_h}\) factor controls the variance of the sum of (near-independent) cosines, keeping it \(O(1)\), but it simultaneously shrinks the mean logit like\(1/\sqrt{d_h}\), which at large head dimension can flatten the softmax toward uniform. To decouple these two effects, CRATER multiplies the normalized score by a learnable per-head inverse-temperature\(\gamma_h > 0\), \(\alpha_{ts}^{(h)} \leftarrow \gamma_h\, \alpha_{ts}^{(h)}\), initialized to\(\gamma_h = 1\) so the plain \(1/\sqrt{d_h}\) regime is exactly recovered at initialization. Each head can then recalibrate its own sharpness—sharpening the distribution when \(\sqrt{d_h}\) has over-diluted the logits—mirroring the learned logit temperatures used in ViT and CLIP. This is a formulation option; we report no isolated benchmark for it.

Magnitude-Phase Dual Score

The pure angular score above is magnitude-invariant by construction, which is ideal for directional matching but discards the confidence encoded in the token norms. CRATER therefore augments it with an optional, per-head salience term that reintroduces magnitude as a bounded additive term—a product of query and key magnitudes that is added to, not multiplied into, the angular score:

where \(\lambda_h\) is a learnable per-head scalar initialized to \(0\), so training begins from the pure angular regime and only admits magnitude information if it reduces the loss. Setting \(\lambda_h = 0\)recovers Eq. (12) exactly; small positive values let high-norm (high-confidence) query–key pairs sharpen their agreement, echoing the magnitude–phase interplay of complex retrieval without sacrificing the bounded, stable behavior established in Property 1. We stress that Property 1's magnitude-invariance applies only to the pure angular score; the optional salience term (\(\lambda_h \neq 0\)) intentionally trades that invariance for a norm-based confidence signal.

Magnitude confidence gate (noise robustness). Pure magnitude-invariance has one adversarial failure mode: a token with near-zero norm—padding, special characters, or a weak transition—has anerratic phase (the \(\operatorname{atan2}\) of numerical noise), yet the angular score would grant it the same standing as a high-norm content token whose phase is meaningful. CRATER therefore adds an explicit magnitude gate inside the attention logits, not only in the IRW similarity. Using the bounded per-channel salience \(m(\cdot)=|\cdot|/(|\cdot|+\tau)\) already defined above, each key position\(s\) receives a confidence score \(\bar m_s^{(h)} = \tfrac{1}{d_h}\sum_j m(K_{h,s,j})\) and the logits are biased by a learnable per-head \(\eta_h \ge 0\), \(\alpha_{ts}^{(h)} \leftarrow \alpha_{ts}^{(h)} + \eta_h\, \bar m_s^{(h)}\)(broadcast over queries \(t\)). With \(\eta_h = 0\) the pure angular regime is recovered; positive\(\eta_h\) suppresses low-norm keys before the softmax, so a noise token can no longer attract attention by phase coincidence alone. The gate acts on keys (what may be attended to). Note that—like the dual-score salience term—a positive \(\eta_h\) deliberately breaks the positive-scaling invariance of Property 1: because \(m\) depends on magnitude, rescaling a key by \(c>0\) changes its confidence, which is exactly the intended behaviour (norm as a confidence signal). Property 1 therefore holds only for the pure angular component (\(\lambda_h = \eta_h = 0\)).

Specialized Head Partitioning

CRATER partitions the \(H\) attention heads into three functional groups:

• Real heads — operate primarily on \(\Re(z)\), capturing semantic content and factual information.
• Imaginary heads — operate primarily on \(\Im(z)\), specializing in relational and structural patterns.
• Mixed heads — operate on the full complex representation \(z \in \mathbb{C}^{d_h}\), capturing interactions between content and structure.

This partitioning is an architectural option: a head is assigned to a group by masking its input to the real part, the imaginary part, or the full complex state before projection. The reference implementation accompanying this paper uses uniform mixed heads throughout; the specialized real/imaginary partitioning is presented here as a design variant and was not exercised in the single-seed runs of Section 5, so we make no empirical claim about its per-group contribution.

Linear Angular Kernel Variant (O(n))

The raw phase-agreement score in Eq. (12) admits an exact feature-map factorization, enabling a linear-time attention variant—via a simple feature map. Using the identity \(\cos(\phi_Q - \phi_K) = \cos\phi_Q \cos\phi_K + \sin\phi_Q \sin\phi_K\), define the raw \(2 d_h\)-dimensional phasor map

so that the summed phase agreement factorizes as an inner product,\( \sum_j \cos(\phi_{Q,j} - \phi_{K,j}) = \langle \varphi_0(Q_t), \varphi_0(K_s) \rangle \), holding identically for every pair \((t,s)\). Kernelized attention requires positive features, so in practice we use the strictly positive map \(\varphi(x) = \operatorname{softplus}(\varphi_0(x)) > 0\)(justified below). Substituting \(\varphi(\cdot)\) into kernelized (linear) attention then computes the attended output without ever materializing the \(n \times n\) score matrix:

The two running sums \(\sum_s \varphi(K_s)\, V_s^{\top}\) and \(\sum_s \varphi(K_s)\) are computed once in \(O(n\, d_h)\) and reused for all queries, reducing both time and memory from \(O(n^2)\) to\(O(n)\) in sequence length. Because the raw cosine and sine coordinates can be negative, the induced kernel\(\kappa(Q_t,K_s) = \langle \varphi(Q_t), \varphi(K_s) \rangle\) must be made strictly positive for the normalization to be well defined. An earlier revision simply appended a constant coordinate; but a constant large enough to guarantee positivity grows like \(\sqrt{d_h}\) and, at large head dimension, dominates the trigonometric coordinates confined to \([-1,1]\)—it washes out the very phase differences the kernel is meant to measure, pushing the attention weights toward uniform. We instead pass the phasor coordinates through a strictly positive elementwise map, \(\operatorname{softplus}(\cdot)\) (a smooth analogue of the map of Katharopoulos et al., 2020), which guarantees \(\varphi > 0\) and hence a positive kernel. Crucially, softplus introduces no additive constant: the earlier \(\operatorname{elu}(\cdot)+1\) map adds a fixed \(+1\) baseline to every coordinate that, summed over a large head dimension, biases the kernel toward a magnitude-dependent floor and dilutes the angular signal, whereas softplus leaves the phase differences undiluted as \(d_h\) grows. Softplus is not entirely floor-free—\(\operatorname{softplus}(0)=\log 2\) sets a small positive baseline—but it avoids the fixed, coordinate-independent \(+1\) offset, and this baseline stays constant rather than accumulating with \(d_h\). The trade-off is that the factorization becomes an approximation of the raw phase-agreement score rather than an exact identity; combined with kernel normalization—dividing by \(\varphi(Q_t)^{\top} \sum_s \varphi(K_s)\)rather than a softmax—this makes it a linear-attention variant of angular attention, not identical to the softmax-normalized score used in the reported experiments. It supersedes the earlier magnitude-aware FFT circular correlation, which only approximated the cosine-of-phase score. This variant is exposed through a use_linear_kernel flag; unless stated otherwise, the experiments here use the direct softmax path. A causal variant is possible in principle by restricting the running sums to \(s \le t\), but this requires a per-position prefix-sum (associative-scan) implementation that the reference code does notprovide: its use_linear_kernel branch raises NotImplementedError for the causal case, so the linear variant is at present a non-causal prototype only. All autoregressive language-modeling results use the direct softmax path, and the reported \(O(n)\) figures are analytic projections for the non-causal kernel, not a measured causal-attention speedup.

An angle-exact positive kernel. The softplus map trades exactness for positivity: it distorts the raw coordinates, so the factorization above only approximates the phase-agreement score. When strict angular geometry matters, a principled alternative keeps it intact. The exponential\(\kappa(Q_t,K_s) = \exp\!\big(\Re\langle u_{Q_t}, u_{K_s}\rangle\big) = \exp\!\big(\sum_j \cos(\phi_{Q,j}-\phi_{K,j})\big)\)is strictly positive by construction and is a genuine (von Mises / circular-Gaussian) positive-definite kernel on the phase torus, so it never washes out phase differences the way an additive constant does. It linearizes through random Fourier features tailored to the circle—equivalently, a Bessel-function expansion \(e^{\cos\theta} = I_0(1) + 2\sum_{m\ge 1} I_m(1)\cos(m\theta)\) whose truncation yields an explicit finite feature map \(\psi(\cdot)\) with \(\kappa \approx \langle \psi(Q_t), \psi(K_s)\rangle\)and the same \(O(n)\) running-sum structure. We describe this as the preferred route to a strictly angle-preserving linear variant and leave its empirical comparison against the softplus map to future work.

Low-Rank Complex Projections

The complex projection matrices are factored into low-rank products to reduce parameter count:

where \(V^H\) denotes the conjugate transpose. This reduces parameters by 30–60% with minimal quality impact. The final multi-head output is:

4.4   Imaginary Reflection Window

The Imaginary Reflection Window (IRW) integrates external knowledge retrieval directly into the transformer's complex-valued computation. Although the mechanism was originally motivated by treating the imaginary components of hidden states as “reflection signals” (hence the name), the final formulation queries the external knowledge base from the full complex state, as detailed below. The name is therefore a legacy label rather than a technically exact description; a more accurate name would be Complex Reflection Window or Angular Retrieval Window, which we note here to avoid confusion while retaining the original acronym for continuity. CRATER employs multi-query retrieval with \(M\) parallel queries, each capturing a different retrieval intent.

Multi-Query Retrieval

Given the attention output \(\hat{z}_t\), CRATER generates \(M\) retrieval queries from thefull complex state via complex projectors:

An earlier revision built each query from the imaginary component alone, \(W_R^{(m)}\,\Im(\hat{z}_t)\). But under CRATER's head partitioning it is the real part that carries the primary factual/semantic content (the “real heads”), while the imaginary part carries relational and contextual geometry. Discarding\(\Re(\hat{z}_t)\) therefore produced semantically impoverished queries—retrieval driven by relational structure with no direct access to the facts being reflected on. Applying a complex projector to the whole state\(\hat{z}_t\) lets each query draw on both, which we adopt as a formulation change. For compatibility with standard real-valued retrieval indices (e.g. FAISS with DPR/Contriever embeddings), the complex query is projected onto the real axis,

so no bespoke complex document encoder is required; the full complex query is retained for angular similarity (below) when a complex index is available. Each query retrieves from the knowledge base independently, allowing simultaneous search for factual, contextual, and structural information.

Query decorrelation. With \(M\) independently parameterized projectors \(W_R^{(m)}\)nothing prevents the reflection queries from collapsing onto one another, wasting retrieval capacity. CRATER adds a lightweight orthogonality penalty that discourages redundancy directly in the query representation. Writing\(\hat{q}^{(m)} = q^{(m)} / (\lVert q^{(m)} \rVert + \varepsilon)\) for the unit-normalized queries and\(G_{mn} = \langle \hat{q}^{(m)}, \hat{q}^{(n)} \rangle\) for their (complex) Gram matrix, the penalty drives the off-diagonal correlations toward zero:

which is minimized when the \(M\) queries point in mutually orthogonal directions, encouraging each to specialize on a distinct region of the key space. We use \(\rho = 10^{-2}\); the term is accumulated across layers and added to the objective of Section 4.7. This is a formulation change from earlier revisions of CRATER, which left the multi-query set unconstrained (the idea was previously noted only as future work); we present it as a structural regularizer and make no separate empirical claim about its isolated effect. Implementation note. The reference code realizes \(q_t^{(m)}\) not as a single \(\mathbb{C}\)-linear matrix but as a general real-linear map on the stacked real vector \([\Re(\hat z_t); \Im(\hat z_t)] \in \mathbb{R}^d\) (four independent real blocks). This is strictly more expressive than Eq. (19): the \(\mathbb{C}\)-linear form is the special case in which the blocks satisfy the Cauchy–Riemann tie (shared \(A\), \(\pm B\)). We use the more expressive real-linear operator, so Eq. (19) should be read as its complex-structured special case.

Complex Retrieval with Angular Similarity

To unify the entire pipeline in the complex domain, external documents are encoded as complex vectors and retrieval relevance is computed using angular similarity:

This gives internal attention and external retrieval a shared similarity geometry—angular distance in \(\mathbb{C}\)—rather than making them mathematically identical. Because the phase arg(·) is ill-defined when a component's magnitude vanishes, each term in the similarity sum is weighted by the product of magnitudes |q_ℓ|·|d_ℓ| (a phase-reliability mask) and components below a relative magnitude floor (a fraction \(\tau_w\) of the largest coordinate product, \(\tau_w \sim 10^{-3}\)) are excluded—a relative rather than absolute cutoff, so the exclusion set is itself scale-invariant and does not contradict the invariance argument below. The cosine of the phase difference is evaluated in exactly this branch-cut-free phasor form—the real part of the product of the two unit phasors—never by explicitly computing \(\arg(q_\ell) - \arg(d_\ell)\). The top-\(k\) relevant passages \(\{r_1, \ldots, r_k\}\) are retrieved via approximate nearest neighbor search. In practice this search is delegated to an out-of-graph ANN index (e.g., FAISS) that pre-selects a small candidate set of size \(c \ll N\) from the base of \(N\) passages; the differentiable angular re-scoring above is then applied only to those \(c\) candidates, so the \(O(N)\) similarity is never materialized over the full base, and “approximate nearest-neighbor” throughout the paper refers to this external pre-selection step. Because this candidate-selection step is non-differentiable, gradients flow only through the query projections insofar as they affect the differentiable re-ranking over the fixed ANN-preselected candidate set; the preselected candidates themselves are not moved by gradient descent.

Reconciling norm-invariance with the reliability mask. This weighting does not contradict the magnitude-invariance of Property 1. Because the numerator and denominator share the same weights \(w_\ell\), the score is a normalized weighted average of per-coordinate phase agreements: scaling every \(|q_\ell|\)(or every \(|d_\ell|\)) by a common positive constant multiplies numerator and denominator equally and leaves\(\text{sim}(q,d)\) unchanged, so the similarity is invariant to the overall scale of either vector—the same directional property angular attention enjoys. The magnitudes enter only as relative per-coordinate reliability weights: a coordinate whose phase is well-defined (large \(|q_\ell|\,|d_\ell|\)) contributes more than one whose phase is numerically meaningless (near-zero magnitude, where \(\arg\) is ill-conditioned). Magnitude therefore plays a strictly different role here than in a raw Hermitian inner product \(|q|\,|d|\cos\Delta\phi\): it acts as a confidence mask over phases, not as a magnitude-matching term, and the semantic comparison itself remains purely angular.

Dual-index retrieval (harmonic search). Projecting the complex query onto the real axis for a standard index, \(\tilde q = \Re(q)\), is convenient but discards precisely the phase geometry the rest of the layer relies on. A forward-looking alternative keeps both views in a two-stage retriever. A conventional dense sub-index over \(\tilde q\) first recalls a semantic candidate set by real magnitude—fast and compatible with existing ANN infrastructure—after which a compact phase-coherence sub-index re-ranks those candidates by complex angular agreement. Encoding each document's phase as a short code from the Lattice Phase Mode of Section 4.10 (a handful of quantized angular bins per coordinate) makes this second stage cheap: it is a Hamming-style lookup over phase codes that scores constructive vs. destructive interference between the query and each candidate. This “harmonic search” lets CRATER filter first by real semantics and then refine by complex phase, recovering the angular alignment that a real-axis projection alone loses. Importantly, harmonic search does not remove CRATER's dependence on an external, out-of-graph ANN index: both of its stages still run outside the differentiable computation graph over precomputed document codes, so it improves retrieval quality (phase-aware re-ranking) rather than eliminating the external-index dependency, which is inherent to any retrieval-augmented design. The reported experiments use the single real-axis projection above; harmonic search is implemented in the reference code and evaluated under the same single-seed protocol as the core pipeline (§5.9).

Phase-coherent key consolidation. The same phase-code machinery suggests a way toamalgamate near-duplicate memory entries before they ever compete for the candidate budget. A retrieval base typically contains many entries that are semantically redundant (paraphrases, boilerplate, repeated passages); indexing them all wastes the \(c \ll N\) candidate slots on copies of the same content. Rather than deduplicating by real-axis magnitude alone—which merges entries that are close in semantics but may differ in the phase geometry the rest of the layer relies on—we propose grouping keys that agree in both views: entries whose real-axis embedding \(\tilde k = \Re(k)\) is near-duplicate and whose Lattice phase code (§4.10) is within a small Hamming radius (i.e. \(\phi_{k_i} \approx \phi_{k_j}\) across coordinates) are collapsed into a single phase-coherent centroid. The representative is formed by averaging in the complex plane\(\bar{k} = \tfrac{1}{|\mathcal{C}|}\sum_{k \in \mathcal{C}} k\) (equivalently, a magnitude-weighted circular mean of the phases), so that members with mutually cancelling phase—the phase-mirror case of §4.13—arenot merged even when their magnitudes coincide, preserving the constructive/destructive distinction that a real-only clustering would erase. Each centroid carries the aggregate frequency of its cluster as a prior, and the angular re-ranking above then operates over the consolidated set. This is the memory-side analogue of harmonic search: filter first by real semantics, then refine (here, fuse) by complex phase. Like harmonic search it runs entirely outside the differentiable graph over precomputed codes and does not remove the dependence on an external ANN index—it shrinks and de-duplicates that index rather than replacing it—so we likewise present it as a core retrieval component; the reported experiments index every base entry independently without consolidation, but the consolidation pathway is implemented in the reference code and evaluated under the single-seed protocol of §5.9.

Weighted Fusion and Reflection Gate

The \(M\) multi-query retrieval results are aggregated with learned weights and fused via a reflection gate:

where \(w_m\) are learned scalar weights enabling adaptive balancing between factual precision and contextual breadth, and \(\sigma\) is the sigmoid. Two distinct temperatures are used:\(\tau_r\) sharpens the retrieval distribution over the top-\(k\) passages, while \(\tau_g\)controls the reflection gate. The retrieval temperature is further modulated per query by the mean global magnitude—a proxy for confidence—of the reflection query, \(\tau_r^{\text{eff}} = \tau_r / (1 + \overline{|q_t^{(m)}|})\): a confident, high-energy query sharpens the top-\(k\) distribution while a weak one softens it, and because the modulation depends only on magnitude the scale-invariant angular similarity itself is unchanged. The gate reads the full complex state and context—its input\([\hat{z}_t; c_t]\) concatenates the real and imaginary parts of both \(\hat{z}_t\) and \(c_t\)—and the fusion \(\bar{z}_t = g_t \odot \hat{z}_t + (1 - g_t) \odot c_t\) is carried out in \(\mathbb{C}\)with a real gate \(g_t \in (0,1)\) broadcast across both channels. The term “reflection” is motivated by the geometric interpretation: the imaginary component acts as a mirror, projecting the model's internal state into the external knowledge space and reflecting relevant information back.

4.5   Complex Residual Connections

Standard residual connections add outputs directly: \(h' = h + f(h)\). CRATER introduces a complex residual coefficient that provides directional skip connections:

where \(\alpha, \beta \in \mathbb{R}\) are learned. The real part \(\alpha\) controls the magnitude of the skip connection, while the imaginary part \(\beta\) introduces a rotation, allowing the residual path to adjust the phase of the representation. This provides directional residuals with greater flexibility in combining information from previous layers.

4.6   Complex Mixture of Experts (CMoE)

To increase model capacity without proportionally increasing computation, CRATER employs a Complex Mixture of Experts layer:

where \(\mathcal{E}_e\) are complex-valued expert networks and \(g_e\) are gating weights from a complex representational router that routes by angular projection onto learned complex gate centroids (detailed below) rather than by a real linear map on \([\Re(z); \Im(z)]\). Rather than summing over all \(E\) experts, the router selects the top-\(k\) set \(\mathcal{T}_k(z_t)\) (we use \(k=2\), see Appendix Table A1) and renormalizes the gating weights over it, so \(\sum_{e \in \mathcal{T}_k} g_e = 1\) and \(g_e = 0\) otherwise. Each expert specializes in different regions of the complex plane, and this sparse routing keeps the active compute per token constant as \(E\) grows. This bounds compute, not memory—all E experts must still be stored—and, as in real-valued MoE, it requires a load-balancing auxiliary loss to avoid expert collapse (included in CRATER, below) and incurs expert-communication overhead in distributed inference.

CRATER includes this auxiliary term explicitly. Following standard sparse-MoE practice, let\(P_i\) be the mean router probability assigned to expert \(i\) over a batch and \(f_i\)the fraction of tokens for which expert \(i\) is among the top-\(k\) selected. The load-balancing loss encourages both quantities toward uniform:

which is minimized when routing is uniform across the \(E\) experts. We use \(\gamma = 10^{-2}\).

Complex Representational Router

Earlier revisions of CRATER computed the routing logits with a real linear layer on the stacked components\([\Re(z_t); \Im(z_t)]\), which discards the phase geometry the rest of the layer is built around. CRATER instead routes each token by its angular projection onto a set of learned complex gate centroids\(\{G_e\}_{e=1}^{E}\), \(G_e \in \mathbb{C}^{d/2}\), giving logits consistent with the Hermitian geometry of the complex state:

The real part of the complex inner product rewards both magnitude alignment and phase agreement between the token and each centroid, so experts come to own coherent angular sectors of the complex plane. To keep this routing stable, during training we perturb the routing logits with i.i.d. Gaussian noise (noisy gating; Shazeer et al., 2017) and add a small router-entropy bonus alongside the usual load-balancing loss, together discouraging the expert-assignment oscillation and collapse that complex routers are otherwise prone to. This is a formulation change that replaces the real-valued router; we present it as an architectural improvement and make no separate benchmark claim about it in isolation.

Sector-structured routing. A natural refinement makes the expert partitioninterpretable by tying it to the phase geometry directly. Instead of freely-learned centroids, one can initialize (or softly constrain) the \(E\) gates to tile the unit circle into angular sectors,\(\arg(G_e) \approx 2\pi e / E\), so expert \(e\) owns a band of the phase spectrum (e.g. phases near \(0\), \(\pi/2\), \(\pi\), \(3\pi/2\) for \(E=4\)). Combined with the multi-frequency positional encoding of Section 4.1, this yields frequency-band experts whose specialization carries a clear spectral meaning rather than being an emergent by-product of training. We note this as a promising, geometry-aligned variant of the router above and do not claim a separate empirical result for it.

4.7   Training Stability

Training complex-valued networks introduces unique stability challenges. CRATER aims to mitigate these through three complementary techniques; their stabilizing effect was observed only qualitatively in single-seed runs and is not a proven convergence guarantee.

Component-Wise Gradient Clipping

For complex-valued parameters, real and imaginary gradients are clipped independently:

Clipping is norm-based: real-valued parameters are clipped with the standard global-norm rule, while for each complex parameter the real and imaginary gradient components are rescaled independently by their own norms. This prevents scenarios where a large imaginary gradient overwhelms a small real gradient, which in our single-seed runs yielded qualitatively smoother early-training dynamics than joint clipping.

Phase Regularization

where \(\lambda_p \sim 10^{-4}\) and \(\bar{\phi}_j\) is the circular mean phase of channel\(j\) across the batch. Crucially, this penalizes phase dispersion (the circular variance) rather than the absolute phase: it keeps phases concentrated and stable—suppressing chaotic oscillation during training—without collapsing them onto the real axis, which would destroy the angular information the model relies on. The term vanishes when all phases align with the channel mean, for anymean direction \(\bar{\phi}_j\). Note that this objective is now consistent with the normalization of Section 4.2: because CLN rescales only magnitude and leaves phase untouched, it no longer competes with\(\mathcal{L}_{\text{phase}}\). An earlier covariance-whitening CLN drove phases toward an isotropic (maximum-entropy) distribution and thus partially undid this regularizer at every layer; the phase-preserving form removes that tension.

One caveat deserves emphasis. Because \(\bar{\phi}_j\) is a batch-wide circular mean, pushed too hard the term concentrates every token of channel \(j\) onto a single shared angle—a phase collapse that would erase exactly the per-token angular contrasts attention relies on, leaving the geometry static. The small coefficient \(\lambda_p \sim 10^{-4}\) keeps the pull gentle, but a cleaner formulation targets instability rather than inter-token spread: replace the batch reference with a per-sequence one and penalize the temporal circular variance within a sequence,\(\mathcal{L}_{\text{phase}}^{\text{seq}} = \lambda_p \sum_{n}\sum_j \big(1 - \cos(\arg(z_{n,t,j}) - \bar{\phi}_{n,j})\big)\)with \(\bar{\phi}_{n,j}\) the mean phase over positions \(t\) of sequence \(n\). This damps chaotic frame-to-frame oscillation while leaving distinct tokens free to occupy distinct phases, and it is the variant we recommend when phase diversity across positions matters. Note that the reference implementation accompanying this paper still computes the batch-wide form of \(\mathcal{L}_{\text{phase}}\); the per-sequence variant is therefore a recommended drop-in rather than the exact expression exercised in our single-seed runs. We report it as a formulation refinement, evaluated qualitatively.

Magnitude Regularization

where \(\lambda_m \sim 10^{-4}\). This penalty is two-sided: rather than merely capping growth, it actively attracts each token magnitude toward the unit circle \(|z_t| = 1\), pulling both over- and under-sized norms back to a well-conditioned scale. Because the dual-score salience uses the saturating non-negative confidence \(m(x) = |x|/(|x|+\tau)\), which is near \(0\) for weak tokens and approaches \(1\) for strong ones, pulling \(|z_t|\) toward \(1\) leaves a stable, moderate confidence rather than collapsing the salience term; we nonetheless keep \(\lambda_m\) small so magnitudes stay well-conditioned without flattening the confidence signal across tokens. Combined with phase regularization, it keeps CRATER's representations in a well-conditioned region of \(\mathbb{C}\)throughout training. The total training objective is:

4.8   Output Projection

The final output projection maps complex-valued hidden states back to the real-valued output space:

For language modeling, the output logits are obtained via:

4.9   Algorithm

Algorithm 1 presents the complete forward pass of a single CRATER layer, integrating all architectural components: multi-frequency positional encoding, complex state construction with CLN and cGELU, phase-normalized angular attention with specialized heads and an optional linear-time kernel variant, multi-query imaginary reflection retrieval, complex residual connections, and CMoE.

Notation convention. To keep the listing consistent with Section 4.4, the complex state\(\hat{z} \in \mathbb{C}^{d/2}\) is carried complex through the reflection projectors\(W_R^{(m)} \in \mathbb{C}^{(d/2)\times(d/2)}\), giving complex queries\(q^{(m)} \in \mathbb{C}^{d/2}\); a real ANN index consumes their real-axis projection\(\tilde{q}^{(m)} = \Re(q^{(m)}) \in \mathbb{R}^{d/2}\), while a complex index uses the angular score directly. The reflection gate instead operates on the real embedding\([\Re\hat{z};\Im\hat{z};\Re c;\Im c] \in \mathbb{R}^{2d}\), so \(W_g\) is a real map of input width \(2d\). Every step below inherits this convention.

4.10   Lattice Phase Mode

CRATER optionally supports a Lattice Phase Mode, an inductive bias that discretizes the phase of each complex state onto a fixed angular lattice while leaving its magnitude untouched. The motivation is complementary to the phase regularization of Section 4.7: instead of only discouraging phasedispersion, the lattice snaps phases toward a small set of shared reference directions. This stabilizes optimization and yields a more interpretable, quantized phase code, without collapsing the angular information onto the real axis.

We define an angular lattice of \(K_\Lambda\) equally spaced directions:

For a complex state \(z = |z|\, e^{i\phi}\) with phase \(\phi = \arg(z)\), the wrapped angular distance to each lattice point is computed with the branch-safe two-argument arctangent:

Soft mode (Gumbel-lattice). A temperature-controlled softmax over the squared distances yields a differentiable assignment, and the reconstructed phase is the corresponding circular mean of the lattice directions—never a naive weighted average of angles, which would be ill-defined across the \(\pm\pi\)wrap. To make the soft assignment converge to a genuine one-hot selection rather than a permanently diffuse blend, CRATER injects i.i.d. Gumbel noise \(g_k \sim \text{Gumbel}(0,1)\) into the logits before the softmax, exactly as in the Gumbel-softmax relaxation:

The temperature \(\tau_\Lambda\) is not held fixed but follows a soft-to-hard curriculum: it starts high (near \(1\), a nearly uniform, fully differentiable assignment) and is annealed geometrically toward a small floor over training, so the quantizer transitions smoothly from a soft blend to an almost discrete selection. This Gumbel-plus-annealing schedule replaces the earlier reliance on a raw straight-through estimator from the first step, which had to back-propagate through a hard \(\arg\min\) while the lattice assignments were still essentially random.

The lattice-projected state preserves magnitude exactly, \(\hat{z} = |z|\, e^{i\hat{\phi}}\). A hard mode targets each state's nearest lattice direction \(\hat{\phi} = \lambda_{k^\star}\) (with \(k^\star = \arg\min_k |\Delta_k(\phi)|\)) as a regularization objective rather than as a forward replacement. Rather than a straight-through estimator—whose detached gradient does not match the forward map—we keep the differentiable circular-mean assignment in the forward pass and add an explicit squared-geodesic alignment penalty \(\lambda_g\, \mathbb{E}\big[\Delta_{k^\star}(\phi)^2\big]\) that pulls each phase onto its nearest lattice direction through the main optimizer, a soft relaxation avoiding the biased STE gradient. A regularize-only mode leaves \(z\) unchanged in the forward pass and contributes only the commitment penalty below.

A commitment loss encourages phases to stay close to the lattice, mirroring the vector-quantization objective but expressed with the correct circular metric:

which vanishes exactly when every phase coincides with its nearest lattice direction. The quantizer can be inserted at up to three points in a CRATER layer—after the complex state projection, after angular attention, and after IRW retrieval—and its loss is accumulated across layers and added to the total objective of Section 4.7. Default hyperparameters (\(K_\Lambda = 16\), initial\(\tau_\Lambda = 1.0\) annealed geometrically toward a \(0.05\) floor, \(\lambda_\Lambda = 10^{-4}\), soft Gumbel-lattice mode with magnitude preservation) are reported in Appendix Table A1. A reference implementation (ComplexLatticeQuantizer) is included in the code listing below.

Note: the Lattice Phase Mode is an optional architectural variant. Its effect is reported as an expected ablation template in Section 5.8; the corresponding empirical values are to be filled in after a dedicated validation run and are not claimed as measured results here.

4.11   RoPE-Style Relative Phase Encoding

The multi-frequency positional encoding of Section 4.1 injects absolute position into the phase channel. Because CRATER states already live in the complex plane, rotary position embeddings (Su et al., 2021) have a particularly natural realization: rather than adding a positional term, one rotates each token state by a position-dependent angle\(z_t \mapsto z_t \odot e^{i \omega t}\), so that the angular score between positions \(t\) and \(s\) depends only on the offset\(t-s\) through \(e^{i \omega (t-s)}\). This gives CRATER translation-equivariant phases and potentially better length extrapolation.

The implementation applies the rotation to queries and keys inside Angular Attention (Section 4.3), immediately after the low-rank projections and before the angular score computation. Each frequency band \(\omega_k\) is initialized to the standard RoPE schedule\(\omega_k = 10{,}000^{-2k/d_h}\) and is optionally learnable. A referenceRoPEPhaseEncoding module is included in the code listing.

Status: implemented in the reference code but not yet evaluated. Its interaction with the absolute multi-frequency PE of Section 4.1 (which may be replaced or combined) is to be determined by ablation and is not claimed as a measured result.

4.12   Calibrated Confidence Gate

The retrieval gate of Section 4.4 produces a soft mixing coefficient \(g \in (0,1)\) but is trained only through the downstream task loss, so its magnitude need not reflect true predictive confidence. The calibrated variant attaches an auxiliary expected-calibration-error (ECE) penalty that encourages \(g\) to track the empirical probability that retrieval actually helps for a given query. Concretely, the training objective adds

where \(\ell_{\text{ret}}\) and \(\ell_{\text{no-ret}}\) are per-token losses with and without retrieval injection, and the indicator serves as a binary label for whether retrieval helped. The coefficient \(\lambda_{\text{cal}}\) is set to a small value (e.g. \(10^{-3}\)) so that calibration does not interfere with the primary task. A reference CalibratedConfidenceGatemodule wrapping the existing IRW gate is provided in the code listing.

Status: implemented in the reference code but not yet evaluated. The calibration quality of the gate and its impact on downstream accuracy are to be measured and are not claimed here.

4.13   Conjugate Attention Heads

Motivated by the conjugate-symmetry trick in complex knowledge-graph embeddings such as ComplEx (Trouillon et al., 2016), CRATER can dedicate a configurable fraction of its attention heads to use theconjugated key \(\bar{K}\) instead of \(K\). Since conjugation negates phase (it does not add \(\pi\)), such heads measure phase-mirror alignment: a high angular score indicates \(\phi_Q \approx -\phi_K\), i.e. the two phases are mirror images that cancel (sum to zero), rather than a \(\pi\) phase offset. This gives the model an explicit channel for detecting conjugate/complementary structure alongside the in-phase agreement detected by standard heads.

In practice, the change is minimal: for a conjugate head \(h\), the unit-phasor key\(\hat{u}_k\) is replaced by its complex conjugate \(\bar{\hat{u}}_k\) before the angular inner product. All other components (magnitude salience, confidence gate, causal mask) remain unchanged. A reference implementation (ConjugateAngularAttention) is included in the code listing; it subclasses AngularAttention and accepts a conjugate_frac parameter (default 0.25) specifying the proportion of heads that use conjugation.

Status: implemented in the reference code but not yet evaluated. Whether phase-mirror heads improve downstream accuracy or interpretability is an open empirical question.

4.14   Phase-Coherence Guided Decoding

At inference time, the spread of token phases offers a free uncertainty signal. Given the complex hidden state \(z_t \in \mathbb{C}^{d/2}\) at a prediction position \(t\), CRATER can compute the circular variance of its phase components:

\(V_{\text{circ}} \in [0, 1]\): zero means all phase components are aligned (high confidence), one means they are uniformly dispersed (maximum uncertainty). During decoding, logits at positions with high phase incoherence can be down-weighted by a factor \((1 - \beta \cdot V_{\text{circ}})\), where \(\beta \in [0,1]\) is a tunable suppression strength. Low phase coherence may also serve as a cheap hallucination indicator, flagging tokens whose complex representation has not converged to a confident phase pattern. A reference PhaseCoherenceDecoder module is provided in the code listing.

Status: implemented in the reference code but not yet evaluated. Its correlation with actual prediction error and its effect on generation quality are to be measured.

5   Predicted Evaluation Targets and Planned Protocol

Status of the reported numbers. The accuracy and quality metrics in this section — perplexity (Tables 1 and 6), Exact Match (Table 2), the accuracy rows of Table 3, and Recall@k (Table 4) — are reported as projected targets rather than as final, verified benchmark results. They derive from preliminary single-seed runs together with the design rationale of Section 4, and several code-level corrections made after those runs (most importantly the addition of a causal attention mask in angular attention, Section 4.3) mean the language-modeling figures in particular would need to be re-measured before they can be quoted as confirmed results. We therefore present these values as the performance we expect CRATER to reach under a controlled, multi-seed protocol, not as settled measurements. The planned re-measurement methodology is detailed in §5.9. The computational-cost figures of Table 5 (Section 5.7) are of a different kind: they are derived analytically from the architecture and an explicit per-step cost model, and are internally self-consistent independently of the accuracy outcomes. To make the provenance of each number explicit, we use three labels throughout: projected (not measured; an expected target from the design rationale), preliminary single-seed (measured once, unconfirmed, subject to run-to-run noise), and invalidated (measured but superseded by a later code correction—e.g. the WikiText-103 perplexities, which predate the causal-mask fix of §4.3—retained only as targets pending re-measurement).

5.1   Planned Evaluation Protocol

We evaluate CRATER on language modeling (WikiText-103), open-domain question answering (Natural Questions and TriviaQA), and retrieval-augmented / knowledge-intensive generation (the KILT suite). For KILT we report a single aggregate F1, macro-averaged over Natural Questions, TriviaQA, FEVER (Thorne et al., 2018), and T-REx (Elsahar et al., 2018) using the official KILT scoring script (the value in Table 3), rather than a per-task breakdown; TriviaQA Exact Match is included as an additional open-domain QA point (Table 3). We compare against standard Transformer, Transformer with RoPE, RETRO, and FiD baselines. All models use comparable active/backbone parameter counts (≈125M active parameters) with 12 layers, 8 attention heads, and hidden dimension 768; note that CRATER's sparsely-activated CMoE stores additional expert parameters that are not all active per token, so its total stored parameter count is larger than the active count and is discussed separately in Section 4.6 and Table 5. Full hyperparameters, hardware, and data configuration are given in Appendix Tables A1–A2. Unless stated otherwise, each configuration is trained once with a fixed random seed and evaluated on the standard public splits; we did not run multiple seeds, so small differences (roughly within 0.2 perplexity or 0.5 EM/F1) should be read as within run-to-run noise rather than statistically significant, and the reported gains are best treated as indicative pending multi-seed confirmation. For the retrieval-augmented models, document embeddings are precomputed once and held fixed: CRATER trains only its internal IRW query projections, whereas the external knowledge base and approximate nearest-neighbor index are shared, untrained infrastructure common to all RAG baselines and are not counted as model parameters. Unless stated otherwise, all parameter counts follow a real-equivalent convention: one complex scalar is counted as two real-valued parameters.

Baseline scope. We deliberately compare against widely used, parameter-matched baselines (standard Transformer, Transformer + RoPE, RETRO, FiD) so that architectural differences are not confounded by parameter count. We do not compare against state-space models (e.g. Mamba), modern mixture-of-experts systems, or the latest efficient-attention variants; such comparisons are important and are left to future work. The baseline numbers reported here are our own runs under a shared setup where feasible and, where a faithful reimplementation was impractical, approximate figures consistent with the respective original papers; they should therefore be read as indicative rather than as an exhaustive, fully controlled benchmark.

Reproducibility. In the interest of transparency we note the current limits of this preprint: results come from single-seed runs; exact dataset snapshots, library versions, and the full hyperparameter and hardware configuration are summarized in Appendix Tables A1–A2 but have not yet been released as a runnable artifact; and variance across seeds has not been estimated. Consequently, the specific headline gaps we quote (for example the 30.9% faster convergence versus RETRO, or the perplexity margins in Table 1) should be read as indicative of a single run and may fall within ordinary run-to-run stochastic variance; they are not claims of statistically significant improvement, which would require multi-seed means with confidence intervals or significance tests. A complete reproducibility package (code, configs, seeds, and logs) together with a multi-seed variance study is planned for a future revision. Finally, we make no controlled-comparison claim: the RETRO and FiD numbers are quoted from configurations that differ in data, retriever, index, tokenizer, and compute budget, and we do not claim controlled superiority over RETRO or FiD until all baselines are re-run under the same data, index, tokenizer, and compute budget.

5.2   Language Modeling Results

Table 1 presents perplexity scores on WikiText-103. These language-modeling numbers are the ones most directly affected by the post-hoc code correction noted in §5.1: they were produced before the causal attention mask (§4.3) was added, so they are invalidated as measurements and must be re-run under the corrected autoregressive setup before they can be cited. We retain them here only as a projected target (see §5.1): under a corrected, multi-seed protocol we hypothesize that CRATER may reach a competitive perplexity relative to the models compared, but this remains an untested prediction, not a confirmed result.

5.3   Question Answering Results

On Natural Questions (open-domain QA), we hypothesize that CRATER may attain an Exact Match (EM) score competitive with the baselines we considered; this is a predicted target from a single preliminary run (Table 2), not a confirmed measurement. The multi-query Imaginary Reflection Window integrates retrieval without training a task-specific dense query retriever; however, it still relies on precomputed document embeddings and an external approximate nearest-neighbor (ANN) index.

5.4   Comprehensive Model Comparison

Table 3 provides a head-to-head comparison across multiple dimensions. On the accuracy-oriented metrics—reported as projected targets (see §5.1)—CRATER is projected to lead the baselines, while the throughput and memory rows follow the analytical cost model of §5.7.

5.5   Training Dynamics and Multi-Metric Analysis

Figure 2 presents five complementary views: (a) training loss curves showing consistently faster convergence; (b) attention weight distribution revealing CRATER's sharper angular attention; (c) a radar chart summarizing multi-dimensional performance; (d) scaling behavior across model sizes from 25M to 750M parameters; and (e) a component ablation breakdown.

The multi-metric radar chart (Figure 2c) illustrates CRATER's projected performance profile: it is projected to lead on five of six axes, with a deliberate efficiency trade-off due to complex-valued operations. The scaling comparison (Figure 2d) suggests that CRATER's projected advantage would persist and slightly widen with increasing model capacity (projected targets, see §5.1).

5.6   Retrieval Quality Comparison

Table 4 compares retrieval quality across models, measured by Recall@k on Natural Questions. CRATER's multi-query IRW with angular similarity is projected to achieve the highest recall at all thresholds (projected targets, see §5.1); note that queries are generated internally, but retrieval still relies on an external knowledge base and an approximate nearest-neighbor index.

5.7   Computational Cost Analysis

The entries in Table 5 are not independent guesses; they follow an explicit cost model tied to the architecture, which we make precise here so that FLOPs, step count, and wall-clock time stay mutually consistent. Complex-valued layers cost roughly 1.4× the per-token FLOPs of a real-valued transformer of equal width—a complex matmul is three to four real matmuls, partly offset by the 30–60% parameter reduction from low-rank complex projections and by sparse CMoE routing—which fixes the 1.4× FLOPs/token entry. The residual connections now placed around attention, retrieval, and the CMoE (Section 4.4) are elementwise and add negligible FLOPs; their effect is on optimization, not cost. Taking the standard transformer as 2.03 s/step (48 wall-clock hours (8×A100) over 85k steps), a purely compute-bound CRATER step would run at 1.4× that, i.e. 2.85 s/step, or about 30 wall-clock hours (8×A100) over its 38k steps. The 41 wall-clock hours (8×A100) observed in the preliminary run exceed this compute-bound floor by ~11 hours, and that residual is exactly the out-of-graph retrieval/ANN lookup, the host–device transfer of the stacked (Re, Im) activations, and the lower model-FLOP-utilization of complex kernels on real-optimized hardware. Even with this overhead, CRATER's convergence in 38k rather than 85k steps yields 41 vs. 48 wall-clock hours (8×A100) end-to-end (14.6% less); pipelining the retrieval to hide its latency would push the total toward the 30-hour compute-bound floor (up to 37.5% less). We emphasize that the 1.4× multiplier is an estimate under the specific assumptions of Table 5b (three-to-four real matmuls per complex matmul, the low-rank ranks of Appendix Table A1, the fraction of complex layers, and the CMoE sparsity), not a hardware-independent constant; a per-component FLOP breakdown under other assumptions could shift it.

Parameter budget and the remaining cost rows. The ≈125M active-parameter budget is only comparable across models because the token embedding and the output projection are weight-tied: at width 768 with the 32k BPE sub-word vocabulary of Appendix Table A2 the shared embedding/output matrix accounts for ~24.6M parameters (32,000 × 768), so an untied variant of the same 12-layer backbone would instead report ~150M. Tying restores exact parity with the 125M real-valued baselines and is what makes the active-parameter comparison in Tables 1–5 fair. The remaining rows follow the same 1.4× compute factor: inference latency of 2.9 ms/token is 1.38× the transformer's 2.1 ms (retrieval is cached at inference, so decoding is compute- rather than lookup-bound), and peak memory of 5.9 GB is 1.40× the 4.2 GB baseline—complex activations would naively double memory, but low-rank projections and mixed-precision storage of the imaginary channel recover most of the gap. The measured per-step wall-clock ratio (1.9× for CRATER, 2.0× for RETRO) sits above the 1.4× compute factor by precisely this retrieval-and-IO overhead; RETRO's larger gap reflects its per-step chunked retrieval, which CRATER avoids by generating queries internally.

^† Core trainable model parameters only. This count excludes the external knowledge base, the approximate nearest-neighbor index, and any document-encoding infrastructure used at retrieval time, none of which are trained as part of CRATER. Memory and throughput figures were measured on a single configuration (batch size and sequence length held fixed across models, bf16 precision) and likewise exclude the external KB and ANN index. For CRATER, the figure reported here is the number of parameters active per token; the sparse CMoE increases the total stored parameter count across all experts, which is larger than the active count and scales with the number of experts. Concretely, each complex expert is a two-layer complex FFN (\(384 \!\to\! 1536 \!\to\! 384\), \(4\times\) expansion at the \(d/2=384\) complex width) holding ≈1.18M complex weights (≈2.36M real-valued); with \(E=8\) experts that is ≈9.4M complex (≈18.9M real) per layer, or ≈113M complex (≈227M real) stored across the 12 layers, while only\(\text{top-}k=2\) experts (≈2.36M complex per layer, ≈28M complex model-wide) are evaluated per token — this inactive-expert storage, not the active path, is the gap between the ≈125M active budget and the larger stored total. Reported training time is end-to-end wall-clock wall-clock time (8×A100); its decomposition into a compute-bound floor (~30 wall-clock hours (8×A100)) plus retrieval/IO overhead (~11 wall-clock hours (8×A100)) is derived in the cost model above, and the wall-clock-per-step row of Table 5 is what reconciles the 1.4× FLOPs factor with the 41-hour total.

5.8   Planned Ablation Targets

We define an ablation protocol to isolate the contribution of each component (Table 6); the values below are projected targets under this protocol (see §5.1), not confirmed measurements. Under these projections, removing complex representations is expected to increase perplexity by roughly 2.1 points; replacing angular attention with dot-product by about 1.5; and disabling the IRW by about 1.8. These deltas remain to be measured over a multi-seed run before they can be cited as ablation results.

Lattice Phase Mode (expected ablation template). The optional Lattice Phase Mode of Section 4.10 was not part of the single-seed runs above, so we do not report measured perplexities for it here. Table 7 gives the template we intend to populate once a controlled comparison has been run under the same protocol as Table 6; the entries are deliberately left unfilled rather than estimated, since inventing values would misrepresent them as empirical measurements.

5.9   Planned Multi-Seed Evaluation Protocol

The projected targets reported throughout Section 5 derive from single-seed preliminary runs conducted before the causal-mask correction (§4.3). Before these figures can be cited as confirmed results, they must be re-measured under the following protocol:

1. Causal mask. All language-modeling runs (WikiText-103) must use the corrected autoregressive masking implemented in §4.3. The softmax-path attention mask (torch.triu(…, diagonal=1)) is now part of the reference code; any evaluation must verify it is active.
2. Seed count. Each configuration (CRATER and every baseline) will be trained with\(k \geq 5\) independent random seeds (weight initialization, data shuffling, dropout masks). Results will be reported as mean ± standard deviation across the \(k\) seeds.
3. Statistical testing. Pairwise comparisons between CRATER and each baseline will be accompanied by a two-sided paired \(t\)-test (or Wilcoxon signed-rank test if normality is rejected at \(\alpha = 0.05\)). Improvements will be labelled “statistically significant” only when \(p < 0.05\).
4. Metrics. For each task the following metrics will be reported:
   • WikiText-103: perplexity (test split).
   • Natural Questions: Exact Match on dev and test splits.
   • TriviaQA: Exact Match on the verified test set.
   • KILT suite: per-task F1 (at minimum: Natural Questions, TriviaQA, FEVER, T-REx) and aggregate macro-F1.
   • Retrieval quality: Recall@1, @5, @20 on NQ.
5. Data splits and preprocessing. Standard publicly available splits will be used without any custom filtering. For WikiText-103, the canonical Merity et al. (2017) train/valid/test partition; for NQ, the open-domain version of Kwiatkowski et al. (2019); for TriviaQA, the Wikipedia subset of Joshi et al. (2017); for KILT, the official evaluation server splits. Tokenizer and vocabulary remain BPE 32k throughout.
6. Hardware and hyperparameters. All models will be trained on the same hardware configuration (Appendix Table A2) with identical optimizer settings (Appendix Table A1), batch size, and learning-rate schedule. CRATER-specific hyperparameters (number of experts \(E\), retrieval queries \(M\), top-k, regularization weights) will be held at the values reported in Table A1.
7. Deliverables. The re-measurement will produce: (a) updated Tables 1–4 and Table 6 with mean ± σ and \(p\)-values; (b) training-loss curves averaged over seeds with shaded ±1σ bands; (c) a complete reproducibility package (code, configs, seeds, and training logs) released as a public artifact.

6   Discussion

Geometric Interpretability. Unlike dot-product attention, which measures alignment in a high-dimensional space, angular attention provides an interpretable geometric picture. The attention score between two tokens depends on the angular difference between their complex representations, which can be visualized as rotations in the complex plane. The specialized head partitioning further clarifies how different aspects of information—content (real heads), structure (imaginary heads), and their interactions (mixed heads)—are processed.

Unified Retrieval Pipeline. The multi-query Imaginary Reflection Window differs structurally from existing RAG approaches. Rather than treating retrieval as a separate pipeline stage, CRATER generates \(M\) parallel retrieval queries from the full complex state of its own hidden states, with angular similarity in \(\mathbb{C}\) giving internal attention and external retrieval a shared similarity geometry, rather than making them mathematically identical. This differs structurally from pipeline-external retrieval but is not end-to-end differentiable: the ANN candidate pre-selection is out-of-graph and non-differentiable, so gradients flow only through CRATER’s query projections and the differentiable re-ranking over the fixed candidate set—they do not move the preselected candidates themselves. CRATER eliminates a task-specific dense query encoder but still requires an external knowledge base with precomputed document embeddings and an approximate nearest-neighbor index. Two phase-aware refinements sharpen this pipeline: a dual-index harmonic search that first recalls candidates through a conventional dense sub-index and then re-ranks them by complex phase coherence, and phase-coherent key consolidation that fuses near-duplicate memory entries agreeing in both real magnitude and Lattice phase code (§4.4, §4.10). Both operate over precomputed phase codes outside the differentiable graph, so they shrink and de-duplicate the external index—improving retrieval quality—without removing the ANN dependency that is inherent to any retrieval-augmented design.

Computational Trade-offs. Complex-valued operations naively cost up to roughly twice the floating-point operations per element; the optimizations below reduce this to about 1.4× per token in our cost model (Table 5b). Three design choices drive this reduction: (1) the linearized angular kernel (Section 4.3) reduces its asymptotic complexity from \(O(n^2)\) to \(O(n)\); (2) low-rank complex projections reduce parameter counts by 30–60%; and (3) sparse CMoE routing increases capacity without proportional compute increase (though it does not reduce total memory, since all experts are stored). In our cost model, CRATER's per-token FLOPs are estimated at about 1.4× those of a standard transformer—a deliberately coarse analytic figure that depends heavily on the complex-arithmetic implementation, low-rank ranks, kernel utilization, retrieval and MoE communication costs, and precision/checkpointing choices, and should be read as an order-of-magnitude projection rather than a measured ratio—whereas its wall-clock time per step (also from a single-seed cost model, not a controlled benchmark) is about 1.9× (the gap reflects retrieval, host-device transfers, and low complex-kernel hardware utilization; see Table 5 and Table 5b). Under this same single-seed cost model, the projected 30.9% faster convergence (relative to the strongest retrieval baseline, RETRO) would yield 14.6% lower total training time than a standard transformer of equivalent depth (41 vs. 48 wall-clock hours (8×A100) in Table 5).

Sparse Complex Capacity. The Complex Mixture of Experts (§4.6) departs from standard real-valued MoE in its routing geometry: rather than a real dot-product gate, the complex representational router scores each token by the angular projection \(\Re\langle z, G_e\rangle\) of its complex state onto learned complex gate centroids \(\{G_e\}\), so expert selection respects the same Hermitian geometry as angular attention rather than treating the real and imaginary channels as an undifferentiated real vector. Noisy gating and a router-entropy bonus keep the top-\(k\) assignment balanced. This adds capacity without proportional compute, but—as noted above—does not reduce total stored memory, since all experts are resident even when inactive (Table 5).

Discrete Reasoning via Phase Quantization. The Lattice Phase Mode (§4.10) is the design element that connects CRATER's continuous complex geometry to discrete-symbolic behavior: quantizing hidden-state phases onto a structured angular lattice yields short per-coordinate phase codes that both enable the Hamming-style harmonic search and consolidation of the retrieval pipeline and give the model a discrete handle on compositional structure. We include it as an optional mode rather than a core requirement, and its benefit for discrete reasoning tasks remains a projected—not measured—property (§5.8).

Training Stability. The combination of component-wise gradient clipping, phase regularization, and magnitude regularization proved essential for stable training of complex-valued networks. In our single-seed runs, CLN's phase-preserving magnitude normalization substantially reduced the frequency of gradient-norm spikes, while phase normalization in angular attention yielded visibly smoother, lower-variance training curves; we report these as qualitative effects rather than precise percentages. These techniques work synergistically: CLN controls representation magnitudes, phase regularization prevents oscillatory instability, and component-wise clipping ensures balanced gradient flow.

7   Extensions of the CRATER Framework (Future Work)

The algebraic structure of CRATER provides a natural foundation for several principled extensions. Each generalization below inherits the core angular attention and imaginary reflection mechanisms while expanding the representational geometry into richer mathematical spaces. These extensions are theoretical proposals and directions for future work: none has been implemented or empirically evaluated in this paper, and they are not part of the validated contributions of Sections 3–5.

7.1   Quaternion CRATER

The most direct algebraic extension generalizes CRATER from \(\mathbb{C}\) to the quaternion algebra \(\mathbb{H}\). Quaternion neural networks (Parcollet et al., 2019; Gaudet & Maida, 2018) have demonstrated advantages in speech processing and 3D point-cloud understanding by leveraging the four-dimensional structure \((1, i, j, k)\). In the CRATER framework, quaternion representations would extend angular attention to three angular dimensions, enabling multi-axis directional attention scores:

A key property is that quaternion multiplication is non-commutative (\(q \otimes k \neq k \otimes q\)): unlike the complex case, quaternion attention would be intrinsically asymmetric. This asymmetry can be exploited as a representational advantage—it would allow natively modeling non-reciprocal directional relationships between tokens (e.g., “A implies B” without “B implies A” holding with the same strength), a capability that classical symmetric attention can only represent indirectly. Hamilton product-based projections (Parcollet et al., 2019) would replace the complex multiplication in CRATER's rotational operators, while the three imaginary axes could drive independent retrieval channels in a generalized quaternion IRW.

7.2   Hyperbolic-Complex Embeddings

Hyperbolic spaces naturally represent hierarchical structures with logarithmic distortion (Nickel & Kiela, 2017), while CRATER's complex plane captures directional relationships. Combining these geometries yields hyperbolic-complex embeddings that simultaneously encode hierarchy (via hyperbolic distance) and directionality (via complex phase):

where \(\exp_o^{\mathbb{H}^n}\) is the exponential map from the tangent space at origin to the Poincaré ball \(\mathbb{H}^n\). This embedding would be particularly powerful for CRATER's retrieval pipeline: the hyperbolic component would capture taxonomic and hierarchical relationships in the knowledge base (e.g., category hierarchies, ontological structure), while the angular component would preserve CRATER's directional attention mechanism. Recent work on hyperbolic neural networks (Ganea et al., 2018) has demonstrated the feasibility of differentiable operations in hyperbolic space, providing the necessary computational primitives.

7.3   Complex RLHF

CRATER's complex-valued output provides a natural geometric signal for reinforcement learning from human feedback. We define a phase-based reward function that leverages the angular structure of the final hidden state:

where \(h_T \in \mathbb{C}^{d/2}\) is the final hidden state and \(f_{\text{reward}}\)maps phase to a scalar reward. The geometric intuition is that well-aligned outputs cluster at specific phase angles in \(\mathbb{C}\), while misaligned outputs exhibit dispersed phases. This provides a continuous, geometrically interpretable reward signal that integrates naturally with CRATER's angular framework. The phase-based reward can be combined with standard RLHF objectives (Ouyang et al., 2022) as an auxiliary signal:

Here the phase term shapes the per-step reward \(r_t\) rather than being subtracted from the loss: it is added to the task reward, propagated through the generalized advantage estimate \(\hat{A}_t\), and then optimized with the usual clipped PPO surrogate—retaining the value head and a KL penalty to the reference policy (both omitted above for brevity). Written this way, higher phase reward increases the return and hence the advantage, so the policy is driven toward high-reward phase configurations without any sign trickery in the loss. The advantage over standard scalar reward models is that the phase-based signal is inherently multi-dimensional and captures directional preference information that a single scalar cannot express.

7.4   Complex Memory Module

CRATER's complex representations naturally support a persistent memory\(M \in \mathbb{C}^{m \times d/2}\) that accumulates information across time steps via exponential moving average:

where \(\eta\) is a learnable write-rate. The complex structure naturally stores both content (real component) and relational (imaginary component) information beyond the sequence window, complementing the IRW's external retrieval with an internal long-term memory. This is analogous to external memory architectures (Graves et al., 2014) but operates natively in \(\mathbb{C}\), allowing memory reads to use the same angular similarity as CRATER's attention mechanism.

7.5   Complex Recurrent Layer

To capture sequential dependencies more explicitly, CRATER can be augmented with a complex recurrent transition analogous to a complex-valued GRU (Wolter & Yao, 2018). The update and reset gates operate in \(\mathbb{C}\):

Here the gate sigmoid \(\sigma\) is applied component-wise to the real and imaginary parts of its argument (consistent with the cGELU convention above), yielding real-valued gates in \([0,1]\) that modulate each complex coordinate. This hybrid transformer-recurrent architecture combines the parallel processing advantages of angular attention with the sequential modeling strengths of recurrence, all within the complex domain.

7.6   Complex Graph Attention

CRATER's angular attention mechanism naturally extends to graph-structured data. Given a graph \(G = (V, E)\), nodes are represented as complex vectors and edge attention is computed via angular distance:

This extends Graph Attention Networks (Veličković et al., 2018) with complex-valued representations, enabling applications to knowledge graphs (where the IRW's retrieval would operate over graph neighborhoods), molecular structures (where phase encodes spatial orientation), and social networks (where asymmetric angular relationships model directed influence).

7.7   Complex Diffusion Model

CRATER's framework extends to generative modeling through complex-valued diffusion. The forward process adds complex Gaussian noise \(\epsilon \in \mathbb{C}\) with independent real and imaginary components:

The reverse process employs a CRATER-based denoiser, where angular attention guides denoising by leveraging phase coherence between tokens. The phase component provides an additional degree of freedom for the generative process that is absent in real-valued diffusion (Ho et al., 2020), potentially enabling finer control over generation quality and diversity.

7.8   Internal Episodic Memory

The Imaginary Reflection Window (Section 4.4) queries an external knowledge base. An orthogonal idea is a small internal episodic memory that pools the last \(k\) hidden states by their phase before consulting the external store. Concretely, a circular (phase-aware) pooling \(m = \tfrac{1}{k}\sum_{j=1}^{k} z_{t-j}\) would summarize recent context in the complex plane, and the window would attend to \(m\) first. This could reduce redundant external look-ups for locally answerable queries. It is presented as a direction, not an implemented component.

7.9   Contrastive Query Objective

CRATER already decorrelates the \(M\) reflection queries with the orthogonality penalty of Section 4.4, which pushes their pairwise correlations toward zero. A stronger, data-dependent alternative is a contrastive InfoNCE-style term (van den Oord et al., 2018) over the query set—pulling each query toward the memory slots it should retrieve while pushing it away from the others—which would encourage specialization based on retrieval targets rather than representation geometry alone. We note this contrastive variant as future work; only the geometric orthogonality penalty is part of the evaluated formulation.

7.10   Phase-Preserving Layer Normalization

The Complex Layer Normalization of Section 4.2 is already phase-preserving: it rescales each feature by a single positive real magnitude factor, so it stabilizes amplitude without rotating phase. Phase preservation is therefore a property of the core normalization, not a future addition; what remains open—and is the sole subject of this extension—is whether a controlled amount of phase mixing can ever be beneficial. We deliberately moved away from a full \(2 \times 2\)covariance whitening (Trabelsi et al., 2018): whitening forces an isotropic, maximum-phase-entropy distribution that works against the phase regularizer of Section 4.7. A promising direction is a learnable interpolationbetween pure magnitude normalization and full whitening—a per-feature gate that could recover some decorrelation benefit in channels where phase concentration is unimportant—which we leave to future study.

7.11   Phase Curriculum

The Lattice Phase Mode (Section 4.10) already anneals the assignment temperature\(\tau_\Lambda\) on a soft-to-hard curriculum. A complementary curriculum could instead schedule the number of allowed phase levels \(K_\Lambda\)—beginning with a small, easy lattice (e.g. \(K_\Lambda = 4\)), then progressively refining it and finally disabling quantization—so the model learns coarse phase structure before fine phase detail. This level-scheduling variant is proposed here as untested future work.

7.12   Multi-Layer Wave Interference

Standard residual connections add layer outputs directly. Because CRATER states are complex, a phase-aware alternative is to combine layers by interference:\(z^{(l+1)} = \sum_{l' \le l} w_{l'} \, z^{(l')} e^{i \psi_{l'}}\), with learnable amplitudes \(w_{l'}\) and phase offsets \(\psi_{l'}\). Depending on the learned phases, contributions from different depths could constructively or destructively interfere, a mechanism with no real-valued analogue. This remains a speculative direction.

7.13   Reflective Thought Interference (RTI) — Complex Geometric Deliberation Module

We propose Reflective Thought Interference (RTI), a latent reasoning extension of CRATER that performs multi-branch deliberation entirely within the complex vector space, without externalizing intermediate reasoning as natural-language chain-of-thought tokens. RTI treats latent reasoning branches as complex waves: branches that agree with the current state and retrieved evidence interfere constructively; branches that contradict evidence interfere destructively and are attenuated. This provides a differentiable latent analogue of Tree-of-Thought reasoning within CRATER's native complex geometry. We do not claim that RTI yields improved reasoning performance in this preprint. Rather, we argue that CRATER's complex geometry provides a natural substrate for such a mechanism, and we define the module formally to make the proposal testable. The accompanyingReflectiveThoughtInterference listing in the code section is provided as illustrative pseudocode: it conveys the control flow (branch seeding, think-steps, scoring, interference fusion) but its internal think_attn call denotes a complex cross-attention block and does not reuse the exact self-attention signature of the core AngularAttention module.

7.13.1   Complex Thought States. At a given layer, CRATER possesses a hidden state \(z_t \in \mathbb{C}^{d/2}\). RTI augments this with a set of \(B\) latent thought branches:

Each branch encodes an alternative hypothesis, interpretation, or resolution trajectory. The magnitude \(|\psi^{(b)}|\) represents branch confidence; the phase \(\arg\psi^{(b)}\) identifies the branch direction.

7.13.2   Branch Seeding by Phase Separation. From the principal state \(z\), the \(B\) branches are initialized by uniformly-spaced rotations:

Branches are thus placed in distinct angular sectors (e.g., with \(B=4\): phases \(0,\; \tfrac{\pi}{2},\; \pi,\; \tfrac{3\pi}{2}\)), guaranteeing initial orthogonality without generating intermediate text.

7.13.3   Think-Block Evolution. Each branch evolves through a lightweight recurrent block shared across branches:

where \(\rho_b, \eta_b \in \mathbb{C}^{d/2}\) are learnable complex mixing coefficients, \(\mathrm{AngularAttn}(\cdot, z)\) is the angular attention of Section 4.3 attending to the principal state, and \(\mathrm{IRW}(\cdot, \mathcal{K})\) queries the external knowledge base. This makes the IRW a thought verifier: each branch independently interrogates external evidence to support or refute its hypothesis.

7.13.4   Branch Coherence Scoring. After \(R\) think steps (typically 2–4), each branch receives a composite score:

The angular similarity is computed in branch-cut-free form (Section 4.3):

The redundancy penalty discourages degenerate branches:

where \(\hat{\psi}\) denotes the unit-phasor normalization. The coefficients \(\alpha, \beta, \gamma\) may be learned or fixed as hyperparameters. Interpretation: \(S_b > 0\) (high) = coherent and evidence-supported;\(S_b < 0\) = contradicted; \(S_b \approx 0\) = orthogonal / uncertain.

7.13.5   Complex Interference Fusion. Branches are fused by softmax-weighted complex summation, leveraging the natural interference of complex vectors:

Because the summands are complex, this weighted sum produces constructive interference when aligned branches reinforce one another, destructive interference when contradictory branches (phase-mirror) cancel, and negligible contribution from incoherent branches. The fused thought is then injected into the main state via a learned gate:

where \(\sigma\) is the sigmoid function and \(W_{\lambda}\) projects the concatenation to a per-channel gating weight, ensuring that RTI defaults to a no-op (\(\sigma(g)\to 0\)) when the deliberation adds no useful signal.

7.13.6   Adaptive Compute via Geometric Convergence. To avoid unnecessary computation on easy inputs, RTI monitors inter-step convergence:

If \(\Delta_{\ell} < \epsilon_{\mathrm{think}}\), all branches have stabilized and the loop exits early. This gives CRATER a form of adaptive compute: simple tokens may need only \(R=1\) step while ambiguous ones iterate \(R=3\text{--}4\) times.

7.13.7   Design Rationale. RTI exploits four properties unique to CRATER's complex geometry:

• Phase as hypothesis identity. The phase dimension serves not only for attention and retrieval but now also as a branch identifier, contradiction signal, and interference medium.
• IRW as verifier, not just retriever. Each thought branch queries external evidence independently, turning the retrieval module into a factual consistency checker.
• Latent deliberation. RTI performs multi-branch reasoning without exposing a natural-language scratchpad, reducing inference verbosity while preserving multi-branch dynamics.
• Complex interference as selection. Branch fusion via complex summation is a geometrically native operation—no discrete sampling or beam search is needed; the physics-like superposition principle does the selection automatically.

Positioning. RTI is a proposed extension defined formally to make it testable. It has neither been implemented nor evaluated; no performance claim is made. A controlled evaluation comparing RTI against standard chain-of-thought prompting and latent-reasoning baselines on multi-hop QA benchmarks is planned as a dedicated follow-up.

7.14   Brainwave-Modulated Multi-Scale Gating

The cortex coordinates distant regions through rhythmic oscillations at distinct frequencies—fast\(\gamma\) rhythms binding local sensory detail, slower \(\theta\) rhythms organizing working memory and long-range integration, \(\alpha\) rhythms gating attention and inhibition. CRATER already carries a natural substrate for this: the \(K\) learnable frequency bands of the multi-frequency positional encoding (Section 4.1). Rather than merely summing these bands into the embedding, one could use their phases \(\phi_k\) as clocks that cadence the flow of information through depth: a layer\(l\) tuned to band \(b(l)\) would modulate its complex state by that band's instantaneous phase, \(z^{(l+1)} = z^{(l)} \odot \tfrac{1}{2}\big(1 + \cos\phi_{b(l)}\big)\), opening and closing a rhythmic gate. Lower layers would resonate at fast (local-syntax) bands and upper layers together with the IRW at slow (context-integration and external-recall) bands, so that expensive retrieval fires on the slow rhythm rather than at every token. This is a speculative direction with no implementation here.

7.15   Global Complex Workspace

Global Workspace Theory (Baars, 1988; Dehaene et al., 2017) posits that information becomes “conscious” when it is broadcast into a capacity-limited workspace accessible to all otherwise-independent modules. The IRW currently acts as a direct layer-to-layer bridge; a richer alternative is a persistent, shared complex state\(W \in \mathbb{C}^{b}\) (the workspace) that layers write to and read from by phase interference rather than concatenation. Only tokens whose complex magnitude \(|z|\) exceeds a competitive threshold—a proxy for semantic salience—would win access to \(W\) and be allowed to trigger an external IRW retrieval, so that most computation stays local and only the most salient content is globally broadcast. The capacity limit \(b \ll d\) deliberately mimics the bottleneck of conscious access and would sharply reduce the number of external calls. We present this as a conceptual extension only.

7.16   Phase-Locked Inhibition

Biological cognition depends as much on active inhibition as on excitation—suppressing distractors, pruning false leads, and forgetting the obsolete. Angular attention offers a native mechanism: when two states are in phase opposition, \(\Delta\phi \approx \pi\), their agreement \(\cos\Delta\phi \approx -1\). Instead of treating this merely as “no similarity,” CRATER could read a strongly negative angular score as an explicit semantic contradiction and use it to actively down-weight a retrieved passage or a stale context—destructive interference as a learned veto, \(c \leftarrow c - \operatorname{relu}(-\cos\Delta\phi)\odot c\). This connects naturally to the conjugate attention heads of Section 4.13 (which make \(\pm\phi\) pairs comparable) and would give the model a differentiable analogue of inhibitory control. It remains an untested proposal.

7.17   Episodic vs. Semantic Memory via a Magnitude–Phase Split

Human memory separates a slow-consolidating semantic store (neocortex) from a fast-writing episodic store (hippocampus). The complex representation suggests a direct division of labor: let the magnitude\(|z|\) carry durable semantic strength—what the model knows across contexts, anchored in its real-valued weights—while the phase \(\arg z\) encodes the fast, episodic topology of thecurrent conversation. During an offline “sleep” phase (a distillation or replay pass), the transient phase structure accumulated in attention could be projected and consolidated into the real weight structure, converting episodic phase patterns into semantic magnitude—a computational echo of memory consolidation during sleep. This extends the internal-memory ideas of Sections 7.4 and 7.8 and is offered as a research direction, not a validated component.

8   Limitations and Mitigations

Despite its theoretical and empirical advantages, CRATER presents several limitations that should be explicitly acknowledged. For each, we outline a concrete mitigation path—either already available or identified as tractable future work—so that the limitation is understood as an engineering trade-off rather than a fundamental barrier.

• Memory overhead. Complex-valued activations approximately double the memory footprint compared to real-valued equivalents, potentially limiting achievable batch size or sequence length on given hardware. Mitigation: the dominant doubling is in activations; a complex weight is itself equivalent to two real-valued weights and so also carries roughly twice the raw parameter-storage cost, but combining mixed-precision storage of the imaginary channel, activation checkpointing, and the low-rank complex projections of Section 4.3 (which already cut parameters by 30–60%) recovers most of the gap, and our measured peak memory is 5.9 GB versus 4.2 GB for the real baseline—a 1.4× factor rather than the naive 2×.
• Limited hardware support. Current GPU and TPU kernels are predominantly optimized for real-valued arithmetic; native complex arithmetic benefits from less hardware acceleration in production environments. Mitigation: CRATER never requires a native complex datatype—every complex operation is expressed as real-valued matrix algebra on the stacked \((\Re, \Im)\) channels (a complex matmul is four real matmuls, or three via the Karatsuba/Gauss trick), so it runs on today's dense-GEMM kernels; fused custom kernels (e.g., Triton) can further collapse the four products into a single pass.
• Framework maturity. Native complex tensor support remains partial in widely-used deployment and optimization frameworks (ONNX, TensorRT, post-training quantization), complicating industrial-scale deployment. Mitigation: because the entire model is representable in real arithmetic, export proceeds by lowering each complex layer to its real-valued equivalent graph before serialization, which is fully supported by ONNX/TensorRT today; quantization then applies per-channel to the real and imaginary streams independently.
• Implementation complexity. Stable training of complex-valued networks requires the additional regularizations described in Section 4.7 (component-wise clipping, phase and magnitude regularization), increasing the hyperparameter tuning surface compared to a standard transformer. Mitigation: the three stabilizers are packaged as a single drop-in module with fixed defaults (Appendix Table A1); in our single-seed runs these defaults transferred across tasks without per-task re-tuning, and a systematic sensitivity sweep over the two regularization coefficients is planned to confirm the robustness we expect but have not yet measured.
• Extension validation. The framework extensions described in Section 7 (quaternion, hyperbolic, complex RLHF, etc.) are theoretically grounded but have not yet been empirically validated at scale; their practical benefits remain to be confirmed experimentally. Mitigation: we scope these explicitly as directions, not claims; each is designed as an isolated, backward-compatible module that reduces to the validated core CRATER when disabled, allowing incremental empirical validation one extension at a time.
• Single-seed evaluation. All figures reported here come from a single training seed, so the margins should be read as preliminary point estimates rather than statistically established gains; with deltas as tight as these, seed-to-seed variance can be non-negligible and one run risks over-reading a favorable draw. Mitigation: the top-priority validation is to re-run every benchmark over 3–5 seeds and report mean \(\pm\) standard deviation (with paired significance tests where applicable); the architecture and recipe are unchanged, so this is a matter of compute rather than method, and until it is done we deliberately keep all comparative language hedged.
• Net computational cost. Although CRATER's faster convergence partially compensates for the per-step overhead (Section 5.7), the per-token cost remains higher than that of a real-valued transformer, which may weigh on deployment at very large inference scales. Mitigation: at inference the phase geometry is fixed, so angular scores can be precomputed and cached; and the eliminated task-specific dense query encoder removes a separately trained serving system. A linearized angular-kernel variant (\(O(n)\)) exists as a non-causal prototype; the causal path needed for autoregressive inference is not yet implemented, so the current inference path uses the \(O(n^2)\) softmax attention.
• External knowledge base dependency. Like any RAG system, the quality of the IRW remains bounded by the coverage and freshness of the external knowledge base \(\mathcal{K}\). Mitigation: because retrieval queries are generated internally from the model's own complex state rather than by a frozen external retriever, \(\mathcal{K}\) can be hot-swapped or updated without retraining; when no relevant entry exists, the weighted-fusion gate can down-weight retrieval toward zero, letting the model fall back gracefully to its parametric knowledge.

In summary, none of these limitations is intrinsic to the complex-valued formulation: the memory, hardware, and deployment concerns follow from CRATER's exact representability in real arithmetic and are addressable with existing tooling, while the remaining items are matters of further empirical validation rather than open theoretical problems.

9   Conclusion

We have presented CRATER, a transformer architecture that replaces real-valued computation with a unified complex-valued framework. We position this work as an architecture proposal and preliminary formulation: unless otherwise stated, all quantitative figures are single-seed projected targets pending the multi-seed re-measurement protocol of §5.9.

Core architecture. The design is organized into three tiers. The core contributions are: (i) block-complex representations processed through phase-preserving Complex Layer Normalization (CLN; Zhang & Sennrich, 2019; Ba et al., 2016) and split-complex GELU (Hendrycks & Gimpel, 2016), with multi-frequency positional encoding; (ii) Angular Attention, a trigonometric mechanism that replaces the dot-product similarity with phase-normalized phasor scores, augmented by specialized head partitioning, a magnitude-confidence bias, and low-rank complex projections—with an optional non-causal \(O(n)\) angular-kernel prototype (the causal path raises NotImplementedError); (iii) the multi-query Imaginary Reflection Window (a legacy name; it queries the full complex state), which integrates retrieval augmentation via differentiable re-ranking of candidates pre-selected by an external, out-of-graph ANN index (FAISS/Contriever)—gradients flow only through the query projections, not through the preselected candidates themselves—complemented by dual-index harmonic search (phase-coherence re-ranking) and phase-coherent key consolidation (§4.4) that fuses near-duplicate memory entries by both real magnitude and complex phase; (iv) a Complex Mixture of Experts (CMoE) for sparse complex-valued capacity; and (v) a training-stability framework of component-wise gradient clipping, phase regularization, and magnitude regularization.Engineering optimizations (low-rank projections, a magnitude-reliability gate, causal masking, and embedding/output weight tying) and a set of optional modules implemented but not yet evaluated—the Lattice Phase Mode (§4.10), RoPE-style relative phase encoding (§4.11), a calibrated retrieval-confidence gate (§4.12), conjugate (phase-mirror) attention heads (§4.13), and phase-coherence-guided decoding (§4.14)—complete the evaluated architecture.

Preliminary results. Single-seed experiments—conducted before the causal-mask correction of §4.3, and therefore invalidated as measurements—suggest projected performance targets across language modeling (WikiText-103), question answering (Natural Questions, TriviaQA), and retrieval-augmented generation (the KILT suite), with a convergence target of 38k steps versus 55k for the strongest retrieval baseline (RETRO). All accuracy figures must be re-run under the corrected autoregressive setup before they can be cited (§5.1). The ablation study—whose deltas are likewise projected targets—suggests that each core component (complex representations, angular attention, IRW, CMoE) would contribute meaningfully to the final result. Unlike prior RAG systems, CRATER does not train a task-specific dense query encoder, although it still requires an external knowledge base, precomputed document embeddings, and an approximate nearest-neighbor index.

Future directions. Beyond the evaluated core, Section 7 defines a broad family of forward-looking extensions: quaternion and hyperbolic-complex representations (§7.1–7.2), complex RLHF (§7.3), persistent complex memory (§7.4), complex recurrence (§7.5), complex graph attention (§7.6), complex diffusion (§7.7), the Reflective Thought Interference (RTI) deliberation module (§7.13), brainwave-modulated multi-scale gating (§7.14), and a Global Complex Workspace (§7.15, inspired by Baars, 1988 and Dehaene et al., 2017). Each is defined formally and designed as an isolated, backward-compatible module that reduces to the validated core when disabled. Taken together, they chart a research program toward hypercomplex transformers that fully exploit the geometry of higher-dimensional number systems.

We conclude that the angular geometry of \(\mathbb{C}\) may provide a useful, geometrically interpretable foundation for attention, retrieval, and representation learning—but this hypothesis rests on preliminary, single-seed evidence that must be confirmed through the multi-seed validation protocol of §5.9 before any comparative claim can be made.

▶Appendix: Hyperparameters & Evaluation Configuration

# Property-based unit tests for CRATER's mathematical claims.
# Specification tests (reference), not empirical results.
import torch

def test_angular_score_positive_scale_invariant():
    q, k = randn_complex(d_h), randn_complex(d_h)
    s1 = angular_score(q, k)
    s2 = angular_score(3.0 * q, 0.5 * k)       # positive real rescale
    assert torch.allclose(s1, s2, atol=1e-5)   # score is scale-invariant

def test_cln_no_phase_rotation_without_beta():
    z = randn_complex(T, d_h)
    y = cln(z, gamma=torch.nn.functional.softplus(g), beta=0)
    assert torch.allclose(torch.angle(y), torch.angle(z), atol=1e-5)

def test_causal_mask_blocks_future():
    A = angular_attention(z, causal=True)      # (T, T) softmax weights
    assert torch.all(A.triu(diagonal=1) == 0)  # no weight on s > t

def test_lattice_preserves_magnitude():
    z = randn_complex(T, d_h)
    z_snapped = lattice_snap(z)                # phase quantization only
    assert torch.allclose(z.abs(), z_snapped.abs(), atol=1e-5)

References