{"id":986,"date":"2026-07-31T06:55:12","date_gmt":"2026-07-31T06:55:12","guid":{"rendered":"https:\/\/blog.agentsarchitects.ai\/?p=986"},"modified":"2026-07-31T07:30:35","modified_gmt":"2026-07-31T07:30:35","slug":"why-nope-is-replacing-rope-in-modern-ai-architectures","status":"publish","type":"post","link":"https:\/\/blog.agentsarchitects.ai\/index.php\/2026\/07\/31\/why-nope-is-replacing-rope-in-modern-ai-architectures\/","title":{"rendered":"Why NoPE Is Replacing RoPE in Modern AI Architectures"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n


NOPE (No Position Encoding) is a transformer design in which attention layers receive no rotary, absolute, or bias-based positional signal. Moonshot AI applies NOPE to the global Multi-Head Latent Attention (MLA) layers of Kimi Linear. Removing RoPE restores the matrix-absorption property that lets MLA cache a single latent tensor at decode time, eliminating the decoupled rotary dimensions and their separate cache. The Kimi Linear report attributes a 75 percent KV cache reduction and roughly 6x faster decoding at 1M-token context to this hybrid design. Position is not deleted: it is relocated into the per-channel decay gate of Kimi Delta Attention (KDA).<\/p>\n\n\n\n

The spelling NoPE, from Kazemnejad et al. (2023), and NOPE refer to the same design. This article uses NOPE throughout.<\/p>\n\n\n\n

Key takeaways<\/h3>\n\n\n\n
    \n
  1. Causal masking already encodes order.<\/strong> A decoder-only transformer attends to a prefix of exactly t<\/em> tokens, so softmax normalisation leaks a 1\/t<\/em> signal into head statistics. Position does not have to be injected for a decoder to learn it.<\/li>\n\n\n\n
  2. RoPE welds position into the KV tensors.<\/strong> This is the reason DeepSeek-V2’s MLA needs decoupled rotary dimensions cached separately from the latent.<\/li>\n\n\n\n
  3. NOPE removes that coupling.<\/strong> With no rotation between W_Q and W_UK, the up-projections absorb cleanly and decode becomes exactly Multi-Query Attention (MQA) over one latent cache.<\/li>\n\n\n\n
  4. Position moves, it does not vanish.<\/strong> KDA’s per-channel decay gate Diag(\u03b1_t) supplies a learned, input-conditioned set of timescales in place of RoPE’s fixed spectrum \u03b8_i = b^(-2i\/d).<\/li>\n\n\n\n
  5. The claim is narrower than “position encoding is obsolete.”<\/strong> Kimi Linear uses NOPE in one quarter of its layers, on top of a recurrent positional prior supplied by the other three quarters.<\/li>\n<\/ol>\n\n\n\n

    What is NOPE (No Position Encoding)?<\/h3>\n\n\n\n

    NOPE is a transformer configuration in which no explicit positional information is added to token representations or attention scores. There is no rotary position embedding, no learned or sinusoidal absolute embedding, and no distance-dependent attention bias such as ALiBi. Order information is left to arise from the causal mask and, in hybrid architectures, from recurrent state dynamics in neighbouring layers.<\/p>\n\n\n\n

    Kazemnejad et al. (2023) introduced the term and showed that decoder-only transformers with no positional encoding matched or exceeded absolute position embeddings, ALiBi, and RoPE on length-generalisation tasks in reasoning and mathematics at comparable small scale. Haviv et al. (2022) had already shown that language models trained without positional encodings develop absolute position information that a probe can recover from their hidden states.<\/p>\n\n\n\n

    Moonshot AI’s Kimi Linear applies NOPE at production scale, in the global full-attention layers of a hybrid architecture.<\/p>\n\n\n\n

    Why does removing position encoding work in a decoder-only transformer?<\/h3>\n\n\n\n

    Because a causal decoder was never permutation invariant. The “attention needs position” argument describes an encoder operating over an unordered set. A causal decoder is a different object: the token at index t<\/em> attends to a prefix of exactly t<\/em> elements, so the computation graph carries a strict partial order before any embedding is added.<\/p>\n\n\n\n

    That order reaches the representation through a mechanism worth stating plainly. When a head distributes attention broadly across its prefix, softmax normalisation makes each weight scale roughly as 1\/t<\/em>. The norm and mixing statistics of that head’s output therefore vary systematically with depth into the sequence. Two such heads and a subtraction give a usable positional counter.<\/p>\n\n\n\n

    The implication is that explicit positional encoding is not what makes decoders order-aware. It supplies a prior about how order should be represented. Priors can help. Priors can also constrain, and RoPE’s constraint shows up at extension time.<\/p>\n\n\n\n

    What does RoPE cost at long context?<\/h3>\n\n\n\n

    RoPE applies a rotation R(\u03b8, m) to queries and keys using fixed frequencies \u03b8_i = b^(-2i\/d). Two costs follow, and neither is a deployment detail.<\/p>\n\n\n\n

    Cost one: the frequency spectrum is load-bearing.<\/strong> Barbero et al. (2024) analysed how trained models use rotary embeddings and found that high-frequency components carry positional and local attention behaviour, while low-frequency components carry semantic content with position largely factored out. The popular account, that RoPE provides smooth long-range decay, is not the mechanism doing the work.<\/p>\n\n\n\n

    This reframes context extension. Position Interpolation, NTK-aware scaling, and YaRN all remap the frequency spectrum. If the high-frequency band is where the model encoded its local structural machinery, extension is not a free reparameterisation. It is a targeted distribution shift in the channels responsible for adjacency and local syntax. That explains why extension recipes are sensitive to base-frequency choice and to how much continued training is affordable.<\/p>\n\n\n\n

    Cost two: position is welded into the KV tensors.<\/strong> This is the systems argument, and it is the core of the case below.<\/p>\n\n\n\n

    Why does RoPE break Multi-Head Latent Attention?<\/h3>\n\n\n\n

    RoPE breaks MLA’s inference-time compression because the rotation depends on token indices and therefore cannot be folded into a fixed weight matrix.<\/p>\n\n\n\n

    In Multi-Head Latent Attention, introduced with DeepSeek-V2, keys and values are reconstructed from a low-rank latent c_t, and only that latent is cached. At decode time the up-projections absorb into neighbouring weights: W_UK folds into W_Q, and W_UV folds into W_O. Attention then reduces to exactly Multi-Query Attention over the cached latent. That absorption is where the memory saving comes from.<\/p>\n\n\n\n

    RoPE prevents it. The relative rotation R(m – n) sits between W_Q and W_UK and depends on the query and key indices, so the product cannot be precomputed once as a static matrix. DeepSeek’s solution was decoupled RoPE: a small set of additional head dimensions that carry rotary information and are cached separately alongside the latent. It works, at the cost of extra cache, extra kernel complexity, and extra interaction with every context-extension procedure.<\/p>\n\n\n\n

    Remove positional encoding from those layers and the coupling disappears. No decoupled rotary dimensions, no second cache tensor, and the absorbed form is exactly MQA over one latent. NOPE is therefore not merely neutral for MLA. It restores the algebraic property that MLA’s compression was designed to exploit.<\/p>\n\n\n\n

    Moonshot AI’s Kimi Linear report attributes a 75 percent KV cache reduction and roughly 6x faster decoding at 1M-token context to the resulting architecture, relative to a full-attention MLA baseline, while remaining competitive or better on short-context and long-context evaluations. Those figures are from the report, under its own experimental conditions.<\/p>\n\n\n\n

    If RoPE is gone, what encodes position in Kimi Linear?<\/h3>\n\n\n\n

    The per-channel decay gate inside Kimi Delta Attention. Kimi Linear interleaves KDA layers with full MLA layers at a 3:1 ratio. The KDA layers own positional responsibility; the MLA layers handle global retrieval free of positional distortion.<\/p>\n\n\n\n

    KDA belongs to the delta-rule lineage: linear attention as fast weights (Katharopoulos et al. 2020; Schlag et al. 2021), DeltaNet’s chunkwise parallel delta rule (Yang et al. 2024), and gated variants that follow. Its state update, using the source’s transpose convention, is:<\/p>\n\n\n\n

    S_t = (I - \u03b2_t k_t k_t\u1d40) \u00b7 Diag(\u03b1_t) \u00b7 S_{t-1} + \u03b2_t k_t v_t\u1d40\n        <\/pre>\n\n\n\n
    with the decay produced from the token itself:<\/p>\n\n\n\n
    \u03b1_t = \u03c3(W_up \u00b7 Swish(W_down \u00b7 x_t))\n        <\/pre>\n\n\n\n
    The generalisation that matters is Diag(\u03b1_t). A scalar gate gives a layer one forgetting timescale per token, so the entire state ages at a single rate. A per-channel gate gives every feature dimension its own timescale, so one layer can hold fast channels for local structure and slow channels for document-level state at the same time.<\/p>\n\n\n\n
    Now compare that to RoPE. RoPE also supplies a spectrum of timescales, through \u03b8_i = b^(-2i\/d). The difference is origin: RoPE’s spectrum is fixed at design time by a base hyperparameter, and KDA’s is learned and conditioned on the input. The contribution is not the removal of a frequency spectrum. It is the replacement of a hand-specified spectrum with a data-dependent one. Nothing requires rescaling at 1M tokens because nothing was pinned to a training-time index.<\/p>\n\n\n\n
    The engineering cost is real. General diagonal-plus-low-rank state transitions are more expensive than scalar-gated ones, and the report’s kernel contribution is a chunkwise formulation that keeps the per-channel gate inside roughly the cost envelope of scalar-gated baselines.<\/p>\n\n\n\n
    How do RoPE, ALiBi, and NOPE plus KDA compare?<\/h3>\n\n\n\n
    Property RoPE ALiBi NOPE with KDA gating Positional signal Fixed rotation frequencies \u03b8_i = b^(-2i\/d) Fixed linear distance penalty Learned per-channel decay Diag(\u03b1_t), input-conditioned Extension beyond training length Needs PI, NTK-aware scaling, or YaRN Extrapolates, with quality loss on retrieval No index-tied parameters to rescale Timescales available per layer Fixed spectrum, set by base b<\/em> One, set by slope per head One per feature channel, learned Interaction with MLA absorption Blocks it, requires decoupled rotary dims plus separate cache Additive bias, no KV coupling Clean absorption, decode is MQA over one latent Explicit control over position at serving time High: reindex, interpolate, shift caches Moderate Low: no index to manipulate Primary risk Frequency misalignment after extension Recency bias, weak long-range retrieval Recency bias from decay, entangled ablation attribution<\/p>\n\n\n\n
    What are the limits of this result?<\/h3>\n\n\n\n
    Five caveats belong with any citation of this work.<\/p>\n\n\n\n
      \n
    1. This is not global NOPE.<\/strong> It is NOPE in one quarter of the layers, sitting on a recurrent positional prior supplied by the other three quarters. Stripping RoPE from a pure full-attention stack is a different claim with weaker support.<\/li>\n\n\n\n
    2. Ablation attribution is entangled.<\/strong> NOPE, KDA, the 3:1 layer ratio, the mixture-of-experts configuration, and the data recipe all co-vary. Published comparisons are architecture-level, not clean single-factor isolations.<\/li>\n\n\n\n
    3. A decay gate is a recency prior.<\/strong> Exact retrieval at extreme distance depends on the global layers and on the model learning to route around decay. Needle-in-a-haystack tests are necessary and not sufficient; multi-hop and repository-scale tasks are the discriminating evaluations.<\/li>\n\n\n\n
    4. Explicit position is a control surface, and you lose it.<\/strong> With RoPE you can reindex a KV cache, interpolate, shift a cached prefix, or reorder retrieved chunks. Implicit position removes those levers. Cached-chunk reuse becomes less index-sensitive, which helps, but serving stacks that perform prefix caching or KV surgery have less deterministic control over how position is interpreted.<\/li>\n\n\n\n
    5. Small-scale length-generalisation results do not transfer automatically.<\/strong> The 2023 NOPE evidence came from small models on reasoning tasks. Frontier-scale behaviour is an empirical question per training recipe, not a theorem.<\/li>\n<\/ol>\n\n\n\n
      What should long-context practitioners do differently?<\/h3>\n\n\n\n
        \n
      • If your long-context path depends on YaRN to reach 2x training length, treat the positional scheme as the primary suspect, not the attention kernel.<\/li>\n\n\n\n
      • If you build on MLA, the RoPE decision is not a modelling detail. It determines whether your decode path is truly MQA over a single latent cache.<\/li>\n\n\n\n
      • When comparing hybrid linear-attention stacks, ask for the layer ratio and the gate structure (scalar or per-channel) before comparing benchmark numbers. Those two choices dominate.<\/li>\n\n\n\n
      • When extending an existing RoPE checkpoint, treat frequency remapping as a shift in the model’s local-structure channels and budget continued training accordingly.<\/li>\n\n\n\n
      • When training from scratch for very long context, the question is not “which positional encoding?” but “which component owns order, and does it represent order in a length-independent way?”<\/li>\n<\/ul>\n\n\n\n
        Frequently asked questions<\/h3>\n\n\n\n
        Does NOPE mean transformers no longer need positional information?<\/strong> No. It means positional information does not need to be added explicitly to token representations. In a decoder-only model, causal masking supplies an implicit order signal, and in Kimi Linear the recurrent decay gates of the KDA layers supply a learned positional prior. Order is still represented, through the computation graph rather than through an embedding.<\/p>\n\n\n\n
        Is NOPE the same as NoPE?<\/strong> Yes. NoPE is the spelling used by Kazemnejad et al. (2023) and in the Kimi Linear report. NOPE is a common stylisation of the same term. Both refer to attention layers with no positional encoding of any kind.<\/p>\n\n\n\n
        Why not simply remove RoPE from an existing model?<\/strong> Because a pretrained model has already allocated capacity to RoPE’s frequency channels, particularly the high frequencies that carry local structure. Removing them post hoc changes the input distribution to every attention head. Kimi Linear does not retrofit NOPE onto a rotary model: it trains a hybrid in which other layers own positional representation from the start.<\/p>\n\n\n\n
        Does NOPE remove the need for YaRN or Position Interpolation?<\/strong> Within this architecture, yes, because there are no index-tied frequency parameters to rescale. That property comes from the architecture as a whole, not from the absence of RoPE alone. A model with no positional encoding and no recurrent gating has no length-independent positional representation to fall back on.<\/p>\n\n\n\n
        How much of the reported speedup comes from NOPE versus from linear attention?<\/strong> The report does not isolate them cleanly, and neither should a citation. The KV cache reduction has two sources: linear-attention layers that do not cache per-token keys and values, and MLA layers whose absorption is unblocked by NOPE. Attribute the figure to the architecture, not to one component.<\/p>\n\n\n\n
        Does this apply to encoder models or bidirectional attention?<\/strong> No. The argument depends on causal masking to supply implicit order. A bidirectional encoder over an unordered set is permutation invariant in the way the textbook description assumes, so it still requires explicit positional information.<\/p>\n\n\n\n
        References<\/h3>\n\n\n\n
        Architectural details and performance figures below belong to their primary sources. Consult those sources for exact experimental conditions before citing any figure.<\/p>\n\n\n\n
          \n
        1. Kimi Team (Moonshot AI). Kimi Linear: An Expressive, Efficient Attention Architecture.<\/em> arXiv:2510.26692, 2025.<\/li>\n\n\n\n
        2. Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., Reddy, S. The Impact of Positional Encoding on Length Generalization in Transformers.<\/em> arXiv:2305.19466, NeurIPS 2023.<\/li>\n\n\n\n
        3. Haviv, A., Ram, O., Press, O., Izsak, P., Levy, O. Transformer Language Models without Positional Encodings Still Learn Positional Information.<\/em> Findings of EMNLP 2022, arXiv:2203.16634.<\/li>\n\n\n\n
        4. Su, J., Lu, Y., Pan, S., Wen, B., Liu, Y. RoFormer: Enhanced Transformer with Rotary Position Embedding.<\/em> arXiv:2104.09864, 2021.<\/li>\n\n\n\n
        5. Barbero, F. et al. Round and Round We Go! What Makes Rotary Positional Encodings Useful?<\/em> 2024.<\/li>\n\n\n\n
        6. Press, O., Smith, N., Lewis, M. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.<\/em> arXiv:2108.12409, 2021.<\/li>\n\n\n\n
        7. Chen, S., Wong, S., Chen, L., Tian, Y. Extending Context Window of Large Language Models via Positional Interpolation.<\/em> arXiv:2306.15595, 2023.<\/li>\n\n\n\n
        8. Peng, B., Quesnelle, J., Fan, H., Shippole, E. YaRN: Efficient Context Window Extension of Large Language Models.<\/em> arXiv:2309.00071, 2023.<\/li>\n\n\n\n
        9. Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.<\/em> arXiv:2006.16236, 2020.<\/li>\n\n\n\n
        10. Schlag, I., Irie, K., Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers.<\/em> arXiv:2102.11174, 2021.<\/li>\n\n\n\n
        11. Yang, S., Wang, B., Zhang, Y., Shen, Y., Kim, Y. Parallelizing Linear Transformers with the Delta Rule over Sequence Length.<\/em> arXiv:2406.06484, 2024.<\/li>\n\n\n\n
        12. Yang, S. et al. Gated Delta Networks: Improving Mamba2 with Delta Rule.<\/em> 2024.<\/li>\n\n\n\n
        13. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.<\/em> arXiv:2405.04434, 2024.<\/li>\n<\/ol>\n\n\n\n
          #NOPE #RoPE #LongContext #KimiLinear #LinearAttention<\/p>\n","protected":false},"excerpt":{"rendered":"
          Discover how NoPE (No Position Encoding) improves long-context AI by removing RoPE from Multi-Head Latent Attention, reducing KV cache, accelerating inference, and enabling more efficient transformer architectures.<\/p>\n","protected":false},"author":1,"featured_media":987,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-986","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence"],"_links":{"self":[{"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/posts\/986","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/comments?post=986"}],"version-history":[{"count":3,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/posts\/986\/revisions"}],"predecessor-version":[{"id":991,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/posts\/986\/revisions\/991"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/media\/987"}],"wp:attachment":[{"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/media?parent=986"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/categories?post=986"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.agentsarchitects.ai\/index.php\/wp-json\/wp\/v2\/tags?post=986"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}