Research report ·

TLDR

    Key insights

    Proposed hardware spec

    SUMMARY

    A chip built only for batch-size-1 decode: about 15–30 TOPS (INT4 equivalent), three memory tiers with no off-package HBM, and a 5W power budget. The target is 8–15 tokens/s on a heavily sparse model with 100–500B total parameters.

    Memory hierarchy

    TIER 0
    SRAM on-die32–64 MB · >10 TB/s Holds shared experts, the router, and the draft model.
    TIER 1
    LPDDR6 on-package24–48 GB · 150–250 GB/s · 4–6 pJ/bit Holds active experts and the KV or state-space state. Its speed and energy set the token rate.
    TIER 2
    NAND cold store256 GB – 1 TB · ~4 GB/s Stores the model only. Too slow (40–80×) to read during decode.

    Power budget (5W)

    Tokens per second by model and memory bandwidth

    Hover a row.
    ScenarioGB / token@100 GB/s@273 GB/s @614 GB/s@3 TB/s@5W energy-limited
    POWER AT WHICH THE LIMIT CHANGES
    P* = Bandwidth × J/byte
    This does not depend on model size. With 100 GB/s phone memory and a system-level memory energy of 0.16 nJ/byte, P* is about 16W. Below that power the device is limited by energy; above it, by bandwidth. A 5W handheld is well below 16W, so the 5W column is the one that applies.

    Citation graph

    Drag to pan · scroll to zoom · double-click a node to focus

    Timeline

    Drag to orbit · X time · Y citations · Z angle lane

    Next steps

      Idea deck

      Report quality

      Citations