TurboEXL3: Why EXL3 should be the default model formats for modern Macs.

@ TheDavidTai4h24 min read

0 Replies. Reply

3

19

2.1K

Abstract

Mixture-of-experts models for Apple silicon ship their expert weights almost exclusively as block-scaled 4-bit quants: MLX's affine 4-bit, MXFP4 or NVFP4. Accuracy per byte measures how close a quantized model stays to the original per byte of weights. By that measure these formats trail trellis-coded quantization, as implemented in turboderp's EXL3 format. At 4.05 bits per weight, EXL3's KL divergence to BF16 is 35% lower than that of NVFP4 at about 4.8 bits, even with NVFP4's activations kept at 16 bits.

EXL3 has stayed off Macs because it is slow to serve:

• Each weight is a trellis code that has to be decoded inside the matrix multiply.
• Each projection is wrapped in Hadamard rotations.
• A mixture-of-experts layer splits every batch into small per-expert multiplies.

On an M5 Max, Sushi, the existing exact EXL3 engine for MLX, prefills a 5.05-bit pack 15 to 37% slower than the standard MLX configuration.

turboexl3 is a library of fused EXL3 kernels for the M5 GPU and its neural accelerators:

• Prefill: it decodes each slice of an expert's weights once into on-chip memory and multiplies it against up to 128 tokens on the neural accelerators.
• Decode: when speculative decoding verifies 2 to 4 tokens, it runs a layer's routed experts, shared expert and router in five launches.

With turboexl3, full EXL3 packs prefill within 7% of the standard mlx-serve configuration from 8K tokens up. At the same speculative-decoding depth, the 3.05- and 4.05-bit packs decode 4 to 17% faster at every context length from 1K to 256K tokens. The 2.05- and 5.05-bit packs decode at about the same speed, between 2% slower and 7% faster.

The format can then be chosen on accuracy per byte, which offers two trades:

1. Keep a 4-bit quant's size and gain accuracy. The 4.05-bit pack is 9% smaller than the standard Q4 pack and peaks within 2 GiB of it in memory, with 35% lower KL divergence than NVFP4.
2. Keep standard Q4 accuracy and cut the size. The 3.05-bit pack solves 984 of 1,084 EvalPlus problems, against 987 for the standard pack, ddalcu's iQ-MLX 4.7 bpw quant. It takes 30% less disk and 14 GiB less memory.

1. Introduction

Qwen3.8-Flash-Next is a mixture-of-experts model with 512 experts per layer, and it fits on a 128 GB Mac only after quantization. The usual choice for its expert weights, which hold most of its bytes, is a block-scaled 4-bit format, picked because its kernels are simple and fast. The measure that should decide the format is accuracy per byte, and on that measure trellis-coded quantization is ahead (Section 2). What has kept it off Macs is serving speed (Section 3).

The rest of this note:

• presents turboexl3, a kernel library that removes the speed problem on the M5 (Section 4);
• measures full EXL3 packs at four rates through mlx-serve, against the standard configuration and against the existing EXL3 engine: kernel timings, throughput, memory, coding accuracy and speculative-decoding acceptance (Sections 5 and 6);
• argues that models for M5-class Macs should move to full EXL3 (Section 7).

2. The accuracy problem

2.1 Block-scaled 4-bit formats

A block-scaled format stores each weight as a 4-bit code, and each small block of weights shares a scale. Dequantizing a weight takes one multiply, plus an add for affine formats. Three formats cover the expert weights of most quantized models on Apple silicon:

• MLX affine 4-bit stores 4-bit integers with a 16-bit scale and a 16-bit bias for every 64 weights, which comes to 4.5 bits per weight.
• MXFP4 [3] stores 4-bit floating-point values with an 8-bit power-of-two scale (E8M0) for every 32 weights, which comes to 4.25 bits per weight. The values are E2M1: the magnitudes 0, 0.5, 1, 1.5, 2, 3, 4 and 6, each with a sign.
• NVFP4 [4] stores the same E2M1 values with an 8-bit floating-point scale (E4M3) for every 16 weights, which comes to 4.5 bits per weight. It also keeps one 32-bit scale per tensor, which holds the block scales inside E4M3's range. Its smaller blocks and finer scales fit each block more closely than MXFP4's power-of-two scales, so it is the more accurate of the two FP4 formats.

All three spend their bits the same way:

• A quarter to half a bit of every weight goes to scales.
• Each weight is rounded to its block's grid independently of its neighbours.

Each format also has a single operating point. A smaller model means a different format, and a more accurate one means 8 bits.

2.2 NVFP4 with 4-bit and with 16-bit activations

NVFP4 was introduced with NVIDIA's Blackwell GPUs, whose tensor cores multiply FP4 values directly. An NVFP4 checkpoint can therefore run in two ways, and turboderp measured both:

• W4A4 also quantizes the activations to NVFP4, on the fly, with a scale for every 16 values. Both operands of each matrix multiply are then FP4, and Blackwell's tensor cores run the multiply at twice their FP8 rate. This is the case the format was designed for. It adds an error that quantizing the weights alone does not: every activation is rounded to 4 bits on every forward pass.
• W4A16 keeps the activations in 16 bits. The kernel expands each weight to 16 bits and multiplies at 16-bit precision, so only the weights are approximated. It is the more accurate variant, and it is the one available on hardware without FP4 arithmetic.

On turboderp's trace, quantizing the activations multiplies NVFP4's KL divergence by 2.4, from 0.0102 to 0.0244, and raises its perplexity from 1.3964 to 1.4201 (Table 1). His chart places the checkpoint at about 4.8 bits per weight by his count, against the format's nominal 4.5.

On a Mac, NVFP4 is a W4A16 format. MLX [7] stores NVFP4 weights with their 16-weight E4M3 scales and multiplies them with 16-bit activations. The format's arithmetic advantage on Blackwell therefore does not carry over. On Apple silicon NVFP4 offers its size, and size against accuracy is the comparison this note makes.

2.3 Trellis-coded quantization and EXL3

Trellis-coded quantization shifts the whole size–accuracy curve. QTIP [1] first rotates each weight matrix with random Hadamard transforms, so that its values are close to Gaussian. It then encodes long runs of weights as paths through a bitshift trellis, and a small codebook function decodes each path back into values. The encoder chooses each path to minimize the error over the whole run instead of rounding each weight on its own. No bits go to block scales. turboderp's EXL3 format in exllamav3 [2] applies this method to current models.

EXL3 multiplies each weight matrix on both sides by sign-randomized Hadamard transforms, applied in blocks of 128. It encodes 16×16 tiles of the result as trellis paths. The only scales are two 16-bit vectors per matrix, with one entry per input channel and one per output channel. At inference time, a short bit window slides along each path, and a codebook function (MUL1 or MCG) maps each window to a weight. The rate is set per projection and can be any multiple of 1/16 bit from 1 to 8 bits per weight.

turboderp publishes Qwen3.8-Flash-Next at 2.05, 3.05, 4.05, 5.05 and 6.05 bits per weight. I call the first four K2 to K5, after the trellis rate K.

A *full* pack quantizes every linear layer with EXL3: attention, the linear-attention layers, the shared expert, the routed experts and the output head. turboderp's quality measurements and every result in this note are for full packs.

2.4 Accuracy per bit

Table 1 and Figure 1 place the EXL3 packs on turboderp's measurements, next to NVFP4 and the GGUF quants on the same trace. At every rate, EXL3 has the lowest KL divergence. Three comparisons carry this note's argument:

• At 4.05 bits, EXL3's KL divergence (0.0066) is 35% below NVFP4 W4A16's (0.0102), with 0.7 fewer bits per weight. It is also 21% below that of the 5.2-bit GGUF quant.
• At 3.05 bits, EXL3's KL divergence (0.0174) is within 8% of the 4.1-bit GGUF quant's (0.0162), with three-quarters of the bits. It is 29% below NVFP4 W4A4's (0.0244).
• NVFP4 W4A16 is more accurate than EXL3 at 3.05 bits, but it uses about 1.7 more bits per weight.

Neither MXFP4 nor MLX affine 4-bit was measured on this model. MXFP4 uses NVFP4's 4-bit values with coarser power-of-two scales over blocks twice as large, so NVFP4 is the stronger FP4 baseline.

Table 1. KL divergence to BF16 and perplexity on turboderp's trace (lower is better). Bits per weight exclude embeddings and the output head. Values for formats other than EXL3 are read from his chart.

| Format | Bits per weight | KL divergence | Perplexity | | --- | --- | --- | --- | | EXL3 K2 | 2.05 | 0.0681 | 1.4871 | | EXL3 K3 | 3.05 | 0.0174 | 1.4074 | | EXL3 K4 | 4.05 | 0.0066 | 1.3897 | | EXL3 K5 | 5.05 | 0.0040 | 1.3848 | | NVFP4 W4A16 (16-bit activations) | ≈4.8 | 0.0102 | 1.3964 | | NVFP4 W4A4 (4-bit activations) | ≈4.8 | 0.0244 | 1.4201 | | GGUF UD-IQ3_XXS | ≈3.3 | 0.0349 | 1.4339 | | GGUF UD-IQ4_XS | ≈4.1 | 0.0162 | 1.4021 | | GGUF UD-Q4_K_XL | ≈5.2 | 0.0084 | 1.3929 | | BF16 | 16 | 0 | 1.38474 |

Figure 1. KL divergence to BF16 against bits per weight. Data: turboderp, measured with exllamav3.

3. The speed problem

3.1 What an EXL3 projection costs

Every EXL3 projection runs in three stages:

1. Prepare: scale the input by the input-channel vector and apply the Hadamard transform, one 128-element block at a time.
2. Inner product: multiply by the weight matrix, decoding each weight from its trellis code.
3. Finish: apply the Hadamard transform to the output, then scale it by the output-channel vector.

Each stage adds work that an affine 4-bit projection does not have:

• Decoding. An affine 4-bit weight costs a shift, a mask and a multiply-add. An EXL3 weight needs a bit window read from the trellis stream at a position that depends on the tile, followed by the codebook function: an integer multiply and a few bit operations. Prefill is limited by arithmetic, so this work adds directly to its time.
• Rotations. The prepare and the finish are passes over the activations that an affine projection does not need. When they run as separate launches, each one reads and writes the activations in device memory.
• Small multiplies. Qwen3.8-Flash-Next routes each token to 10 of 512 experts. An 8,192-token prefill chunk therefore gives each expert about 160 tokens on average, and each expert's weights must be decoded for that small batch. In decode, speculative verification passes 1 to 4 tokens. A layer's work then splits into many small launches that wait on one another.

3.2 The existing engine

Sushi is the existing exact EXL3 engine for MLX. Its prefill multiplies each expert's tokens in 32-row windows on the neural accelerators. Each lane decodes its weights into registers, so the whole expert is decoded again for every window. At about 160 tokens per expert, each weight is decoded five times per chunk.

Sushi's own notes record three attempts to share the decode at this model's shapes:

• one decode feeding a 64-row window;
• a double-buffered decode through threadgroup memory;
• decoding the next tile ahead of each multiply.

Each ran 7 to 30% slower, because the kernel is bound by registers and occupancy.

Measured end to end, Sushi leaves EXL3 behind the standard configuration where prompts are long (Table 2):

• From 8K tokens up, every pack prefills more slowly than the standard pack, K5 by 15 to 24%.
• At 256K tokens, K5's time to first token is 125.0 seconds against 104.5.
• Below 8K tokens, K2 and K3 prefill faster than the standard configuration and K5 is 19 to 37% slower.

Decode depends on the speculative-decoding depth. mlx-serve's default runs the standard pack at depth 6. On these prompts, that is 7 to 30% slower than the same pack at depth 3, and against that default Sushi's packs decode faster almost everywhere. At the same depth of 3, though:

• K4 decodes 4 to 10% slower than the standard pack;
• K2 decodes up to 11% slower, and K5 up to 7% slower;
• K3 ranges from 2% slower to 8% faster.

Table 2. Sushi against the standard configuration, MTP on: range of the speed difference by pack, and time to first token at 256K tokens (standard: 104.5 s). Decode is compared two ways:

• with the standard pack at the same MTP depth of 3 (for K5, its depth-3 run with the pack's earlier MTP mixer);
• with the standard pack at mlx-serve's default depth of 6.

The ranges leave out the runs flagged as noisy (Section 5).

| Pack | Prefill, 8K–256K | Time to first token at 256K | Decode | | --- | --- | --- | --- | | EXL3 K2 | −6.5% to −2.0% | 111.8 s | −2.5% to +30.0% | | EXL3 K3 | −8.3% to −5.7% | 114.0 s | +6.8% to +37.0% | | EXL3 K4 | −8.9% to −6.0% | 114.2 s | −1.5% to +16.4% | | EXL3 K5 | −23.8% to −14.7% | 125.0 s | +3.6% to +32.0% |

4. turboexl3

turboexl3 is a Zig library of Metal kernels for EXL3 projections on Apple GPUs. MLX launches the kernels, and the library reaches mlx-serve as a quant plugin. Its design answers each cost in Section 3:

• It decodes each weight as few times as possible, as close as possible to the arithmetic that uses it.
• It folds the rotations into the neighbouring launches.
• It merges the steps of each layer into as few dependent launches as possible.

4.1 Prefill: decode once, multiply many times

Above 16 rows, a layer's routed experts run as one chain of eight launches in six dependent levels:

1. Route. One launch counts each expert's tokens, groups the tokens by expert and builds the table of tiles. It replaces a sort and a separate table pass.
2. Prepare. The gate and up inputs are rotated, in grouped order and in the layout the neural accelerators read.
3. Gate and up. One GEMM computes both projections, one 64-deep stage of the input dimension at a time:
4. The threadgroup's simdgroups decode the stage's 16 trellis tiles into threadgroup memory, in the neural accelerators' fragment order.
5. The neural accelerators multiply 64 or 128 tokens by that stage, with 32-bit accumulation.
6. Each expert's full runs of 128 tokens take 128-row tiles. A remainder of at most 32 tokens takes a register path in the same launch, where each simdgroup decodes straight into its own fragment registers.
7. Mid. One launch runs the gate and up finishes, SwiGLU in 32-bit, and the rotation of the down projection's input.
8. Down. The same GEMM body computes the down projection.
9. Finish. One launch runs the down projection's finish and the score-weighted sum of each token's ten expert outputs.

The decoded weights never reach device memory. The GEMMs read K bits per weight and write no 16-bit copy. An expert's weights are decoded once per 64 or 128 tokens instead of once per 32. The MUL1 codebook's final affine map is applied once per output, in the finish, instead of once per weight.

4.2 Decode: an MoE layer in five launches

At 1 to 16 rows, the routed experts run as a chain of split-K GEMV launches. The served form merges every pair of steps that share data, giving five launches:

1. one launch prepares the gate and up inputs;
2. one runs the gate and up GEMVs;
3. one runs the mid;
4. one runs the down GEMV;
5. one runs the down finish together with the weighted sum.

That makes 5 dependent levels from two rows. At one row the prepare joins the GEMV launch, giving 4. The unfused chain needs 11 launches in 8 levels. Two tokens that chose the same expert share one decode of its weights. The split-K partial sums keep enough threadgroups in flight at one row.

At 2 to 4 rows, the shared expert and the router fold into the same launches. That range covers speculative verification up to depth 3:

• The shared expert's GEMVs run as extra grid rows of the routed gate, up and down launches, and its finishes ride the routed finishes.
• The router's top-k and softmax run inside the prepare launch.

The host's chain for the same work is the routed chain plus three dense projections, SwiGLU, a sigmoid gate, a multiply and an add. At one row, the shared expert stays in that chain.

4.3 The trunk

The attention, linear-attention, shared-expert, output-head and MTP projections are dense EXL3 projections. turboexl3 picks a path by the number of rows:

• 1 to 4 rows: one GEMV launch. For each output block, the threadgroup that arrives last runs the finish.
• 5 to 16 rows: a three-launch GEMV chain that decodes each weight once per chunk of up to 8 rows.
• 17 to 32 rows: a register-path GEMM on the neural accelerators.
• More rows: the weights are decoded to 16 bits and multiplied with MLX's matmul. mlx-serve can keep the decoded copies after the first call, which trades memory for prefill time.

At prefill widths, the attention projections also absorb neighbouring work:

• The input projection computes q, k and v, with the QK normalization and the partial rotary embedding in its finish.
• The output projection forms the sigmoid-gated attention output inside its prepare.

4.4 Numerics

turboexl3 holds every kernel to one reference, \format.project\, a CPU port of Sushi's EXL3 reference. Its kernels fall into two classes:

• Byte-identical kernels replace a chain of turboexl3's own kernels and reproduce that chain's output exactly. Examples are the fused decode forms, the shared-expert and router folds, and the trunk's one-launch GEMV.
• Tolerance kernels sum in their own order, such as the GEMMs on the neural accelerators. They round where the reference rounds and stay within a stated tolerance of it. On sampled rows of the routed prefill chain, the largest error is at most 5×10⁻³ of the output's root mean square in 32-bit. In bf16 it is at most 1.5 times Sushi's own error, plus 2×10⁻³.

Accumulation is 32-bit throughout, and every sum runs in a fixed order, so the same request gives the same output on every run. The GPU tests compare bits for the first class and tolerances for the second. The builds measured here also carry Sushi's exact path, so the two engines can be compared on one binary and pack (Section 6.2).

5. Experimental setup

Hardware and software. One Apple M5 Max with 128 GB of unified memory, macOS 27.0.1, mlx-serve [5].

Model and packs. Qwen3.8-Flash-Next has 512 routed experts per layer with top-10 routing, a shared expert, and an MTP head for speculative decoding.

• EXL3 packs: turboderp's plain quants at 2.05, 3.05, 4.05 and 5.05 bits per weight, repacked into mlx-serve's file layout without requantization.
• Baseline ("standard Q4"): ddalcu's iQ-MLX 4.7 bpw pack, mlx-serve's recommended quant for 128 GB Macs. Its experts are affine 4-bit with group size 64, quantized with an importance matrix collected on the BF16 model. The remaining layers are 8-bit. By ddalcu's count, that comes to 4.68 bits per weight over the quantized weights.

Configurations.

• turboexl3: K2 to K4 ran on one build at MTP depth 3, and K5 on the final build at depth 4.
• Sushi, the exact engine, ran on the same binaries.
• Standard: upstream mlx-serve at its default MTP depth, which ran at 6. It carries two load-time switches that let a 100 GB pack load on a 128 GB machine.

All MTP runs used typical acceptance with a threshold of 0.2.

Kernel timings. One test binary holds both turboexl3 and Sushi, with no server. It times one MoE layer's routed experts per call, at every rate from K2 to K8:

• decode at 1 to 4 rows, as the median of 31 interleaved rounds;
• prefill at 1,328 and 8,192 rows, as the median of 15 rounds.

Routing is captured from mlx-serve serving a 17,743-token coding prompt. Decode is also timed on synthetic routing.

Context sweep. Each configuration served the same prompts at nine lengths from 1K to 256K tokens. Each prompt was Python source followed by one coding task, run with thinking on, greedy decoding, seed 1234 and a 1,024-token answer cap. Each run used a fresh server, started after a warm-up boot and a 40 °C cool-down gate. There was one run per cell. Rates and draft acceptance come from the server's log. Three single runs were flagged as noisy and not re-run: K2 at 8K, and K4 at 1K and 2K, on both engines. The tables show them, and the ranges quoted in the text leave them out.

Task accuracy. EvalPlus 0.3.1 [6]: 164 problems each in HumanEval and HumanEval+, and 378 each in MBPP and MBPP+, for 1,084 in all. Greedy pass@1, reasoning effort low, up to 16,384 new tokens, seed 1234, MTP on. The four EXL3 packs ran on one turboexl3 build at MTP depth 4, and the standard pack on the standard configuration. One answer reached the token cap (K4 on Mbpp/56) and still passed. Every other answer finished on its own.

Quality per bit. KL divergence to BF16 and perplexity are turboderp's measurements with exllamav3 [2]. He measured them on a self-generated in-domain trace of 14,496 input and 45,588 output tokens. On this trace, BF16's perplexity is 1.38474 and the KL divergence noise floor is 0.00262.

6. Results

6.1 Kernels against Sushi

Table 3 and Figure 2 time the routed experts of one MoE layer, one call at a time.

Prefill. turboexl3's served path is faster than Sushi's at every rate from K2 to K8:

• 1.02 to 1.32 times on a full 8,192-token chunk;
• 1.07 to 1.98 times on a 1,328-token final chunk.

The margins are largest at K5 to K7, where Sushi's own prefill slows down.

Decode. On routing captured from a real prompt, turboexl3's served chain at the published rates (K2 to K5) runs at:

• 1.00 to 1.23 times Sushi's speed for a 1-row call;
• 1.03 to 1.49 times for verification calls of 2 to 4 rows.

The peak at K4 comes from Sushi's own times, which rise at that rate. turboexl3 decodes an expert's weights once for all the rows of a call that chose it, and real prompts repeat experts. A 4-row call's 40 expert slots hold 28 distinct experts on average, against 39 on synthetic routing. On synthetic routing, Sushi is faster in 11 of the 28 decode cells from K2 to K8, by up to 4%.

These timings cover the routed experts alone. They leave out the rest of each layer, including the shared expert and the router, which the served forward folds into the same launches at 2 to 4 rows. They show where the per-call gains come from; Section 6.2 measures the end-to-end effect.

Table 3. Per-call speed of turboexl3 relative to Sushi for one MoE layer's routed experts: Sushi's median time divided by turboexl3's (above 1, turboexl3 is faster). Routing is captured from a real prompt, and decode uses turboexl3's served form. K6 to K8 are in Figure 2.

| Rate | Decode, 1 row | 2 rows | 3 rows | 4 rows | Prefill, 1,328 rows | 8,192 rows | | --- | --- | --- | --- | --- | --- | --- | | K2 | 1.03 | 1.05 | 1.12 | 1.16 | 1.07 | 1.02 | | K3 | 1.00 | 1.03 | 1.08 | 1.13 | 1.16 | 1.06 | | K4 | 1.23 | 1.36 | 1.42 | 1.49 | 1.17 | 1.04 | | K5 | 1.02 | 1.05 | 1.09 | 1.14 | 1.97 | 1.31 |

Figure 2. turboexl3 against Sushi, one MoE layer call at a time, by trellis rate: decode with the served form at 1 to 4 rows, and prefill with the served path at 1,328 and 8,192 rows.

6.2 End to end against Sushi

Sushi and turboexl3 on the same binary and pack isolate the kernels (Table 4, Figure 3). With MTP on, turboexl3 is faster at almost every length on every pack:

• Prefill: the gains are largest on the larger packs, up to 40% on K5.
• Decode: the gains are 2 to 22%.

The prefill gains change the comparison with the standard configuration. With Sushi, the K5 pack prefills 15 to 37% slower than the standard configuration at every length (Table 2). With turboexl3, it is within 7% from 8K tokens up, and its time to first token at 256K tokens falls from 125.0 to 111.3 seconds.

The one regression is decode with MTP off, 1.3 to 2.1% behind Sushi.

Table 4. turboexl3 against Sushi on the same binary and pack: range of the speed difference across the nine prompt lengths. The ranges leave out the runs flagged as noisy, whose values are given in parentheses.

| Pack | Prefill | Decode | | --- | --- | --- | | EXL3 K2, MTP on | −3% to +5% (8K: +17%) | +2% to +11% | | EXL3 K3, MTP on | −0.5% to +11% | +4% to +9% | | EXL3 K4, MTP on | +3% to +12% (1K: +34%, 2K: −26%) | +14% to +22% | | EXL3 K5, MTP on | +5.5% to +40% | +2% to +12% | | EXL3 K5, MTP off | +10% to +24% | −2.1% to −1.3% |

Figure 3. turboexl3's speedup over Sushi on the same binary and pack.

6.3 Throughput against the standard configuration

Decode (Table 5, Figure 4). The comparison depends on the speculative-decoding depth. mlx-serve's default runs the standard pack at depth 6, and on these prompts that is 7 to 30% slower than the same pack at depth 3. At the same depth of 3, at all nine prompt lengths:

• K2 decodes at the standard pack's speed, from 2% slower to 4% faster;
• K3 is 4 to 13% faster;
• K4 is 5 to 17% faster;
• K5 is level, from 1% slower to 7% faster. This is its depth-3 run, with the pack's earlier MTP mixer.

On the final build, K5 ran at depth 4. Against the standard pack at depth 4 it is 0.5 to 38% faster, but depth 4 is itself a slow setting for the standard pack, 1 to 22% slower than depth 3.

Against the default depth of 6, the leads are larger: K2 by 5 to 35%, K3 by 14 to 44%, K4 by 15 to 35%, and K5 by 9 to 39%.

At the same depth, Sushi decodes K2, K4 and K5 at or below the standard pack's speed, and K3 from 2% slower to 8% faster (Table 2). turboexl3's 2 to 22% on the same binary (Table 4) is what puts every pack level or ahead. K2 reads the fewest bytes per token, but it accepts fewer drafts: 3.13 tokens per round at depth 3, against the standard pack's 3.47 (Section 6.6).

Prefill (Table 6, Figures 5 and 6). From 8K tokens up, EXL3 prefill is within 7% of the standard configuration in either direction, and 5 to 6% behind at 256K. Below 8K tokens:

• K2 and K3 are 8 to 16% faster.
• K4 is 9% slower at 4K; its 1K and 2K runs are flagged as noisy.
• K5 is 11 to 21% slower.

The slower short-prompt cells add 0.1 to 0.3 seconds of time to first token. At 256K tokens, time to first token is 110 to 111 seconds for EXL3 and 104.5 for the standard pack.

Figure 4. Decode throughput with MTP on, by prompt length, at equal MTP depth: every pack and the standard pack at depth 3 (top); K5 on the final build against the standard pack at depths 4 and 6 (bottom).

Figure 5. Prefill throughput by prompt length.

Figure 6. Time to first token by prompt length.

Table 5. Decode throughput with MTP on, tokens per second, with each run's MTP depth.

| Pack | MTP depth | 4K | 16K | 64K | 256K | | --- | --- | --- | --- | --- | --- | | EXL3 K2 | 3 | 125.3 | 120.6 | 110.9 | 96.0 | | EXL3 K3 | 3 | 135.2 | 132.3 | 116.6 | 99.4 | | EXL3 K4 | 3 | 132.3 | 137.6 | 119.5 | 105.5 | | Standard Q4 | 3 | 120.9 | 117.5 | 109.7 | 95.6 | | EXL3 K5 | 4 | 138.2 | 124.2 | 113.6 | 95.2 | | Standard Q4 | 4 | 100.3 | 111.9 | 99.3 | 94.7 | | Standard Q4 | 6 (default) | 100.1 | 101.7 | 96.6 | 87.4 |

Table 6. Prefill throughput, tokens per second.

| Pack | 4K | 16K | 64K | 256K | | --- | --- | --- | --- | --- | | EXL3 K2 | 2555 | 2772 | 2680 | 2362 | | EXL3 K3 | 2498 | 2834 | 2660 | 2348 | | EXL3 K4 | 2116 | 2830 | 2639 | 2344 | | EXL3 K5 | 1987 | 2712 | 2551 | 2329 | | Standard Q4 | 2312 | 2770 | 2686 | 2481 |

6.4 Memory

Peak memory follows pack size (Table 7, Figure 9). Against the standard pack, at every prompt length:

• K2 peaks 29 to 31 GiB lower;
• K3 peaks 14 to 17 GiB lower;
• K4 peaks within 2 GiB of it;
• K5 peaks 13 to 16 GiB higher.

Table 7. Size on disk without the n-gram table, which streams from SSD, and peak server memory (phys_footprint, Metal allocations included).

| Pack | Size on disk | vs standard Q4 | Peak at 16K | Peak at 256K | | --- | --- | --- | --- | --- | | EXL3 K2 | 36.9 GB | −51% | 45.4 GiB | 55.6 GiB | | EXL3 K3 | 52.7 GB | −30% | 60.2 GiB | 70.4 GiB | | EXL3 K4 | 68.6 GB | −9% | 75.1 GiB | 84.8 GiB | | EXL3 K5 | 84.4 GB | +12% | 90.1 GiB | 100.1 GiB | | Standard Q4 | 75.3 GB | <br> | 74.2 GiB | 84.8 GiB |

Figure 9. Peak server memory by prompt length.

6.5 Coding accuracy

Table 8. EvalPlus 0.3.1, greedy pass@1, with problems solved in parentheses.

| Pack | HumanEval | HumanEval+ | MBPP | MBPP+ | Solved of 1,084 | | --- | --- | --- | --- | --- | --- | | EXL3 K2 | 0.970 (159) | 0.927 (152) | 0.958 (362) | 0.807 (305) | 978 | | EXL3 K3 | 0.982 (161) | 0.939 (154) | 0.950 (359) | 0.820 (310) | 984 | | EXL3 K4 | 0.988 (162) | 0.963 (158) | 0.958 (362) | 0.796 (301) | 983 | | EXL3 K5 | 0.988 (162) | 0.945 (155) | 0.955 (361) | 0.812 (307) | 985 | | Standard Q4 | 0.988 (162) | 0.945 (155) | 0.958 (362) | 0.815 (308) | 987 |

EvalPlus does not separate the packs. Every EXL3 pack is within two points of the standard pack on every suite. Their totals run from 978 to 985 problems, against the standard pack's 987.

• Largest gap: K4 on MBPP+, 301 against 308 problems (1.9 points). K4 leads by 1.8 points on HumanEval+.
• Significance: paired problem by problem against the standard pack, no difference is significant. The smallest exact McNemar p-value is 0.14.
• Ceiling: the benchmark is near its ceiling for every pack, with HumanEval between 0.97 and 0.99.

6.6 Speculative-decoding acceptance

mlx-serve drafts tokens with the model's MTP head and verifies them with the full model. A quantization that shifts the model's predictions therefore shows up as rejected drafts (Table 9, Figure 10). Both measures depend on the draft depth. Each later draft is conditioned on more speculative tokens, so as depth rises, per-draft acceptance falls and drafts accepted per round rise. Per-draft rates are comparable only at equal depth.

At depth 3, the standard Q4 pack accepts 86.1% of drafts and makes 3.47 tokens per round. K3 and K4 are level with it: 86.0% and 86.9% per draft, and 3.44 and 3.53 tokens per round. K2 accepts 72.9% and makes 3.13 tokens per round, the clearest effect of its higher KL divergence.

K5 ran at depth 4, where it accepts 81.6% per draft and makes 4.07 tokens per round. The standard pack at depth 4 accepts 84.9% per draft and makes 3.71 tokens per round.

Tokens per round also depend on how many drafts mlx-serve's planner makes each round, up to the depth, which it chooses from measured round costs. At depth 4 the planner made 3.75 drafts per round on average for K5, and 3.21 for the standard pack.

At mlx-serve's default depth of 6, the standard pack accepts 81.5% per draft and makes 4.13 tokens per round. That is more tokens per round than at depth 3, yet the pack decodes 7 to 30% slower (Section 6.3), so each deeper round takes longer than its extra tokens repay.

Figure 10. MTP draft acceptance per pack: per-draft rate by prompt length, and the mean over the sweep with drafts accepted per round.

Table 9. MTP draft acceptance over the nine prompt lengths, by draft depth: mean per-draft rate with its range, and mean tokens per verification round (accepted drafts plus the token the full model adds).

| Pack | MTP depth | Per-draft acceptance | Tokens per round | | --- | --- | --- | --- | | EXL3 K2 | 3 | 72.9% (68.7–76.7) | 3.13 | | EXL3 K3 | 3 | 86.0% (81.6–89.0) | 3.44 | | EXL3 K4 | 3 | 86.9% (82.4–90.2) | 3.53 | | Standard Q4 | 3 | 86.1% (81.2–89.6) | 3.47 | | EXL3 K5 | 4 | 81.6% (77.1–85.6) | 4.07 | | Standard Q4 | 4 | 84.9% (79.7–90.5) | 3.71 | | Standard Q4 | 6 (default) | 81.5% (72.7–86.8) | 4.13 |

6.7 Two trades

Option one: Q4 size, more accuracy. K4's pack is 68.6 GB on disk, 9% smaller than the standard Q4 pack, and its peak memory is within 2 GiB of the standard pack's at every length (Table 7). At that size:

• its KL divergence is 35% below NVFP4 W4A16's and 59% below that of the 4-bit GGUF quant;
• its perplexity is the closest to BF16's of any format near 4 bits (Table 1);
• at the same MTP depth, it decodes 5 to 17% faster than the standard configuration (Table 5).

EvalPlus cannot show the accuracy gain. K4 solves 983 problems against the standard pack's 987, a difference within the noise of one run (Section 6.5).

Option two: Q4 accuracy at K=3. K3's KL divergence is within 8% of the 4-bit GGUF quant's with three-quarters of the bits, and 29% below NVFP4 W4A4's (Table 1). On EvalPlus, K3 solves 984 of 1,084 problems against 987 for the standard Q4 pack, within 0.8 points on every suite (Table 8).

The K3 pack is 30% smaller on disk (52.7 GB against 75.3 GB), and its peak memory is 14 to 17 GiB lower at every prompt length. On a 128 GB machine, that is room for a longer context or a second model.

Figure 11. EvalPlus problems solved (of 1,084) against size on disk, for the four EXL3 packs and the standard Q4 pack.

7. Discussion

Block-scaled 4-bit is the default for expert weights on Macs because it is cheap to decode, not because it is accurate per byte (Section 2). The more accurate format cost speed (Section 3): prefill on long prompts, and decode at equal speculative-decoding depth. turboexl3 removes most of that cost (Sections 4 and 6):

• prefill is within 7% from 8K tokens up;
• at the same depth, K3, K4 and K5 decode faster than the standard configuration at every length, and K2 matches it.

The standard configuration also leaves decode speed unused on this model. At depth 3 instead of its default of 6, the standard pack decodes 7 to 30% faster on these prompts.

The format can then be chosen on accuracy per byte, where EXL3 is ahead at every rate measured. Which trade to take depends on what limits the deployment: K4 when accuracy does, K3 when memory does.

The NVFP4 comparison is the most direct one for Macs. On Blackwell, NVFP4 gives up accuracy for FP4 arithmetic when it quantizes the activations. On Apple GPUs, it runs with 16-bit activations and offers only its size. At that size, the 4.05-bit EXL3 pack has 35% lower KL divergence, with 0.7 fewer bits per weight.

What EXL3 still costs, from Section 6:

• slower prefill below 8K tokens for K4 and K5, adding 0.1 to 0.3 seconds of time to first token;
• about 5% slower prefill at 256K tokens;
• 1 to 2% slower decode than the exact engine with MTP off.

I think models for M5-class Macs should move to full EXL3: 4.05 bits where accuracy is the priority, 3.05 bits where memory is. Block-scaled 4-bit remains the choice for Apple GPUs that turboexl3 hasn't been measured on.

8. Limitations

• Scope: one machine, one model, one run per cell.
• Speed comparisons with the standard configuration are between complete configurations on different builds. The standard pack was measured at its default MTP depth of 6 and at the EXL3 runs' depths of 3 and 4. Only the turboexl3–Sushi comparison isolates the kernels.
• Quality data is turboderp's, measured with exllamav3 rather than through mlx-serve. MXFP4 and MLX affine 4-bit have no KL divergence measurement on this model. ddalcu measured the standard pack at a median of 0.0052 over 290 positions, on the top 1,024 tokens. His method differs from turboderp's, so that figure is not comparable with Table 1.
• Hardware: turboexl3's prefill path requires the M5 generation's neural accelerators. Older Apple GPUs were not measured.
• Task accuracy covers coding only.

9. Conclusion

EXL3 gives more accuracy per byte than the block-scaled 4-bit formats that Mac models ship their experts in. That includes NVFP4 in the 16-bit-activation form it takes on Apple GPUs. Its obstacle was speed.

turboexl3 decodes EXL3 weights inside fused kernels on the M5's GPU and neural accelerators. With it, EXL3 prefills within 7% of the standard MLX path from 8K tokens up. At 3 and 4 bits, it decodes faster than the standard path at every context length, at the same speculative-decoding depth. Full EXL3 should be the default for models shipped to M5-class Macs.

Acknowledgments

turboderp created the EXL3 format and exllamav3, published the Qwen3.8-Flash-Next quants used here, and measured their KL divergence and perplexity. ddalcu maintains mlx-serve and published the baseline pack. Sushi provided the exact EXL3 reference engine.

Availability

• turboexl3, with full result tables and charts: https://github.com/davidtai/turboexl3
• EXL3 packs for mlx-serve: https://huggingface.co/OpensourceWTF
• turboderp's EXL3 quants and quality charts: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
• mlx-serve: https://github.com/ddalcu/mlx-serve

References

1. A. Tseng, Q. Sun, D. Hou, C. De Sa. QTIP: Quantization with Trellises and Incoherence Processing. NeurIPS 2024.
2. turboderp. exllamav3 and the EXL3 format; Qwen3.8-Flash-Next EXL3 quants and measurements. https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
3. Open Compute Project. OCP Microscaling Formats (MX) Specification, v1.0. 2023.
4. NVIDIA. NVFP4, a 4-bit floating-point format with FP8 block scales, introduced with the Blackwell architecture. 2025.
5. ddalcu. mlx-serve. https://github.com/ddalcu/mlx-serve
6. J. Liu, C. S. Xia, Y. Wang, L. Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (EvalPlus). NeurIPS 2023.
7. Apple. MLX: an array framework for Apple silicon. https://github.com/ml-explore/mlx

TheDavidTai

@TheDavidTai

Founder, hacker, stochastic tool. @ microsoft escapee. Currently @ eigenlabs. Checkout my github actions emulator: [notdownhub.com](https://t.co/TPIOeqH6U7 "https://notdownhub.com")