Can we make LLMs generate faster than their current speed?

Question

The problem

Most AI models write text one word at a time.

That keeps answers good, but makes them slow.

Some models try to write everything at once to be faster.

But then:

• they lose memory efficiency

• the text feels less connected

• compute cost goes up

So currently it’s fast vs good, not both.

The solution: ReFusion

ReFusion doesn’t work word by word.

It works in small chunks of text, called slots.

How it works:

• First, the model plans which chunks can be written together

• Then it writes those chunks in parallel

For example: Sentence to write:

“The cat sat on the mat.”

Step 1: Plan slots

The model decides:

Slot 1: “The cat”

Slot 2: “sat on”

Slot 3: “the mat”

Step 2: Write slots in parallel

All three are generated at the same time:

Slot 1 → “The cat”

Slot 2 → “sat on”

Slot 3 → “the mat”

Step 3: Combine

Final sentence:

“The cat sat on the mat.”

The result

• Faster than the strong models we use today

• Quality stays almost the same

Do you think decoding speed and not model size might be the next big frontier?