In typical ReAct deployments, LLM agents spend an excessive amount of runtime doing nothing. While an agent waits for external tools, databases, or environment APIs to return execution feedback, reasoning locks up entirely. A research team from Singapore Management University—Zhensu Sun, Chengran Yang, Yunbo Lyu, Jieke Shi, and David Lo—has tackled this bottleneck with Second Thought, an inference framework designed to exploit idle I/O windows for asynchronous speculative reasoning.
The system is completely training-free, avoiding the operational overhead and compute cost of fine-tuning model weights. Once an agent finishes its initial Thought phase, Second Thought immediately forks decoding into four auxiliary branches that run off the main sequential trajectory. When the environment observation finally arrives, the runtime truncates auxiliary streams and blends valid precomputed reasoning directly back into subsequent turns.
Empirical benchmarks across three evaluation suites and three reasoning LLMs show that speculative offloading pays off. Second Thought reduced main-thread sequential decoding by up to 43% in six out of nine benchmark pairings, dropped total turn counts across all nine, and trimmed median per-task wall-clock latency by 10.9%. Along the way, it posted Pass@1 accuracy gains of +12.4 and +10.2 percentage points in select configurations. For engineering teams operating autonomous pipelines at scale, converting I/O wait states into preemptive inference offers a practical throughput boost without altering underlying model architectures.