Using 500,000 Real-World Calls, I Cut Daily Network Waiting Time from 3 Hours to 3 Minutes

2026-09-08

Data source: 513,336 production calls from 2026-07-16 to 2026-09-08

I. The Bottom Line

If you don't have time to read the whole thing, here's the key takeaway:

Based on data from 500,000 real API calls, I reduced daily network transfer wait time from approximately 3 hours to 3 minutes.

How? — By separating inference from tool execution, keeping high-volume traffic on the internal network and sending only low-volume traffic across the international link.

Here's the complete process and the real data.

II. Background: Why the Old Architecture Was Slow

Initially, my primary machine was deployed in mainland China, running CodeX. CodeX is a complete AI application development framework responsible for calling the OpenAI API, managing conversation context, executing tool calls, and more—all on a single machine.

Its network topology looked like this:

flowchart LR
    A[CN Primary Machine<br>CodeX] -->|Public network request<br>VPN + Cloudflare| B[OpenAI API]
    B -->|Inference response<br>~794 KB / call| A

Every inference request had to travel from China through a VPN and Cloudflare, cross the Pacific to reach OpenAI, and then return with the result. This network path had several unfavorable characteristics:

• Bandwidth: 10 Mbps (actual effective throughput of approximately 0.75 MB/s)
• Latency: Approximately 300 ms RTT, with frequent fluctuations
• Stability: Occasional packet loss and retransmissions

Under this architecture, with 10,000 calls per day, simply waiting for network data transfers took approximately 3 hours.

During those 3 hours, CodeX wasn't doing anything—it was just waiting for the network.

III. The Data Speaks

To understand exactly where those “3 hours” were going, I retrieved all access logs from the old architecture between July 16 and September 8, 2026—a total of 513,336 valid calls. This sample size was large enough to clearly demonstrate the issue.

I analyzed the traffic composition of a single call:

The two groups differed by 58×. Inference traffic accounted for 98.3% of total traffic, while tool control signals accounted for only 1.7%.

Then I calculated the time cost:

Inference traffic crossing the international link:

7.94 GB = 7,940 MB ÷ 0.75 MB/s ≈ 10,587 seconds ≈ 2.94 hours

Tool control traffic crossing the international link (if transmitted separately):

0.14 GB = 140 MB ÷ 0.75 MB/s ≈ 187 seconds ≈ 3 minutes

The problem was clear: nearly 3 hours every day were spent transmitting inference traffic, even though this data should never have needed to cross the international link.

IV. The Solution: From an “All-in-One” System to a Distributed Deployment

Since the problem was that “everything ran on the primary machine in China, so all traffic had to cross the international link,” the solution was to split the workload.

New Architecture: Inference in AWS, Tools in China

I separated the inference capabilities from CodeX and deployed them to AWS US:

flowchart LR
    subgraph CN["🇨🇳 CN Primary Machine"]
        T[Tool execution environment]
        C[HTTP/2 client]
    end

    subgraph AWS["☁️ AWS US"]
        R[Inference device<br>Rust + SQLite]
        O[OpenAI API]
    end

    C -->|Tool parameters downstream<br>591.59 Bytes| R
    C -->|Tool results upstream<br>13,387.34 Bytes| R
    R <-->|Inference traffic<br>7.94 GB/day via internal network| O

The key changes in this new architecture were:

1. The inference device—a lightweight Rust server with SQLite—was deployed in AWS US, in the same region as OpenAI.
2. Traffic between the inference device and OpenAI uses the AWS internal backbone: latency below 10 ms and bandwidth at the Gbps level.
3. The primary machine in China no longer calls OpenAI directly. It is responsible for only two tasks:
   • Receiving tool call parameters pushed by the inference device (591.59 Bytes)
   • Sending the results (13,387.34 Bytes) back to the inference device after executing the tools
4. SQLite maintains the conversation context on the inference device, while the primary machine in China remains stateless.

Here's a comparison of the international link traffic in the old and new architectures:

V. Final Results

VI. Data Reliability Notes

• Statistical period: 2026-07-16 16:50 to 2026-09-08 09:51 (China Standard Time)
• Total sample size: 513,336 calls; complete production logs, not a sample
• Traffic measurement: Based on the actual sizes of HTTP request and response bodies
• Tool call parameters: 591.59 Bytes / call
• Tool call responses: 13,387.34 Bytes / call
• Bandwidth estimate: Calculated using a measured effective throughput of 0.75 MB/s (rather than the theoretical value of 1.25 MB/s)
• RTT estimate: Calculated using the measured value of 300 ms, including the combined overhead of the VPN and Cloudflare

All original logs can be exported for verification.

VII. Summary

I didn't upgrade the bandwidth, switch to a more powerful machine, or make OpenAI run faster.

I did just one thing: separate inference from tool execution.

• Inference (high-volume traffic) stays in AWS US, next door to OpenAI
• Tool execution (lightweight control signals) stays on the primary machine in China

The result: 3 hours of network waiting every day became 3 minutes.

Data-driven architecture optimization involves no magic—just tangible numbers.

Data source: Production inference device access logs, with a sample size of 513,336 calls.
Statistical period: 2026-07-16 16:50 to 2026-09-08 09:51 (CST).