It is currently early morning, Sunday, January 11, 2026.
Yesterday, I had lunch with Hobo. It had been a long time since we last met, and we talked a lot over the meal. He was very concerned about our recent situation and work. We exchanged many thoughts.
I envy that he, working in a foreign company, can use LLMs like GPT and Claude Opus without limits for work assistance and efficiency improvement. In contrast, in our domestic work environment, there are still many restrictions and inconveniences in using these tools.
Our consensus is that, in current coding work, code written by LLMs cannot go directly into the production environment—it is very, very unreliable.
Observability
I asked him, if we limit it to a single module and it passes strict unit tests and benchmark tests, can it be used? He added that excellent observability is also needed, as long-term service stability must be considered. Furthermore, the cost of splitting a large system into many such well-defined small modules is inherently very high.
This is indeed a point I had previously overlooked. In this article, I mentioned that after passing interface style tests, unit tests, and benchmark tests, humans can trust LLM-generated code. I once constructed a benchmark test where, considering CPU and memory usage, ordinary benchmark tests alone could not detect performance issues in LLM-generated code. You must apply stress tests in advance to uncover problems. Stress tests also cannot truly simulate the various complex scenarios in a production environment, so ultimately, excellent observability is required to use LLM-generated code in production.
But how should observability be designed and tested?
Observability itself is a tool for testing whether actual conditions meet expectations, but it operates in the production environment rather than the testing environment.
Moreover, it may need to intrude into the implementation code to collect sufficient information. (Intrusive instrumentation typically implies higher maintenance costs.)
If we simply collect some metric data from outside the interface and environmental information, we can often only uncover a portion of the issues. For example, we cannot observe whether the internal state of the module is correct, whether it is consuming excessive resources, whether there are memory leaks, deadlocks, etc.
Furthermore, observability metrics are often related to resource isolation, such as CPU, memory, I/O, etc. Without excellent resource isolation, it is often difficult to detect issues.
Additionally, the key to observability lies in the alerting system. Hobo once mentioned, "Every instrumentation metric implies it should have a corresponding alerting rule; otherwise, the instrumentation is meaningless."
In practice, alerting rules are typically an operations task, while instrumentation is a development task. Perhaps this practice itself is problematic. Why don't we consider embedding alerting rules directly in the code?
For example, each instrumentation point could carry a definition of an alerting rule, automatically triggering an alert when a metric exceeds a certain threshold. This way, developers can directly consider observability and alerting rules while writing code, thereby improving code quality and reliability.
For instance, while instrumenting, we could also design an assertion mechanism where if an assertion fails, an alert is triggered. This seems similar to an error/warning mechanism. Does logging an error/warning imply it needs attention?
We could start with logs, focusing on recording error/warning logs, treating these logs as part of observability, and combining them with the alerting system to enhance system reliability.
I strongly agree with Hobo's viewpoint: Code deployed to production must have excellent observability; otherwise, long-term stability cannot be guaranteed.
LLM Intelligence Level vs. Engineering Methods
Additionally, Hobo mentioned the issue of how the LLM's own intelligence level affects coding quality. Here, we have some disagreement.
He believes that the LLM's intelligence level is the key factor determining coding quality—insufficient intelligence means the task cannot be completed. I believe that while the LLM's intelligence level is important, the more critical aspect is how to design tasks and testing processes well to ensure the generated code meets expectations.
Hobo leans toward elite capability, a talentist viewpoint; whereas I lean more toward system optimization, a constructivist viewpoint.
Both are correct, but at different stages.
Below the model capability threshold, I am absolutely correct. For the vast majority of current commercial applications, the value of engineering methods far outweighs waiting for the next "smarter" model. A well-designed prompt chain, a comprehensive test suite, and an iterative process can completely enable a moderately capable model to produce stable, usable code. This is the mainstream and successful path for current AI application deployment.
When facing true cognitive limits, Hobo's viewpoint becomes evident. When task complexity reaches a level requiring genuine understanding, abstraction, and innovation (e.g., designing a brand-new algorithm or understanding an extremely vague, contradictory requirement), the model's "intelligence ceiling" becomes an insurmountable obstacle. At this point, no process, no matter how good, can make the model accomplish something it "cognitively cannot do."
I represent the "pragmatism of the engineer," the core driving force for creating value with AI today. Hobo represents the "foresight of the researcher," focusing on future breakthroughs in capability.
The ideal state is "elite-level intelligence" combined with "elite-level engineering methods."
Using the best processes to stimulate and harness the most powerful intelligence. Our disagreement is not about right or wrong but about focus (current optimization vs. fundamental breakthrough) and time scale (short-term deployment vs. long-term evolution).
In a team, this complementary perspective is extremely valuable.
