Infer-forge: Harness, Loop, and Graph Engineering Around SGLang

1. Introduction

Inference optimization may look local in code, but its validity is global. A kernel, communication path, or scheduling change becomes meaningful only at a specific deployment point defined by the model, workload, SLO, serving topology, runtime version, and accelerator platform. The same patch may improve one point and regress another. Agent-assisted exploration can produce more environments, experiments, measurements, and rejected paths, all of which must remain reproducible.

The first requirement is therefore reliable execution. Reproducing a deployment point requires more than model capability: tools, environments, context, memory, Verification, and safety boundaries must remain stable. Harness Engineering turns those surrounding conditions into a reproducible and inspectable execution system—the basis for the abstraction Agent = Model + Harness^1,2,3,6.

Reliable execution must then remain coherent over time. Inference engineering Tasks often span repeated rounds of investigation, implementation, deployment, Evaluation, failure, and recovery. Loop Engineering connects successive executions so that one Task can preserve its Task Contract, incorporate new evidence, and satisfy its Exit Criteria by producing either a verified Deliverable or a reliable Follow-up Handoff^4,5,7.

Project-scale work exceeds the boundary of one Task. Multiple Tasks must proceed in parallel, exchange Deliverables, share state, trigger Rework, and change direction as evidence accumulates. Graph Engineering organizes independently convergent Task Loops into an evolving Task Graph. The graph keeps released and rejected paths connected to their dependencies, constraints, and evidence, so project decisions remain explainable^10,11,12,13.

Infer-forge applies this progression to inference engineering around SGLang. Its scope follows an engineering change through the inference stack: from kernels and communication libraries, through engine integration and deployment, to Evaluation and online diagnosis. One shared workspace and three accumulating execution structures keep that end-to-end path coherent:

• MonoRepo establishes a reproducible workspace for cross-repository engineering.
• Harness supplies reusable execution capabilities, memory, Verification, and safety boundaries.
• Task Loop keeps one long-running Task moving until it reaches its Exit Criteria.
• Task Graph connects independently convergent Tasks into larger objectives, including project delivery and capability evolution.

1.1 Implementation Status and Availability

Infer-forge is an internal engineering system developed independently around SGLang; it is not an official SGLang or LMSYS component.

Scope	Current status
MonoRepo workspace, Task and Journal records, and Task/Graph schemas, CLIs, and validators	Implemented and used internally
Task Loop lifecycle transitions and Handoffs	Partially automated and tool-supported
Task generation and adaptive Task Graph evolution	Implemented and used internally; infer-forge derives Tasks from user goals, connects them into a Task Graph, and updates the Graph during execution as Tasks are accepted, abandoned, or redirected
Capability trigger inputs and lifecycle operations	Implemented and explicitly invoked
Task acceptance or abandonment, Human Gates, and release decisions	Human-directed; these decisions feed back into Task Graph evolution

Infer-forge is not currently open source because its core components are deeply coupled to our internal repositories, infrastructure, workflows, and safety controls. The current codebase would therefore have limited portability outside our environment.

Instead, this article makes the construction methodology public. Our goal is for teams and individuals to give it to an AI coding tool, add the context of their own systems, and rapidly build an infer-forge implementation adapted to their environment.

Infer-forge has moved from workflow design into sustained engineering use. Across one engineer's April–July record, the observed peak number of Tasks in flight rose from 2 to 9. In one DeepSeek-V4-Pro serving project, 38 independently verifiable Task nodes across seven Task Types were coordinated as a Task Graph. Together, these records show infer-forge in sustained use across both a four-month engineering record and a project-scale Task Graph.

A capable Agent can make one execution succeed; an engineering system is designed to make successful work reproducible. Infer-forge does not promise that every Task will finish faster. It provides the structure to preserve the provenance of each deployment point, sustain verifiable work over long-running Tasks, and coordinate evidence across Task boundaries.

2. Inference as a Deployment Space

Figure 1 turns the deployment point introduced above into a concrete chain of constraints. The Model determines the supported modalities and model-specific execution paths. The Serving Scenario translates Modality and Traffic Shape into an SLO. That SLO constrains the Serving Topology, where a Deployment Architecture is combined with a Parallelism strategy. The topology is then realized through a Versioned Runtime Profile that pins the engine configuration and container image for a particular service revision. Finally, the complete runtime must be built and verified on an Accelerator Platform. A deployment point is the full path through this chain—not any one layer in isolation.

There is no context-free inference optimization. Different serving objectives and SLOs can require fundamentally different deployment paths, including Colocated PD, PD Disaggregation, and EPD Disaggregation. Each architecture changes stage boundaries, communication paths, resource balance, and the set of feasible Parallelism strategies. Those decisions propagate into the runtime profile and accelerator-specific implementation that must be verified. A kernel improvement becomes a serving result only when the complete deployment point reproduces it and passes throughput, latency, correctness, and stability gates. Without its deployment point, a performance claim cannot be reproduced, compared, or carried forward.

Infer-forge does not eliminate this combinatorial space; it makes every movement through it explicit and verifiable. Instead of asking an Agent to “optimize DeepSeek-V4-Pro,” we define a Task that records the current deployment point, bounds the subset of dimensions it may change, and fixes the Verification gates before execution begins. A Task might replace the MoE backend while holding the serving scenario, topology, and accelerator constant, then produce either an accepted improvement or a documented rejection. Both outcomes reduce uncertainty for the next Task. Before any of them can be reproduced, however, the exact cross-repository code state behind the deployment point must be fixed. That is the role of the MonoRepo.

3. MonoRepo

3.1 Why a MonoRepo

Inference optimization crosses repository boundaries, but it must ship as one coherent system. A change may begin in a kernel library, depend on a communication backend, enter SGLang through engine integration, and finally require a matching deployment configuration. When these repositories live in separate workspaces, their relationships become transient knowledge that engineers and Agents must repeatedly reconstruct. One missing branch or incompatible revision is enough to invalidate the result.

Infer-forge turns that dependency map into a shared workspace. Git submodules place the relevant repositories under one root while preserving their independent histories, branch policies, access controls, and release processes. The root gives engineers and Agents a stable map of the inference stack and one entry point from which cross-repository work can be developed and verified.

This structure changes cross-repository work in three ways:

• The whole stack stays in view. Repository boundaries remain explicit, but their engineering relationships are visible from one workspace.
• Context stays bounded. An Agent can navigate the full repository map while loading only the repositories required by the current Task.
• Integration starts in place. Related branches can be developed, combined, and verified without repeatedly locating repositories or rebuilding their relationships from memory.

The workspace coordinates change; the Task record makes it durable. Each Task captures the branches, commits, and execution state used for the work, and the Journal archives that record after completion. The repositories can continue evolving without erasing the provenance of work that has already been verified and delivered.

3.2 Repository Map

One workspace does not mean one undifferentiated codebase. Infer-forge separates repositories by the role they play in engineering: the Built-in Workspace coordinates cross-repository work, the Inference Stack Repos contain the serving system being changed, and the Harness Repos carry those changes from execution to verified evidence.

Built-in Workspace

The Built-in Workspace is the coordination layer of infer-forge, not another implementation repository. Code remains in the repository that owns it. The root contains only the mechanisms that need to operate across repository boundaries:

• Task System materializes the workspace required by a Task: the relevant repositories, environment entry points, record locations, and—when isolation is required—a dedicated worktree and Agent session.
• Cross-lib Management makes the repository graph operable. It maintains repository locations, default branches, and the operations used to synchronize and integrate changes. The actual branches and commits used for a piece of work belong to the Task record and are archived into the Journal.
• Skills expose capabilities at the scope where they belong. The root provides Skills for Task lifecycle and cross-repository coordination; each repository retains the Skills specific to its own domain.

Inference Stack Repos

Inference Stack Repos are where serving behavior and performance actually change. SGLang is the center of the stack, while Dynamo organizes SGLang instances into a distributed service. DeepGEMM, FlashMLA, FlashInfer, and Humming provide specialized compute kernels. DeepEP and Mooncake provide expert-parallel communication and cross-node KV transfer. These repositories evolve independently, but one serving result may depend on changes across several of them at once.

Harness Repos

Code does not become an engineering result merely because an Agent can edit it. Harness Repos provide the capabilities that carry a change through the rest of its lifecycle: finding compute resources, preparing environments, deploying the service, running performance and correctness Evaluation, diagnosing failures and online behavior, preserving long-term records, and enforcing safety boundaries. They turn otherwise disconnected operational steps into a repeatable path from code change to verified Deliverable.

Together, the three groups form one engineering path: the Built-in Workspace prepares and coordinates the work, the Inference Stack Repos supply the system under change, and the Harness Repos carry that change to Verification. Infer-forge brings them into one workspace without forcing them into one repository history, ownership model, or release process.

4. Task Loop

The MonoRepo provides the workspace; the Task Loop structures how work advances over time. Inference engineering often requires multiple rounds of Research, implementation, deployment, Evaluation, and recovery. The Task Loop keeps those executions aligned with one Goal and Task Contract until they satisfy their Exit Criteria through a verified Deliverable or a reliable Follow-up Handoff.

“Everything Can Be a Task” applies to independently verifiable units of work, not to every action. A Task needs its own Goal, Scope, Acceptance, Verification path, and Exit Criteria. Commands and intermediate experiments remain inside it as Loop Block instances or tool calls. Once a unit of work can complete or hand off independently, it can become a Task node in a larger Task Graph.

4.1 Overview

A Task Loop keeps its boundary stable while allowing its execution path to adapt. Task Definition establishes the Task Type, Starting Context, Task Contract, and Exit Criteria. The Main Loop advances the work through a sequence of Loop Block instances and uses Task Goal Met? to decide whether to exit or continue. Task Memory preserves the current state, next Sub-target, Execution Records, and Handoff between iterations.

The Harness surrounds the loop rather than acting as another stage inside it. It supplies the resources, methods, memory systems, Verification entry points, and safety boundaries required during execution. Different Tasks can use those capabilities in different orders without being forced through one fixed pipeline.

4.2 Task Definition

A Task Loop begins with a testable commitment, not with activity. Before an agent searches the codebase, launches a deployment, or consumes evaluation capacity, the Task must state what it is trying to accomplish and how the result will be judged. Task Definition establishes four anchors:

• Task Type selects a default Playbook.
• Starting Context records the state from which execution begins.
• Task Contract fixes the Goal, Scope, Acceptance, and Verification.
• Exit Criteria define the durable Deliverable or Follow-up Handoff that execution must produce.

4.2.1 Task Type

A Playbook turns prior practice into a head start. For each Task Type, it provides a lightweight, adaptable template: which repositories are likely to matter, which Skills, Tools & CLI, and Verification entry points are relevant, and which safety boundaries apply. This reduces the space the agent must search before useful work can begin.

A Playbook standardizes the starting approach, not the complete execution path. The agent may change methods, introduce new Loop Block instances, or use capabilities outside the template as the Task Contract and new evidence require. It reduces repeated discovery without replacing engineering judgment.

Each Task Type captures a distinct unit of verifiable work in inference engineering:

• Plan defines objectives, constraints, and decomposition, then delivers an executable plan for downstream Tasks.
• Research investigates a bounded uncertainty, evaluates available evidence, and delivers conclusions with explicit limitations.
• Code implements any scoped code change and delivers the change together with its Verification evidence.
• Integration assembles changes from multiple upstream Tasks, including across repositories or components, and verifies the combined system.
• Evaluation runs functional, performance, accuracy, stress, and stability assessments and delivers evidence against Acceptance.
• Release advances a verified candidate through approval, canary rollout, expansion, or rollback and records the outcome.
• Online Diagnosis investigates a production issue within read-only boundaries and hands required changes to downstream Tasks.
• Capability maintains the versioned Skills & Tools set through Add, Update, Merge, Retire, or No Change.
• + Custom handles verifiable work without a reusable Playbook while preserving the same Task Definition.

The taxonomy is earned through repeated practice. An activity becomes a built-in Task Type only after its starting conventions, Acceptance boundary, and Deliverable are stable enough to guide future work. Standardization follows proven practice; it does not attempt to predict every path in advance.

4.2.2 Starting Context

Starting Context draws the line between known state and assumption. A Task may begin from a blank state, a prepared environment, or an upstream result. What matters is that the source and verification status of that state remain explicit.

• Custom Setup records conditions prepared for the current Task, such as the model, container image, deployment profile, repository versions, dataset, or experiment entry point.
• Imported Context carries forward an upstream environment, intermediate result, or Follow-up Handoff, together with its provenance and verification status.

4.2.3 Task Contract

The Task Contract holds the target still while execution adapts. Goal states the intended outcome, Scope bounds the work, Acceptance states the observable conditions for success, and Verification defines the evidence required to judge them⁶. Without this boundary, an agent can appear to succeed simply by changing the problem after seeing the result.

For an inference optimization Task, the contract may fix the model and weight format, hardware placement, serving topology, runtime versions, workload grid, and baseline. Acceptance can then require better throughput, TTFT, or TPOT without violating accuracy or stability. Changing the topology, GPU class, or workload after observing the result is a contract change—not an optimization result.

Evidence may redirect the Main Loop, but it must not silently move the finish line. A clarification that does not materially change the Goal, Scope, or Acceptance must be explicit and recorded; a material change to any of them starts a new Task.

4.2.4 Exit Criteria

A Task does not end because activity stops. It ends when its state can be verified or safely continued.

• A Deliverable packages the completed result with enough evidence to verify the Task Contract.
• A Follow-up Handoff preserves completed work, current state, unresolved questions, and the next entry point when another Task must continue the work.

Exit Criteria turn execution into durable engineering state: either a verifiable result or a reliable starting point for the next Task.

4.3 Loop Execution

A Task is sized by its engineering objective, not by the duration of a runtime session. The Task Contract defines the complete objective; the Main Loop realizes it through one or more Loop Block instances as evidence emerges. When defining a Task, the user does not need to predict whether it can finish within one sustained runtime loop.

The Task is the unit of continuity; the Loop Block is the unit of execution. Each Loop Block owns one Sub-target, one Exit Condition, one routing decision, and one set of Execution Records. Claude Code /loop and Codex /goal are current execution entry points, but neither determines the scope of the Task.

4.3.1 Main Loop

The Main Loop turns an open-ended Task into a sequence of evidence-producing Loop Blocks:

• Define Loop Block selects the current Sub-target and Exit Condition.
• Execution Routing selects the Model Tier and Agent Topology for that Sub-target.
• Execute Loop Block runs the block through an available sustained-execution entry point.
• Task Goal Met? compares the accumulated state with the Task Contract and decides whether to exit or define another block.

Completing a Loop Block is not the same as completing the Task. A block may confirm a hypothesis, reject an approach, or expose a new constraint. Its Exit Condition closes that local unit of work; Task Goal Met? determines whether the accumulated evidence satisfies the whole Task Contract. The Yes branch proceeds to Exit Criteria, while the No branch defines the next Loop Block from the evidence already produced.

4.3.2 Task Memory

Task Memory makes the Loop Block sequence durable and traceable. It preserves completed Loop Blocks and the active Current Loop Block, including its Sub-target and Exit Condition. A Next Loop Block is recorded only after the current block ends and Task Goal Met? determines that the Task must continue.

Execution Records capture what the agent did at each step and what result it produced. They make the execution auditable, traceable, and reproducible.

A Loop Block Handoff transfers state inside one Task; a graph-level Handoff edge connects independent Task node instances.

4.3.3 Execution Routing

Execution Routing makes two orthogonal choices for the current Loop Block:

• Model Tier follows reasoning difficulty. Uncertainty, reasoning depth, and required expertise determine whether the block uses Lower-tier, Mid-tier, or Strongest Available.
• Agent Topology follows content volume and expected context length. Work that fits reliably within one context uses Single Agent; work that must be divided across contexts uses Multi-Agent¹⁴.

A difficult but compact problem may use Strongest Available with Single Agent. A large but routine evaluation matrix may use Lower-tier or Mid-tier with Multi-Agent. Model Tier supplies reasoning capability; Agent Topology manages context load.

Figure 5 shows one possible Multi-Agent topology: a Coordinator works with Infra, Code, and Eval, while Reviewer examines the result outside the main execution chain⁷. The roles are illustrative; the actual decomposition follows the content and context requirements of the current Loop Block.

If model-tier selection or subagents are unavailable, the Loop Block proceeds with the capabilities provided by the runtime, with the fallback preserved in its Execution Records.

4.4 Harness

A Task Loop can sustain progress only if the Harness can sustain execution. Without a stable Harness, every Loop Block must rediscover machines, reconstruct environments, locate commands, recover evidence, and renegotiate safety boundaries. Infer-forge instead provides one execution substrate built from Node Registry, Skills & Tools, Journal, and Safety Guard.

The value lies in composition, not in any capability alone. After Task Definition, an agent can inspect resource state, prepare an environment, deploy a workload, modify and evaluate the system, and recover on another Node when necessary. Node state, operational methods, execution history, and safety constraints remain connected throughout the process, so a human does not have to rebuild the path between steps.

4.4.1 Node Registry

Resource autonomy must begin with provable ownership, not guesswork. The Node Registry is a Git-backed ledger that records every Task-to-Node claim. Collectors periodically write runtime-process and GPU-activity observations into the Registry, creating a continuous, auditable history. GPU telemetry is one signal; no single sample can establish that a Task has finished.

Cleanup fails closed. A claim becomes eligible only when fresh, gap-free observations show both runtime and GPU activity continuously idle for a configurable policy window. Missing, stale, gapped, or conflicting evidence blocks cleanup; runtime-active/GPU-idle Nodes require review.

Eligibility is not execution, and cleanup is not reuse. Cleanup must be triggered explicitly and removes only the Registry claim. It does not stop the workload or establish Task completion. Before a Node is reused, its current machine state must be validated again.

The Node Registry is not a scheduler and does not launch workloads. Its role is narrower—and foundational: maintain an auditable resource state on which cleanup, allocation, and deployment decisions can rely.

4.4.2 Skills & Tools

A capable agent should not rediscover how to perform the same engineering work for every Task. Skills preserve reusable approaches across SGLang Upstream, Cross-lib, Task, and Ops. Tools & CLI expose stable, recordable interfaces for deployment, build, weight and code synchronization, Evaluation, profiling, online diagnosis, and monitoring.

A Playbook selects the relevant subset for the current Task Type. Skills narrow the decision space; Tools & CLI turn the selected decision into an action whose inputs and outputs can enter Execution Records. The agent starts from accumulated practice without being forced through a fixed path.

Skills & Tools is a versioned capability baseline, not a growing pile of instructions. Ordinary Tasks use the current baseline. A Capability Task uses Journal evidence and revalidation after a Model or runtime change to decide what should be added, updated, merged, retired, or left unchanged.

4.4.3 Journal

Evidence compounds only when the next Task can find and reuse it. Task Memory preserves the execution state of one Task; the Journal carries evidence across Tasks. LLM-wiki connects Task records into a knowledge network, while Multi-dim Index organizes them by dimensions such as model, Task Type, and GPU. Together, they support Retrieve, Compare, and Filter without forcing each Task to rediscover the same facts.

In an earlier version of SGLang's OpenAI-compatible chat benchmark handler, streamed delta.reasoning_content was not included in TTFT and output accounting (fixed in sgl-project/sglang#23954); non-stream handling was addressed separately in #25298. The Journal surfaced this historical record in a later Task, preventing a measurement error from being mistaken for an engine regression.

The Journal preserves evidence; it cannot promote evidence directly into executable capability. Any change to the Skills & Tools baseline must pass through a Capability Task and Verification.

4.4.4 Safety Guard

As agent execution becomes longer and more concurrent, an unenforced mistake can travel farther. Safety Guard therefore applies constraints that no Playbook or Loop Block may bypass:

• Push Guard and Traceable Path constrain how code changes move.
• Env Isolation and Production Read-only Access constrain environments and production operations.
• Secrets and Data constrain access to credentials and governed datasets.
• Human Gate requires approval for high-risk actions.
• Cross-Model Adversarial Review applies reviewer ≠ coder to reduce self-review blind spots⁶.

Safety Guard constrains actions; Verification constrains claims. In one kernel optimization Task, a candidate appeared to reach 72.30 TFLOPS, a 5.7% improvement over its comparison point. A later Verification exposed a race: aggregate statistics looked stable while individual elements were corrupted. The candidate was rejected before Integration.

Reliable autonomy is measured not only by what it completes, but also by what it refuses to advance. In inference engineering, stopping a false performance win can be more valuable than producing another patch.

5. Task Graphs

A project does not scale merely by creating more Tasks. It scales when dependencies, shared state, and control decisions become explicit. A Task Loop gives one Task an independent path to convergence; a Task Graph connects those convergent units without dissolving their Verification boundaries.

A Task Graph is not a fixed workflow template or a closed taxonomy. Whenever multiple Tasks need explicit dependencies, verified Handoffs, shared state, or control relationships, they can be organized into a Task Graph. The graph can serve one project, one investigation, one release, or any other objective that exceeds a single Task boundary.

Delivery Graph and Capability Graph are two recurring examples, not the only valid graph structures. We use them to demonstrate the same graph language in two common situations: coordinating Tasks that jointly change the inference system, and maintaining the capabilities used by future Tasks.

Graph correctness begins by separating execution from state and external control. Only a Task node executes work and runs a Task Loop. A Shared repo stores persistent state without running a loop. An External system represents people or platforms that interact with the graph from outside infer-forge.

The edge type states what crosses each boundary:

• A Handoff edge carries a verified Deliverable or Follow-up Handoff that becomes Imported Context for the successor Task.
• A State edge represents a read from or write to a Shared repo; it does not imply Task completion or Verification.
• A Control edge triggers work, returns it for revision, or changes its direction without carrying a verified result.

A write is not a Handoff; a trigger is not a Deliverable; connectivity alone is not verified progress.

5.1 Delivery Graph

Inference delivery is a convergence problem, not a checklist. A Plan Task can divide the objective into Prefill, Decode, kernel, communication, and deployment workstreams. Research and Code Tasks then advance independently and in parallel, each producing evidence under its own Task Contract.

Integration is where parallel work becomes one runnable deployment point. It assembles changes across the engine, libraries, image, and deployment configuration. The combined candidate then enters Evaluation under fixed model, Serving Scenario, hardware, performance, accuracy, and stability conditions. A failed result returns through Rework rather than advancing on partial success.

Release is a verified state transition, not the last box in a diagram. Only a candidate that passes Evaluation can reach Release through a Follow-up Handoff. External Signals may trigger Online Diagnosis; its findings return through Feedback or start a Code Task through Hotfix. These Control edges can redirect work, but they cannot bypass Integration or Evaluation.

A Delivery Graph makes parallelism useful by forcing every path to converge through evidence.

5.2 Capability Graph

The Delivery Graph changes the inference system; the Capability Graph changes the system that performs the work. Task Execution uses the current Skills & Tools baseline and records practical evidence in the Journal. Across Tasks, repeated steps, reusable commands, workflow patterns, and proven fixes become candidates for capability maintenance.

The Journal is evidence, not authority. Journal commit event, Scheduled trigger, and Model or runtime change can wake a Capability Task, but none of them proves that executable behavior should change. The Task scans the Journal delta, reads the current baseline, and uses Verification to produce Add, Update, Merge, Retire, or No Change.

Every capability has a carrying cost and a shelf life. Some Skills preserve durable project knowledge; others compensate for a particular model or runtime. As those systems improve, old scaffolding may consume Context, conflict with newer behavior, or constrain judgment. The Claude Code team reports removing more than 80% of its system prompt for newer models with no measurable loss on its coding evaluations¹⁵. Boris Cherny has separately advocated periodically pruning CLAUDE.md files, Skills, and hooks¹⁶.

A Model or runtime change therefore triggers revalidation of the current baseline. Capability removal is an evidence-driven ablation method, not blanket deletion; safety boundaries and verified invariants remain unless evidence supports changing them. A capability set that supports only Add does not evolve—it accumulates debt.

If a candidate requires implementation, Evaluation, or Release work, the Capability Task produces a Follow-up Handoff to the corresponding Task. The Capability Graph prevents raw experience from mutating executable behavior while ensuring that verified experience does not remain trapped in the Journal.

6. One Engineer, Multiple Loops

This chapter follows one AI infrastructure engineer from sustained Task execution to concurrent work and project-scale coordination.

6.1 From Execution to Judgment

AI infrastructure engineering changes when execution can continue while the engineer is elsewhere. Once an engineer has completed Task Definition and established the necessary Human Gates, an Agent can carry the work through environment setup, deployment, Evaluation, recovery, and iteration over hours or days. The Task Loop keeps that execution connected to the Task Contract, its evidence, and the next decision.

Figure 13 shows the resulting working mode. While one Task Loop is still executing, the engineer can define another Task, review completed evidence, or return to unrelated engineering work. The engineer no longer has to advance every command sequence personally; scarce attention shifts from execution to judgment: setting constraints, evaluating evidence, resolving tradeoffs, and deciding what should proceed next.

One engineer can therefore direct multiple sustained Task Loops without blurring the boundary of each Task. The Harness carries execution forward; the engineer concentrates attention where it changes the project's direction.

6.2 Observed Tasks in Flight

Figure 14 shows how the engineer's Tasks in flight changed over four months. Each bar spans from a Task's creation timestamp to its archive timestamp, and the peak is the maximum number of these intervals that overlap at any instant. Of the 90 Tasks created during the observation window, 86 archived Tasks had valid timestamps and are included; three were still open at the cutoff, and one record was excluded because its archive timestamp preceded its creation timestamp.

Across the April–July record, the median archived lifetime rose month over month: approximately 10 hours in April, 14 hours in May, 20 hours in June, and 28 hours in July. The monthly peak number of Tasks in flight was 2, 2, 6, and 9, respectively.

6.3 Project-Scale Coordination

One engineer coordinated Pushing the Limits of Serving DeepSeek-V4-Pro¹⁸ through 38 independently verifiable Task nodes across seven Task Types. The graph organized four serving workstreams—short-context Prefill, long-context Prefill, low-latency Decode, and high-throughput Decode—so they could progress independently while sharing the decisions, constraints, and Verification evidence required for one Release. The four workstreams converged on four different deployment points rather than one universal optimum.

One Decode Task remained open for nine days while other workstreams continued elsewhere in the graph. The graph prevented this long-running Task from blocking the project. No single Agent or Task had to retain the whole project context: each workstream could converge against its own Task Contract, then contribute a verified Deliverable to the larger Release.

The project produced four released serving profiles and preserved seven paths that did not enter the final Release. Those paths were part of the engineering result: they recorded what had been evaluated, why it did not advance, and what downstream Tasks should not need to rediscover.

This is how the operating model scales: one engineer can direct multiple sustained Task Loops, converge them through a Task Graph, and preserve both the released system and the decisions behind it.

7. Layers Accumulate

Harness turns execution-local Context into durable structure for work that must continue. Context holds the goal, code, current state, and working judgments needed for one execution. A Harness selects what deserves to outlive that execution, externalizes it for reuse, and supplies stable tools, environments, memory, verification entry points, and safety boundaries⁹. Figure 16 is an accumulation, not a succession: Context becomes the execution core of the Harness; the Harness remains the load-bearing foundation of each Task Loop; and Task Graphs connect those Task Loops into project-scale work. Every higher layer extends the reach of the layers below without removing their capabilities or constraints.

Abstractions can change in months; reliability is earned layer by layer. The representative essays cited here helped crystallize Harness Engineering, Loop Engineering, and Graph Engineering as a shared vocabulary in just over five months^2,4,13, but their dependency is fundamental rather than chronological. Incomplete Context can compromise one execution. A weakness in the Harness—stale knowledge, a faulty tool, or a missing constraint—can recur throughout a Task Loop and propagate across a Task Graph. Higher layers amplify both the strengths and the weaknesses beneath them. Loop and Graph Engineering do not supersede Harness Engineering; they increase how much depends on it.

8. Lessons Learned

8.1 Task Granularity

A Task boundary should be drawn where Verification can stand on its own. A Task needs its own Task Contract—Goal, Scope, Acceptance, and Verification—and its own Exit Criteria. Work that changes only the Sub-target or Exit Condition while the Task Contract remains intact belongs in another Loop Block. Work that requires a materially different Goal, Scope, or Acceptance belongs in a new Task. A Task that is too large hides independent evidence; one that is too small turns execution into coordination overhead.

The lesson from the nine-day Decode Task was not that nine days is inherently too long. Experiments governed by the same performance, accuracy, and stability contract belonged in successive Loop Blocks; directions requiring an independent Goal, Scope, or Acceptance should have become separate Tasks. A practical test is to ask where execution should return after failure: the next Loop Block, or a new Task Definition?

Split on a change in the contract, not the passage of time.

8.2 Evidence-Driven Task Graphs

A Task Graph should absorb evidence, not freeze assumptions. A Plan Task defines the most credible initial decomposition, but the graph must be able to evolve as evidence changes¹³. Research may invalidate a path, Evaluation may produce Rework or a new Code Task, and Feedback from Online Diagnosis may change the Scope of what follows. In response to verified evidence, the graph may add, remove, or reorder Task nodes.

A downstream Task may be planned earlier, but it must not import an upstream result until that result has passed Verification and crossed a Handoff edge. In the DeepSeek-V4-Pro graph, the rejected directions, Rework paths, and cross-workstream constraint did not exist in the initial plan; they entered the graph only after execution produced evidence.

A graph that cannot change after planning records intention, not engineering.

8.3 Evidence Integrity

Evidence is only as trustworthy as the Harness that produces it. Evaluation tools, configurations, and data paths are part of the result. They require the same versioning, review, and Verification as the code under test.

A Deliverable must preserve a reproduction path, not just a conclusion. For inference work, it should contain enough information to recreate the deployment point and compare the baseline with the candidate: exact code and runtime versions, image, deployment and workload configurations, commands, results, and failed attempts. A Follow-up Handoff states what is complete, where the evidence lives, which judgments still hold, and where the next Task should begin. Once that package is verified and imported, it becomes Imported Context.

If the next Task must reconstruct the environment or trust an assertion, the work has not been delivered.

9. Conclusion

Infer-forge does not reduce the complexity of inference engineering; it makes work across that complexity inspectable, resumable, and verifiable. It keeps each change tied to its deployment point, code provenance, and evidence across repositories, execution rounds, and Follow-up Handoffs, so verified Deliverables and documented rejections can be reproduced, revisited, and carried forward.

Infer-forge supports an operating model in which one AI infrastructure engineer can direct multiple sustained Task Loops. Our four-month record shows the observed peak number of Tasks in flight increasing from 2 to 9. We are exploring increasingly autonomous Task Graph coordination while preserving Verification boundaries, Human Gates, and engineering judgment. We hope the practices embodied in infer-forge help teams and individuals apply Harness, Loop, and Graph Engineering to large-scale, complex systems.

Acknowledgments

We thank the SGLang Team and the broader SGLang community for developing and openly sharing the Skills that support agent-assisted SGLang development⁸. We are also grateful to Peng Zhang of the SGLang community. We especially thank Xiaoyu Zhang (BBuf) for creating and sharing the AI-Infra-Auto-Driven-SKILLS collection¹⁷.

We also thank the researchers and engineering teams cited in this article. Their work on Harness Engineering, Loop Engineering, and Graph Engineering helped shape the concepts and methods behind infer-forge.

References

1. Vivek Trivedy — The Anatomy of an Agent Harness, LangChain, March 10, 2026.
2. Ryan Lopopolo — Harness engineering: leveraging Codex in an agent-first world, OpenAI, February 11, 2026.
3. Birgitta Böckeler — Harness engineering for coding agent users, Martin Fowler, April 2, 2026.
4. Addy Osmani — Loop Engineering, June 7, 2026.
5. Sydney Runkle — The Art of Loop Engineering, LangChain, June 16, 2026.
6. Prithvi Rajasekaran — Harness design for long-running application development, Anthropic, March 24, 2026.
7. Erik Schluntz and Barry Zhang — Building Effective AI Agents, Anthropic, December 19, 2024.
8. SGLang Team — Agent-Assisted SGLang Development: An Initial Exploration, LMSYS Org, July 2, 2026.
9. Lilian Weng — Harness Engineering for Self-Improvement, Lil'Log, July 4, 2026.
10. Boye Niu et al. — Flow: Modularized Agentic Workflow Automation, arXiv:2501.07834, 2025.
11. Andy Xu and Yu-Wing Tai — Meta-Agent: From Task Descriptions to Verified Multi-Agent Systems, arXiv:2605.25233, 2026.
12. Ao Li et al. — GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving, arXiv:2605.22566, 2026.
13. Sydney Runkle and Harrison Chase — 3 Years of Graph Engineering with LangGraph, LangChain, July 22, 2026.
14. Nelson F. Liu et al. — Lost in the Middle: How Language Models Use Long Contexts, TACL 2023 / arXiv:2307.03172.
15. Thariq Shihipar — The New Rules of Context Engineering for Claude 5 Generation Models, Claude, July 24, 2026.
16. Boris Cherny and Diana Hu — Boris Cherny: Building Claude Code, Y Combinator Startup School, July 27, 2026.
17. Xiaoyu Zhang (BBuf) — AI-Infra-Auto-Driven-SKILLS, GitHub.
18. Tianyu Zhang, Yusong Gao, Yun Zhang — Pushing the Limits of Serving DeepSeek-V4-Pro on Compute-Constrained NVIDIA H20, LMSYS Org, August 19, 2026.