Research / Program Active
Self-Adaptive
Infrastructure.
How can computing infrastructure safely learn from its own runtime behaviour and adapt its operational policies under uncertainty?
Why this question
Every system already has
the data it needs.
A queue knows how long its jobs took. A pool knows when it saturated. A retry knows what it cost. That information is almost always discarded, and the policies that govern the system are instead chosen once, by a person, from defaults — and then left alone for years while the workload underneath them changes completely.
The interesting question is not whether a better policy exists for a given workload. It usually does, and finding it offline is a well-studied problem. The question is how a running system can find one from its own data, decide the evidence is sufficient, act inside limits that bound the damage of being wrong, and then measure whether it actually helped.
That is harder than it sounds, and the hard part is not the learning. It is knowing when the evidence is good enough to justify moving, and arranging matters so that a wrong move is cheap and reversible. Most of this program is about that, rather than about algorithms.
Observe → model → decide → act → measure → learn
Six steps,
none of them optional.
A system that skips measure is not adaptive; it is merely changeable. Most production automation stops at act, which is why it so often cannot say whether it helped.
Record what actually happened: durations, queue depths, failures, saturation, latency distributions.
Estimate the workload and environment. Distributions, not averages; error bars, not point estimates.
Determine whether any policy change is justified by the evidence, and which one.
Apply the change inside explicit limits, with a rollback path defined before the change is made.
Compare against the baseline that was fixed beforehand. Attribute the effect, or admit it cannot be attributed.
Feed the outcome back. A change that did not help is information, and is kept.
This resembles feedback control, and MAPE-K is the obvious reference point. Neither framework is imposed on every experiment — a scheduling question and a retry-budget question have genuinely different structure, and forcing one vocabulary onto both tends to hide the difference rather than illuminate it.
Earned, not configured
A system should climb
one rung at a time.
The ladder exists because the failure mode of adaptive infrastructure is not a bad model. It is a system granted authority it had not yet demonstrated it deserved. Each rung is a claim that has to be supported by evidence gathered on the rung below it.
The program is at Level 0, in simulation. Nothing here runs against production infrastructure, and no company system is being adapted, observed or instrumented by this work.
Mechanisms, not implementations
Where an adaptive policy
could live.
These are directions the question opens onto, not systems that exist. Only the first is under active investigation; the rest are listed so the shape of the program is legible, and so a later experiment has somewhere to attach.
- Job ordering
- Duration prediction
- Preemption
- Fairness
- Tail latency
- Queueing
- When to scale
- How much
- Predictive vs reactive
- Hysteresis
- Worker counts
- Connection pools
- Adaptive pool sizing
- Saturation
- Retry timing
- Adaptive backoff
- Retry budgets
- Failure amplification
- Adaptive thresholds
- Recovery conditions
- Context sensitivity
- Capacity estimation
- Admission control
- Prioritisation
- Fairness
- Dynamic TTL
- Eviction
- Warming
- Predictive caching
- CPU and memory
- Workers
- Bandwidth
- Heterogeneous resources
- Queue-aware routing
- Latency-aware routing
- Capacity-aware routing
- Pool sizing
- Batch sizing
- Maintenance scheduling
- Adaptive resource control
- Detection
- Diagnosis
- Controlled intervention
- Rollback verification
- What to sacrifice
- Preserving critical SLOs
- Behaviour under pressure
Real experiments only
The first
experimental domain.
Scheduling was chosen first because it is the mechanism where the gap between what a system knows and what it uses is widest, and because queueing theory provides closed-form results to validate a simulator against. A simulator that cannot reproduce M/M/1 is not measuring anything.
Adaptive Scheduling
Under Uncertainty.
ActiveHow should a scheduler adapt when job duration is initially unknown but becomes statistically predictable through repeated execution? A deterministic discrete-event simulator, validated against closed-form queueing theory, sweeping parameters continuously and attacking its own conjectures.
Open the research notebook →- What should run first when job duration is unknown?
- When are imperfect duration predictions useful?
- When should a long-running job be preempted to execute a shorter one?
- When does additional parallelism stop being beneficial?
- Can a scheduler dynamically choose between policies as the workload changes?
- How should a scheduler balance exploration and exploitation while learning workload behaviour?
Each with its evidence status
What is currently
believed, and how firmly.
Published from the machine running the work. A falsified conjecture is kept with its counterexample — the record of what failed is the part that stops the same wrong idea returning.
Loading findings…
Same skeleton every time
What an experiment
has to contain.
Question. Hypothesis. Background. Baseline. Model. Intervention. Metrics. Data. Result. Threats to validity. Failure modes. Conclusion. Next experiment.
The sections that get skipped when nobody enforces them are threats to validity and failure modes, which is precisely why they are on the list. An experiment that cannot name how it might be fooling itself has not been finished.
Data, simulation, measurement, statistics, formal analysis, proof and reproducible computation. AI is used throughout — literature search, hypothesis generation, code, criticism, hunting for counterexamples — but it is not a source of truth, and no claim on this site rests on a language model having asserted it.
This program works in simulation, on synthetic workloads and public datasets. Any experiment touching real company infrastructure would require explicit organisational approval first, and is not assumed to exist. Were one ever approved, it would carry:
- anonymisation, and no confidential customer data in any public artifact
- no monitoring of individual people's work
- reversible interventions with a defined rollback
- guardrails, a controlled rollout and metrics agreed in advance
Recent events in this program
The working
record.
Loading activity…