Cowrie Research

oracle-migration-quarantine-and-canary-design

15 min read

Oracle Migration Quarantine and Canary Design

  • Name: oracle migration quarantine and canary design
  • Category: oracle reference / rollout-safety checklist / configuration-change containment note
  • Summary: Many oracle incidents are not really price feed bugs in the narrow sense. They are rollout failures. A protocol changes feed composition, wrapper interpretation, caps, fallback order, asset-to-feed binding, or deposit-admission thresholds and immediately lets the new path control liquidations, borrowing, minting, or collateral admission at full size. Healthy oracle operations treat migrations and major config changes like new production risk surfaces: they narrow blast radius first, prove the new answer against independent references, and only then let the new path become full solvency truth. This note exists to keep that operational design target explicit instead of leaving Rho, Gamma, Moonwell, and Aave CAPO to imply it only through postmortems.
  • Why this note exists:
    • Oracle incidents often preserve the same hidden sentence: the protocol activated a new price path before it had earned full trust.
    • The failure is rarely just one wrong number. It is usually wrong number plus immediate full-size authority over borrowing, liquidation, or minting.
    • The analytical win is to ask how should a protocol stage new oracle truth into production? rather than only what arithmetic bug happened afterward?

What this note means by quarantine and canary design

  • Quarantine means a new oracle path, wrapper migration, cap layer, or parameter regime does not immediately inherit the protocol’s full balance sheet.
  • Canary design means the protocol first exposes the new path to bounded observation, bounded value, or bounded user impact before letting it become general solvency truth.
  • The design goal is not never change the oracle. It is oracle changes should fail small while the protocol learns whether the new path is actually coherent.

Three rollout-authority promotions operators should keep separate

  1. A new oracle path or config regime exists in production

    • A feed binding, fallback order, wrapper interpretation, CAP overlay, or threshold set has been deployed and can now publish some local answer.
    • This is only the activation question: the new path is reachable at all.

  2. That path has earned the right to count as canonical solvency truth for this exact market object

    • The protocol has shown that the path still refers to the right asset, quote currency, decimals, chronology, and paired-state object under the exact live activation state.
    • This is the canary question: the newly reachable path is actually coherent balance-sheet truth rather than merely a syntactically valid config.

  3. That accepted truth deserves full borrowing, liquidation, minting, or collateral authority right now

    • Even a coherent-looking new path should not automatically inherit the whole market’s downstream authority budget on the same block it goes live.
    • This is the staged-trust question: what bounded live window exists before this path gets to govern the whole balance sheet?

The main rollout controls

1. Activation-time market quarantine

  • New oracle paths should usually begin with borrowing disabled, liquidations disabled, strict supply caps, or sharply reduced collateral factors.
  • If the protocol cannot tolerate a short observation window with limited utility, it probably should not be changing the oracle live.
  • This matters most for wrapper migrations, new market launches, CAPO-style overlays, and asset/feed rebinding changes.

2. Independent answer canaries

  • The protocol or operator stack should compare the newly proposed final onchain answer against:
    • the old answer
    • at least one independent external reference
    • the expected unit / quote-currency interpretation
  • The canary question is not just did the transaction succeed? but does this new answer still describe the same asset, quote, decimal scale, and time basis we think it does?

3. Paired-state atomicity checks

  • If a price path depends on multiple jointly meaningful fields, they should move together or fail together.
  • Examples include:
    • anchor plus timestamp
    • ratio plus quote leg
    • cap parameter plus reference state
    • feed address plus asset binding
  • A partial state update is often more dangerous than no update because it looks fresh while encoding an impossible reference point.

4. Borrow-side and liquidation-side throttles during migration

  • A new oracle path should not instantly control the deepest downstream actions.
  • Good rollout discipline often means temporarily narrowing:
    • borrow caps
    • liquidation close factors
    • market-level debt ceilings
    • collateral factors
    • mint quotas for share-priced or wrapper-priced assets
  • The point is to keep an integration mistake from becoming immediate balance-sheet loss.

5. Wrapper and collateral distrust windows

  • If the migrated asset is itself a wrapper, vault share, LP token, or rate-bearing derivative, the protocol should assume the new price path may be locally coherent yet still unsafe as immediate collateral truth.
  • Delayed collateral admission, temporary haircuts, or explicit quarantine windows belong inside the oracle boundary for these assets.
  • A standards-compliant wrapper plus a brand-new oracle route is still two fresh trust assumptions landing at once.

6. Explicit fallback-precedence review

  • Many oracle changes do not fail because the primary source is wrong. They fail because fallback, cap, or protection logic silently overrides a good base feed.
  • Reviewers should be able to answer:
    • which branch wins right now?
    • under what condition does it override the primary feed?
    • what exact number would publish if that branch triggered today?
  • If nobody can answer that concretely, the migration is not ready.

7. Fast rollback and scoped kill switches

  • Oracle migrations should ship with a narrow reversal path before activation.
  • Good rollback design means operators can freeze one market, one asset, one overlay, or one wrapper integration without halting the entire protocol.
  • Good recovery design is narrower too: after a bad cutover, the protocol should be able to reopen into observation-only, repay-only, or repay-and-supply-only operation before any restored path regains ordinary borrowing or liquidation authority.
  • A rollback path that is slower than liquidator or borrower reaction time is mostly theater.

What sound design looks like

  • The final solvency object is written down plainly before activation.
    • Reviewers can state this market will publish <asset>/<quote> in <decimals>, derived as <leg A> × or ÷ <leg B> with <cap/fallback/anchor rules>.
  • The protocol stages trust instead of assuming it.
    • New oracle paths begin under smaller caps or narrower rights than mature paths, so path exists, path seems coherent, and path gets full balance-sheet authority remain separate promotions instead of one activation event.
  • Observation happens under bounded exposure.
    • The first live window is meant to catch integration drift while losses are still bounded.
  • Operators compare concrete answers, not just code diffs.
    • The important artifact is often the emitted calldata, deployed feed binding, or generated threshold values.
  • Rollback is part of the launch plan, not an afterthought.
    • If reverting the change requires slower governance than exploiting it, the migration is underdesigned.
  • Recovery should be staged rather than binary.
    • Market resumed or feed restored is not the same thing as ordinary borrowing and liquidation authority came back safely; healthy restart ladders restore unwind paths before full credit powers.
  • Wrapper-priced assets get extra skepticism.
    • If the migrated oracle path prices vault shares, LP tokens, rebasing wrappers, or capped exchange-rate assets, collateral admission should lag oracle activation rather than travel with it by default.

Useful positive design shapes

1. Stage first, trust later

  • The clean design target is not perfect pre-activation certainty. It is a narrow first window where the new price path can be observed without controlling the whole market.
  • Borrow caps, collateral haircuts, delayed liquidation enablement, or temporary mint pauses are all legitimate canary tools.
  • The important habit is that the protocol’s balance sheet does not need to accept full-size risk on the same block the new oracle path goes live.

2. Canary answers should test identity, units, and chronology together

  • A healthy canary compares more than number close enough.
  • It should ask:
    • is this still the right asset?
    • is it still the right quote currency?
    • is the scale still correct?
    • do all state variables refer to the same reference moment?
  • Rho, Moonwell, and Aave CAPO all preserve different ways one of those questions can fail while the others still look superficially fine.

3. Wrapper migrations need collateral quarantine, not just oracle correctness

  • A wrapper or share-based asset can have a technically correct local exchange-rate read and still be unsafe as immediate solvency truth during rollout.
  • The positive design move is to separate:
    • the protocol can read a value
    • the protocol may instantly lend or liquidate against that value at scale
  • This is where oracle rollout safety starts touching vault-share-truth-security-object and the narrower downstream-solvency companion collateral-admission-lag-and-borrow-side-quarantine.

4. Defensive overlays should be treated as new oracle systems

  • CAP layers, anti-manipulation wrappers, bounded-rate modules, and deposit guards are not merely protection code.
  • Once they can override or gate the underlying answer, they are themselves oracle systems and deserve rollout quarantine like any other migration.
  • A defensive layer is not safer just because its purpose is safety.

Negative exemplar cluster: what no quarantine looks like

  • Rho Markets — a wrong asset/feed binding went live with immediate borrow authority, so one deployment-time mapping error controlled real debt creation before humans could intervene.
  • Gamma Strategies — a safety threshold intended to guard deposits was mis-set wide enough to authorize manipulated minting, and the bad envelope became production truth immediately.
  • Moonwell cbETH — a wrapper migration treated cbETH / ETH as if it were already cbETH / USD, and live liquidations consumed the dimensional mistake before governance-scale reaction could catch up.
  • Aave CAPO — a defensive overlay update pushed paired state out of alignment, and the protocol let the half-updated cap immediately drive healthy-user liquidations.
  • The common lesson is not just bugs happen. It is a fresh oracle path should not inherit full liquidation and borrowing authority before the protocol proves the path is coherent under live conditions.

Red flags that usually mean the rollout is underdesigned

  • The code path is audited, so we can enable it at full size immediately
  • We can always pause later when liquidation bots or borrowers are faster than the pause path
  • Fallback only matters in edge cases even though no one can say which branch would win right now
  • The asset is the same, only the wrapper/feed changed as if interpretation changes were not real security events
  • The migration only affects a safety overlay as if caps and anti-manipulation guards were not themselves price-authoring systems
  • We checked the feed address without checking quote currency, decimals, or asset binding
  • The script output looked reasonable without comparing the final published onchain answer to independent references under the exact activation calldata

Failure-prevention checklist

  • Treat every oracle migration, wrapper rebinding, cap overlay change, and fallback-precedence edit as a new solvency-authoring system, not as routine config.
  • Launch fresh oracle paths under quarantine by default: lower collateral factors, disabled fresh borrowing, capped minting, delayed liquidation enablement, or sharply bounded supply/debt ceilings until the path earns trust.
  • Write the exact published solvency object down before activation: asset, quote currency, decimals, arithmetic composition, cap logic, fallback order, and the state variables that must stay synchronized for the answer to make sense.
  • Compare the final onchain answer against at least one independent reference and against the displaced path using the exact activation calldata, not only code review or script previews.
  • Review fallback and overlay precedence explicitly so operators can say which branch would publish now, what conditions trigger it, and how far that branch can move protocol rights if it fires.
  • Keep paired oracle state atomic wherever possible; if cap, anchor, quote leg, timestamp, or wrapper binding cannot move together safely, fail closed instead of publishing a half-updated solvency object.
  • Separate oracle activation from immediate balance-sheet authority for wrapper-priced, LP-priced, vault-share, and rate-bearing assets; collateral admission should lag new price publication when interpretation risk is fresh.
  • Pre-stage narrow rollback and kill-switch paths per market / asset / overlay before launch so responders can reverse one bad migration faster than bots can consume it.
  • Predefine the restart ladder too: freeze or revert first, reopen into observation-only / repay-only / repay-and-supply-only lanes if needed, then restore bounded borrowing and only later ordinary liquidation and full credit authority.
  • Monitor first-live-window drift aggressively: branch selection, unit mismatches, price divergence vs independent references, liquidation anomalies, and unexpected debt growth should all page operators.
  • Rehearse rollback and staged trust restoration so quarantine is an operating mode the protocol already knows how to use, not a plan improvised during loss.

Audit prompts

  • What exact published solvency object will this migration authorize onchain, and how does each field differ from the previous live path?
  • Which branch actually wins today: primary, cap, anchor, fallback, wrapper exchange-rate path, or some mixed precedence rule that only appears under specific conditions?
  • If the new answer is wrong by asset identity, quote currency, decimal scale, chronology, or paired-state alignment, what is the maximum downstream loss before operators can narrow authority?
  • Which downstream rights stay disabled or reduced during the first live window, and which still turn on immediately at full size?
  • Are all jointly meaningful state variables updated atomically, or can the protocol briefly publish an impossible combination such as new cap with old anchor or new wrapper binding with old quote leg?
  • What independent references, old-path comparisons, or shadow checks prove the new path is coherent under the exact activation calldata and not just in isolated tests?
  • Can operators roll back one market, one asset, or one overlay faster than liquidators, borrowers, or MEV bots can exploit a wrong answer?
  • If the cutover looks wrong or degraded, can the protocol reopen narrowly for exits and repayments before any restored path regains ordinary borrowing and liquidation authority?
  • Does the migration change real source independence, chronology, or wrapper semantics even if the visible feed address count barely changes?
  • For wrapper, LP, or vault-share collateral, what separate distrust window exists between we can read a value and we will lend or liquidate against that value at scale?
  • What telemetry would show a bad rollout within minutes: branch flips, divergence alarms, fresh bad debt, healthy-user liquidation spikes, or over-minting against the newly activated path?

Lessons that should already be standard

  • An oracle migration is not complete when the transaction succeeds; it is complete when the new path has survived a bounded live window without disproving its own assumptions.
  • Defensive overlays are still oracle systems. A CAP, anti-manipulation wrapper, or threshold guard that can override the published answer deserves the same rollout paranoia as the underlying feed.
  • Same asset, different wrapper/feed is a real security event because asset identity, quote currency, and chronology semantics often change at the same time.
  • The first live block is the wrong moment to grant full borrowing and liquidation authority to a fresh solvency object.
  • Rollback speed is part of oracle design. If the protocol cannot reverse one bad activation faster than the market can monetize it, the migration is underdesigned.
  • Feed restored or market resumed is not the same thing as full credit restored; healthy restart paths bring back unwind capacity before they bring back ordinary borrowing and liquidation rights.
  • Wrapper correctness and collateral correctness are different claims. A locally coherent wrapper price can still be too fresh to trust as full-size solvency truth.
  • Fallback precedence is part of the security model. Teams that cannot say which branch would publish right now are already operating an oracle they do not fully understand.
  • Good rollout policy assumes change will sometimes be wrong and arranges for that wrongness to fail small.

Design questions worth asking first

  • What exact new solvency object is being activated, and how does it differ from the old one?
  • During the first live window, what is the maximum bad debt, over-borrow, forced-liquidation, or over-minting damage this path could cause?
  • Which downstream powers are temporarily narrowed until the new path proves itself under live conditions?
  • If the migration must be unwound, what is the staged recovery path: freeze, observation-only, repay-only, repay-and-supply-only, bounded borrowing, then ordinary liquidation and full credit restoration?
  • What independent answer will the protocol or operators compare against before and after activation?
  • Which state variables must remain synchronized for the path to make sense, and what prevents partial updates?
  • Can operators roll back one market, one asset, or one overlay faster than the market can exploit a bad answer?

How this note complements the exploit corpus

  • Start with oracle-configuration failure when the question is how did the protocol publish a false price even though the market itself may have stayed honest?
  • Use this note when the sharper question becomes how should a protocol stage oracle change into production without giving the fresh path full balance-sheet authority immediately?
  • Pair with collateral-admission-lag-and-borrow-side-quarantine when the sharper downstream question is when a freshly published wrapper or overlay price should be allowed to create debt or trigger liquidations.
  • Pair with vault-share-truth-security-object when the migration prices wrappers, LP tokens, vault shares, or rebasing collateral objects whose own exchange-rate logic needs a second distrust boundary.
  • Pair with unit-normalization failure when the core risk is dimensional, and with oracle-source-independence failure when the migration claims fallback or source diversity that is not truly independent.

Internal linkages

Control surface

Graph View

Backlinks