Cowrie Research

exact-production-artifact-attestation-and-rollback-discipline

16 min read

Exact Production Artifact Attestation and Rollback Discipline

  • Name: exact production artifact attestation and rollback discipline
  • Category: release-integrity reference / software supply-chain defense note / wallet-facing artifact-attestation checklist / rollback-verification design note
  • Summary: Healthy wallet-facing and signer-adjacent software delivery does not stop at we reviewed the repo or even we pinned the package version. It proves one exact production artifact lineage as a single security object: reviewed source, reproducible build output, published tarball, version-resolution policy, runtime service identifiers, CDN or loader delivery, and the exact bytes browsers or backends actually execute. It also proves rollback at that same layer, instead of treating good version published as if it automatically meant users now run good bytes. This note exists to keep the positive design target explicit instead of making readers reconstruct what recovery should have looked like only by inverting release-provenance and software-supply-chain incidents.

Fast classifier: which artifact-drift shape are you trying to make recoverable?

  • same package name, different custody semantics — the package identity still looks canonical, but the published artifact changed what backend or signer-adjacent code actually does with secret material or authority. Solana web3.js is the clean exemplar.
  • same real site, different loader-resolved bytes — the downstream domain stays honest, but signer-critical execution is still delegated to a mutable loader, package, or CDN path. Ledger Connect Kit is the clean exemplar.
  • same familiar runtime URL, different operator-resolved bytes — the source-controlled script tag or service identifier survives, but ownership or request-time output behind that URL changes underneath it. Polyfill.io is the clean exemplar.
  • same canonical signer site, different bytes for one wallet or signer cohort — the trusted domain stays real and looks normal for most users, but request-time or cohort-targeted delivery serves different signer-critical JavaScript only when a specific wallet, signer address, or transaction path is present. Bybit is the clean signer-targeted exemplar.
  • Use this note when the sharper positive-design question is how one attestation and rollback discipline should prove recovery across all four shapes at the layer users or operators actually execute.
  • Why this note exists:
    • The practical failure in many wallet-drainer and signer-SDK incidents is not just someone published malware; it is that defenders authenticated the wrong object along the path from source to execution.
    • The practical recovery failure is similarly specific: teams often fix one layer of the chain while users or operators are still executing poisoned bytes, resolving poisoned endpoints, or trusting secrets that the bad artifact already exposed.
    • The useful design move is to make both forward integrity and rollback proof machine-checkable at the same layer users and operators actually run.

The seven joints a healthy production-artifact lineage should keep fused

  1. Reviewed source identity
    • The system must know which repository state or source tree humans actually reviewed.
    • Main branch looked fine is not a release object.
    • This is only the first joint, not the whole trust boundary.
  2. Reproducible build artifact
    • The team should know what deterministic output that reviewed source is supposed to produce.
    • If the shipped tarball or browser bundle cannot be tied back to one expected build result, the release object is already split.
  3. Published package or tarball identity
    • The exact artifact uploaded to npm or another registry is its own security object.
    • Same package name or same semver line is not enough if the tarball bytes differ from the reviewed build.
  4. Version-resolution and loader policy
    • The system must know how production chooses which artifact to execute: exact lockfile, pinned import map, mutable dist-tag, runtime CDN loader, or some other policy.
    • If downstream execution depends on latest, mutable semver ranges, or remote loader rules, that resolution layer is part of wallet or signer authority.
  5. Runtime service identifiers and remote dependencies
    • Embedded project IDs, RPC endpoints, remote config URLs, wallet-connect IDs, or similar runtime selectors belong inside the same release object once the artifact trusts them to shape user flows.
    • Clean package bytes pointing at attacker-controlled runtime services is still broken production truth.
  6. Actually fetched execution bytes
    • The real security object is what the browser or backend fetched and executed in production, not what a release dashboard says should have been there.
    • Multi-region CDN responses, cache state, edge rewrite rules, and loader-side transformations all matter here.
  7. Rollback proof plus exposed-secret disposition
    • Recovery is not complete until operators can prove both that expected bytes are now executing and that any secrets touched by the poisoned artifact have been rotated, revoked, or otherwise de-risked.
    • We published a fixed version and the exploit path is gone are different claims.

Three artifact-lineage questions healthy systems should answer separately

  1. What exact production artifact lineage exists?
    • Teams should be able to name one concrete lineage from reviewed source through deterministic build output, published artifact, runtime selector, remote service identifiers, and finally executed bytes.
    • If that answer changes depending on whether you ask release engineering, security, or incident response, the lineage is already split.
  2. What proves that lineage is still canonical execution truth?
    • The healthy question is not only did we publish a clean package? but what proves the browser or backend still resolves the artifact we meant to bless?
    • Multi-region fetches, loader-state verification, CDN-edge sampling, and remote-identifier checks belong here because they decide whether the named lineage still describes reality.
  3. What live authority does that accepted lineage still deserve?
    • Even once lineage is authentic, teams should still say what approvals, key access, signer budgets, remote-service reach, or wallet-prompt authority those bytes are allowed to exercise right now.
    • That final authority question is why rollback and secret rotation belong in the same discipline: clean lineage restored and dangerous authority retired are related but separate claims.

What sound design looks like

  • One release object, many checkpoints, no silent reinterpretation
    • Source review, package publication, runtime selection, browser delivery, and recovery verification should all refer back to one explicit expected artifact lineage.
    • If each layer uses a looser notion of identity than the one before it, the security object has already drifted. Healthy operators also keep separate answers for what lineage exists, what proves it is canonical execution truth, and what authority that accepted lineage still deserves.
  • Production should prove bytes, not only names
    • Package names, domains, and version labels are routing hints, not integrity proofs.
    • Healthy systems can answer what exact bytes should production be running right now? and verify the answer independently.
    • For signer-critical surfaces, the stronger question is often which exact bytes should each signer cohort, wallet cohort, and release channel be running right now, and can we prove no hidden targeted variant exists?
  • Rollback must be verified at the execution layer
    • Recovery should be declared only after sampling the same CDN edges, loader paths, browser fetches, or backend resolution flows that real users hit.
    • A clean tarball in the registry does not prove a clean browser session.
  • Shared artifact lineage should trigger one grouped rollback object
    • If the same poisoned package, loader, domain-owned script, or remote project identifier fans out across many apps, wallets, or signer surfaces, responders should open one family-level rollback and secret-rotation sheet rather than many independent app tickets.
    • The useful question is not did this one frontend republish? but which downstream consumers still resolve the same compromised artifact lineage or trust the same exposed authority material?
  • Remote configuration belongs inside release integrity
    • If a runtime artifact trusts remote configuration or service identifiers to shape wallet prompts, signing endpoints, or connection paths, those runtime selectors are part of the release object.
    • Keeping code clean while leaving service-routing truth mutable is only partial containment.
  • Secret rotation is part of rollback when signer-adjacent code was touched
    • If the poisoned artifact could observe mnemonics, raw private keys, API secrets, or session credentials, then code rollback without secret disposition is incomplete recovery.

Healthy operational shapes

1. Local immutable bundling for wallet-critical paths

  • The safest browser-facing shape is often to bundle exact reviewed artifacts locally and serve them with integrity metadata rather than resolving shared wallet code dynamically at page load.
  • This shrinks the distance between reviewed artifact and executed artifact.
  • It also makes rollback more legible because one deploy can replace one known bundle.

2. Explicit artifact manifests

  • Teams should publish one machine-readable manifest that pins source commit, built artifact hash, package tarball hash, intended resolved version, expected service identifiers, and expected browser-facing bundle hashes.
  • The point is not bureaucracy; it is giving incident response one canonical object to compare every layer against.

3. Multi-path runtime verification

  • Production verification should sample what different users or backends actually fetch: multiple regions, multiple CDN edges, multiple loader paths, or multiple runtime environments.
  • One successful local fetch is not enough if caches or edge behavior can still differ elsewhere.
  • For signer-adjacent flows, verification should also sample across wallet cohorts, signer cohorts, authenticated-versus-unauthenticated sessions, and transaction-context branches so targeted payload substitution cannot hide behind everyone else saw the normal app.

4. Fast fail-closed release controls

  • Wallet-facing systems should be able to pin exact versions, disable runtime loaders, revoke remote project IDs, or fall back to a known-good local bundle quickly.
  • The best rollback is often stop serving mutable upstream code entirely until the lineage is re-proven.

5. Coupled rollback and secret-response playbooks

  • When the compromised artifact touched private keys or other signer authority, rollback and secret response should be one workflow.
  • A package fix without key rotation, credential revocation, or signer migration leaves the earlier compromise half alive.

Design questions worth asking first

  • What exact artifact lineage did the team think production was authenticating?
  • Can you map reviewed source, built artifact, published tarball, resolved runtime version, embedded service identifiers, and actually fetched bytes onto one explicit object?
  • Which layer can still change production behavior without a downstream redeploy: dist-tags, CDN edges, runtime loaders, remote config, or service ownership?
  • If you published a clean rollback right now, what evidence would prove that browsers and backends have actually stopped executing the poisoned artifact?
  • If the bad artifact touched keys or other authority material, what exact rotation or revocation steps are required before recovery is honest?
  • Could two users in different regions still receive materially different execution bytes after you declare the incident fixed?

Red flags that usually mean the lineage is still broken

  • The repo is clean now used as if source state proved runtime state.
  • We published a fixed version with no evidence about dist-tags, loader policy, CDN cache state, or actual browser fetches.
  • The package name is canonical with no tarball-hash or runtime-byte verification.
  • We reverted the bad commit even though the build output, remote config, or service identifiers can still drift underneath the same release label.
  • No further malicious code is in the registry while previously exposed signer secrets or API credentials remain valid.
  • One internal fetch looked good treated as if multi-region user delivery were now proven clean.

Failure-prevention checklist

  • Publish one machine-readable release manifest for every signer-critical build that pins the reviewed source commit, expected deterministic build output hash, published package or bundle hash, intended runtime-selected version, allowed loader path, and expected remote service identifiers. Make that manifest answer three separate questions up front: what lineage exists, what runtime checks prove it is still canonical, and what authority budget that lineage is allowed to keep if those checks pass.
  • Promote releases only after proving the executed artifact, not just the repo or tarball. Browser-facing and backend-facing signer routes should have outside-in verification that the bytes production resolves match the reviewed lineage.
  • Remove mutable resolution from signer-critical paths wherever possible: no broad semver ranges, no latest, no silent dist-tag promotion, and no wallet-critical runtime loaders that can drift without a downstream redeploy.
  • Treat remote configuration, project IDs, RPC endpoints, CDN loader URLs, analytics toggles that affect signer flows, and other runtime selectors as part of the same release object as the code itself.
  • Inventory every step that can rewrite the final shipped artifact after review: CI packaging, transitive dependencies, minification, bundling, CDN transforms, edge workers, and request-time script generation.
  • Make rollback proof symmetric with release proof. If the compromise lived in package publication, loader selection, CDN delivery, or runtime service identity, recovery should verify those same layers before incident closure.
  • Pair artifact rollback with secret disposition. If poisoned code could observe private keys, wallet session tokens, API secrets, or signer-adjacent credentials, rotation or revocation belongs in the same runbook.
  • Sample live delivery across multiple regions, caches, browsers, backend environments, and signer-facing cohorts before declaring recovery; one good local fetch is not evidence that production execution is clean.
  • Detect differential delivery explicitly. If signer-critical bundles, API responses, or transaction-builder assets can vary by wallet address, signer address, session state, or transaction context, that branching logic must be versioned, attested, and monitored like code rather than treated as harmless personalization.
  • Keep fast fail-closed controls ready: disable remote loaders, pin a known-good local bundle, revoke project IDs, force cache busting, and move signer-critical routes into maintenance mode if lineage cannot be re-proven quickly.
  • Pre-build a family-level rollback sheet for shared packages, loaders, and remote service identifiers. When one compromised artifact fans out across many apps or customers, responders should already know how to enumerate every downstream consumer, runtime selector, cached copy, and exposed-secret class in one grouped response object.
  • Preserve forensic evidence at the artifact layer: tarballs, CDN responses, lockfiles, service-identifier snapshots, and runtime hashes should be archived so postmortems can answer what users or operators actually executed.

Audit prompts

  • What exact object does this team believe production authenticated: repo commit, deterministic build output, published tarball, loader-selected version, embedded service identifier, or the bytes a browser or backend actually fetched?
  • Can the team prove that reviewed source, build output, published artifact, runtime selector, and executed bytes all name one lineage, or is equivalence being assumed between adjacent stages?
  • Which mutable selectors can still change signer-facing behavior without a downstream deploy: dist-tags, semver ranges, loader aliases, CDN edge rules, remote config, or service ownership?
  • Which build or delivery steps can rewrite final wallet-facing or signer-facing bytes after human review has completed?
  • If a rollback were declared right now, what evidence would prove that browsers and backends in multiple regions have stopped executing the poisoned artifact?
  • If the same package, loader, or remote service identifier was reused across many downstream apps, what single grouped rollback view would enumerate every consumer still in scope?
  • Which secrets, credentials, session tokens, or authority objects might already have been exposed by the compromised artifact, and what concrete revocation or rotation path exists for each?
  • Are embedded service identifiers versioned and attested alongside code, or can a familiar package or URL still point at materially different live behavior?
  • Does incident response distinguish registry state is clean, CDN state is clean, runtime execution is clean, and previously exposed authority is retired, or are those being collapsed into one vague recovery claim?
  • What logging or artifact capture exists to answer the later postmortem question what exact bytes did users or operators execute during the compromise window?
  • If a maintainer, CDN operator, or runtime-service owner changed one thing tonight, which release-stage joint would fail first and how quickly would the team know?

Lessons that should already be standard

  • Reviewed source is not the custody object; the custody object is the exact artifact lineage that production resolves and executes.
  • Package names, domains, version labels, and script URLs are routing hints, not integrity proofs. Trust should terminate at bytes plus the runtime selectors those bytes depend on.
  • Rollback has to be proved where the victim executed code. Publishing a clean tarball or merging a clean commit is only one stage of recovery.
  • Mutable loaders, dist-tags, remote config, and service identifiers are authority surfaces whenever they can still steer wallet prompts, signing routes, or secret-handling behavior.
  • Secret exposure outlives code exposure. If the bad artifact could see authority material, clean bytes without credential rotation are not honest containment.
  • Shared artifact lineage means shared incident until proven otherwise. If the same compromised package, runtime loader, or service-owned URL fans out across many consumers, rollback should start from one family-level blast-radius map rather than from isolated app postmortems.
  • Browser delivery and backend package resolution are the same design family: both fail when the familiar software identity drifts away from the authority-bearing artifact actually running.
  • Teams that cannot answer what exact artifact did production trust? in one sentence will usually restore the wrong layer first during an incident. The next failure after that is usually treating artifact is authentic again as if it automatically meant artifact still deserves the same live signer authority.
  • Recovery evidence should be symmetrical with compromise evidence. Whatever layer made the attack real is the layer that must later prove the attack is gone.

Where this note helps most

  • Start here when the sharper question is what should healthy source-to-runtime attestation and rollback proof have looked like?
  • Drop into Release-provenance failure when the exploit-side question is how reviewed source, published package, loader-selected version, and delivered runtime bytes drifted apart in the first place.
  • Drop into Software supply-chain compromise when the broader exploit-side question is how a poisoned shared dependency or loader became wallet or signer authority for many downstream apps at once.
  • Pair with Front-end compromise approval phishing when the artifact-integrity failure mainly cashes out through browser-served wallet prompts.
  • Pair with Secret-material observability leakage when the recovery-side question includes whether secrets already crossed into attacker-visible telemetry or runtime logs.

Internal linkages

Control surface

Graph View

Backlinks