Cowrie Research

release-provenance-failure

17 min read

Release-Provenance Failure / Source-to-Runtime Artifact Drift

  • Name: release-provenance failure / source-to-runtime artifact drift
  • Category: exploit pattern / software supply-chain compromise / release-integrity failure / artifact-identity drift
  • Summary: Some crypto supply-chain incidents are easiest to misunderstand as npm got hacked or the CDN served bad JavaScript. The sharper recurring failure is that teams authenticated one software object while users or operators executed another. Security review may cover repository source, but production risk often lives in the published tarball, dist-tag resolution, runtime loader, CDN-cached bundle, embedded service identifiers, or the exact browser-fetched bytes. Once reviewed source drifts away from executed artifact, uncompromised domains and uncompromised downstream contracts can still become theft surfaces. This note is the deeper release-integrity subfamily of software-supply-chain-compromise: not every supply-chain compromise needs source-to-runtime drift to matter, but the nastiest ecosystem-wide cases often do.

Quick classifier

  • Stay here when the sharpest preserved lesson is which exact release-stage object drifted away from the thing people thought they had authenticated? That object might be a reviewed source tag, published tarball, loader-selected version, CDN response, embedded service identifier, or the exact browser/runtime bytes that actually executed.

  • Use the shorthand same package name, different custody semantics for backend or library-centric cousins like Solana web3.js, where the familiar dependency identity survived but the published artifact changed what secret-handling or signing code actually did.

  • Use the shorthand same real site, different loader-resolved bytes for browser/runtime loader cousins like Ledger Connect Kit, where downstream domains stayed honest but signer-critical execution was still delegated to mutable package, loader, or CDN identities.

  • Use the shorthand same familiar runtime URL, different operator-resolved bytes for service-identity drift cousins like Polyfill.io, where the trusted object in source control was a script URL or service identifier whose ownership and request-time output changed underneath it.

  • Pivot first to frontend-compromise-approval-phishing when one protocol’s own registrar / DNS / CDN / edge / HTML delivery stack is the decisive compromise surface and the shared-upstream release chain is not the main analytical lift.

  • Pivot first to software-supply-chain-compromise when the decisive compromise lived upstream in a shared dependency or loader ecosystem that many otherwise honest downstream apps inherited; then use this note when the sharper follow-up becomes which source / tarball / loader / CDN / runtime object drifted, and how would rollback have been proved at the executed-artifact layer?

  • Pivot first to canonical-domain runtime-byte integrity when the design question is what the real domain should have proved before a user trusted its wallet prompt.

  • Pivot first to third-party runtime allowance and integrity pinning when the design question is why a signer-facing route was still willing to execute mutable remote code at all, even if no broader source-to-runtime artifact drift needs to be reconstructed.

  • Ledger Connect Kit is the clean seam case: the compromise first lived upstream in a shared package / loader ecosystem, many downstream dApps inherited it, and the sharper follow-up is then which exact loader-selected / CDN-cached / browser-fetched artifact users were still executing after the nominal fix.

  • What it is:

    • A team or downstream integrator believes it has authenticated the software that will shape wallet prompts or touch signing keys.
    • In reality, the reviewed object and the executed object differ somewhere along the release path: source tag, package tarball, runtime-loaded bundle, CDN edge response, loader-selected version, or dependent remote configuration.
    • That drift lets attacker-controlled code inherit the trust attached to a real package name, real domain, or real release process.
    • The resulting theft path may be browser-side wallet-drainer prompts, build-time wallet implantation, or backend secret exfiltration, but the preserved design failure is that release identity was not pinned end to end.

  • Typical preconditions:

    • Teams audit source repositories or semver package names more carefully than the exact artifacts production users actually fetch.
    • Runtime loaders, mutable dist-tags, CDN paths, or automatic latest-version resolution can change behavior without an explicit downstream redeploy.
    • Browser-delivered or backend-resolved artifacts are not pinned with strong integrity checks and are not routinely compared against the reviewed source or built artifact.
    • Shared dependencies have authority to construct signing prompts, handle approvals, or touch raw private keys.
    • Incident response can publish a clean replacement faster than it can prove all caches, loaders, service identifiers, and exposed secrets have been cleared.

  • Common vulnerable surfaces:

    • package-registry publication versus source-repository review
    • mutable dist-tags, semver drift, and runtime latest resolution
    • package tarballs or build artifacts that differ from the reviewed source tag
    • CDN-backed wallet SDK loaders and browser-fetched bundles
    • embedded project IDs, RPC endpoints, remote config URLs, or service identifiers that can redirect wallet traffic even when the package name stays the same
    • build pipelines where one trusted dependency can rewrite downstream shipped artifacts
    • rollback paths that republish good code but leave poisoned bytes or stolen secrets live elsewhere

  • Exploit sequence:

    1. The attacker gains control over some release-stage object: publish account, transitive dependency, runtime loader, CDN-served artifact, or equivalent distribution surface.
    2. The apparent software identity remains plausible: same package name, same real dApp domain, same familiar wallet flow, or same reviewed repository.
    3. Downstream users or operators execute the attacker-shaped artifact actually resolved in production, not the benign object they thought they had authenticated.
    4. That artifact shapes wallet prompts, rewrites bundled output, or exfiltrates secret material from hot code paths.
    5. The attack can persist beyond first disclosure because clean version published and all environments now execute clean bytes with rotated secrets are not the same state.

  • Variants:

    • Transitive build-artifact substitution — a malicious nested dependency rewrites the final shipped wallet or app artifact during the victim’s own release process, as in Copay.
    • Runtime-loader drift — downstream sites load whatever package version or bundle a shared loader/CDN serves at page load, as in Ledger Connect Kit.
    • Runtime-URL / service-identity drift — the same familiar script URL or service endpoint survives in source code while domain ownership, request-time generation, or live delivery behavior changes underneath it, as in Polyfill.io.
    • Publish-account artifact swap — the package name and version stream remain canonical, but a malicious release changes key-handling logic inside a widely trusted SDK, as in Solana web3.js.
    • Rollback-lag persistence — maintainers restore a clean release, but browsers, CDNs, service credentials, or already-exposed keys preserve the attack path after the nominal fix.
    • Remote-configuration identity drift — the artifact itself may be unchanged, but embedded project IDs, endpoints, or remotely fetched configuration redirect signing or wallet-connection behavior.

  • Detection heuristics:

    • Compare source tags, published tarball hashes, loader-selected versions, domain / service ownership, CDN responses, and browser-fetched bytes; treat disagreements as a security incident, not a packaging nuisance.
    • When the sharper question is how healthy teams should attest that whole lineage and later prove rollback at the same execution layer users hit, use exact production artifact attestation and rollback discipline.
    • Alert on unexpected dist-tag moves, package publication by unusual accounts, or sudden runtime-version drift in wallet-critical code.
    • Sample live front-end artifacts from multiple regions or CDN edges before declaring a rollback complete.
    • Diff spender addresses, typed-data domains, key-handling code paths, remote endpoints, and service identifiers across package updates instead of only diffing visible UI behavior.
    • Ask which exact object downstream users are trusting: repo commit, tarball, locally bundled artifact, loader URL, or whatever the browser fetched this minute.

  • Defenses:

    • Pin exact versions and prefer locally bundled immutable artifacts over mutable runtime loaders for wallet-critical or signer-critical paths.
    • Use integrity checks that bind the runtime artifact, not just the repository review outcome: content-addressing, subresource integrity, signed artifacts, import-map pinning, or equivalent controls.
    • Separate and monitor the full release chain: source review, tarball publication, version resolution, domain / service ownership, CDN distribution, remote config, and runtime fetch.
    • Treat embedded service identifiers and remote endpoints as part of the same release-integrity boundary as the package code itself.
    • Rehearse incident response that includes cache purges, forced version pins, loader disablement, remote-config revocation, and key rotation where secrets may have been exposed.
    • Require evidence at the layer users actually execute before declaring recovery: not just fixed commit merged, but expected bytes are what browsers and backends now resolve. Make the release review state the three promotions explicitly: what familiar object exists, what proves it is still the canonical production artifact, and what surviving authority that artifact is still permitted to exercise.

Three production-artifact promotions healthy systems should not collapse

  1. A familiar release-stage object exists
    • A repository tag, package name, semver line, tarball, CDN URL, loader alias, or service identifier can exist and still be only a routing hint.
    • Same package, same script URL, or same version family does not yet mean the signer-critical thing downstream users execute is the object defenders meant to bless.
  2. That object counts as canonical production truth for this signer surface
    • Healthy systems separately prove that the reviewed source, built artifact, published tarball, runtime-selected version, embedded service identifiers, and finally fetched bytes still describe one production object.
    • This is the step Copay, Ledger Connect Kit, Polyfill.io, and Solana web3.js each collapsed in different ways: something familiar stayed reachable, but it stopped meaning the artifact we actually authenticated.
  3. That accepted artifact still deserves live wallet or signer authority right now
    • Even a genuinely canonical artifact should not automatically keep broad authority forever. The remaining question is what spender prompts, permit flows, hot-key access, remote service reach, or signer budget that artifact is still allowed to exercise today.
    • This is why rollback has to prove more than clean bytes published: defenders must also prove caches, loaders, service identifiers, and any secrets or approvals the bad artifact touched are no longer live authority.

What sound design looks like

  • A healthy release path names one production object all the way through: reviewed source, reproducible build output, published tarball or bundle hash, runtime-selected version, embedded service identifiers, and the exact bytes a browser or backend will actually execute. The stronger companion on that positive-design side is exact production artifact attestation and rollback discipline: this family is what breaks when that one object silently splits.

  • Source review and runtime trust should meet at an explicit attestation boundary, not at hope. A secure team can answer which exact artifact did production trust? without switching vocabulary between repo commits, semver tags, CDN URLs, loader aliases, and browser-cached bundles mid-sentence, and without collapsing this object exists, this object is canonical production truth, and this object still deserves signer authority into one casual claim.

  • Mutable selectors should be treated as authority, not convenience. Dist-tags, runtime loaders, CDN aliases, project IDs, remote config endpoints, and service-owned script URLs are safe only when they are either pinned into the same reviewed production object or fail closed before they can redirect signer-facing behavior.

  • Rollback should be symmetric with compromise. If the attack path lived in tarball publication, loader resolution, CDN cache state, or remote service identity, recovery should prove the same layers have returned to one clean object rather than stopping at good code was republished.

  • Secret exposure should be modeled as part of release provenance, not as a separate afterthought. If the bad artifact could observe mnemonics, private keys, API tokens, session credentials, or wallet-connection project IDs, a healthy design assumes rollback is incomplete until those authorities are rotated or revoked.

  • Browser-facing and backend-facing release integrity should share the same habit of proof. Wallet prompts, bot signers, relayers, and custody backends all need to know whether the runtime bytes still match the reviewed lineage; the only difference is what kind of authority those bytes can spend once drift occurs.

  • The clean design question is: if users or operators later say we trusted version X, what exact artifact, selector, and service identity does that sentence name, and how would the team prove that production really executed that same object during both rollout and rollback?

  • High-signal incidents:

Failure-prevention checklist

  • Define one production artifact manifest for every signer-critical release that pins reviewed source commit, build output hash, published tarball hash, intended resolved version, allowed loader behavior, expected service identifiers, and expected browser/backend runtime hashes.
  • Require promotion gates at the executed-artifact layer, not only the source or package layer. A wallet-facing release should not go live until operators can prove what exact bytes browsers and backends will resolve in production.
  • Remove mutable version selection from signer-critical paths wherever possible: no latest, broad semver ranges, or unpinned runtime loaders on code that can shape wallet prompts or touch keys.
  • Treat embedded service identifiers, remote config URLs, wallet-connect project IDs, RPC selectors, and CDN loader endpoints as part of the same release object as the package code.
  • Keep rollback evidence symmetrical with release evidence. If the incident fix was good version published, recovery is incomplete until caches, dist-tags, CDN edges, remote config, and runtime fetches all prove the same clean object.
  • Couple artifact rollback with secret disposition. If the poisoned artifact could observe mnemonics, raw private keys, session tokens, or API credentials, require rotation / revocation before closing the incident.
  • Build outside-in verification that samples multi-region browser and backend fetches and alerts on runtime hash drift, unexpected dist-tag movement, or changed remote dependency graphs.
  • Prefer self-hosted immutable bundles or strongly pinned local build artifacts for wallet-critical flows; if remote loaders remain, document the exact exception and fail-closed conditions.
  • Inventory which build steps or transitive dependencies can rewrite final shipped artifacts, and require attestation that the released bundle still matches the reviewed lineage after those steps complete.
  • Rehearse emergency controls that can disable runtime loaders, pin known-good versions, revoke remote config, and switch signer-critical routes into read-only or maintenance mode quickly.

Audit prompts

  • What exact object does the team believe production authenticated: repo commit, built bundle, published tarball, resolved package version, CDN-served asset, embedded service identifier, or the exact bytes a real browser fetched?
  • Where can production behavior still change without a downstream redeploy: dist-tags, semver ranges, runtime loaders, remote config, CDN edge rules, service ownership, or build-time transitive dependencies?
  • Can you prove that the published tarball and runtime-resolved bundle match the reviewed build output, or are those being assumed to be equivalent?
  • Which dependencies or services on signer-critical routes still have authority to change wallet prompts, typed-data rendering, spender destinations, or key-handling behavior after source review?
  • If a clean rollback were published right now, what evidence would prove that users in multiple regions and caches have stopped executing poisoned bytes?
  • Which secrets, credentials, session tokens, or keys may already have been exposed by the compromised artifact, and what exact rotation path exists for each?
  • Do incident runbooks distinguish registry/package state is clean from runtime execution state is clean, or is rollback still being judged at the wrong layer?
  • Are embedded project IDs, RPC endpoints, or service identifiers versioned and attested alongside the code, or can they drift under a stable package identity?
  • Which release accounts, CI steps, or transitive dependencies can still modify the final artifact after the human review step has completed?
  • How would the team answer a postmortem question of the form what exact bytes were users or operators executing during the compromise window?

Lessons that should already be standard

  • Package names, domains, and semver lines are not integrity proofs. In signer-critical systems, the real custody object is the exact executed artifact plus the runtime selectors it trusts.
  • Reviewed source and safe production are only loosely correlated unless the build, publication, resolution, and runtime-delivery layers are all pinned into one explicit lineage.
  • Recovery must be proven where the victim executed code, not where the maintainer published a fix. Registry cleanup without runtime verification leaves the exploit path half open.
  • Shared loaders, mutable dist-tags, and remote config are deployment authority even when teams talk about them as convenience plumbing.
  • Secret exposure outlives code exposure. Once a bad artifact could touch keys or credentials, rollback without rotation is not honest containment.
  • Browser-facing and backend-facing supply-chain incidents are the same design family more often than teams admit: both fail when a familiar software identity is allowed to drift away from the exact authority-bearing bytes.
  • If the team cannot state what exact artifact did production trust? in one sentence, incident response will improvise under pressure and likely restore the wrong layer first. The recurring review failure is collapsing object existence, artifact authenticity, and live signer authority into one vague we trust this version story.

Internal linkages

Control surface

Graph View

Backlinks