Cowrie Research

replayable-secret-non-observability-and-telemetry-exclusion-discipline

14 min read

Replayable-Secret Non-Observability and Telemetry-Exclusion Discipline

  • Name: replayable-secret non-observability and telemetry-exclusion discipline
  • Category: wallet-secret handling reference / telemetry-boundary design note / signer-runtime hardening checklist / observability-exclusion safety reference
  • Summary: Healthy wallet, signer, and custody-adjacent systems do not try to keep raw mnemonics, private keys, seed bytes, or signer-equivalent session artifacts safe after generic app runtimes, crash reporters, analytics SDKs, traces, or support bundles have already seen them. They design the boundary earlier and more bluntly: replayable authority should either never enter those surfaces at all, or it should exist only behind non-extractable handles that those surfaces cannot serialize into future signing power. The review mistake to keep explicit is that replayable authority exists in general-purpose runtime space, some nearby observability or support lane can observe that state, and the resulting observed copy can still persist or replicate as live signer authority are three different approvals, not one vague don't log secrets slogan. This note exists to keep the positive design target explicit instead of making readers reconstruct what correct key / mnemonic handling should have looked like only by inverting incidents like Slope Wallet and the Solana web3.js npm compromise.

Fast classifier: which secret-leak shape are you trying to make impossible?

  • wallet or signer code accidentally logged raw authority — crash reporting, analytics, or debug helpers serialized mnemonic or private-key material during import, creation, or signing. Slope Wallet is the clean wallet-side exemplar.
  • release compromise made signer code exfiltrate raw authority — the application already handled replayable secrets in general-purpose runtime memory, and a malicious package or update simply added the export path. Solana web3.js is the clean backend-runtime exemplar.
  • support / debugging surface quietly became custody — diagnostic bundles, verbose logs, screenshots, traces, or support tooling observed enough secret material that whoever could query them effectively held wallet authority.
  • Use this note when the sharper positive-design question is how one signer-secret boundary should keep replayable authority out of observability, support, and generic runtime surfaces before redaction folklore or backend hardening ever become relevant.

Why this note exists

  • Many wallet-side and signer-runtime incidents are misremembered as keys got stolen when the sharper failure was earlier: the system let replayable authority become normal application data.
  • The practical question is often not was telemetry encrypted? or was Sentry self-hosted? but why could an observability or support surface ever see something that could later sign?
  • The useful design move is to force one blunt invariant across the whole stack: nothing that could later authorize value should be emitted to logs, traces, crash reports, analytics payloads, support bundles, screenshots, or general-purpose debugging helpers.
  • This note is adjacent to exact production artifact attestation and rollback discipline, but the focus here is earlier in the custody story: not which bytes were delivered? but what authority those bytes were ever allowed to hold in ordinary runtime space.

Three custody-boundary promotions healthy systems should stop earlier

  1. Replayable authority exists in ordinary runtime space
    • The first question is whether raw mnemonic, private-key, seed, keystore-plaintext, or long-lived signer-session material ever appears in general-purpose app memory at all.
    • Non-extractable handles, enclave APIs, hardware-backed keys, and remote signers are strong because they prevent the initial promotion from needs to sign to generic runtime holds replayable authority.
  2. Observability or support surfaces can observe that state
    • The second question is whether logs, traces, analytics SDKs, crash reporters, support bundles, screenshots, or convenience serializers are allowed to see or stringify nearby signer state.
    • This is the decisive design boundary for Slope-style incidents: the failure is not only that a secret existed, but that generic debugging machinery was allowed to treat it as ordinary application data.
  3. Observed copies still survive as live authority
    • The third question is whether whatever was observed can persist, replicate, queue, export, or remain queryable in a form that still cashes out as signer power later.
    • A secret-bearing telemetry event that can still be searched, restored, replayed, or exported is not mere metadata residue; it is displaced custody.

The six joints a healthy signer-secret boundary should keep fused

  1. Secret form
    • The system should distinguish replayable authority from non-secret metadata explicitly.
    • Raw mnemonic phrases, private keys, seed bytes, keystore plaintext, long-lived session tokens, and signer-export API credentials are not ordinary application values.
    • If they can be printed, stringified, or serialized like ordinary values, the type boundary is already too weak.
  2. Runtime residency
    • Replayable authority should spend as little time as possible inside general-purpose process memory.
    • Hardware-backed keys, enclave APIs, remote signing, or non-extractable handles are healthier than flows that load raw secret bytes into a mobile app, browser extension, or backend worker and then hope surrounding code behaves.
  3. Observability exclusion
    • Logging, tracing, analytics, crash reporting, and support instrumentation need an allowlist of safe facts rather than a blacklist of dangerous guesses.
    • The security target is not best effort redaction after broad capture; it is secret-bearing paths are structurally non-observable.
  4. Downstream replication boundary
    • A single telemetry event can fan out into queues, data lakes, dashboards, SIEMs, vendor backends, support exports, and contractor-visible tools.
    • Healthy systems treat first capture and later copies as one custody problem rather than pretending the primary log sink is the only risk.
  5. Artifact trust and secret trust
    • Runtime integrity and secret non-observability are coupled. If code that handles replayable secrets can change without strong attestation, a later artifact compromise can turn secret residency into secret exfiltration.
    • This is why Slope and Solana web3.js belong in adjacent design conversations even though one was accidental logging and the other a malicious package release.
  6. Recovery semantics
    • If replayable authority ever crosses the observability boundary, the honest response is secret retirement: wallet migration, key rotation, session revocation, or seed abandonment.
    • We removed the log statement is a code fix, not a recovery claim for previously exposed authority.

What sound design looks like

  • Good designs answer the three promotions aggressively and early: replayable authority is absent from ordinary runtime space where possible, signer-adjacent observability is structurally non-observable where runtime contact still exists, and any remaining metadata path is too narrow to preserve or reconstruct live signer power.
  • Replayable authority is rare and hard to observe
    • The healthiest systems minimize how often raw secrets even exist in generic runtime space and make secret-bearing types impossible to dump accidentally.
    • Observability tools see explicit non-secret facts only: version, device class, route name, error code, maybe a public transaction id.
  • Telemetry policy is allowlist-first on signer paths
    • Wallet import, recovery, signing, key derivation, session restoration, and keystore handling paths should default to no body capture, no object dumps, no stack-local collection, and no breadcrumb payload capture.
    • If a team must observe those paths, the observed schema should be tiny, typed, and reviewed like custody code.
  • Support tooling is treated as a custody-adjacent surface
    • Debug archives, user-submitted diagnostics, screenshots, and remote troubleshooting should be designed under the assumption that they can become exfiltration paths just as easily as formal telemetry.
    • Healthy support flows either exclude signer authority by construction or use one-time, explicitly bounded collection paths with strict review and short retention.
  • Runtime integrity and secret exclusion reinforce each other
    • A healthy signer stack both proves the delivered artifact lineage and constrains what that artifact could observe or emit even if a nearby subsystem misbehaves.
    • The clean target is not trust this app not to log; it is this class of app is not allowed to produce replayable-secret telemetry in the first place.
  • Recovery assumes authority burn, not cosmetic cleanup
    • Once a mnemonic, key, or long-lived signer token may have crossed the line, the system should assume that authority is no longer trustworthy even if evidence of active theft is incomplete.
    • The stronger recovery question is how fast can we enumerate affected authority and retire it?

Healthy architectural shapes

1. Non-extractable key handles by default

  • Prefer hardware-backed keys, enclaves, HSM/KMS signing APIs, passkey-like handles, or remote signers that expose sign but not export secret.
  • This turns nearby telemetry or runtime compromise into a narrower incident than anyone who saw process memory now owns the wallet.
  • The best observability control is often not a better redaction rule, but a design where the runtime never sees replayable authority bytes at all.

2. Secret-bearing types that fail closed on serialization

  • Mnemonic, private-key, keystore-plaintext, and signer-session objects should not implement generic debug printers, JSON serializers, or convenience toString() methods.
  • Attempts to stringify them should fail, redact by construction, or produce clearly non-replayable placeholders.
  • This moves the burden from developer discipline to type-system or interface discipline.

3. Field-level allowlisted telemetry schemas

  • Signer-adjacent telemetry should be tiny and predeclared: route id, failure code, device class, build hash, maybe the public address already onchain.
  • No raw request bodies, no free-form exception object dumps, no automatic stack-local capture, and no generic breadcrumb recording on secret-handling routes.
  • The useful property is that developers cannot accidentally widen the schema with one helper call.

4. Support and debug collection as a separate ceremony

  • If user support or incident response ever needs richer client state, collect it through an explicit break-glass path with one-time approval, strong review, minimal scope, short retention, and clear proof that signer authority is excluded.
  • Upload all logs is not a safe support primitive for wallet or signer software.

5. Secret-retirement-first response playbooks

  • Incident response should start from which authority might have crossed the line? rather than can we keep using these wallets now that the code is fixed?
  • Fast enumerations of affected wallets, sessions, seeds, service credentials, and downstream copies matter more than writing the prettiest redaction patch.

Design questions worth asking first

  • Which exact paths ever load replayable authority into general-purpose runtime memory?
  • What instrumentation can observe those paths today: logs, traces, crash reporters, analytics SDKs, support-bundle generators, screenshots, session replay, clipboard capture, or remote debugging?
  • What proves that signer-adjacent telemetry is allowlist-only rather than broad capture plus redaction hope?
  • If one telemetry or support event were captured right now, how many downstream copies could it create and who could query them?
  • Could a later release compromise turn existing secret residency into active exfiltration without any new architectural boundary being crossed?
  • If a mnemonic or key crossed the boundary for one minute, what exact rotation or migration workflow would fire and how would affected users or operators be identified?

Red flags that usually mean replayable authority is still being treated like normal app data

  • We redact sensitive fields without proving that the secret-bearing path is allowlist-only and structurally non-observable.
  • It is fine because the telemetry is self-hosted even though raw authority already left the signer boundary.
  • The object is encrypted in transit used as if transport security made capture itself acceptable.
  • Only support engineers can see it when support systems were never governed like signer custody.
  • The package compromise only matters if you already had keys in memory stated as if keeping raw keys in memory were not already the main architectural problem.
  • We fixed the logging bug without immediate key rotation, session revocation, or seed abandonment for anything already exposed.

Failure-prevention checklist

  • Review signer paths in promotion order: why does replayable authority exist here at all?, what nearby surface could observe it?, and could any observed copy still survive as live authority? If a design answer starts at retention or backend access control, it is already too late in the sequence.

  • Draw the replayable-authority map explicitly: where mnemonics, private keys, seed bytes, keystore plaintext, long-lived signer sessions, and signer-export credentials can exist in memory, at rest, or in transit.

  • Replace raw-secret residency with non-extractable handles wherever possible: hardware wallets, enclaves, remote signers, HSM/KMS APIs, and scoped signing services are healthier than runtime-held secret bytes.

  • Make signer-adjacent telemetry default-deny and allowlist-only. Disable body capture, object dumps, generic serialization, stack-local collection, and automatic breadcrumbs on wallet import, recovery, signing, and session-restoration paths.

  • Remove generic serializers from secret-bearing types. Mnemonic and key objects should fail closed on debug printing, JSON encoding, and convenience stringification.

  • Treat support bundles, screenshots, debug archives, session replay, and remote troubleshooting as custody-adjacent collection surfaces, not operational afterthoughts.

  • Minimize downstream replication and retention for any signer-adjacent telemetry that remains. If safe metadata must be collected, keep retention short, exports narrow, and query rights tightly reviewed.

  • Couple artifact integrity review with secret-boundary review. Any release that changes signer-path code, SDKs, logging helpers, error handling, or telemetry libraries should be reviewed as both a runtime-integrity and a custody-boundary change.

  • Rehearse secret-retirement response now: enumerate affected wallets or sessions quickly, migrate funds or rotate keys, revoke service credentials, and prove primary plus replicated stores no longer retain accessible secret-bearing events.

  • Alert on the impossible: mnemonic-shaped strings, keystore JSON, base58/base64 key lengths, wallet import payloads, or signer-session secrets observed anywhere in telemetry or support systems should page incident response immediately.

Audit prompts

  • Which signer-paths still blur runtime secret residency, observability access, and downstream live-authority persistence into one undifferentiated secret handling answer instead of proving each boundary separately?

  • Which exact signer-path functions ever materialize replayable secret bytes in application memory, and why do they need to?

  • Can any crash reporter, analytics SDK, tracing layer, support-bundle generator, or debugging helper automatically capture those objects or nearby request bodies?

  • Do secret-bearing types have safe interfaces by construction, or can one helper call stringify or serialize them into future signing authority?

  • What downstream systems can receive one signer-adjacent event: queues, vendors, self-hosted mirrors, SIEMs, analytics warehouses, support dashboards, or contractor tools?

  • How would responders determine whether a later artifact compromise had access to runtime-held secrets even if no direct exfiltration evidence survived?

  • If telemetry or support collection paused and resumed later, could stale secret-bearing events surface after the client-side fix shipped?

  • Which roles can query raw telemetry or support artifacts, and are those privileges governed like signer-adjacent access rather than routine ops access?

  • What exact user or operator migration path exists if one mnemonic, key, or long-lived signer token is suspected to have crossed the boundary?

  • Can the system still function if signer-adjacent telemetry is reduced to explicit field-level safe metadata only? If not, which debugging assumption is currently competing with custody safety?

Lessons that should already be standard

  • Raw secret existed, tooling could observe it, and the observed artifact stayed usable as authority are separate security promotions. Healthy systems force review to stop at the first or second boundary rather than gambling on the third.

  • Telemetry, support tooling, and debugging helpers are part of the custody boundary the moment they can observe replayable authority.

  • Encrypted log transport and self-hosted monitoring are not meaningful defenses if the system already emitted signer-equivalent material.

  • Blacklist redaction is too weak for secret-bearing paths. The safe default is non-observability plus explicit allowlisted metadata only.

  • Runtime integrity and secret residency are one design family. If raw authority sits in ordinary app memory, a later package or frontend compromise can cash that out immediately.

  • Recovery begins with authority retirement, not with prettier logging policy. Once a mnemonic or key may have crossed the line, continued trust in that authority is already a policy gamble.

  • Support and observability convenience routinely compete with signer safety unless the boundary is made structural rather than procedural.

  • The easiest way to keep telemetry from becoming a signer is to keep replayable secrets from existing in general-purpose runtime space at all.

Where this note helps most

  • Start here when the sharper question is what should healthy wallet- or signer-secret handling have made impossible before logs, traces, or support tooling entered the story?
  • Drop into debug-telemetry exfiltration when the exploit-side question is how a telemetry or observability path actually became a signer.
  • Pair with exact production artifact attestation and rollback discipline when the next question is how a compromised package, SDK, or browser-delivered artifact turned existing secret residency into active exfiltration.
  • Pair with software supply-chain compromise when the broader exploit-side family is malicious shared code delivery rather than accidental telemetry design.

Internal linkages

Control surface

Graph View

Backlinks