Fail Closed or Don't Ship the Webhook

Unsigned success is worse than downtime.

If a payment, KYC, or chat endpoint accepts traffic when the shared secret is missing, you did not “stay flexible.” You taught production to trust strangers. On Muchiround that lesson showed up the honest way: one rail did fail-closed correctly, several adapters still failed open, and a pentest made the asymmetry impossible to ignore.

This is the field note — not a threat-model essay. Copy the contract. Fix the stubs. Wire secrets into Kamal before you celebrate the merge.

Product framing

Webhook handlers are public write surfaces with a private language: HMAC, Ed25519, provider headers. The product promise is “only the provider can mutate this state.”

When verification is optional in production, the promise becomes:

• anyone who finds the URL can mark a charge paid
• anyone can approve or decline KYC
• anyone can inject chat events as if they were the bot platform

That is not a P2 hygiene ticket. For a wallet and identity product it is a trust bug. Users in Zimbabwe, South Africa, and the diaspora do not get a second chance to believe your ledger after a spoofed credit.

Related reading on the payment path itself: One Fiat Process Path and Custom Currencies Without Lying to Your Ledger. This article is only about the door on the way in.

The positive control: Paystack

After SPEC-005 and the C1 security pass, Paystack verification lives in one module with a single secret resolver. Blank secret does not mean “skip check” outside local environments.

# Payments::WebhookSignature — fail-closed outside development/test
module Payments
  module WebhookSignature
    module_function

    def valid_paystack?(payload, signature)
      secret = Pay::Paystack.webhook_secret.to_s
      if secret.blank?
        return allow_unsigned_paystack?
      end
      return false if signature.blank?

      expected = OpenSSL::HMAC.hexdigest("SHA512", secret, payload.to_s)
      ActiveSupport::SecurityUtils.secure_compare(expected, signature.to_s)
    end

    def allow_unsigned_paystack?
      return true if Rails.env.test? || Rails.env.development?
      # Explicit allow-unsigned only honored in local-class envs, never as a silent default in prod
      false
    end
  end
end

Controllers and adapters call this — they do not re-implement HMAC with a private “if secret missing, continue” branch. Controllers that used to fork verification logic were how secrets drifted (Pay::Paystack.webhook_secret vs a second ENV name nobody deployed).

Rules we keep:

1. One resolver for the provider secret.
2. Blank secret → reject in production/staging.
3. Blank signature → reject when a secret exists.
4. secure_compare, always.
5. Specs for blank-secret production env, bad HMAC, good HMAC — not only the happy path.

Cloudflare Stream’s TV webhook follows the same spirit: missing secret raises; it does not soft-accept. Twilio lead inbound validators return false in production when the auth token is missing, and they refuse to honor a skip switch in production.

Failure modes we still own

The uncomfortable part of a real codebase: not every mount matched Paystack yet when security walked the tree.

1. Blank-secret early return

DidIt KYC verification still had the classic stub:

def self.verify_signature!(raw_body, signature)
  secret = ENV.fetch("DIDIT_WEBHOOK_SECRET", nil)
  return if secret.blank? # dev fallback — warn but don't block

  expected = OpenSSL::HMAC.hexdigest("SHA256", secret, raw_body)
  raise SignatureError unless ActiveSupport::SecurityUtils.secure_compare(expected, signature.to_s)
end

In development that feels kind. In production it means deploy without the Kamal secret and the endpoint still mutates KYC state. The fix shape is the Paystack shape: blank secret outside dev/test raises or returns false; never return success.

2. verify always true

Discord external-chat adapter:

def verify_webhook(request)
  # Discord verifies webhooks with Ed25519 signature
  # Stub: implement when credentials are configured
  true
end

A stub that returns true is an open door with a comment. Prefer one of:

• implement Ed25519 verification, or
• unmount / feature-flag the route until verification exists, or
• fail closed when the public key ENV is blank outside local envs.

Telegram was a softer variant of the same bug: return true if expected.nil? — missing bot secret accepts every request.

3. Secret in docs, not in deploy

A runbook that names DIDIT_WEBHOOK_SECRET while deploy.yml only wires API keys is how fail-open code survives “we thought prod was protected.” Ops and eng share one checklist:

Check	Pass means
Code fail-closed	blank secret rejects in Rails.env.production? (and staging if it is internet-facing)
Kamal env.secret	name present; container presence-only check after deploy
Provider dashboard	webhook URL + signing secret match what the app reads
Spec	production-env example with blank secret expects 401/403/raise
Log hygiene	no secret values, no full unsigned bodies in info logs

Presence checks only — never paste secret values into Kanban, Basecamp, or PR comments.

The contract (copy this)

For every new inbound webhook:

1. Route is public by design — treat it as hostile.
2. Secret from ONE resolver (gem config or ENV), not two.
3. if secret.blank?
     reject unless local-class env (test/development)
4. if signature.blank? → reject
5. constant-time compare
6. bind business fields server-side (amount, user, purpose) — signature alone is not enough for money
7. idempotency key before side effects
8. if you cannot verify yet → do not mount the route in production

Money paths add the extra bind from One Fiat Process Path: amount and currency on the webhook must match the local charge before credit. Signature proves the provider spoke; your charge row proves what you meant to sell.

Checklist for the next PSP (and the next chat rail)

Use this when you add EcoCash, Paynow, another KYC vendor, or another bot platform:

• Signature module with explicit allow_unsigned? that is false in production
• Controller before-action returns 401/403 on failure (no half-processed jobs)
• Factory/spec helpers for valid and invalid signatures
• Production boot or deploy assert: if flag enabled, secret must be present
• Fake/test adapter hard-disabled outside test/development
• Kamal secret name in env.secret + secrets example (no values in git)
• Runbook: rotate secret, dual-run old/new if provider supports it
• Discord-class case: no true stubs on internet-facing mounts

What “done” looks like operationally

Shipping the Ruby change is half the work. Done means:

1. PR merged under a real gate (empty checks are not green).
2. Secrets present on the deploy host.
3. Deploy image includes the fail-closed commit.
4. Smoke: signed request accepted; unsigned or blank-secret rejected.
5. Residual cards for every remaining return true / return if secret.blank? with an owner.

We still track DidIt/Telegram fail-closed and Discord verify-or-unmount as security work — naming them in public is intentional. Hiding open stubs does not make them safer.

Tradeoffs

Fail-closed can page you. A missing secret after a bad deploy rejects every KYC callback until ops fixes ENV. That is the correct failure: loud, finite, recoverable. Silent acceptance is quieter and expensive.

Local DX. Test and development may allow unsigned traffic so feature work does not require a full provider sandbox. Keep that branch tiny and environment-gated. Never copy if secret.blank?; return true; end into a shared helper without the env gate.

Provider quirks. Some vendors send multiple signature schemes or rotate headers. Still fail closed on “no secret configured.” Complexity belongs in the compare step, not in “skip when confused.”

What’s next

If you maintain Rails webhooks in African fintech or marketplace products:

1. rg "return if secret.blank|return true$" app/ and read every hit.
2. Promote one provider (your money path) to a shared signature module.
3. Make blank-secret production specs the merge tax for any new mount.
4. Pair every code fix with a Kamal secret line — code without deploy wiring is a story you tell yourself.

Fail closed, or do not ship the webhook. Everything else is a comment on an open door.