A plaintext password showing up in logs during an on-call situation highlights the gaps in even the strictest no-secrets policies.
Why This Happens: Logging is all over the place, leading to leaks:
- Middleware logs errors and exposes credentials.
- Auto-serialization prints out sensitive stuff.
- Debug logs accidentally capture inputs.
- Third-party tools go overboard with logging.
- Tracing spreads secrets everywhere.
- Debugging tools don’t get turned off.
- Mixed content slips past redactors.
Why Common Fixes Don't Cut It:
- Enforcing policies isn’t easy.
- Regex can miss things or flag false positives.
- Allow/deny lists have trouble with changing headers.
- Encryption doesn’t stop log exposure.
- Unexpected components might log bodies.
- Configurations drift without notice.
- Scrubbing isn’t consistent.
- No checks for regression.
What Actually Works: Go for smart design and proactive strategies:
- Typed Secrets:
- Use secret types that don’t serialize or show redacted versions.
- Default-Deny High-Risk Logs:
- Avoid logging request/response bodies by default. Hash sensitive headers.
- Centralized Structured Logging:
- Set up log normalization early with a redaction policy.
- Control Logging in Dependencies:
- Wrap HTTP clients/servers. Turn off parameter logging in DB/ORMs.
- Contextual Logging:
- Pay attention to event intent, using identifiers instead of full values.
- Continuous Testing and Detection:
- Use secret fixtures in tests. Keep scanning logs for secret patterns.
- Governance and Ergonomics:
- Provide a centralized logging SDK with a sensitivity guide.
Success Measures:
- No plaintext secrets in logs for 30 days.
- Full compliance with centralized logging policy.
- No unauthorized logging of bodies.
- Strict control over data retention and access.
Implementation Steps:
- Audit your log emitters, stop logging bodies, and use a centralized logging agent. Go for structured logs and test regularly.
If Breaches Happen:
- Isolate logs, change credentials, address sources, and update policies.
In Short: Credential leaks happen because of messy logging. Focus on smart system design, precise typing, and organized logging to stop them.
