Passwords might end up in logs during system issues, like when a 401 error causes sensitive data to leak through a serializer. Traditional fixes such as rotating keys, purging logs, and regex redaction don't always work well.
Default log management settings often make things worse under stress:
- Request logging grabs URLs and bodies.
- Exception serialization logs sensitive data.
- Structured logging might capture entire objects with secrets.
- Middleware logs headers.
- CLI tools and libraries might accidentally log passwords.
- Changes in data structure can slip through redaction.
Treat Logs Like a Data Product:
-
Design for Minimization:
- Log events only, not full objects. Use allowlisted paths.
- Block bodies on sensitive routes and omit query strings.
-
Edge Enforcement:
- Use a logger API to remove sensitive info and standardize values.
- Avoid capturing raw bodies; log only selected metadata.
- Redact headers at load balancers.
-
Testing and Automation:
- Use analysis tools and perform log contract tests.
- Canary tokens can help with monitoring.
-
Operational Control:
- Limit changes to log levels and retention of sensitive data.
- Enforce strict access controls and ensure proper maintenance.
- Have incident playbooks ready.
Practical Tips:
- Log only necessary fields, not entire objects.
- Prefer dynamic allowlists instead of simple key-based redaction.
- Steer clear of stack traces that have sensitive data.
- Configure third-party tools to avoid default full logging.
Implementation Strategies:
- Build a logging module that complies with your schema.
- Hash stable identifiers and truncate data that's too long.
- Use interceptors and gateways for monitoring.
Adjustments for Languages and Frameworks:
- Restrict problematic logging defaults in languages like Go, Java, Python, Node.js, Ruby, and frameworks like gRPC.
APM and Proxy Settings:
- Limit details in full request captures; adjust attribute settings accordingly.
Success Metrics:
- Cut down on data sources that emit too much and reduce canary token alerts.
- Stick with approved logging methods and quickly respond to leaks.
Aim for better handling of structured data, acknowledge some loss in debugging detail, and roll out key practices within 30 days: set up allowlist-friendly logging, disable unnecessary logs, and thoroughly test to ensure robust log management.
