devops

What this skill does

# DevOps Discipline

**Declarative, reproducible, observable, secure by default.**

Every infrastructure failure traces to one of four root causes:

- Undeclared state — infrastructure exists that isn't in code
- Unverified changes — changes deployed without testing or rollback plan
- Invisible systems — infrastructure that can't be monitored or debugged
- Assumed security — security treated as a later hardening step instead of a default

This skill prevents all four.

## References

- **IaC principles** — [`${CLAUDE_SKILL_DIR}/references/iac-principles.md`]: GitOps v1.0 spec, 12-factor methodology,
  state management comparison (stateful/stateless/GitOps), configuration design patterns, IaC maturity data
- **Observability** — [`${CLAUDE_SKILL_DIR}/references/observability.md`]: Three pillars deep dive, SLI/SLO/error budget
  framework, DORA metrics, toil reduction patterns, common observability gaps, sampling strategies
- **Change management** — [`${CLAUDE_SKILL_DIR}/references/change-management.md`]: Deployment strategy comparison table,
  release engineering principles, rollback requirements checklist, progressive delivery tooling
- **Security posture** — [`${CLAUDE_SKILL_DIR}/references/security-posture.md`]: Zero trust NIST domains, JIT access
  patterns, certificate lifecycle, machine identity governance, supply chain integrity, policy-as-code tooling
- **Disaster recovery** — [`${CLAUDE_SKILL_DIR}/references/disaster-recovery.md`]: RTO/RPO sizing by business impact,
  service tiering, recovery architecture trade-offs, chaos engineering approaches and tooling
- **Testing** — [`${CLAUDE_SKILL_DIR}/references/testing.md`]: Testing pyramid layers with tool lists, IaC quality
  metrics, runbook format and automation maturity levels, incident response patterns

## Core Principles

These apply to all infrastructure work regardless of tool.

### Everything in Code

- All infrastructure is defined in version-controlled, declarative code
- No manual changes through CLI, console, or SSH. If it's not in code, it doesn't exist
- Configuration is separated from code — config varies between environments, code does not
- Changes go through code review before deployment
- Every change is auditable: who changed what, when, why

### Idempotent and Reproducible

- Applying configuration twice produces zero changes on the second run. If the second apply shows diffs, the
  configuration is broken
- Given the same inputs, the system produces identical infrastructure every time
- Pin all dependency versions — tools, providers, modules, images. "Latest" is not a version
- Builds are hermetic: independent of the machine running them

**Pinning example:**

- Bad: `image: nginx:latest` / `version: ">=2.0"` / `hashicorp/consul:*`
- Good: `image: nginx:1.25.4@sha256:6a5...` / `version: "2.3.1"` / `hashicorp/consul:1.17.2`

### Immutable Over Mutable

- Prefer replacing infrastructure over patching in place
- Servers are cattle, not pets — replaceable units provisioned from known definitions
- Mutable infrastructure accumulates drift, undocumented changes, and snowflake state
- When mutation is unavoidable (stateful systems), treat it as a managed exception with explicit drift detection

### Secure by Default

- Least privilege everywhere — no wildcard permissions, no shared credentials, no "tighten later". 99% of cloud
  identities use less than 5% of granted permissions — right-size from the start
- Zero trust — network location does not grant trust. Authenticate and authorize every request regardless of origin.
  Verify across all NIST domains: identity, device, network, application workload, data, visibility/analytics,
  governance
- Assume breach — treat all traffic (including internal) as potentially malicious. Focus on internal monitoring,
  session-based auth, limiting lateral movement
- Just-in-time access — elevated permissions are temporary (4-8 hour maximum lifespan) and expire automatically. Static
  elevated permissions accumulate into permission sprawl (~25% growth per quarter without lifecycle management)
- Encrypt in transit (TLS everywhere, no "internal-only" exceptions) and at rest
- Never commit secrets to version control. Use a secrets manager. Rotate on schedule
- Pin dependencies, verify checksums and signatures, scan for vulnerabilities. See
  [`${CLAUDE_SKILL_DIR}/references/security-posture.md`] for supply chain integrity requirements

**Secrets example:**

- Bad: `db_password: "hunter2"` in a YAML file committed to git
- Bad: `ENV DB_PASS=hunter2` baked into a Dockerfile
- Good: `db_password: "{{ vault_db_password }}"` with Ansible Vault
- Good: `password_file: /run/secrets/db_pass` with Docker secrets or SOPS

### Observable

- Every component exposes the four golden signals: latency, traffic, errors, saturation
- Collect all three observability pillars: metrics, logs, and traces. Metrics for alerting and trends, logs for event
  context, traces for distributed request paths
- Monitoring (reactive, predefined thresholds) is necessary but not sufficient. Observability (proactive, infer internal
  state from outputs) is required for distributed systems where failure modes are emergent and unpredictable
- Alert on symptoms (what's broken), not causes (why). Use causes for debugging
- Every alert must be actionable — if the response is "ignore it," delete the alert. Alert fatigue is the primary
  obstacle to fast incident response
- Use distributions and percentiles, not averages. Averages hide tail latency
- Monitor saturation proactively — latency spikes are leading indicators of saturation
- Manage observability configuration as code — dashboards, alerts, SLOs, and notification policies live in Git and
  deploy via CI/CD

### Recoverable

- Every stateful component has a backup strategy with defined RTO and RPO
- Backups are automated, encrypted, stored off-site, and regularly test-restored
- Every change has a defined rollback path before deployment
- Recovery procedures are documented as runbooks with specific commands, not prose
- Validate resilience through chaos engineering — controlled fault injection to verify recovery procedures work before
  real incidents occur

## Deliverable Standards

<deliverable-checklist>
An infrastructure deliverable is not done until all of these are true:

- [ ] **Defined in code** — declarative, version-controlled, reviewed
- [ ] **Idempotent** — second apply produces zero changes
- [ ] **Tested** — static analysis passes, integration verified in test environment
- [ ] **Secrets managed** — no secrets in code or config; injected at runtime
- [ ] **Monitoring in place** — golden signals exposed, alerts configured, dashboard exists
- [ ] **Rollback defined** — documented procedure, tested, with known duration
- [ ] **Backup strategy** — automated backups for stateful components, restore tested
- [ ] **Documented** — topology, access, dependencies, runbooks, recovery procedures
- [ ] **Environment parity** — dev/staging/prod use the same definitions with environment-specific variables.
      Differences are explicit and minimized
- [ ] **Security reviewed** — least privilege, encryption, access control verified
- [ ] **Supply chain verified** — dependencies pinned, checksums validated, images signed where applicable,
      vulnerability scanning in CI/CD
- [ ] **Compliance enforced** — policy-as-code checks pass in CI/CD pipeline; organizational policies (naming, tagging,
      allowed regions) are automated
- [ ] **Drift detection active** — continuous monitoring for configuration drift with alerting or auto-remediation
- [ ] **Deprovisioning defined** — teardown procedure exists, handles dependencies in correct order, verified to leave
      no orphaned resources </deliverable-checklist>

## Change Management

### Before Deploying

- Review the plan diff — what will be created, modified, destroyed
- Verify rollback path exists and is documented
- Confirm monitoring will detect problems during rollout
- Assess