emergency-release-workflow

What this skill does

# Emergency Release Workflow Skill

## Summary
Fast-track workflow for critical production issues requiring immediate deployment. Covers urgency assessment, expedited PR process, deployment verification, and post-incident analysis.

## When to Use
- Critical production bugs affecting users
- Security vulnerabilities (CVEs)
- Urgent business requirements
- Data integrity issues
- Service outages
- Payment processing failures

## Urgency Assessment

### Priority Levels
| Level | Type | Response Time | Deployment | Example |
|-------|------|---------------|------------|---------|
| **P0** | Security vulnerability | < 2 hours | Immediate to production | Auth bypass, data leak, active exploit |
| **P1** | Production down | < 4 hours | Same day | App crash, complete feature failure, payment down |
| **P2** | Major bug | < 24 hours | Next business day | Critical feature broken, significant user impact |
| **P3** | Business critical | < 48 hours | Scheduled release | Marketing campaign blocker, partner deadline |

### P0 Criteria (Immediate Action)
- Authentication/authorization bypass
- Data breach or exposure
- Remote code execution vulnerability
- Production service completely unavailable
- Data corruption affecting multiple users
- Payment processing completely broken

### P1 Criteria (Same Day)
- Critical feature completely broken
- Error affecting majority of users
- Revenue-impacting bug
- Database connectivity issues
- Third-party integration failure (critical service)

### P2 Criteria (Next Business Day)
- Major feature partially broken
- Affects specific user segment
- Workaround available but not ideal
- Performance degradation (not complete failure)

---

## Emergency Release Process

### 1. Create Hotfix Branch
```bash
# Branch from current production (main)
git checkout main
git pull origin main

# Create hotfix branch
git checkout -b hotfix/ENG-XXX-brief-description

# Example:
git checkout -b hotfix/ENG-1234-fix-auth-bypass
```

### 2. Implement Minimal Fix
```
⚠️ CRITICAL: Minimal change only

DO:
✅ Fix the immediate issue
✅ Add regression test
✅ Document root cause in comments

DON'T:
❌ Refactor surrounding code
❌ Fix unrelated issues
❌ Add new features
❌ Update dependencies (unless that's the fix)
```

### 3. Test Thoroughly
```bash
# Run full test suite
pnpm test

# Type check
pnpm tsc --noEmit

# Build verification
pnpm build

# Manual testing checklist:
# - [ ] Reproduce original issue
# - [ ] Verify fix resolves issue
# - [ ] Test happy path
# - [ ] Test edge cases
# - [ ] Verify no new issues introduced
```

### 4. Create PR with Labels
```bash
git add .
git commit -m "fix: [brief description of fix]

Fixes critical issue where [description].
Root cause: [explanation].

Ticket: ENG-XXX
Priority: P0"

git push origin hotfix/ENG-XXX-brief-description
```

**Use clear labels in PR title:**
- `[RELEASE]` - Direct to production
- `[HOTFIX]` - Critical fix, expedited review
- `[P0]` or `[P1]` - Priority indicator

---

## PR Template for Hotfixes

### Hotfix PR Template
```markdown
## 🚨 [RELEASE] ENG-XXX: Brief description of fix

### Urgency
- [x] P0 - Security vulnerability
- [ ] P1 - Production down
- [ ] P2 - Major bug
- [ ] P3 - Business critical

### Impact
**Users affected**: [All users / Premium tier / Specific region / etc.]

**Severity**: [Choose one]
- [ ] Service completely unavailable
- [ ] Critical feature broken
- [ ] Security vulnerability
- [ ] Data integrity issue
- [ ] Degraded performance

**User impact**:
Describe how this affects end users.

### Root Cause
[Brief explanation of what caused the issue]

**How it happened:**
1. [Step 1]
2. [Step 2]
3. [Result: issue manifested]

**Why it wasn't caught:**
- [ ] Missing test coverage
- [ ] Race condition in production
- [ ] External service behavior changed
- [ ] Recent deployment introduced regression
- [ ] Other: [explain]

### The Fix
[What this PR changes to resolve the issue]

**Changes made:**
- Modified `file.ts` to [specific change]
- Added validation for [specific case]
- Fixed logic in [specific function]

**Why this fixes it:**
[Explanation of how the change resolves the root cause]

### Testing
- [ ] ✅ Reproduced issue locally
- [ ] ✅ Verified fix resolves issue
- [ ] ✅ Regression test added
- [ ] ✅ No other functionality affected
- [ ] ✅ Tested edge cases
- [ ] ✅ Deployed to staging and verified

### Regression Test
```typescript
// Test added to prevent recurrence
describe('ENG-XXX: Auth bypass fix', () => {
  it('should reject expired tokens', async () => {
    const expiredToken = generateExpiredToken();
    const response = await fetch('/api/protected', {
      headers: { Authorization: `Bearer ${expiredToken}` }
    });
    expect(response.status).toBe(401);
  });
});
```

### Rollback Plan
**If this causes issues:**

```bash
# Option 1: Revert commit
git revert <commit-hash>
git push origin main

# Option 2: Deploy previous version
vercel rollback  # or your platform's rollback command

# Option 3: Feature flag
Set FEATURE_FIX_XXX=false in environment
```

**Monitoring:**
- [ ] Error rate in Sentry
- [ ] API response times in monitoring dashboard
- [ ] User reports in support channels

### Deploy Checklist
- [ ] PR approved by at least one reviewer (waive for P0 if necessary)
- [ ] All CI checks pass
- [ ] Deployed to staging and verified
- [ ] Monitoring alerts configured
- [ ] On-call engineer notified
- [ ] Ready for production deployment

### Post-Deploy Verification
**Immediately after deploy:**
- [ ] Verify fix in production (test endpoint directly)
- [ ] Check error tracking (Sentry, etc.)
- [ ] Monitor for new errors
- [ ] Confirm user reports stop coming in

**Metrics to watch:**
- Error rate (should drop)
- API latency (should remain stable)
- User activity (should normalize)

### Follow-Up
- [ ] Update Linear ticket with resolution
- [ ] Schedule post-incident review (if P0/P1)
- [ ] Create tickets for proper fix (if this was a band-aid)
- [ ] Update runbook/documentation
```

---

## Deployment Steps

### Pre-Deployment
```bash
# 1. Merge PR to main
# (After approval or P0 emergency waiver)

# 2. Pull latest
git checkout main
git pull origin main

# 3. Verify commit
git log -1
# Confirm this is your hotfix commit

# 4. Tag release (if using semantic versioning)
git tag -a v2.3.5 -m "Hotfix: Fix auth bypass vulnerability"
git push origin v2.3.5
```

### Deployment (Platform-Specific)

#### Vercel
```bash
# Trigger production deployment
vercel --prod

# Or use Vercel dashboard:
# Deployments → Select commit → Deploy to Production

# Monitor deployment
vercel logs --follow
```

#### Netlify
```bash
# Deploy via CLI
netlify deploy --prod

# Or trigger from dashboard:
# Deploys → Select commit → Publish deploy
```

#### Railway
```bash
# Push to main triggers deployment automatically
# Monitor in dashboard: railway.app/project/logs
```

#### AWS/GCP/Azure
```bash
# Follow platform-specific deployment process
# Example for AWS Elastic Beanstalk:
eb deploy production --staged

# Monitor:
eb logs --follow
```

### Post-Deployment Verification

#### 1. Smoke Test
```bash
# Test the specific fix
curl -X POST https://api.production.com/auth/login \
  -H "Content-Type: application/json" \
  -d '{"token": "expired_token"}'

# Expected: 401 Unauthorized
```

#### 2. Monitor Error Tracking
```
✅ Check Sentry/Rollbar/etc.:
- Error rate should drop
- No new errors introduced

⏱️ Monitor for 15-30 minutes after deployment
```

#### 3. Verify Metrics
```
Check monitoring dashboard:
- API response times (should be normal)
- Error rates (should drop)
- Database performance (should be stable)
- Third-party service health
```

#### 4. Check User Reports
```
Monitor support channels:
- Support tickets
- In-app chat
- Social media
- Status page comments
```

---

## Communication

### Internal Communication

#### Slack/Teams Message Template
```
🚨 **Production Hotfix Deployed**

**Issue**: [Brief description]
**Ticket**: ENG-XXX
**Priority**: P0
**Status**: ✅ Res