data-backup

What this skill does

# Smart Backup System with Skill Integration

> **Supporting files in this directory:**
> - [MANIFEST_BACKUPS.md](./MANIFEST_BACKUPS.md) -- MANIFEST-aware intelligent backups
> - [FULL_PROJECT_BACKUPS.md](./FULL_PROJECT_BACKUPS.md) -- Full project backups, selective inclusion/exclusion, path verification
> - [ADVANCED_USAGE.md](./ADVANCED_USAGE.md) -- Custom scripts, multiple file backups, real-world examples

## When to Use This Skill

Use this skill when:
- Working on any project with files that change over time
- Jupyter notebooks, data files (CSV/TSV), HackMD presentations, or mixed projects
- Need intelligent cleanup before backup (clear outputs, remove debug code)
- Want to track what changed when (data provenance)
- Need professional backup workflow for collaboration or publication
- Want context-aware backups that use other skills intelligently

## The Problem

Long-running data enrichment projects risk:
- Losing days of work from accidental overwrites
- Unable to revert to previous data states
- No documentation of what changed when
- Running out of disk space from manual backups
- Confusion about which version is current

## Solution: Smart Two-Tier Backup System with Skill Integration

### Core Features

1. **Intelligent Detection** - Automatically detects project type and files to backup
2. **Skill Integration** - Uses jupyter-notebook, hackmd, and other skills for pre-backup cleanup
3. **Daily backups** - Rolling 7-day window (auto-cleanup)
4. **Milestone backups** - Permanent, compressed (gzip ~80% reduction)
5. **CHANGELOG** - Automatic documentation of all changes
6. **Session Integration** - Prompts for backup when exiting Claude Code session

### Smart Detection & Integration

The backup system automatically detects your project type and applies appropriate cleanup:

**Jupyter Notebooks** (uses `jupyter-notebook` skill):
- Detects: `*.ipynb` files
- Pre-backup cleanup: Clear all cell outputs, remove cells tagged 'debug' or 'remove', validate notebooks

**HackMD/Presentations** (uses `hackmd` skill):
- Detects: `*.md` files with `slideOptions:` frontmatter
- Pre-backup cleanup: Validate SVG elements, check slide separators, verify YAML frontmatter

**Data Files** (native handling):
- Detects: `*.csv`, `*.tsv`, `*.xlsx` files
- Pre-backup cleanup: Validate file integrity, check for corruption

**Python Projects** (uses `managing-environments` skill):
- Detects: `requirements.txt`, `environment.yml`, `venv/`, `.venv/`
- Pre-backup cleanup: Remove `.pyc`, `__pycache__`, `.pytest_cache`, clean build artifacts

**Mixed Projects**: Detects all of the above and applies appropriate cleanup for each file type.

### Directory Structure

**For data-only projects:**
```
project/
├── your_data_file.csv          # Main working file
├── backup_project.sh           # Smart backup script
└── backups/
    ├── daily/                  # Rolling 7-day backups
    ├── milestones/             # Permanent compressed backups
    ├── CHANGELOG.md            # Auto-generated change log
    └── README.md               # User documentation
```

**For mixed projects (notebooks + data):**
```
project/
├── analysis.ipynb              # Jupyter notebooks
├── data.csv                    # Data files
├── backup_project.sh           # Smart backup script
└── backups/
    ├── daily/                  # Rolling 7-day backups
    │   └── backup_2026-01-17/
    │       ├── notebooks/      # Cleaned (no outputs)
    │       └── data/
    ├── milestones/             # Permanent compressed backups
    ├── CHANGELOG.md
    └── README.md
```

### Storage Efficiency

- **Daily backups**: ~5.4 MB (7 days x 770KB)
- **Milestone backups**: ~200KB each compressed (80% size reduction with gzip)
- **Total**: <10 MB for complete project history
- **Auto-cleanup**: Old daily backups delete after 7 days

## Implementation

### Quick Start with `/backup` Command

**First time - Setup the backup system:**
```
/backup
```
This will:
- Detect your project type (notebooks, data files, presentations, etc.)
- Set up appropriate backup scripts with smart cleanup
- Create backup directory structure
- Optionally configure automated backups

**Daily usage - Create backups:**
```
/backup                    # Daily backup with smart cleanup
/backup milestone "desc"   # Milestone backup
/backup list              # View all backups
/backup restore DATE      # Restore from backup
```

### What Happens During Backup

**Smart cleanup before backup:**
1. **Detects file types** in your project
2. **Applies skill-specific cleanup:**
   - Notebooks: Clear outputs, remove debug cells
   - HackMD: Validate SVG, check formatting
   - Python: Remove `.pyc`, `__pycache__`
   - Data: Validate integrity
3. **Creates organized backup** with cleaned files
4. **Updates CHANGELOG** with what was backed up

### Manual Script Usage (Alternative)

```bash
./backup_project.sh                           # Daily backup
./backup_project.sh milestone "description"   # Milestone
./backup_project.sh list                      # List backups
./backup_project.sh restore 2026-01-23        # Restore
```

### When to Create Milestones

- After adding new data sources (GenomeScope, karyotypes, external APIs)
- Before major data transformations or filtering
- When completing analysis sections
- Before submitting/publishing
- Before sharing with collaborators
- After recovering missing data

## Key Features

### Safety Features

1. **Never overwrites without asking** - Prompts before overwriting existing backups
2. **Safety backup before restore** - Creates backup of current state before any restore
3. **Automatic cleanup** - Old daily backups auto-delete (configurable)
4. **Complete audit trail** - CHANGELOG tracks everything
5. **Milestone protection** - Important versions preserved forever (compressed)

### CHANGELOG Tracking

The CHANGELOG.md automatically documents:
- Date of each backup
- Type (daily vs milestone)
- Description of changes (for milestones)
- Major modifications made to data

**Example CHANGELOG:**
```markdown
## 2026-01-23
- **MILESTONE**: Recovered VGP accessions (backup created)
  - Added columns: `accession_recovered`, `accession_recovered_all`
  - Recovered 5 VGP accessions from NCBI
- Daily backup created at 2026-01-23 15:00:00

## 2026-01-22
- Enriched GenomeScope data for 21 species from AWS repository
- Added column: `genomescope_path` with direct links to summary files
```

## Using `/backup` Command

**Setup mode (first run):** `/backup` -- Detects project type, sets up scripts, creates directory structure.

**Daily backup mode:** `/backup` -- Quick daily backup.

**Milestone mode:** `/backup milestone "description of changes"` -- e.g., `/backup milestone "added heterozygosity data"`

**List and restore:**
```
/backup list              # Show all available backups
/backup restore 2026-01-23 # Restore from specific date
```

**Configuration:** Edit `backup_project.sh` to change retention days (default: 7), backup directory location, or custom cleanup rules.

## Benefits for Data Analysis

- **Data Provenance**: CHANGELOG documents every modification; clear audit trail for methods sections in papers
- **Confidence to Experiment**: Easy rollback encourages trying different approaches safely
- **Professional Workflow**: Matches publication standards; reviewers can verify data processing steps
- **Collaboration-Ready**: Team members can understand data history and enrichment process

## Session Integration with `/safe-exit`

When you end a Claude Code session with `/safe-exit`, the system automatically:

1. **Detects if backup system exists** in the current project
2. **Prompts for backup** if system is configured (daily, milestone, skip, or cancel)
3. **Performs cleanup and backup** if requested
4. **Prompts for Obsidian session summary** (if obsidian skill is available)
5. **Exits session** cleanly

This ensures you never forget to backup AND document your work at the end of your session!

## Example Workflow