sed

What this skill does

# GNU sed Skill

All output assumes **GNU sed** (`sed --version` shows GNU). POSIX compatibility is
not a goal; use `-E` (extended regex) by default unless basic regex is clearly
sufficient.

---

## Core Principles

1. **Prefer one-liners.** Multi-line sed scripts are a last resort.
2. **Use `-E`** for extended regex (avoids backslash noise).
3. **Use `-i` for in-place edits**; always show the command with a backup suffix
   suggestion (`-i.bak`) if the operation is destructive.
4. **Dry-run first.** When editing files, show the `sed` without `-i` before the
   in-place version so the user can verify.
5. **Chain with pipes** when sed alone isn't the cleanest tool.
6. **Never use sed to parse structured formats** (JSON, YAML, XML). Redirect to
   `jq`, `yq`, or `xmllint` and say why.

---

## Flags to Always Know

| Flag         | Meaning                                          |
|--------------|--------------------------------------------------|
| `-E`         | Extended regex (ERE): `+`, `?`, `|`, `()` work without backslash |
| `-i[SUFFIX]` | Edit in place (GNU: `-i` or `-i''` for no backup, `-i.bak` for backup) |
| `-n`         | Suppress automatic print; use with `p` to print selectively |
| `-e`         | Multiple expressions in one invocation           |
| `-f FILE`    | Read commands from a script file                 |
| `--sandbox`  | Disallow `e`, `r`, `w` commands (safe mode)      |

---

## Command Quick Reference

### Substitution (`s`)

```bash
# Basic replace (first occurrence per line)
sed 's/foo/bar/'

# Replace all occurrences on each line
sed 's/foo/bar/g'

# Case-insensitive replace (GNU extension)
sed 's/foo/bar/gI'

# Replace only on lines matching a pattern
sed '/error/s/foo/bar/g'

# Replace on line ranges
sed '10,20s/foo/bar/g'

# Use & to reference the matched text
sed 's/[0-9]\+/(&)/g'   # wraps every number in parens

# Capture groups with \1, \2
sed -E 's/(foo)(bar)/\2\1/'

# Delimiter swap (useful when pattern contains /)
sed 's|/usr/local|/opt|g'
```

### Delete lines (`d`)

```bash
# Delete lines matching a pattern
sed '/^#/d'

# Delete blank lines
sed '/^[[:space:]]*$/d'

# Delete line range
sed '5,10d'

# Delete from pattern to end of file
sed '/START/,$d'

# Delete lines NOT matching (invert with !)
sed '/keep/!d'
```

### Print / Extract (`p`, `-n`)

```bash
# Print only matching lines (like grep)
sed -n '/error/p'

# Print line numbers with matches
sed -n '/error/='

# Print a line range
sed -n '10,20p'

# Print from pattern to pattern (inclusive)
sed -n '/START/,/END/p'

# Print only the captured group (GNU extension)
sed -En 's/.*key=([^ ]+).*/\1/p'
```

### Insert / Append / Change (`i`, `a`, `c`)

```bash
# Insert a line before match
sed '/pattern/i\NEW LINE'

# Append a line after match
sed '/pattern/a\NEW LINE'

# Replace the entire matching line
sed '/pattern/c\REPLACEMENT LINE'
```

### Transform characters (`y`)

```bash
# Like tr: map chars one-to-one
sed 'y/abc/ABC/'
```

### Multi-expression

```bash
sed -E -e 's/foo/bar/g' -e 's/baz/qux/g'
```

---

## Patterns & Addresses

```
/regex/         — lines matching regex
/re1/,/re2/     — line range from re1 to re2 (inclusive)
N               — line number N
N,M             — line numbers N through M
N~STEP          — every STEP lines starting at N  (e.g. 1~2 = odd lines)
$               — last line
/re/!           — lines NOT matching
```

---

## Common Use Cases

### Shell scripting: rewrite config values

```bash
# Dry-run
sed -E 's/^(TIMEOUT=).*/\1120/' config.env

# In-place with backup
sed -E -i.bak 's/^(TIMEOUT=).*/\1120/' config.env
```

### Code refactoring: rename symbol across files

```bash
# Preview
grep -rl 'OldClass' src/ | xargs sed -E 's/\bOldClass\b/NewClass/g'

# In-place
grep -rl 'OldClass' src/ | xargs sed -E -i.bak 's/\bOldClass\b/NewClass/g'
```

Use `\b` for word boundaries (GNU sed supports `\b` in ERE).

### Log processing: extract fields

```bash
# Extract timestamps from nginx logs
sed -En 's/.*\[([^\]]+)\].*/\1/p' access.log

# Show only lines between two timestamps (approximate)
sed -n '/2024-01-01 10:00/,/2024-01-01 11:00/p' app.log

# Strip ANSI color codes
sed -E 's/\x1B\[[0-9;]*[mK]//g'
```

### File batch editing: add/remove lines

```bash
# Add a shebang to all .sh files missing one
find . -name '*.sh' | xargs sed -i '1{/^#!/!i\#!/usr/bin/env bash
}'

# Remove trailing whitespace
sed -E -i 's/[[:space:]]+$//' file.rb

# Comment out lines matching a pattern
sed -i 's/^FEATURE_X=/#FEATURE_X=/' .env
```

### Delete lines between markers (exclusive)

```bash
sed '/START/,/END/{/START/!{/END/!d}}' file.txt
```

### Number lines (print with line numbers)

```bash
sed = file.txt | sed 'N;s/\n/\t/'
```

---

## Combining with Other Tools

```bash
# sed + xargs: batch replace in files found by grep
grep -rl 'pattern' . | xargs sed -i 's/pattern/replacement/g'

# sed + awk: sed for line filtering, awk for field extraction
sed -n '/error/p' app.log | awk '{print $1, $NF}'

# sed in a pipeline
cat access.log | sed -n '/POST/p' | cut -d' ' -f7 | sort | uniq -c
```

---

## Output Format

When responding to a sed request:

1. **Show the one-liner** in a code block, ready to copy-paste.
2. If editing files in-place, show **dry-run first**, then the `-i` version.
3. Add a **one-line comment** explaining what each part does if non-obvious.
4. If the task is better served by `awk`, `perl`, `jq`, or `ripgrep`, say so briefly
   and optionally provide that alternative instead.

---

## Gotchas

- **macOS ships BSD sed**, not GNU sed. `-i` requires a mandatory suffix (`-i ''`
  is valid on BSD, `-i.bak` on both). If the user is on macOS, note this.
- Regex in sed is greedy by default; there is no non-greedy quantifier in GNU sed
  ERE. Workaround: use negated character classes (`[^x]*` instead of `.*?`).
- `\w` is a GNU sed extension, not portable sed. Use `[[:alnum:]_]` for portable
  word characters, and `[0-9]` or `[[:digit:]]` for digits. Use Perl (`perl -pe`)
  when you need PCRE shorthands such as `\d`.
- Multiline matching across line breaks requires the `N` command or a hold-space
  trick. If the user needs real multiline regex, recommend `perl -0777 -pe`.