Scripts de Backup: S3 + Lifecycle Policies

📋 Contexto

Los backups son como los seguros: nadie quiere pagarlos hasta que los necesita. En este post, cómo implementé un sistema de backups automatizado para mi home lab usando S3, con lifecycle policies para optimizar costos y scripts bash para automatización completa.

🎯 Estrategia de Backup (3-2-1 Rule)

• 3 copias: Original + 2 backups
• 2 medios diferentes: Disco local + cloud (S3)
• 1 copia offsite: S3 en región diferente

💰 Optimización de Costos con S3 Storage Classes

🛠️ Script de Backup Completo

#!/bin/bash
# backup-to-s3.sh - Automated backup script with lifecycle management

set -euo pipefail
IFS=$'\n\t'

##############################################################################
# Configuration
##############################################################################

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly BACKUP_NAME="homelab-backup"
readonly S3_BUCKET="my-backups-$(aws sts get-caller-identity --query Account --output text)"
readonly S3_PREFIX="homelab/"
readonly BACKUP_DIRS=(
    "/home/mytechzone/n8n-lab"
    "/home/mytechzone/scripts"
    "/etc/nginx"
    "/etc/systemd/system"
)
readonly EXCLUDE_PATTERNS=(
    "*.log"
    "*.tmp"
    "node_modules/"
    ".git/"
    "__pycache__/"
)
readonly TEMP_DIR="/tmp/backups"
readonly LOG_FILE="/var/log/backup-s3.log"
readonly RETENTION_DAYS=90

##############################################################################
# Logging
##############################################################################

log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] [INFO] $*" | tee -a "$LOG_FILE"
}

error() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] [ERROR] $*" | tee -a "$LOG_FILE" >&2
}

##############################################################################
# Validation
##############################################################################

check_prerequisites() {
    log "Checking prerequisites..."
    
    # Check AWS CLI
    if ! command -v aws &> /dev/null; then
        error "AWS CLI not installed"
        exit 1
    fi
    
    # Check authentication
    if ! aws sts get-caller-identity &> /dev/null; then
        error "Not authenticated with AWS"
        exit 1
    fi
    
    # Check disk space (need at least 5GB free)
    local available=$(df /tmp | tail -1 | awk '{print $4}')
    if [ "$available" -lt 5242880 ]; then
        error "Not enough disk space in /tmp"
        exit 1
    fi
    
    log "✓ Prerequisites OK"
}

##############################################################################
# S3 Bucket Setup
##############################################################################

setup_s3_bucket() {
    log "Setting up S3 bucket: $S3_BUCKET"
    
    # Create bucket if doesn't exist
    if ! aws s3 ls "s3://$S3_BUCKET" &> /dev/null; then
        log "Creating bucket..."
        aws s3 mb "s3://$S3_BUCKET" --region us-east-1
        
        # Enable versioning
        aws s3api put-bucket-versioning \
            --bucket "$S3_BUCKET" \
            --versioning-configuration Status=Enabled
        
        # Enable encryption
        aws s3api put-bucket-encryption \
            --bucket "$S3_BUCKET" \
            --server-side-encryption-configuration '{
                "Rules": [{
                    "ApplyServerSideEncryptionByDefault": {
                        "SSEAlgorithm": "AES256"
                    }
                }]
            }'
    fi
    
    log "✓ Bucket ready"
}

##############################################################################
# Lifecycle Policy
##############################################################################

configure_lifecycle() {
    log "Configuring lifecycle policy..."
    
    cat > /tmp/lifecycle-policy.json << EOF
{
    "Rules": [
        {
            "Id": "BackupRetention",
            "Status": "Enabled",
            "Prefix": "$S3_PREFIX",
            "Transitions": [
                {
                    "Days": 7,
                    "StorageClass": "STANDARD_IA"
                },
                {
                    "Days": 30,
                    "StorageClass": "GLACIER"
                },
                {
                    "Days": 90,
                    "StorageClass": "DEEP_ARCHIVE"
                }
            ],
            "Expiration": {
                "Days": $RETENTION_DAYS
            }
        }
    ]
}
EOF
    
    aws s3api put-bucket-lifecycle-configuration \
        --bucket "$S3_BUCKET" \
        --lifecycle-configuration file:///tmp/lifecycle-policy.json
    
    rm /tmp/lifecycle-policy.json
    
    log "✓ Lifecycle configured"
}

##############################################################################
# Create Backup
##############################################################################

create_backup() {
    log "Creating backup archive..."
    
    mkdir -p "$TEMP_DIR"
    
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local backup_file="$TEMP_DIR/${BACKUP_NAME}_${timestamp}.tar.gz"
    
    # Build exclude options
    local exclude_opts=""
    for pattern in "${EXCLUDE_PATTERNS[@]}"; do
        exclude_opts+=" --exclude='$pattern'"
    done
    
    # Create tar archive
    log "Archiving directories..."
    eval tar czf "$backup_file" \
        $exclude_opts \
        "${BACKUP_DIRS[@]}" \
        2>> "$LOG_FILE" || {
            error "Tar failed"
            return 1
        }
    
    # Get size
    local size_mb=$(du -m "$backup_file" | cut -f1)
    log "✓ Backup created: $backup_file ($size_mb MB)"
    
    echo "$backup_file"
}

##############################################################################
# Upload to S3
##############################################################################

upload_to_s3() {
    local backup_file=$1
    local filename=$(basename "$backup_file")
    local s3_path="s3://$S3_BUCKET/$S3_PREFIX$filename"
    
    log "Uploading to S3: $s3_path"
    
    # Upload with progress
    aws s3 cp "$backup_file" "$s3_path" \
        --storage-class STANDARD \
        --metadata "backup-date=$(date -Iseconds),hostname=$(hostname)" \
        2>&1 | tee -a "$LOG_FILE"
    
    if [ ${PIPESTATUS[0]} -eq 0 ]; then
        log "✓ Upload successful"
        return 0
    else
        error "Upload failed"
        return 1
    fi
}

##############################################################################
# Cleanup Old Backups (Local)
##############################################################################

cleanup_local() {
    log "Cleaning up local temp files..."
    
    find "$TEMP_DIR" -name "${BACKUP_NAME}_*.tar.gz" -type f -mtime +7 -delete
    
    log "✓ Local cleanup done"
}

##############################################################################
# Verify Backup
##############################################################################

verify_backup() {
    local s3_path=$1
    
    log "Verifying backup in S3..."
    
    # Check if file exists
    if aws s3 ls "$s3_path" &> /dev/null; then
        log "✓ Backup verified in S3"
        return 0
    else
        error "Backup not found in S3"
        return 1
    fi
}

##############################################################################
# Notification (Optional)
##############################################################################

send_notification() {
    local status=$1
    local message=$2
    
    # Option 1: Email via SES
    # aws ses send-email ...
    
    # Option 2: SNS
    # aws sns publish --topic-arn "..." --message "$message"
    
    # Option 3: Webhook (Discord, Slack)
    # curl -X POST -H 'Content-type: application/json' \
    #      --data "{\"text\":\"$message\"}" \
    #      "$WEBHOOK_URL"
    
    log "Notification sent: $status"
}

##############################################################################
# Main
##############################################################################

main() {
    log "=========================================="
    log "Starting backup process"
    log "=========================================="
    
    # Validate
    check_prerequisites
    
    # Setup infrastructure
    setup_s3_bucket
    configure_lifecycle
    
    # Create and upload backup
    local backup_file
    if backup_file=$(create_backup); then
        if upload_to_s3 "$backup_file"; then
            local s3_path="s3://$S3_BUCKET/$S3_PREFIX$(basename "$backup_file")"
            
            if verify_backup "$s3_path"; then
                log "✅ Backup completed successfully"
                send_notification "SUCCESS" "Backup completed: $(basename "$backup_file")"
                
                # Cleanup
                cleanup_local
                rm -f "$backup_file"
                
                exit 0
            fi
        fi
    fi
    
    error "❌ Backup failed"
    send_notification "FAILURE" "Backup failed - check logs"
    exit 1
}

# Run
main "$@"

⏰ Automatización con Cron

# /etc/cron.d/backup-s3
# Run backup daily at 2 AM
0 2 * * * mytechzone /home/mytechzone/scripts/backup-to-s3.sh

# Weekly full backup on Sunday
0 3 * * 0 mytechzone /home/mytechzone/scripts/backup-to-s3.sh --full

# Monthly deep backup (1st day of month)
0 4 1 * * mytechzone /home/mytechzone/scripts/backup-to-s3.sh --monthly

📊 Script de Restore

#!/bin/bash
# restore-from-s3.sh - Restore backup from S3

set -euo pipefail

readonly S3_BUCKET="my-backups-123456789012"
readonly S3_PREFIX="homelab/"
readonly RESTORE_DIR="/tmp/restore"

list_backups() {
    echo "Available backups:"
    echo ""
    
    aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX" \
        --recursive \
        --human-readable \
        --summarize | \
    grep -E '\.tar\.gz$' | \
    nl -w2 -s'. '
}

download_backup() {
    local backup_file=$1
    local s3_path="s3://$S3_BUCKET/$S3_PREFIX$backup_file"
    local local_path="$RESTORE_DIR/$backup_file"
    
    echo "Downloading $backup_file..."
    
    mkdir -p "$RESTORE_DIR"
    
    aws s3 cp "$s3_path" "$local_path" \
        --storage-class STANDARD  # Restore from Glacier if needed
    
    echo "✓ Downloaded to: $local_path"
}

restore_backup() {
    local backup_file=$1
    local target_dir=${2:-/}
    
    echo "Restoring to: $target_dir"
    
    tar xzf "$RESTORE_DIR/$backup_file" -C "$target_dir" --verbose
    
    echo "✓ Restore complete"
}

main() {
    if [ $# -eq 0 ]; then
        list_backups
        echo ""
        read -p "Enter backup filename to restore: " backup_file
    else
        backup_file=$1
    fi
    
    download_backup "$backup_file"
    
    read -p "Restore to / ? (yes/no): " confirm
    if [ "$confirm" = "yes" ]; then
        restore_backup "$backup_file"
    else
        echo "Backup downloaded to $RESTORE_DIR/$backup_file"
        echo "Extract manually with: tar xzf $RESTORE_DIR/$backup_file -C /target/path"
    fi
}

main "$@"

💡 Monitoring de Backups

#!/bin/bash
# check-backup-status.sh - Monitor backup health

set -euo pipefail

readonly S3_BUCKET="my-backups-123456789012"
readonly S3_PREFIX="homelab/"
readonly MAX_AGE_HOURS=28  # Alert if no backup in 28 hours

get_latest_backup() {
    aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX" \
        --recursive | \
    grep -E '\.tar\.gz$' | \
    sort -k1,2 | \
    tail -1 | \
    awk '{print $1, $2, $4}'
}

check_backup_age() {
    local latest=$(get_latest_backup)
    
    if [ -z "$latest" ]; then
        echo "❌ No backups found!"
        return 1
    fi
    
    local backup_date=$(echo "$latest" | awk '{print $1, $2}')
    local backup_file=$(echo "$latest" | awk '{print $3}')
    local backup_timestamp=$(date -d "$backup_date" +%s)
    local current_timestamp=$(date +%s)
    local age_hours=$(( (current_timestamp - backup_timestamp) / 3600 ))
    
    echo "Latest backup: $backup_file"
    echo "Age: $age_hours hours"
    
    if [ $age_hours -gt $MAX_AGE_HOURS ]; then
        echo "⚠️  Backup is too old!"
        return 1
    fi
    
    echo "✅ Backup is current"
    return 0
}

main() {
    echo "=== Backup Status Check ==="
    echo ""
    check_backup_age
}

main "$@"

📈 Estimación de Costos

# Escenario: 10 GB backup diario, retention 90 días

# Month 1:
Days 1-7:   70 GB × $0.023  = $1.61
Days 8-30:  230 GB × $0.0125 = $2.88
Days 31+:   0 GB (no data yet)
Total: $4.49

# Month 2:
Days 1-7:   70 GB × $0.023  = $1.61
Days 8-30:  230 GB × $0.0125 = $2.88
Days 31-60: 310 GB × $0.0036 = $1.12
Total: $5.61

# Month 3 (steady state):
Days 1-7:   70 GB × $0.023  = $1.61
Days 8-30:  230 GB × $0.0125 = $2.88
Days 31-90: 610 GB × $0.0036 = $2.20
Total: $6.69/month

# vs keeping everything in S3 Standard:
900 GB × $0.023 = $20.70/month
Savings: ~68%

🚨 Testing del Restore

#!/bin/bash
# test-restore.sh - Automated restore testing

set -euo pipefail

readonly TEST_DIR="/tmp/restore-test"
readonly S3_BUCKET="my-backups-123456789012"
readonly S3_PREFIX="homelab/"

test_restore() {
    echo "=== Backup Restore Test ==="
    echo ""
    
    # Get latest backup
    local latest_backup=$(aws s3 ls "s3://$S3_BUCKET/$S3_PREFIX" \
        --recursive | \
        grep -E '\.tar\.gz$' | \
        sort | \
        tail -1 | \
        awk '{print $4}')
    
    echo "Testing: $latest_backup"
    
    # Download
    mkdir -p "$TEST_DIR"
    aws s3 cp "s3://$S3_BUCKET/$latest_backup" "$TEST_DIR/"
    
    # Extract
    tar tzf "$TEST_DIR/$(basename "$latest_backup")" &> /dev/null
    
    if [ $? -eq 0 ]; then
        echo "✅ Backup is valid and can be restored"
        rm -rf "$TEST_DIR"
        return 0
    else
        echo "❌ Backup is corrupted!"
        return 1
    fi
}

# Run monthly via cron
test_restore

💡 Best Practices

• ✅ Test restores regularmente: Un backup sin test es un backup sin valor
• ✅ Encripción: Usa S3 SSE o KMS
• ✅ Versioning: Enable S3 versioning para protección contra deletes accidentales
• ✅ Multi-region: Replica backups críticos a otra región
• ✅ Alertas: Notificaciones si backup falla
• ✅ Lifecycle policies: Optimiza costos automáticamente
• ✅ Exclude unnecessary files: logs, tmp, node_modules
• ❌ No hardcodear credenciales: Usa IAM roles o SSO

🔐 IAM Policy Necesaria

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-backups-*",
                "arn:aws:s3:::my-backups-*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket",
                "s3:PutBucketVersioning",
                "s3:PutLifecycleConfiguration"
            ],
            "Resource": "arn:aws:s3:::my-backups-*"
        }
    ]
}

📊 Mi Resultado Final

📚 Recursos

• S3 Lifecycle Management
• S3 Storage Classes
• AWS Pricing Calculator

💭 Conclusión

Los backups no tienen que ser caros ni complicados. Con S3 + lifecycle policies + un buen script bash, tienes un sistema enterprise-grade por menos de $10/mes. La clave es automatización + testing regular.

Recuerda: el mejor backup es el que funciona cuando lo necesitas. Testea tus restores. Siempre.