Production Deployment Guide¶

Complete guide for deploying the Laboratory Management System to a production server.

Table of Contents¶

Prerequisites
Quick Start
Detailed Setup
Deployment Process
SSL Configuration
Monitoring
Backup and Restore
Rollback
Troubleshooting
Security Checklist

Prerequisites¶

Server Requirements¶

Minimum: - OS: Ubuntu 20.04+ or CentOS 8+ - CPU: 2 cores - RAM: 4GB - Storage: 20GB free space

Recommended: - CPU: 4 cores - RAM: 8GB - Storage: 50GB+ free space

Required Software¶

Docker 20.10+ with Docker Compose v2
Git 2.25+
curl (for health checks)
gzip (for backups)

Network Requirements¶

Domain name pointed to your server
Ports 80 and 443 accessible from internet
Port 22 for SSH access
Stable internet connection

Quick Start¶

This section provides a streamlined deployment process for production.

1. Clone Repository¶

cd /opt
sudo mkdir laboratory-system
sudo chown $USER:$USER laboratory-system
cd laboratory-system
git clone <your-repo-url> .

2. Configure Environment¶

# Copy production environment template
cp .env.production.example .env

# Generate SECRET_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(50))"

# Edit configuration
nano .env

Critical variables to set:

# Django Settings
SECRET_KEY=<generated-secret-key>
DEBUG=False
ALLOWED_HOSTS=yourdomain.com,www.yourdomain.com
DOMAIN_NAME=yourdomain.com

# Database
POSTGRES_USER=lab_user
POSTGRES_PASSWORD=<strong-random-password>
POSTGRES_DB=laboratory_db

# Email (see Configuration Guide for SMTP details)
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.yourdomain.com
EMAIL_PORT=587
EMAIL_HOST_USER=noreply@yourdomain.com
EMAIL_HOST_PASSWORD=<email-password>
EMAIL_USE_TLS=True
DEFAULT_FROM_EMAIL=noreply@yourdomain.com

# Security
SESSION_COOKIE_SECURE=True
CSRF_COOKIE_SECURE=True
SECURE_SSL_REDIRECT=True

# Redis & Celery
REDIS_URL=redis://redis:6379/0
CELERY_BROKER_URL=redis://redis:6379/0

See Environment Variables for complete reference.

3. Install Docker (if needed)¶

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Verify installation
docker --version
docker-compose --version

# Re-login for group changes to take effect
exit
# SSH back in

4. Build and Start Services¶

# Build images
docker compose -f compose.yaml -f compose.production.yaml build

# Start database and redis
docker compose -f compose.yaml -f compose.production.yaml up -d postgres redis

# Wait for database to be ready
sleep 10

# Run migrations
make  manage migrate

# Create superuser
make  manage createsuperuser

# Collect static files
make  manage collectstatic --no-input

# Start application services
docker compose -f compose.yaml -f compose.production.yaml up -d web worker beat

5. Initialize SSL Certificates¶

# Initialize Let's Encrypt SSL
./bin/init-letsencrypt.sh yourdomain.com admin@yourdomain.com

# For testing (uses Let's Encrypt staging):
# ./bin/init-letsencrypt.sh yourdomain.com admin@yourdomain.com 1

This will: - Request SSL certificates from Let's Encrypt - Configure Nginx with SSL - Set up automatic certificate renewal

See SSL Certificates for detailed SSL setup.

6. Verify Deployment¶

# Check all services are running
docker compose -f compose.yaml -f compose.production.yaml ps

# Test application health
curl https://yourdomain.com/up

# View logs
docker compose -f compose.yaml -f compose.production.yaml logs -f web

Expected output from health check:

{"status": "ok", "database": "connected", "cache": "connected"}

Detailed Setup¶

Automated Setup (Recommended)¶

For fresh server setup, use the automated setup script:

# Download and run the setup script
curl -fsSL https://raw.githubusercontent.com/your-username/laboratory-system/main/bin/setup-server -o setup-server.sh
chmod +x setup-server.sh

# Run automated setup
./setup-server.sh --repo https://github.com/your-username/laboratory-system.git --domain yourdomain.com

Setup script features: - ✅ Updates system packages - ✅ Installs Docker and Docker Compose - ✅ Configures firewall (UFW) and fail2ban - ✅ Creates application directory - ✅ Clones repository - ✅ Sets up environment configuration - ✅ Configures systemd service - ✅ Sets up log rotation - ✅ Configures automated backups - ✅ Runs initial deployment

Manual Setup¶

If you prefer manual setup or the automated script isn't suitable, follow the Quick Start section above for step-by-step manual instructions.

Deployment Process¶

Pre-Deployment Checklist¶

Before deploying, run the pre-deployment check script:

# Run comprehensive checks
./bin/pre-deploy-check

This verifies: - ✅ Project directory structure - ✅ Git repository status - ✅ Environment configuration - ✅ System resources (disk space, memory) - ✅ Docker installation and status - ✅ Database connectivity - ✅ Redis connectivity - ✅ Pending migrations - ✅ Application health - ✅ Security settings

Deploy to Production¶

Use the deployment script for updates:

# Deploy latest changes
./bin/deploy-production.sh

The deployment script performs these steps in order:

🔍 Validate — Check production mode (DEBUG=False, ALLOWED_HOSTS set) and exit if invalid.
🗄️ Start infrastructure — Bring up Postgres and Redis so backup and migrations can run (idempotent if already up).
💾 Backup — Create a database dump in ./backups (fails if DB is unreachable).
📥 Pull — Fetch and pull latest from origin/main (skips if already up to date).
🏗️ Build — Build Docker images for web, worker, and beat. The Dockerfile runs collectstatic during image build (case-insensitive DEBUG check), so the image always contains a valid WhiteNoise manifest. Set DEPLOY_NO_CACHE=1 to force docker compose build --no-cache (useful when reproducing memory-sensitive deploy issues).
🔄 Migrations — Run migrate --no-input then migrate --check in a one-off web container (fails on migration errors). App services are not started yet.
⏸️ Stop app services — Stop web, worker, and beat if they are running, so the next steps do not compete with Gunicorn/Celery for RAM (avoids intermittent exit 137 / OOM on small VPSs).
📦 Collect static — Run collectstatic --no-input --clear in a one-off container (docker compose run --rm), never via exec into a live web container. Output is persisted to the host via the volume mount at /public_collected (matching STATIC_ROOT).
📚 Build documentation — Run MkDocs in a one-off container and write output into public_collected/docs/ (continues on failure with a warning).
🔁 Start app services — docker compose up -d --force-recreate web worker beat so Gunicorn/WhiteNoise read the fresh staticfiles.json manifest.
🌐 Start proxy — Start (or force-recreate) nginx and certbot so the app is reachable via HTTPS.
✅ Health check — Poll http://localhost/up until success or timeout; exit with error if services do not become healthy.

To force a full image rebuild without Docker layer cache (e.g. to re-test deploy memory pressure):

DEPLOY_NO_CACHE=1 ./bin/deploy-production.sh

Monitor Deployment¶

# Watch application logs in real-time
docker compose -f compose.yaml -f compose.production.yaml logs -f web

# Check all service statuses
docker compose -f compose.yaml -f compose.production.yaml ps

# Test application
curl https://yourdomain.com/up

# View recent errors
docker compose -f compose.yaml -f compose.production.yaml logs --tail=50 web | grep ERROR

SSL Configuration¶

Automatic SSL with Let's Encrypt¶

The system includes automated SSL certificate management:

# Initial setup
./bin/init-letsencrypt.sh yourdomain.com admin@yourdomain.com

Certificate Renewal¶

Certificates automatically renew via cron job. To manually renew:

# Manual renewal
docker compose -f compose.yaml -f compose.production.yaml run --rm certbot certbot renew

# Reload Nginx
docker compose -f compose.yaml -f compose.production.yaml exec nginx nginx -s reload

# Check certificate status
docker compose -f compose.yaml -f compose.production.yaml exec certbot certbot certificates

See SSL Certificates Guide for detailed SSL documentation.

Monitoring¶

View Logs¶

# All services
docker compose -f compose.yaml -f compose.production.yaml logs -f

# Specific service
docker compose -f compose.yaml -f compose.production.yaml logs -f web
docker compose -f compose.yaml -f compose.production.yaml logs -f nginx
docker compose -f compose.yaml -f compose.production.yaml logs -f worker

# Nginx access logs
docker compose -f compose.yaml -f compose.production.yaml exec nginx tail -f /var/log/nginx/laboratory-access.log

# Nginx error logs
docker compose -f compose.yaml -f compose.production.yaml exec nginx tail -f /var/log/nginx/laboratory-error.log

Check Service Status¶

# Service status
docker compose -f compose.yaml -f compose.production.yaml ps

# Resource usage
docker stats

# Application health
curl https://yourdomain.com/up

Automated Monitoring¶

Set up automated health checks:

# Check monitoring logs
tail -f /opt/laboratory-system/logs/monitor.log

# Run manual health check
/opt/laboratory-system/bin/monitor

See Operations Guide for detailed monitoring procedures.

Backup and Restore¶

Automated Backups¶

The system includes automated backup scripts:

# Backups are stored in: /opt/laboratory-system/backups/
ls -lh /opt/laboratory-system/backups/

# Backup retention: 7 daily, 4 weekly, 3 monthly

Manual Backup¶

# Create backup directory if needed
mkdir -p backups

# Backup database
make  db:dump
# Creates: backups/adlab_dump_YYYYMMDD_HHMMSS.sql

# Or using Docker directly:
docker compose -f compose.yaml -f compose.production.yaml exec -T postgres \
  pg_dump -U lab_user laboratory_db | gzip > backups/backup_$(date +%Y%m%d_%H%M%S).sql.gz

Restore from Backup¶

# Stop application services (keep database running)
docker compose -f compose.yaml -f compose.production.yaml stop web worker beat

# Restore database
gunzip -c backups/backup_YYYYMMDD_HHMMSS.sql.gz | \
  docker compose -f compose.yaml -f compose.production.yaml exec -T postgres \
  psql -U lab_user laboratory_db

# Start application services
docker compose -f compose.yaml -f compose.production.yaml up -d web worker beat

See Backup & Restore Guide for detailed backup procedures.

Rollback¶

If deployment issues occur, use the rollback process:

Quick Rollback¶

# Interactive rollback (recommended)
./bin/rollback

Advanced Rollback Options¶

# List available backups
./bin/rollback --list

# Restore specific backup
./bin/rollback --backup db_backup_20241201_143022.sql.gz

# Rollback to specific git commit
./bin/rollback --commit abc1234

# Force rollback (skip confirmations - use with caution)
./bin/rollback --force

Manual Rollback¶

# 1. Stop services
docker compose -f compose.yaml -f compose.production.yaml stop web worker beat

# 2. Revert code to previous commit
git reset --hard HEAD~1

# 3. Rebuild images
docker compose -f compose.yaml -f compose.production.yaml build

# 4. Restore database if needed
# (see Backup and Restore section)

# 5. Start services
docker compose -f compose.yaml -f compose.production.yaml up -d

Troubleshooting¶

SSL Certificate Issues¶

# Check certificate
docker compose -f compose.yaml -f compose.production.yaml exec certbot certbot certificates

# Renew certificate manually
docker compose -f compose.yaml -f compose.production.yaml run --rm certbot certbot renew

# Reload Nginx
docker compose -f compose.yaml -f compose.production.yaml exec nginx nginx -s reload

# Test SSL configuration
curl -vI https://yourdomain.com

Service Not Starting¶

# Check logs for specific service
docker compose -f compose.yaml -f compose.production.yaml logs web

# Check container status
docker compose -f compose.yaml -f compose.production.yaml ps

# Restart specific service
docker compose -f compose.yaml -f compose.production.yaml restart web

# Full restart (if needed)
docker compose -f compose.yaml -f compose.production.yaml down
docker compose -f compose.yaml -f compose.production.yaml up -d

Database Connection Issues¶

# Check database is running
docker compose -f compose.yaml -f compose.production.yaml ps postgres

# Test connection from within container
docker compose -f compose.yaml -f compose.production.yaml exec postgres psql -U lab_user laboratory_db

# Check Django can connect
make  manage check --database default

# View database logs
docker compose -f compose.yaml -f compose.production.yaml logs postgres

Static Files Not Loading¶

# Collect static files again
make  manage collectstatic --no-input

# Check Nginx configuration
docker compose -f compose.yaml -f compose.production.yaml exec nginx nginx -t

# Restart Nginx
docker compose -f compose.yaml -f compose.production.yaml restart nginx

See Troubleshooting Guide for more issues and solutions.

Security Checklist¶

Before going live, verify all security settings:

Environment Security¶

DEBUG=False in production
Strong SECRET_KEY generated (50+ characters)
Strong POSTGRES_PASSWORD set (20+ characters)
ALLOWED_HOSTS configured correctly (no wildcards)
DOMAIN_NAME matches your actual domain

SSL/TLS Security¶

SSL certificates installed and valid
Auto-renewal configured (certbot cron job)
SESSION_COOKIE_SECURE=True
CSRF_COOKIE_SECURE=True
SECURE_SSL_REDIRECT=True
HTTPS working without certificate warnings

Firewall Configuration¶

Firewall enabled (UFW or similar)
Only necessary ports open (80, 443, 22)
SSH port secured or changed
fail2ban configured for SSH protection

Backup & Recovery¶

Automated backups configured
Backup restoration tested successfully
Backups stored securely (offsite if possible)
Retention policy defined and implemented

Application Security¶

Access Control¶

Strong passwords for all admin accounts
SSH key-based authentication enabled
Sudo access restricted
Application user permissions configured correctly

See Security Audit Guide for comprehensive security review.

Post-Deployment Tasks¶

After successful deployment:

Configure Automated Backups

# Set up daily backups via cron
crontab -e
# Add: 0 2 * * * /opt/laboratory-system/bin/backup-database

Set Up Monitoring (optional)
Configure Grafana + Prometheus
Set up log aggregation
Configure alerting
Configure Email
Test email sending: make manage sendtestemail admin@yourdomain.com
Configure SMTP properly
See Email Setup Guide

Set Up Systemd Service (auto-start on boot)

# Copy service file
sudo cp /opt/laboratory-system/bin/laboratory-system.service /etc/systemd/system/

# Enable service
sudo systemctl enable laboratory-system
sudo systemctl start laboratory-system

Test Disaster Recovery
Perform test backup
Perform test restore
Verify rollback procedure
Document Your Setup
Note any custom configurations
Document any deviations from this guide
Update team documentation

Next Steps¶

Configure Email - Set up SMTP for notifications
Operations Guide - Daily operations and maintenance
Troubleshooting - Common issues and solutions
Backup Procedures - Detailed backup documentation

Additional Resources¶

Nginx Setup Guide - Detailed Nginx configuration
SSL Certificate Setup - SSL/TLS configuration
Server Connection - SSH and remote access
Production Checklist - Pre-launch verification

← Back to Deployment Documentation | Documentation Home