# Docker Containerization Guide

Complete guide for running Market Data Service in Docker containers for both development and production environments.

## Table of Contents

- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Development Setup](#development-setup)
- [Production Setup](#production-setup)
- [Configuration](#configuration)
- [Common Operations](#common-operations)
- [Troubleshooting](#troubleshooting)
- [Backup and Restore](#backup-and-restore)

## Prerequisites

- **Docker Desktop** (Mac/Windows) or **Docker Engine** (Linux)
- **Docker Compose** v2.0+ (included with Docker Desktop)
- **Git** (for cloning repository)

No other dependencies required! Everything runs in containers.

## Quick Start

### Development Environment

```bash
# 1. Clone the repository
git clone <repository-url>
cd market-data-service

# 2. Copy environment template
cp .env.example .env

# 3. Edit .env file with your settings (optional for development)
# Default values work for local development

# 4. Start all services
docker-compose up -d

# 5. Check service status
docker-compose ps

# 6. View logs
docker-compose logs -f api
```

The API will be available at:
- **HTTP**: http://localhost
- **API Endpoints**: http://localhost/api/*
- **Health Check**: http://localhost/health
- **WebSocket**: ws://localhost/socket.io/

### Production Environment

```bash
# 1. Set up environment variables
cp .env.example .env
# Edit .env with production values

# 2. Set up SSL certificates (if using HTTPS)
# Place certificates in ./ssl/certs/ and ./ssl/private/
# Or update SSL_CERT_PATH and SSL_KEY_PATH in docker-compose.prod.yml

# 3. Start production services
docker-compose -f docker-compose.prod.yml up -d

# 4. Check status
docker-compose -f docker-compose.prod.yml ps
```

## Development Setup

### Architecture

The development environment consists of three services:

1. **API Service** (`api`)
   - Node.js/Express application
   - Live code reload with volume mounts
   - Runs on port 3000 (internal)
   - Accessible via Nginx on port 80

2. **Database Service** (`db`)
   - PostgreSQL 15
   - Exposed on port 5432 (for local tools)
   - Data persisted in Docker volume

3. **Nginx Service** (`nginx`)
   - Reverse proxy
   - WebSocket support for Socket.io
   - Exposed on port 80

### Starting Development Environment

```bash
# Start all services in detached mode
docker-compose up -d

# Start with logs visible
docker-compose up

# Rebuild containers after code changes
docker-compose up -d --build
```

### Accessing Services

- **API**: http://localhost
- **Database**: localhost:5432
- **API Direct**: http://localhost:3000 (if port mapping enabled)

### Development Workflow

1. **Make code changes** in `src/` directory
2. **Changes are automatically reloaded** (nodemon watches for changes)
3. **View logs**: `docker-compose logs -f api`
4. **Restart service**: `docker-compose restart api`

### Database Migrations

Migrations run automatically on container startup. To run manually:

```bash
# Run migrations
docker-compose exec api npx sequelize-cli db:migrate

# Rollback last migration
docker-compose exec api npx sequelize-cli db:migrate:undo

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status
```

### Accessing Database

```bash
# Connect to PostgreSQL
docker-compose exec db psql -U postgres -d financial_data

# Run SQL commands
docker-compose exec db psql -U postgres -d financial_data -c "SELECT * FROM symbols LIMIT 10;"

# List tables
docker-compose exec db psql -U postgres -d financial_data -c "\dt"
```

## Production Setup

### Architecture

Production environment uses optimized settings:

- **No volume mounts** (code baked into image)
- **Multi-stage builds** for smaller images
- **Health checks** for all services
- **SSL/TLS** support via Nginx
- **Restart policies** for high availability

### Prerequisites for Production

1. **Environment Variables**
   ```bash
   # Set strong passwords and secrets
   DB_PASSWORD=<strong_password>
   JWT_SECRET=<strong_random_secret>
   CORS_ORIGIN=https://your-domain.com
   DOMAIN_NAME=your-domain.com
   SSL_EMAIL=your-email@example.com
   ```
   
   See [CONFIGURATION.md](./docs/CONFIGURATION.md) for complete environment variable documentation.

2. **SSL Certificates**
   SSL certificates are automatically managed via Docker containers. No manual setup required.
   
   See [DEPLOYMENT.md](./docs/DEPLOYMENT.md) for production deployment with SSL certificate setup.

### Starting Production Environment

```bash
# Build production images
docker-compose -f docker-compose.prod.yml build

# Start services
docker-compose -f docker-compose.prod.yml up -d

# Check health status
docker-compose -f docker-compose.prod.yml ps
```

### Production Features

- **Automatic migrations** on startup
- **Health checks** for service monitoring
- **SSL/TLS** encryption
- **Security headers** (HSTS, X-Frame-Options, etc.)
- **Rate limiting** (configurable in nginx.prod.conf)
- **Data persistence** via Docker volumes

## Configuration

### Environment Variables

Key environment variables (see `.env.example` for complete list):

| Variable | Description | Default |
|----------|-------------|---------|
| `DB_HOST` | Database host | `db` (Docker) or `localhost` |
| `DB_PORT` | Database port | `5432` |
| `DB_NAME` | Database name | `financial_data` |
| `DB_USER` | Database user | `postgres` |
| `DB_PASSWORD` | Database password | **Required** |
| `PORT` | API port | `3000` |
| `NODE_ENV` | Environment | `development` or `production` |
| `CORS_ORIGIN` | Allowed origins | `*` |
| `JWT_SECRET` | JWT secret key | **Required** |
| `LOG_LEVEL` | Logging level | `debug` (dev) or `info` (prod) |

### Nginx Configuration

- **Development**: `nginx/nginx.conf` (HTTP only)
- **Production**: `nginx/nginx.prod.conf` (HTTPS with SSL)

Key features:
- WebSocket proxy for Socket.io
- API endpoint routing
- Health check endpoint
- Security headers (production)
- Rate limiting (production)

### Database Configuration

Database is automatically initialized with:
- Database: `financial_data`
- User: `postgres` (or from `DB_USER`)
- Migrations run on API container startup

## Common Operations

### Viewing Logs

```bash
# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api
docker-compose logs -f db
docker-compose logs -f nginx

# Last 100 lines
docker-compose logs --tail=100 api
```

### Restarting Services

```bash
# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart api

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: deletes data)
docker-compose down -v
```

### Accessing Containers

```bash
# Execute command in API container
docker-compose exec api sh

# Execute command in database container
docker-compose exec db sh

# Run Node.js commands
docker-compose exec api node -v
docker-compose exec api npm list
```

### Updating Application

```bash
# 1. Pull latest code
git pull

# 2. Rebuild containers
docker-compose build

# 3. Restart services (migrations run automatically)
docker-compose up -d

# Or for production
docker-compose -f docker-compose.prod.yml build
docker-compose -f docker-compose.prod.yml up -d
```

### Health Checks

```bash
# Check container health
docker-compose ps

# Test API health endpoint
curl http://localhost/health

# Test database connection
docker-compose exec db pg_isready -U postgres
```

## Troubleshooting

### Database Connection Issues

**Problem**: API can't connect to database

**Solutions**:
```bash
# Check database is running
docker-compose ps db

# Check database logs
docker-compose logs db

# Test database connection manually
docker-compose exec db psql -U postgres -d financial_data

# Verify environment variables
docker-compose exec api env | grep DB_
```

### Migration Failures

**Problem**: Migrations fail on startup

**Solutions**:
```bash
# Check migration logs
docker-compose logs api | grep -i migration

# Run migrations manually
docker-compose exec api npx sequelize-cli db:migrate

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status
```

### WebSocket Connection Issues

**Problem**: WebSocket connections fail through Nginx

**Solutions**:
1. Verify Nginx configuration includes WebSocket upgrade headers
2. Check Nginx logs: `docker-compose logs nginx`
3. Test direct connection: `ws://localhost:3000/socket.io/`
4. Verify Socket.io transport is set to 'websocket' only

### Port Conflicts

**Problem**: Port already in use

**Solutions**:
```bash
# Check what's using the port
lsof -i :80
lsof -i :5432

# Change ports in docker-compose.yml
# Update PORT and DB_PORT environment variables
```

### Container Won't Start

**Problem**: Container exits immediately

**Solutions**:
```bash
# Check exit code
docker-compose ps

# View detailed logs
docker-compose logs api

# Check container status
docker ps -a

# Try starting without detached mode
docker-compose up api
```

### Permission Issues

**Problem**: Permission denied errors

**Solutions**:
```bash
# Fix script permissions
chmod +x docker/*.sh

# Check file ownership
ls -la docker/

# Rebuild containers
docker-compose build --no-cache
```

## Backup and Restore

### Database Backup

```bash
# Create backup
docker-compose exec db pg_dump -U postgres financial_data > backup_$(date +%Y%m%d_%H%M%S).sql

# Or using docker directly
docker exec market-data-db pg_dump -U postgres financial_data > backup.sql
```

### Database Restore

```bash
# Restore from backup
docker-compose exec -T db psql -U postgres financial_data < backup.sql

# Or using docker directly
docker exec -i market-data-db psql -U postgres financial_data < backup.sql
```

### Volume Backup

```bash
# Backup database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar czf /backup/db_backup.tar.gz /data

# Restore database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar xzf /backup/db_backup.tar.gz -C /
```

### Automated Backups

Create a cron job or scheduled task:

```bash
# Daily backup script
#!/bin/bash
BACKUP_DIR="./backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
docker-compose exec -T db pg_dump -U postgres financial_data | gzip > $BACKUP_DIR/backup_$DATE.sql.gz
# Keep only last 30 days
find $BACKUP_DIR -name "backup_*.sql.gz" -mtime +30 -delete
```

## MT5 EA Integration

### Development

Update MT5 EA settings:
- `ApiBaseUrl`: `http://market-data.local`
- See [LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md) for local development setup

### Production

Update MT5 EA settings:
- `ApiBaseUrl`: `https://market-price.insightbull.io` (or your production domain)
- SSL certificates are automatically managed
- WebSocket connections use `wss://` protocol
- See [DEPLOYMENT.md](./docs/DEPLOYMENT.md) for production deployment

## Performance Tuning

### Database Optimization

```bash
# Increase shared buffers (edit postgresql.conf in container)
# Or use environment variables in docker-compose.yml
```

### Nginx Optimization

- Adjust `worker_processes` in nginx.conf
- Tune `worker_connections` based on expected load
- Enable caching for static assets (if any)

### Container Resources

Limit resources in `docker-compose.prod.yml`:

```yaml
services:
  api:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 1G
```

## Security Best Practices

1. **Change Default Passwords**: Always set strong `DB_PASSWORD` and `JWT_SECRET`
2. **Use HTTPS in Production**: Configure SSL certificates
3. **Limit CORS Origins**: Set specific `CORS_ORIGIN` in production
4. **Regular Updates**: Keep Docker images updated
5. **Backup Regularly**: Automate database backups
6. **Monitor Logs**: Set up log monitoring and alerting
7. **Use Secrets**: Consider Docker secrets for sensitive data

## Additional Resources

- [Docker Documentation](https://docs.docker.com/)
- [Docker Compose Documentation](https://docs.docker.com/compose/)
- [PostgreSQL Docker Image](https://hub.docker.com/_/postgres)
- [Nginx Documentation](https://nginx.org/en/docs/)

## Support

For issues or questions:
1. Check logs: `docker-compose logs`
2. Review this documentation
3. Check GitHub issues
4. Contact development team