market-data-service/DOCKER.md

Docker Containerization Guide

Complete guide for running Market Data Service in Docker containers for both development and production environments.

Table of Contents

• Prerequisites
• Quick Start
• Development Setup
• Production Setup
• Configuration
• Common Operations
• Troubleshooting
• Backup and Restore

Prerequisites

• Docker Desktop (Mac/Windows) or Docker Engine (Linux)
• Docker Compose v2.0+ (included with Docker Desktop)
• Git (for cloning repository)

No other dependencies required! Everything runs in containers.

Quick Start

Development Environment

# 1. Clone the repository
git clone <repository-url>
cd market-data-service

# 2. Copy environment template
cp .env.example .env

# 3. Edit .env file with your settings (optional for development)
# Default values work for local development

# 4. Start all services
docker-compose up -d

# 5. Check service status
docker-compose ps

# 6. View logs
docker-compose logs -f api

The API will be available at:

• HTTP: http://localhost
• API Endpoints: http://localhost/api/*
• Health Check: http://localhost/health
• WebSocket: ws://localhost/socket.io/

Production Environment

# 1. Set up environment variables
cp .env.example .env
# Edit .env with production values

# 2. Set up SSL certificates (if using HTTPS)
# Place certificates in ./ssl/certs/ and ./ssl/private/
# Or update SSL_CERT_PATH and SSL_KEY_PATH in docker-compose.prod.yml

# 3. Start production services
docker-compose -f docker-compose.prod.yml up -d

# 4. Check status
docker-compose -f docker-compose.prod.yml ps

Development Setup

Architecture

The development environment consists of three services:

1. API Service (api)

   • Node.js/Express application
   • Live code reload with volume mounts
   • Runs on port 3000 (internal)
   • Accessible via Nginx on port 80

2. Database Service (db)

   • PostgreSQL 15
   • Exposed on port 5432 (for local tools)
   • Data persisted in Docker volume

3. Nginx Service (nginx)

   • Reverse proxy
   • WebSocket support for Socket.io
   • Exposed on port 80

Starting Development Environment

# Start all services in detached mode
docker-compose up -d

# Start with logs visible
docker-compose up

# Rebuild containers after code changes
docker-compose up -d --build

Accessing Services

• API: http://localhost
• Database: localhost:5432
• API Direct: http://localhost:3000 (if port mapping enabled)

Development Workflow

1. Make code changes in src/ directory
2. Changes are automatically reloaded (nodemon watches for changes)
3. View logs: docker-compose logs -f api
4. Restart service: docker-compose restart api

Database Migrations

Migrations run automatically on container startup. To run manually:

# Run migrations
docker-compose exec api npx sequelize-cli db:migrate

# Rollback last migration
docker-compose exec api npx sequelize-cli db:migrate:undo

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status

Accessing Database

# Connect to PostgreSQL
docker-compose exec db psql -U postgres -d financial_data

# Run SQL commands
docker-compose exec db psql -U postgres -d financial_data -c "SELECT * FROM symbols LIMIT 10;"

# List tables
docker-compose exec db psql -U postgres -d financial_data -c "\dt"

Production Setup

Architecture

Production environment uses optimized settings:

• No volume mounts (code baked into image)
• Multi-stage builds for smaller images
• Health checks for all services
• SSL/TLS support via Nginx
• Restart policies for high availability

Prerequisites for Production

1. Environment Variables

   # Set strong passwords and secrets
   DB_PASSWORD=<strong_password>
   JWT_SECRET=<strong_random_secret>
   CORS_ORIGIN=https://your-domain.com
   

2. SSL Certificates (for HTTPS) ```bash

   Option 1: Use Let's Encrypt (recommended)

   Certificates will be in /etc/letsencrypt/live/your-domain.com/

# Option 2: Place certificates manually mkdir -p ssl/certs ssl/private # Copy fullchain.pem to ssl/certs/ # Copy privkey.pem to ssl/private/

3. **Update Nginx Configuration**
   - Edit `nginx/nginx.prod.conf`
   - Update SSL certificate paths
   - Update `server_name` if using domain names

### Starting Production Environment

```bash
# Build production images
docker-compose -f docker-compose.prod.yml build

# Start services
docker-compose -f docker-compose.prod.yml up -d

# Check health status
docker-compose -f docker-compose.prod.yml ps

Production Features

• Automatic migrations on startup
• Health checks for service monitoring
• SSL/TLS encryption
• Security headers (HSTS, X-Frame-Options, etc.)
• Rate limiting (configurable in nginx.prod.conf)
• Data persistence via Docker volumes

Configuration

Environment Variables

Key environment variables (see .env.example for complete list):

Variable	Description	Default
DB_HOST	Database host	db (Docker) or localhost
DB_PORT	Database port	5432
DB_NAME	Database name	financial_data
DB_USER	Database user	postgres
DB_PASSWORD	Database password	Required
PORT	API port	3000
NODE_ENV	Environment	development or production
CORS_ORIGIN	Allowed origins	*
JWT_SECRET	JWT secret key	Required
LOG_LEVEL	Logging level	debug (dev) or info (prod)

Nginx Configuration

• Development: nginx/nginx.conf (HTTP only)
• Production: nginx/nginx.prod.conf (HTTPS with SSL)

Key features:

• WebSocket proxy for Socket.io
• API endpoint routing
• Health check endpoint
• Security headers (production)
• Rate limiting (production)

Database Configuration

Database is automatically initialized with:

• Database: financial_data
• User: postgres (or from DB_USER)
• Migrations run on API container startup

Common Operations

Viewing Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api
docker-compose logs -f db
docker-compose logs -f nginx

# Last 100 lines
docker-compose logs --tail=100 api

Restarting Services

# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart api

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: deletes data)
docker-compose down -v

Accessing Containers

# Execute command in API container
docker-compose exec api sh

# Execute command in database container
docker-compose exec db sh

# Run Node.js commands
docker-compose exec api node -v
docker-compose exec api npm list

Updating Application

# 1. Pull latest code
git pull

# 2. Rebuild containers
docker-compose build

# 3. Restart services (migrations run automatically)
docker-compose up -d

# Or for production
docker-compose -f docker-compose.prod.yml build
docker-compose -f docker-compose.prod.yml up -d

Health Checks

# Check container health
docker-compose ps

# Test API health endpoint
curl http://localhost/health

# Test database connection
docker-compose exec db pg_isready -U postgres

Troubleshooting

Database Connection Issues

Problem: API can't connect to database

Solutions:

# Check database is running
docker-compose ps db

# Check database logs
docker-compose logs db

# Test database connection manually
docker-compose exec db psql -U postgres -d financial_data

# Verify environment variables
docker-compose exec api env | grep DB_

Migration Failures

Problem: Migrations fail on startup

Solutions:

# Check migration logs
docker-compose logs api | grep -i migration

# Run migrations manually
docker-compose exec api npx sequelize-cli db:migrate

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status

WebSocket Connection Issues

Problem: WebSocket connections fail through Nginx

Solutions:

1. Verify Nginx configuration includes WebSocket upgrade headers
2. Check Nginx logs: docker-compose logs nginx
3. Test direct connection: ws://localhost:3000/socket.io/
4. Verify Socket.io transport is set to 'websocket' only

Port Conflicts

Problem: Port already in use

Solutions:

# Check what's using the port
lsof -i :80
lsof -i :5432

# Change ports in docker-compose.yml
# Update PORT and DB_PORT environment variables

Container Won't Start

Problem: Container exits immediately

Solutions:

# Check exit code
docker-compose ps

# View detailed logs
docker-compose logs api

# Check container status
docker ps -a

# Try starting without detached mode
docker-compose up api

Permission Issues

Problem: Permission denied errors

Solutions:

# Fix script permissions
chmod +x docker/*.sh

# Check file ownership
ls -la docker/

# Rebuild containers
docker-compose build --no-cache

Backup and Restore

Database Backup

# Create backup
docker-compose exec db pg_dump -U postgres financial_data > backup_$(date +%Y%m%d_%H%M%S).sql

# Or using docker directly
docker exec market-data-db pg_dump -U postgres financial_data > backup.sql

Database Restore

# Restore from backup
docker-compose exec -T db psql -U postgres financial_data < backup.sql

# Or using docker directly
docker exec -i market-data-db psql -U postgres financial_data < backup.sql

Volume Backup

# Backup database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar czf /backup/db_backup.tar.gz /data

# Restore database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar xzf /backup/db_backup.tar.gz -C /

Automated Backups

Create a cron job or scheduled task:

# Daily backup script
#!/bin/bash
BACKUP_DIR="./backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
docker-compose exec -T db pg_dump -U postgres financial_data | gzip > $BACKUP_DIR/backup_$DATE.sql.gz
# Keep only last 30 days
find $BACKUP_DIR -name "backup_*.sql.gz" -mtime +30 -delete

MT5 EA Integration

Development

Update MT5 EA settings:

• ApiBaseUrl: http://localhost or http://localhost:3000

Production

Update MT5 EA settings:

• ApiBaseUrl: https://your-domain.com
• Ensure SSL certificate is valid
• WebSocket connections use wss:// protocol

Performance Tuning

Database Optimization

# Increase shared buffers (edit postgresql.conf in container)
# Or use environment variables in docker-compose.yml

Nginx Optimization

• Adjust worker_processes in nginx.conf
• Tune worker_connections based on expected load
• Enable caching for static assets (if any)

Container Resources

Limit resources in docker-compose.prod.yml:

services:
  api:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 1G

Security Best Practices

1. Change Default Passwords: Always set strong DB_PASSWORD and JWT_SECRET
2. Use HTTPS in Production: Configure SSL certificates
3. Limit CORS Origins: Set specific CORS_ORIGIN in production
4. Regular Updates: Keep Docker images updated
5. Backup Regularly: Automate database backups
6. Monitor Logs: Set up log monitoring and alerting
7. Use Secrets: Consider Docker secrets for sensitive data

Additional Resources

• Docker Documentation
• Docker Compose Documentation
• PostgreSQL Docker Image
• Nginx Documentation

Support

For issues or questions:

1. Check logs: docker-compose logs
2. Review this documentation
3. Check GitHub issues
4. Contact development team