market-data-service/DOCKER.md

Docker Containerization Guide

Complete guide for running Market Data Service in Docker containers for both development and production environments.

Table of Contents

• Prerequisites
• Quick Start
• Development Setup
• Production Setup
• Configuration
• Common Operations
• Troubleshooting
• Backup and Restore

Prerequisites

• Docker Desktop (Mac/Windows) or Docker Engine (Linux)
• Docker Compose v2.0+ (included with Docker Desktop)
• Git (for cloning repository)

No other dependencies required! Everything runs in containers.

Quick Start

Development Environment

# 1. Clone the repository
git clone <repository-url>
cd market-data-service

# 2. Copy environment template
cp .env.example .env

# 3. Edit .env file with your settings (optional for development)
# Default values work for local development

# 4. Start all services
docker-compose up -d

# 5. Check service status
docker-compose ps

# 6. View logs
docker-compose logs -f api

The API will be available at:

• HTTP: http://localhost
• API Endpoints: http://localhost/api/*
• Health Check: http://localhost/health
• WebSocket: ws://localhost/socket.io/

Production Environment

# 1. Set up environment variables
cp .env.example .env
# Edit .env with production values

# 2. Set up SSL certificates (if using HTTPS)
# Place certificates in ./ssl/certs/ and ./ssl/private/
# Or update SSL_CERT_PATH and SSL_KEY_PATH in docker-compose.prod.yml

# 3. Start production services
docker-compose -f docker-compose.prod.yml up -d

# 4. Check status
docker-compose -f docker-compose.prod.yml ps

Development Setup

Architecture

The development environment consists of three services:

1. API Service (api)

   • Node.js/Express application
   • Live code reload with volume mounts
   • Runs on port 3000 (internal)
   • Accessible via Nginx on port 80

2. Database Service (db)

   • PostgreSQL 15
   • Exposed on port 5432 (for local tools)
   • Data persisted in Docker volume

3. Nginx Service (nginx)

   • Reverse proxy
   • WebSocket support for Socket.io
   • Exposed on port 80

Starting Development Environment

# Start all services in detached mode
docker-compose up -d

# Start with logs visible
docker-compose up

# Rebuild containers after code changes
docker-compose up -d --build

Accessing Services

• API: http://localhost
• Database: localhost:5432
• API Direct: http://localhost:3000 (if port mapping enabled)

Development Workflow

1. Make code changes in src/ directory
2. Changes are automatically reloaded (nodemon watches for changes)
3. View logs: docker-compose logs -f api
4. Restart service: docker-compose restart api

Database Migrations

Migrations run automatically on container startup. To run manually:

# Run migrations
docker-compose exec api npx sequelize-cli db:migrate

# Rollback last migration
docker-compose exec api npx sequelize-cli db:migrate:undo

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status

Accessing Database

# Connect to PostgreSQL
docker-compose exec db psql -U postgres -d financial_data

# Run SQL commands
docker-compose exec db psql -U postgres -d financial_data -c "SELECT * FROM symbols LIMIT 10;"

# List tables
docker-compose exec db psql -U postgres -d financial_data -c "\dt"

Production Setup

Architecture

Production environment uses optimized settings:

• No volume mounts (code baked into image)
• Multi-stage builds for smaller images
• Health checks for all services
• SSL/TLS support via Nginx
• Restart policies for high availability

Prerequisites for Production

1. Environment Variables

   # Set strong passwords and secrets
   DB_PASSWORD=<strong_password>
   JWT_SECRET=<strong_random_secret>
   CORS_ORIGIN=https://your-domain.com
   DOMAIN_NAME=your-domain.com
   SSL_EMAIL=your-email@example.com
   

See CONFIGURATION.md for complete environment variable documentation.

1. SSL Certificates SSL certificates are automatically managed via Docker containers. No manual setup required.
   

See DEPLOYMENT.md for production deployment with SSL certificate setup.

Starting Production Environment

# Build production images
docker-compose -f docker-compose.prod.yml build

# Start services
docker-compose -f docker-compose.prod.yml up -d

# Check health status
docker-compose -f docker-compose.prod.yml ps

Production Features

• Automatic migrations on startup
• Health checks for service monitoring
• SSL/TLS encryption
• Security headers (HSTS, X-Frame-Options, etc.)
• Rate limiting (configurable in nginx.prod.conf)
• Data persistence via Docker volumes

Configuration

Environment Variables

Key environment variables (see .env.example for complete list):

Variable	Description	Default
DB_HOST	Database host	db (Docker) or localhost
DB_PORT	Database port	5432
DB_NAME	Database name	financial_data
DB_USER	Database user	postgres
DB_PASSWORD	Database password	Required
PORT	API port	3000
NODE_ENV	Environment	development or production
CORS_ORIGIN	Allowed origins	*
JWT_SECRET	JWT secret key	Required
LOG_LEVEL	Logging level	debug (dev) or info (prod)

Nginx Configuration

• Development: nginx/nginx.conf (HTTP only)
• Production: nginx/nginx.prod.conf (HTTPS with SSL)

Key features:

• WebSocket proxy for Socket.io
• API endpoint routing
• Health check endpoint
• Security headers (production)
• Rate limiting (production)

Database Configuration

Database is automatically initialized with:

• Database: financial_data
• User: postgres (or from DB_USER)
• Migrations run on API container startup

Common Operations

Viewing Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api
docker-compose logs -f db
docker-compose logs -f nginx

# Last 100 lines
docker-compose logs --tail=100 api

Restarting Services

# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart api

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: deletes data)
docker-compose down -v

Accessing Containers

# Execute command in API container
docker-compose exec api sh

# Execute command in database container
docker-compose exec db sh

# Run Node.js commands
docker-compose exec api node -v
docker-compose exec api npm list

Updating Application

# 1. Pull latest code
git pull

# 2. Rebuild containers
docker-compose build

# 3. Restart services (migrations run automatically)
docker-compose up -d

# Or for production
docker-compose -f docker-compose.prod.yml build
docker-compose -f docker-compose.prod.yml up -d

Health Checks

# Check container health
docker-compose ps

# Test API health endpoint
curl http://localhost/health

# Test database connection
docker-compose exec db pg_isready -U postgres

Troubleshooting

Database Connection Issues

Problem: API can't connect to database

Solutions:

# Check database is running
docker-compose ps db

# Check database logs
docker-compose logs db

# Test database connection manually
docker-compose exec db psql -U postgres -d financial_data

# Verify environment variables
docker-compose exec api env | grep DB_

Migration Failures

Problem: Migrations fail on startup

Solutions:

# Check migration logs
docker-compose logs api | grep -i migration

# Run migrations manually
docker-compose exec api npx sequelize-cli db:migrate

# Check migration status
docker-compose exec api npx sequelize-cli db:migrate:status

WebSocket Connection Issues

Problem: WebSocket connections fail through Nginx

Solutions:

1. Verify Nginx configuration includes WebSocket upgrade headers
2. Check Nginx logs: docker-compose logs nginx
3. Test direct connection: ws://localhost:3000/socket.io/
4. Verify Socket.io transport is set to 'websocket' only

Port Conflicts

Problem: Port already in use

Solutions:

# Check what's using the port
lsof -i :80
lsof -i :5432

# Change ports in docker-compose.yml
# Update PORT and DB_PORT environment variables

Container Won't Start

Problem: Container exits immediately

Solutions:

# Check exit code
docker-compose ps

# View detailed logs
docker-compose logs api

# Check container status
docker ps -a

# Try starting without detached mode
docker-compose up api

Permission Issues

Problem: Permission denied errors

Solutions:

# Fix script permissions
chmod +x docker/*.sh

# Check file ownership
ls -la docker/

# Rebuild containers
docker-compose build --no-cache

Backup and Restore

Database Backup

# Create backup
docker-compose exec db pg_dump -U postgres financial_data > backup_$(date +%Y%m%d_%H%M%S).sql

# Or using docker directly
docker exec market-data-db pg_dump -U postgres financial_data > backup.sql

Database Restore

# Restore from backup
docker-compose exec -T db psql -U postgres financial_data < backup.sql

# Or using docker directly
docker exec -i market-data-db psql -U postgres financial_data < backup.sql

Volume Backup

# Backup database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar czf /backup/db_backup.tar.gz /data

# Restore database volume
docker run --rm -v market_data_db_data:/data -v $(pwd):/backup alpine tar xzf /backup/db_backup.tar.gz -C /

Automated Backups

Create a cron job or scheduled task:

# Daily backup script
#!/bin/bash
BACKUP_DIR="./backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
docker-compose exec -T db pg_dump -U postgres financial_data | gzip > $BACKUP_DIR/backup_$DATE.sql.gz
# Keep only last 30 days
find $BACKUP_DIR -name "backup_*.sql.gz" -mtime +30 -delete

MT5 EA Integration

Development

Update MT5 EA settings:

• ApiBaseUrl: http://market-data.local
• See LOCAL_DEV_SETUP.md for local development setup

Production

Update MT5 EA settings:

• ApiBaseUrl: https://market-price.insightbull.io (or your production domain)
• SSL certificates are automatically managed
• WebSocket connections use wss:// protocol
• See DEPLOYMENT.md for production deployment

Performance Tuning

Database Optimization

# Increase shared buffers (edit postgresql.conf in container)
# Or use environment variables in docker-compose.yml

Nginx Optimization

• Adjust worker_processes in nginx.conf
• Tune worker_connections based on expected load
• Enable caching for static assets (if any)

Container Resources

Limit resources in docker-compose.prod.yml:

services:
  api:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 1G

Security Best Practices

1. Change Default Passwords: Always set strong DB_PASSWORD and JWT_SECRET
2. Use HTTPS in Production: Configure SSL certificates
3. Limit CORS Origins: Set specific CORS_ORIGIN in production
4. Regular Updates: Keep Docker images updated
5. Backup Regularly: Automate database backups
6. Monitor Logs: Set up log monitoring and alerting
7. Use Secrets: Consider Docker secrets for sensitive data

Additional Resources

• Docker Documentation
• Docker Compose Documentation
• PostgreSQL Docker Image
• Nginx Documentation

Support

For issues or questions:

1. Check logs: docker-compose logs
2. Review this documentation
3. Check GitHub issues
4. Contact development team