market-data-service

Market Data Service

A high-performance financial data API that provides comprehensive market data for various financial instruments including cryptocurrencies, stocks, forex, and commodities through both RESTful endpoints and real-time WebSocket connections.

MT5 Integration

The service includes an MT5 Expert Advisor (EA) that automatically sends historical candle data and live tick prices to the API.

EA Features

• Automatic symbol registration in the database
• Bulk upload of last 1000 candles per symbol
• Real-time price streaming on tick events
• Error handling with retry logic
• Configurable API endpoint and authentication

Setup Instructions

1. Copy MT5/Experts/MarketDataSender.mq5 to your MT5 Experts directory

2. Configure EA settings:

   • ApiBaseUrl: Your API endpoint (e.g., http://your-server:3000)
   • ApiKey: Optional authentication key
   • HistoricalCandleCount: Number of historical candles to send (default: 1000)
   • HistoricalTimeframe: Timeframe for historical data (default: H1)

3. Attach the EA to any MT5 chart

4. The EA will:

   • Register all available symbols with the API
   • Send historical candles on initialization
   • Stream live prices every second

Notes

• Symbols are automatically created if they don't exist
• Exchange is derived from symbol name (format: EXCHANGE_SYMBOL)
• Default instrument type is forex (customize in EA code if needed)

Features

✅ MT5 Integration

• Automatic retry logic (3 attempts with exponential backoff)
  
• Precision-preserving data transmission
  
• Symbol auto-registration
  

✅ Data Integrity

• Unique constraint on candle timestamps per symbol
  
• 15-decimal precision enforcement
  
• Strict schema validation
  

✅ Reliability

• Database transaction safety
  
• Error recovery mechanisms
  

• Comprehensive test coverage

• Multi-Asset Support: Handles cryptocurrencies, stocks, forex, commodities, and indices

• Real-time Data: Live price feeds with bid/ask spreads

• Historical Data: OHLCV candle data with flexible timeframes

• RESTful API: Well-structured endpoints for all operations

• Data Validation: Comprehensive input validation using Joi

• Error Handling: Robust error handling with detailed responses

• Security: Helmet.js for security headers, CORS support

• Logging: Winston-based logging with multiple transports

• Database: PostgreSQL with Sequelize ORM

• Scalable Architecture: Modular design with controllers, routes, and middleware

Tech Stack

• Backend: Node.js, Express.js
• Database: PostgreSQL
• ORM: Sequelize
• Validation: Joi
• Security: Helmet, CORS
• Logging: Winston, Morgan
• Testing: Jest, Supertest
• Development: Nodemon, ESLint, Sequelize CLI

Project Structure

market-data-service/
├── src/
│   ├── config/
│   │   └── database.js          # Database configuration
│   ├── controllers/
│   │   ├── symbolController.js      # Symbol CRUD operations
│   │   ├── candleController.js      # Candle data operations
│   │   └── livePriceController.js   # Live price operations
│   ├── middleware/
│   │   ├── errorHandler.js          # Global error handling
│   │   └── validation.js            # Request validation
│   ├── models/
│   │   ├── Symbol.js                # Symbol model
│   │   ├── Candle.js                # Multi-timeframe candle model
│   │   ├── LivePrice.js             # Live price model
│   │   └── index.js                 # Model associations
│   ├── routes/
│   │   ├── symbols.js               # Symbol routes
│   │   ├── candles.js               # Candle routes
│   │   └── livePrices.js            # Live price routes
│   ├── utils/
│   │   └── logger.js                # Logging utility
│   ├── app.js                       # Express app configuration
│   └── server.js                    # Server startup
├── tests/                           # Test files
├── schema.sql                       # Database schema
├── .env                             # Environment variables
├── .gitignore                       # Git ignore rules
├── package.json                     # Dependencies and scripts
└── README.md                        # This file

Technical Specifications

Database Constraints

ALTER TABLE candles_1h 
ADD CONSTRAINT unique_symbol_open_time 
UNIQUE (symbol_id, open_time);

Precision Requirements

// All numeric fields require 15 decimal precision
Joi.number().precision(15)

🚀 Quick Start

Local Development

# Clone repository
git clone <your-repo-url>
cd market-data-service

# Start Docker services
docker-compose up -d

# Access API at http://market-data.local

📖 For detailed local development setup, see docs/LOCAL_DEV_SETUP.md

Production Deployment

# Set up environment variables
cp .env.example .env
# Edit .env with your values

# Start production services
docker-compose -f docker-compose.prod.yml up -d

📖 For complete production deployment guide, see docs/DEPLOYMENT.md

---

📚 Documentation

• Local Development Setup - Complete local development guide
• Production Deployment - Production deployment with SSL certificates
• Docker Guide - Docker containerization details
• Configuration Guide - Environment variables and configuration
• API Contract - API endpoint specifications
• MT5 Operation - MT5 Expert Advisor guide

---

🆕 FRESH SERVER DEPLOYMENT (Legacy - See DEPLOYMENT.md)

Step 1: Server Preparation

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install required tools
sudo apt install -y curl wget git htop nano ufw

Step 2: Install Node.js (v18.x LTS)

# Using NodeSource repository
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs

# Verify installation
node --version  # Should show v18.x.x
npm --version   # Should show latest version

Step 3: Install PostgreSQL

# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib

# Start and enable PostgreSQL
sudo systemctl enable postgresql
sudo systemctl start postgresql

# Verify installation
psql --version  # Should show 15.x or higher

Step 4: Deploy Application

# Create application directory
sudo mkdir -p /root/market-data-service
sudo chown $USER:$USER /root/market-data-service
cd /root/market-data-service

# Clone repository (replace with your actual repo URL)
git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git .
# OR copy your project files to this directory

# Install dependencies
npm install --production

# Set up environment variables
cat > .env << EOF
DB_TYPE=postgres
DB_HOST=localhost
DB_PORT=5432
DB_NAME=financial_data
DB_USER=postgres
DB_PASSWORD=your_secure_password_here

PORT=3001
NODE_ENV=production

JWT_SECRET=your_secure_jwt_secret_key_here
CORS_ORIGIN=*

LOG_LEVEL=info
EOF

Step 5: Database Setup

# Switch to postgres user
sudo -u postgres psql

# In PostgreSQL shell, run:
CREATE DATABASE financial_data;
ALTER USER postgres PASSWORD 'your_secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE financial_data TO postgres;
\q

# Exit and run migrations
npx sequelize-cli db:migrate

# Verify tables were created
sudo -u postgres psql -d financial_data -c "\dt"

Step 6: Deploy with Docker (Recommended)

⚠️ Outdated Instructions: This section contains legacy deployment steps. For current production deployment with containerized SSL certificates, see docs/DEPLOYMENT.md

# Navigate to project directory
cd /root/market-data-service

# Copy environment template
cp .env.example .env

# Edit .env file with production values
nano .env

# Build and start production containers
docker-compose -f docker-compose.prod.yml up -d --build

📖 For detailed Docker deployment instructions, see DOCKER.md
📖 For production deployment with SSL, see docs/DEPLOYMENT.md

---

💻 Local Development

Quick Overview: For complete setup instructions, troubleshooting, and MT5 EA configuration, see docs/LOCAL_DEV_SETUP.md.

Prerequisites

• Docker and Docker Compose installed
• Git installed
• Administrator access (script will prompt for password automatically on macOS)

Quick Start

1. Clone the Repository

# Clone from GitHub
git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git
cd market-data-service

2. Setup Local Domain (One-time, Required for MT5 EA)

# Run automated setup script (fully automated on macOS - will prompt for password)
# On Linux, you may need to run with: sudo bash scripts/setup-local-dev.sh
bash scripts/setup-local-dev.sh

Note: On macOS, the script automatically prompts for your administrator password. On Linux, you may need to run with sudo.

Alternative (Manual):

# Add market-data.local to hosts file manually
echo "127.0.0.1 market-data.local" | sudo tee -a /etc/hosts

3. Start Docker Services

# Start all services (database, API, Nginx)
docker-compose up -d

# Verify services are running
docker-compose ps

4. Verify Setup

# Test health endpoint
curl http://market-data.local/health

# Test API endpoints
curl http://market-data.local/api/symbols
curl http://market-data.local/api/health

✅ Setup Complete! Your API is now accessible at http://market-data.local

Why market-data.local?

MT5 Expert Advisors require valid domain URLs and do not allow localhost or 127.0.0.1. The market-data.local domain resolves to your local Docker setup, enabling MT5 EA connectivity.

MT5 EA Configuration

1. Add URL to MT5 Allowed List:

   • Open MT5 → Tools → Options → Expert Advisors
   • Click "Allowed URLs" tab → Add → Enter: http://market-data.local

2. Configure EA:

   • Attach MarketDataSender.mq5 to any chart
   • Set ApiBaseUrl = "http://market-data.local"
   • Other settings as needed

3. Verify Connection:

   • Check MT5 Experts tab for EA logs
   • Look for: ✅ Symbols initialized

Development Workflow

View Logs:

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api
docker-compose logs -f nginx

Code Changes:

• Code in src/ is automatically reloaded (no container restart needed)
• View API logs to see changes: docker-compose logs -f api

Stop Services:

docker-compose down

Restart Services:

docker-compose restart

Need More Help?

📖 For comprehensive setup instructions, troubleshooting, and advanced configuration, see:

• docs/LOCAL_DEV_SETUP.md - Complete local development guide with detailed steps, troubleshooting, and MT5 EA configuration

Alternative: Non-Docker Local Setup

If you prefer to run without Docker (not recommended for MT5 EA):

Click to expand non-Docker setup instructions

#### Prerequisites - **Node.js**: v18.x LTS or higher - **PostgreSQL**: v13.x or higher #### Setup Steps 1. **Install Dependencies:** ```bash npm install ``` 2. **Set Up Database:** ```bash createdb financial_data # Create .env file cat > .env << EOF DB_TYPE=postgres DB_HOST=localhost DB_PORT=5432 DB_NAME=financial_data DB_USER=your_local_username DB_PASSWORD=your_local_password PORT=3001 NODE_ENV=development JWT_SECRET=your_development_jwt_secret_key CORS_ORIGIN=http://localhost:3000 LOG_LEVEL=debug EOF ``` 3. **Run Migrations:** ```bash npx sequelize-cli db:migrate ``` 4. **Start Server:** ```bash npm run dev ``` 5. **Test:** ```bash curl http://localhost:3001/health ``` **Note:** This setup won't work with MT5 EA (requires domain URL). Use Docker setup for MT5 integration.

🔄 UPDATING EXISTING DEPLOYMENT

Quick Update (When Code Changes on GitHub)

# Navigate to project directory
cd /root/market-data-service

# Backup current deployment (optional but recommended)
cp .env .env.backup

# Pull latest changes
git pull origin master

# Rebuild and restart containers (migrations run automatically)
docker-compose -f docker-compose.prod.yml up -d --build

# Verify everything is working
docker-compose -f docker-compose.prod.yml ps
curl https://your-domain.com/health

Full Update (Major Changes)

# Navigate to project directory
cd /root/market-data-service

# Create backup
sudo cp -r /root/market-data-service /root/market-data-service-backup-$(date +%Y%m%d_%H%M%S)

# Stop containers
docker-compose -f docker-compose.prod.yml down

# Update code
git fetch origin
git reset --hard origin/master

# Update environment if needed
nano .env

# Rebuild and start containers (migrations run automatically)
docker-compose -f docker-compose.prod.yml up -d --build

# Verify deployment
docker-compose -f docker-compose.prod.yml ps
curl https://your-domain.com/health

---

🛠️ Troubleshooting Guide

Common Issues & Solutions

1. PostgreSQL User Already Exists

Error: ERROR: role "postgres" already exists

Solution:

# Just set password for existing user
sudo -u postgres psql
ALTER USER postgres PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE financial_data TO postgres;
\q

2. SSL Certificate Not Found

Error: cannot load certificate "/etc/letsencrypt/live/.../fullchain.pem"

Solution:

# Generate certificate first
sudo certbot --nginx -d your-domain.com

# Check if certificates exist
ls -la /etc/letsencrypt/live/your-domain.com/

# Test nginx configuration
sudo nginx -t

3. Docker Container Issues

Problem: Container won't start or keeps restarting

Solution:

# Check container logs
docker-compose -f docker-compose.prod.yml logs api

# Check container status
docker-compose -f docker-compose.prod.yml ps

# Restart specific service
docker-compose -f docker-compose.prod.yml restart api

📊 Monitoring & Maintenance

# Check service status
docker-compose -f docker-compose.prod.yml ps
sudo systemctl status nginx

# View logs
docker-compose -f docker-compose.prod.yml logs -f api
docker-compose -f docker-compose.prod.yml logs -f db
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

SSL Certificate Auto-Renewal

Note: SSL certificates are now automatically managed via Docker containers. See docs/DEPLOYMENT.md for details.

---

🔧 Management Commands

Docker Management

# Check container status
docker-compose -f docker-compose.prod.yml ps

# View logs
docker-compose -f docker-compose.prod.yml logs -f api
docker-compose -f docker-compose.prod.yml logs -f db

# Restart services
docker-compose -f docker-compose.prod.yml restart

# Stop all services
docker-compose -f docker-compose.prod.yml down

# Start services
docker-compose -f docker-compose.prod.yml up -d

# Rebuild and restart
docker-compose -f docker-compose.prod.yml up -d --build

Nginx Management

sudo systemctl status nginx      # Check status
sudo systemctl restart nginx     # Restart nginx
sudo nginx -t                    # Test configuration
sudo tail -f /var/log/nginx/error.log  # View error logs

Database Management

# Connect to database
sudo -u postgres psql -d financial_data

# View tables
\dt

# View table structure
\d table_name

# Backup database
pg_dump financial_data > backup.sql

# Restore database
psql financial_data < backup.sql

---

Environment Variables

See docs/CONFIGURATION.md for complete environment variable documentation.

Quick Reference:

• DB_PASSWORD - Database password (required)
• JWT_SECRET - JWT secret key (required)
• DOMAIN_NAME - Domain for SSL certificates (production)
• SSL_EMAIL - Email for Let's Encrypt (production)
• CORS_ORIGIN - Allowed CORS origins

API Endpoints

See docs/API_CONTRACT.md for complete API documentation.

Quick Reference:

• GET /health - Health check
• GET /api/symbols - Get all symbols
• GET /api/candles - Get candle data
• POST /api/candles/bulk - Bulk create candles
• GET /api/live-prices - Get live prices
• POST /api/live-prices/bulk - Bulk update live prices

Development

Available Scripts

• npm start - Start production server
• npm run dev - Start development server with auto-reload
• npm test - Run Jest test suite
• npm run test:watch - Run tests in watch mode
• npm run migrate - Run database migrations
• npm run migrate:undo - Revert last migration
• npm run lint - Run ESLint
• npm run lint:fix - Fix ESLint issues

Database Migrations

We use Sequelize CLI for database migrations:

# Create new migration
npx sequelize-cli migration:generate --name your-migration-name

# Run pending migrations
npx sequelize-cli db:migrate

# Revert last migration
npx sequelize-cli db:migrate:undo

Deployment

The project is fully containerized using Docker with automatic SSL certificate management.

• Development: See docs/LOCAL_DEV_SETUP.md
• Production: See docs/DEPLOYMENT.md
• Docker Details: See DOCKER.md