market-data-service

Market Data Service

A high-performance financial data API that provides comprehensive market data for various financial instruments including cryptocurrencies, stocks, forex, and commodities through both RESTful endpoints and real-time WebSocket connections.

MT5 Integration

The service includes an MT5 Expert Advisor (EA) that automatically sends historical candle data and live tick prices to the API.

EA Features

• Automatic symbol registration in the database
• Bulk upload of last 1000 candles per symbol
• Real-time price streaming on tick events
• Error handling with retry logic
• Configurable API endpoint and authentication

Setup Instructions

1. Copy MT5/Experts/MarketDataSender.mq5 to your MT5 Experts directory

2. Configure EA settings:

   • ApiBaseUrl: Your API endpoint (e.g., http://your-server:3000)
   • ApiKey: Optional authentication key
   • HistoricalCandleCount: Number of historical candles to send (default: 1000)
   • HistoricalTimeframe: Timeframe for historical data (default: H1)

3. Attach the EA to any MT5 chart

4. The EA will:

   • Register all available symbols with the API
   • Send historical candles on initialization
   • Stream live prices every second

Notes

• Symbols are automatically created if they don't exist
• Exchange is derived from symbol name (format: EXCHANGE_SYMBOL)
• Default instrument type is forex (customize in EA code if needed)

Features

✅ MT5 Integration

• Automatic retry logic (3 attempts with exponential backoff)
  
• Precision-preserving data transmission
  
• Symbol auto-registration
  

✅ Data Integrity

• Unique constraint on candle timestamps per symbol
  
• 15-decimal precision enforcement
  
• Strict schema validation
  

✅ Reliability

• Database transaction safety
  
• Error recovery mechanisms
  

• Comprehensive test coverage

• Multi-Asset Support: Handles cryptocurrencies, stocks, forex, and commodities

• Real-time Data: Live price feeds with bid/ask spreads

• Historical Data: OHLCV candle data with flexible timeframes

• RESTful API: Well-structured endpoints for all operations

• Data Validation: Comprehensive input validation using Joi

• Error Handling: Robust error handling with detailed responses

• Security: Helmet.js for security headers, CORS support

• Logging: Winston-based logging with multiple transports

• Database: PostgreSQL with Sequelize ORM

• Scalable Architecture: Modular design with controllers, routes, and middleware

Tech Stack

• Backend: Node.js, Express.js
• Database: PostgreSQL
• ORM: Sequelize
• Validation: Joi
• Security: Helmet, CORS
• Logging: Winston, Morgan
• Testing: Jest, Supertest
• Development: Nodemon, ESLint, Sequelize CLI

Project Structure

market-data-service/
├── src/
│   ├── config/
│   │   └── database.js          # Database configuration
│   ├── controllers/
│   │   ├── symbolController.js      # Symbol CRUD operations
│   │   ├── candleController.js      # Candle data operations
│   │   └── livePriceController.js   # Live price operations
│   ├── middleware/
│   │   ├── errorHandler.js          # Global error handling
│   │   └── validation.js            # Request validation
│   ├── models/
│   │   ├── Symbol.js                # Symbol model
│   │   ├── Candle1h.js              # 1-hour candle model
│   │   ├── LivePrice.js             # Live price model
│   │   └── index.js                 # Model associations
│   ├── routes/
│   │   ├── symbols.js               # Symbol routes
│   │   ├── candles.js               # Candle routes
│   │   └── livePrices.js            # Live price routes
│   ├── utils/
│   │   └── logger.js                # Logging utility
│   ├── app.js                       # Express app configuration
│   └── server.js                    # Server startup
├── tests/                           # Test files
├── schema.sql                       # Database schema
├── .env                             # Environment variables
├── .gitignore                       # Git ignore rules
├── package.json                     # Dependencies and scripts
└── README.md                        # This file

Technical Specifications

Database Constraints

ALTER TABLE candles_1h 
ADD CONSTRAINT unique_symbol_open_time 
UNIQUE (symbol_id, open_time);

Precision Requirements

// All numeric fields require 15 decimal precision
Joi.number().precision(15)

Installation

1. Clone the repository

   git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git
   cd market-data-service
   

2. Install dependencies

   npm install
   

3. Set up environment variables

   cp .env.example .env
   

   Edit .env with your database credentials and other configuration.

4. Database setup ```bash

   Create PostgreSQL database

   createdb market_data

# Run migrations npx sequelize-cli db:migrate

5. **Start the services**

   ### Windows Setup (with Nginx Proxy) ✅ **RECOMMENDED**
   ```bash
   # Method 1: Manual (as you're currently doing)
   # Terminal 1: Start Node.js server
   node src/server.js

   # Terminal 2: Start Nginx proxy (run as administrator)
   powershell -ExecutionPolicy Bypass -Command "cd nginx-1.24.0; .\nginx.exe"
   # OR: Right-click nginx.exe → "Run as administrator"

   # Method 2: Use batch files (alternative)
   # Right-click and "Run as administrator":
   start_proxy.bat

### Linux/Mac Setup

   # Start development server
   npm run dev

The server will start on http://localhost:3001 and proxy through Nginx on http://market-price.insightbull.io

Windows Nginx Setup Guide ✅ YOUR CURRENT METHOD

Quick Start (Your Working Setup)

1. Start Node.js server:

   node src/server.js
   

2. Start Nginx proxy (run as administrator):

   powershell -ExecutionPolicy Bypass -Command "cd nginx-1.24.0; .\nginx.exe"
   # OR: Right-click nginx.exe → "Run as administrator"
   

3. Configure MT5 EA:

   • Set ApiBaseUrl = "http://market-price.insightbull.io"
   • Use the nginx proxy URL for better performance

4. Stop services: ```bash

   Stop Nginx

   taskkill /f /im nginx.exe

# Stop Node.js taskkill /f /im node.exe

### Alternative: Batch Files
If you prefer using batch files:
- **`start_proxy.bat`**: Start both services (run as administrator)
- **`stop_proxy.bat`**: Stop both services

## Windows Batch Files

The project includes batch files for easy Windows setup:

### `start_proxy.bat`
- **Purpose**: Starts both Node.js server and Nginx proxy
- **Usage**: Right-click → "Run as administrator"
- **What it does**:
  - Adds `market-price.insightbull.io` to hosts file
  - Starts Node.js server on port 3001
  - Starts Nginx proxy on ports 80/8080
  - Tests connectivity

### `stop_proxy.bat`
- **Purpose**: Stops both Node.js server and Nginx proxy
- **Usage**: Double-click or run in terminal
- **What it does**:
  - Stops Nginx processes
  - Stops Node.js processes
  - Frees up ports

### `start_nginx.ps1`
- **Purpose**: PowerShell script to start Nginx only
- **Usage**: `powershell -ExecutionPolicy Bypass -File start_nginx.ps1`

## Environment Variables

Create a `.env` file in the root directory with the following variables:

```env
# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=market_data
DB_USER=your_username
DB_PASSWORD=your_password

# Server Configuration
PORT=3000
NODE_ENV=development

# JWT Configuration (if needed for authentication)
JWT_SECRET=your_jwt_secret_key

# API Keys (if needed for external services)
# BINANCE_API_KEY=your_api_key
# BINANCE_API_SECRET=your_api_secret

# CORS Configuration
CORS_ORIGIN=*

API Endpoints

Health Check

• GET /health - Check service health

Symbols

• GET /api/symbols - Get all symbols (with filtering)
• GET /api/symbols/search - Search symbols by name
• GET /api/symbols/:id - Get symbol by ID
• POST /api/symbols - Create new symbol
• PUT /api/symbols/:id - Update symbol
• DELETE /api/symbols/:id - Delete symbol (soft delete)

Candles

• GET /api/candles - Get candles with filtering
• GET /api/candles/ohlc - Get OHLC data
• GET /api/candles/:symbolId/latest - Get latest candle for symbol
• POST /api/candles - Create new candle
• POST /api/candles/bulk - Bulk create candles

Live Prices

• GET /api/live-prices - Get all live prices
• GET /api/live-prices/exchange/:exchange - Get live prices by exchange
• GET /api/live-prices/type/:type - Get live prices by instrument type
• GET /api/live-prices/:symbolId - Get live price for symbol
• POST /api/live-prices - Create/update live price
• POST /api/live-prices/bulk - Bulk update live prices
• DELETE /api/live-prices/:symbolId - Delete live price

Database Management

The database schema is managed through Sequelize migrations and models:

• Models are defined in src/models/
• Migrations are stored in migrations/
• Associations are configured in src/models/index.js

Key entities:

• Symbol: Financial instrument metadata
• Candle1h: Hourly OHLCV data
• LivePrice: Real-time market prices

To update the database schema:

1. Create a new migration: npx sequelize-cli migration:generate --name description
2. Implement schema changes in the migration file
3. Run migrations: npx sequelize-cli db:migrate

Development

Available Scripts

• npm start - Start production server
• npm run dev - Start development server with auto-reload
• npm test - Run Jest test suite
• npm run test:watch - Run tests in watch mode
• npm run migrate - Run database migrations
• npm run migrate:undo - Revert last migration
• npm run lint - Run ESLint
• npm run lint:fix - Fix ESLint issues

Code Style

This project uses ESLint for code linting. Run npm run lint to check for issues and npm run lint:fix to automatically fix them.

Testing

The project uses Jest with Supertest for endpoint testing. Key features:

• Integration tests with database cleanup
• Test environment database configuration
• Automatic test isolation with { force: true } sync
• Comprehensive controller tests including livePriceController

Run tests:

npm test         # Run full test suite
npm run test:watch  # Run in watch mode

Database Migrations

We use Sequelize CLI for database migrations:

# Create new migration
npx sequelize-cli migration:generate --name your-migration-name

# Run pending migrations
npx sequelize-cli db:migrate

# Revert last migration
npx sequelize-cli db:migrate:undo

Deployment

1. Set NODE_ENV=production in your environment
2. Run npm start to start the production server
3. Consider using a process manager like PM2 for production deployments

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new features
5. Ensure all tests pass
6. Submit a pull request

License

ISC License - see LICENSE file for details.

Support

For support or questions, please contact the development team.