market-data-service

Market Data Service

A high-performance financial data API that provides comprehensive market data for various financial instruments including cryptocurrencies, stocks, forex, and commodities through both RESTful endpoints and real-time WebSocket connections.

MT5 Integration

The service includes an MT5 Expert Advisor (EA) that automatically sends historical candle data and live tick prices to the API.

EA Features

• Automatic symbol registration in the database
• Bulk upload of last 1000 candles per symbol
• Real-time price streaming on tick events
• Error handling with retry logic
• Configurable API endpoint and authentication

Setup Instructions

1. Copy MT5/Experts/MarketDataSender.mq5 to your MT5 Experts directory

2. Configure EA settings:

   • ApiBaseUrl: Your API endpoint (e.g., http://your-server:3000)
   • ApiKey: Optional authentication key
   • HistoricalCandleCount: Number of historical candles to send (default: 1000)
   • HistoricalTimeframe: Timeframe for historical data (default: H1)

3. Attach the EA to any MT5 chart

4. The EA will:

   • Register all available symbols with the API
   • Send historical candles on initialization
   • Stream live prices every second

Notes

• Symbols are automatically created if they don't exist
• Exchange is derived from symbol name (format: EXCHANGE_SYMBOL)
• Default instrument type is forex (customize in EA code if needed)

Features

• Multi-Asset Support: Handles cryptocurrencies, stocks, forex, and commodities
• Real-time Data: Live price feeds with bid/ask spreads
• Historical Data: OHLCV candle data with flexible timeframes
• RESTful API: Well-structured endpoints for all operations
• Data Validation: Comprehensive input validation using Joi
• Error Handling: Robust error handling with detailed responses
• Security: Helmet.js for security headers, CORS support
• Logging: Winston-based logging with multiple transports
• Database: PostgreSQL with Sequelize ORM
• Scalable Architecture: Modular design with controllers, routes, and middleware

Tech Stack

• Backend: Node.js, Express.js
• Database: PostgreSQL
• ORM: Sequelize
• Validation: Joi
• Security: Helmet, CORS
• Logging: Winston, Morgan
• Testing: Jest
• Development: Nodemon, ESLint

Project Structure

market-data-service/
├── src/
│   ├── config/
│   │   └── database.js          # Database configuration
│   ├── controllers/
│   │   ├── symbolController.js      # Symbol CRUD operations
│   │   ├── candleController.js      # Candle data operations
│   │   └── livePriceController.js   # Live price operations
│   ├── middleware/
│   │   ├── errorHandler.js          # Global error handling
│   │   └── validation.js            # Request validation
│   ├── models/
│   │   ├── Symbol.js                # Symbol model
│   │   ├── Candle1h.js              # 1-hour candle model
│   │   ├── LivePrice.js             # Live price model
│   │   └── index.js                 # Model associations
│   ├── routes/
│   │   ├── symbols.js               # Symbol routes
│   │   ├── candles.js               # Candle routes
│   │   └── livePrices.js            # Live price routes
│   ├── utils/
│   │   └── logger.js                # Logging utility
│   ├── app.js                       # Express app configuration
│   └── server.js                    # Server startup
├── tests/                           # Test files
├── schema.sql                       # Database schema
├── .env                             # Environment variables
├── .gitignore                       # Git ignore rules
├── package.json                     # Dependencies and scripts
└── README.md                        # This file

Installation

1. Clone the repository

   git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git
   cd market-data-service
   

2. Install dependencies

   npm install
   

3. Set up environment variables

   cp .env.example .env
   

   Edit .env with your database credentials and other configuration.

4. Set up the database ```bash

   Create PostgreSQL database

   createdb market_data

# Run the schema psql -d market_data -f schema.sql

5. **Start the development server**
   ```bash
   npm run dev

The server will start on http://localhost:3000

Environment Variables

Create a .env file in the root directory with the following variables:

# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=market_data
DB_USER=your_username
DB_PASSWORD=your_password

# Server Configuration
PORT=3000
NODE_ENV=development

# JWT Configuration (if needed for authentication)
JWT_SECRET=your_jwt_secret_key

# API Keys (if needed for external services)
# BINANCE_API_KEY=your_api_key
# BINANCE_API_SECRET=your_api_secret

# CORS Configuration
CORS_ORIGIN=*

API Endpoints

Health Check

• GET /health - Check service health

Symbols

• GET /api/symbols - Get all symbols (with filtering)
• GET /api/symbols/search - Search symbols by name
• GET /api/symbols/:id - Get symbol by ID
• POST /api/symbols - Create new symbol
• PUT /api/symbols/:id - Update symbol
• DELETE /api/symbols/:id - Delete symbol (soft delete)

Candles

• GET /api/candles - Get candles with filtering
• GET /api/candles/ohlc - Get OHLC data
• GET /api/candles/:symbolId/latest - Get latest candle for symbol
• POST /api/candles - Create new candle
• POST /api/candles/bulk - Bulk create candles

Live Prices

• GET /api/live-prices - Get all live prices
• GET /api/live-prices/exchange/:exchange - Get live prices by exchange
• GET /api/live-prices/type/:type - Get live prices by instrument type
• GET /api/live-prices/:symbolId - Get live price for symbol
• POST /api/live-prices - Create/update live price
• POST /api/live-prices/bulk - Bulk update live prices
• DELETE /api/live-prices/:symbolId - Delete live price

Database Schema

The service uses three main tables:

1. symbols - Trading symbols metadata
2. candles_1h - Hourly OHLCV data
3. live_prices - Current market prices

See schema.sql for complete table definitions.

Development

Available Scripts

• npm start - Start production server
• npm run dev - Start development server with auto-reload
• npm test - Run tests
• npm run test:watch - Run tests in watch mode
• npm run lint - Run ESLint
• npm run lint:fix - Fix ESLint issues

Code Style

This project uses ESLint for code linting. Run npm run lint to check for issues and npm run lint:fix to automatically fix them.

Testing

Tests are written using Jest. Add your test files in the tests/ directory.

Deployment

1. Set NODE_ENV=production in your environment
2. Run npm start to start the production server
3. Consider using a process manager like PM2 for production deployments

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new features
5. Ensure all tests pass
6. Submit a pull request

License

ISC License - see LICENSE file for details.

Support

For support or questions, please contact the development team.