# Market Data Service

A high-performance financial data API that provides comprehensive market data for various financial instruments including cryptocurrencies, stocks, forex, and commodities through both RESTful endpoints and real-time WebSocket connections.

## MT5 Integration

The service includes an MT5 Expert Advisor (EA) that automatically sends historical candle data and live tick prices to the API.

### EA Features
- Automatic symbol registration in the database
- Bulk upload of last 1000 candles per symbol
- Real-time price streaming on tick events
- Error handling with retry logic
- Configurable API endpoint and authentication

### Setup Instructions
1. Copy `MT5/Experts/MarketDataSender.mq5` to your MT5 Experts directory
2. Configure EA settings:
   - `ApiBaseUrl`: Your API endpoint (e.g., `http://your-server:3000`)
   - `ApiKey`: Optional authentication key
   - `HistoricalCandleCount`: Number of historical candles to send (default: 1000)
   - `HistoricalTimeframe`: Timeframe for historical data (default: H1)

3. Attach the EA to any MT5 chart
4. The EA will:
   - Register all available symbols with the API
   - Send historical candles on initialization
   - Stream live prices every second

### Notes
- Symbols are automatically created if they don't exist
- Exchange is derived from symbol name (format: `EXCHANGE_SYMBOL`)
- Default instrument type is forex (customize in EA code if needed)

## Features  
✅ **MT5 Integration**  
- Automatic retry logic (3 attempts with exponential backoff)  
- Precision-preserving data transmission  
- Symbol auto-registration  

✅ **Data Integrity**  
- Unique constraint on candle timestamps per symbol  
- 15-decimal precision enforcement  
- Strict schema validation  

✅ **Reliability**  
- Database transaction safety  
- Error recovery mechanisms  
- Comprehensive test coverage  

- **Multi-Asset Support**: Handles cryptocurrencies, stocks, forex, and commodities
- **Real-time Data**: Live price feeds with bid/ask spreads
- **Historical Data**: OHLCV candle data with flexible timeframes
- **RESTful API**: Well-structured endpoints for all operations
- **Data Validation**: Comprehensive input validation using Joi
- **Error Handling**: Robust error handling with detailed responses
- **Security**: Helmet.js for security headers, CORS support
- **Logging**: Winston-based logging with multiple transports
- **Database**: PostgreSQL with Sequelize ORM
- **Scalable Architecture**: Modular design with controllers, routes, and middleware

## Tech Stack

- **Backend**: Node.js, Express.js
- **Database**: PostgreSQL
- **ORM**: Sequelize
- **Validation**: Joi
- **Security**: Helmet, CORS
- **Logging**: Winston, Morgan
- **Testing**: Jest, Supertest
- **Development**: Nodemon, ESLint, Sequelize CLI

## Project Structure

```
market-data-service/
├── src/
│   ├── config/
│   │   └── database.js          # Database configuration
│   ├── controllers/
│   │   ├── symbolController.js      # Symbol CRUD operations
│   │   ├── candleController.js      # Candle data operations
│   │   └── livePriceController.js   # Live price operations
│   ├── middleware/
│   │   ├── errorHandler.js          # Global error handling
│   │   └── validation.js            # Request validation
│   ├── models/
│   │   ├── Symbol.js                # Symbol model
│   │   ├── Candle1h.js              # 1-hour candle model
│   │   ├── LivePrice.js             # Live price model
│   │   └── index.js                 # Model associations
│   ├── routes/
│   │   ├── symbols.js               # Symbol routes
│   │   ├── candles.js               # Candle routes
│   │   └── livePrices.js            # Live price routes
│   ├── utils/
│   │   └── logger.js                # Logging utility
│   ├── app.js                       # Express app configuration
│   └── server.js                    # Server startup
├── tests/                           # Test files
├── schema.sql                       # Database schema
├── .env                             # Environment variables
├── .gitignore                       # Git ignore rules
├── package.json                     # Dependencies and scripts
└── README.md                        # This file
```

## Technical Specifications  

**Database Constraints**  
```sql
ALTER TABLE candles_1h 
ADD CONSTRAINT unique_symbol_open_time 
UNIQUE (symbol_id, open_time);
```

**Precision Requirements**  
```javascript
// All numeric fields require 15 decimal precision
Joi.number().precision(15)
```

## Installation  

1. **Clone the repository**
   ```bash
   git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git
   cd market-data-service
   ```

2. **Install dependencies**
   ```bash
   npm install
   ```

3. **Set up environment variables**
   ```bash
   cp .env.example .env
   ```
   Edit `.env` with your database credentials and other configuration.

4. **Database setup**
   ```bash
   # Create PostgreSQL database
   createdb market_data

   # Run migrations
   npx sequelize-cli db:migrate
   ```

5. **Start the development server**
   ```bash
   npm run dev
   ```

The server will start on `http://localhost:3000`

## Environment Variables

Create a `.env` file in the root directory with the following variables:

```env
# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=market_data
DB_USER=your_username
DB_PASSWORD=your_password

# Server Configuration
PORT=3000
NODE_ENV=development

# JWT Configuration (if needed for authentication)
JWT_SECRET=your_jwt_secret_key

# API Keys (if needed for external services)
# BINANCE_API_KEY=your_api_key
# BINANCE_API_SECRET=your_api_secret

# CORS Configuration
CORS_ORIGIN=*
```

## API Endpoints

### Health Check
- `GET /health` - Check service health

### Symbols
- `GET /api/symbols` - Get all symbols (with filtering)
- `GET /api/symbols/search` - Search symbols by name
- `GET /api/symbols/:id` - Get symbol by ID
- `POST /api/symbols` - Create new symbol
- `PUT /api/symbols/:id` - Update symbol
- `DELETE /api/symbols/:id` - Delete symbol (soft delete)

### Candles
- `GET /api/candles` - Get candles with filtering
- `GET /api/candles/ohlc` - Get OHLC data
- `GET /api/candles/:symbolId/latest` - Get latest candle for symbol
- `POST /api/candles` - Create new candle
- `POST /api/candles/bulk` - Bulk create candles

### Live Prices
- `GET /api/live-prices` - Get all live prices
- `GET /api/live-prices/exchange/:exchange` - Get live prices by exchange
- `GET /api/live-prices/type/:type` - Get live prices by instrument type
- `GET /api/live-prices/:symbolId` - Get live price for symbol
- `POST /api/live-prices` - Create/update live price
- `POST /api/live-prices/bulk` - Bulk update live prices
- `DELETE /api/live-prices/:symbolId` - Delete live price

## Database Management

The database schema is managed through Sequelize migrations and models:

- Models are defined in `src/models/`
- Migrations are stored in `migrations/`
- Associations are configured in `src/models/index.js`

Key entities:
- **Symbol**: Financial instrument metadata
- **Candle1h**: Hourly OHLCV data
- **LivePrice**: Real-time market prices

To update the database schema:
1. Create a new migration: `npx sequelize-cli migration:generate --name description`
2. Implement schema changes in the migration file
3. Run migrations: `npx sequelize-cli db:migrate`

## Development

### Available Scripts

- `npm start` - Start production server
- `npm run dev` - Start development server with auto-reload
- `npm test` - Run Jest test suite
- `npm run test:watch` - Run tests in watch mode
- `npm run migrate` - Run database migrations
- `npm run migrate:undo` - Revert last migration
- `npm run lint` - Run ESLint
- `npm run lint:fix` - Fix ESLint issues

### Code Style

This project uses ESLint for code linting. Run `npm run lint` to check for issues and `npm run lint:fix` to automatically fix them.

### Testing

The project uses Jest with Supertest for endpoint testing. Key features:

- Integration tests with database cleanup
- Test environment database configuration
- Automatic test isolation with `{ force: true }` sync
- Comprehensive controller tests including livePriceController

Run tests:
```bash
npm test         # Run full test suite
npm run test:watch  # Run in watch mode
```

### Database Migrations

We use Sequelize CLI for database migrations:

```bash
# Create new migration
npx sequelize-cli migration:generate --name your-migration-name

# Run pending migrations
npx sequelize-cli db:migrate

# Revert last migration
npx sequelize-cli db:migrate:undo
```

## Deployment

1. Set `NODE_ENV=production` in your environment
2. Run `npm start` to start the production server
3. Consider using a process manager like PM2 for production deployments

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new features
5. Ensure all tests pass
6. Submit a pull request

## License

ISC License - see LICENSE file for details.

## Support

For support or questions, please contact the development team.