# Market Data Service

A high-performance financial data API that provides comprehensive market data for various financial instruments including cryptocurrencies, stocks, forex, and commodities through both RESTful endpoints and real-time WebSocket connections.

## MT5 Integration

The service includes an MT5 Expert Advisor (EA) that automatically sends historical candle data and live tick prices to the API.

### EA Features
- Automatic symbol registration in the database
- Bulk upload of last 1000 candles per symbol
- Real-time price streaming on tick events
- Error handling with retry logic
- Configurable API endpoint and authentication

### Setup Instructions
1. Copy `MT5/Experts/MarketDataSender.mq5` to your MT5 Experts directory
2. Configure EA settings:
   - `ApiBaseUrl`: Your API endpoint (e.g., `http://your-server:3000`)
   - `ApiKey`: Optional authentication key
   - `HistoricalCandleCount`: Number of historical candles to send (default: 1000)
   - `HistoricalTimeframe`: Timeframe for historical data (default: H1)

3. Attach the EA to any MT5 chart
4. The EA will:
   - Register all available symbols with the API
   - Send historical candles on initialization
   - Stream live prices every second

### Notes
- Symbols are automatically created if they don't exist
- Exchange is derived from symbol name (format: `EXCHANGE_SYMBOL`)
- Default instrument type is forex (customize in EA code if needed)

## Features  
✅ **MT5 Integration**  
- Automatic retry logic (3 attempts with exponential backoff)  
- Precision-preserving data transmission  
- Symbol auto-registration  

✅ **Data Integrity**  
- Unique constraint on candle timestamps per symbol  
- 15-decimal precision enforcement  
- Strict schema validation  

✅ **Reliability**  
- Database transaction safety  
- Error recovery mechanisms  
- Comprehensive test coverage  

- **Multi-Asset Support**: Handles cryptocurrencies, stocks, forex, commodities, and indices
- **Real-time Data**: Live price feeds with bid/ask spreads
- **Historical Data**: OHLCV candle data with flexible timeframes
- **RESTful API**: Well-structured endpoints for all operations
- **Data Validation**: Comprehensive input validation using Joi
- **Error Handling**: Robust error handling with detailed responses
- **Security**: Helmet.js for security headers, CORS support
- **Logging**: Winston-based logging with multiple transports
- **Database**: PostgreSQL with Sequelize ORM
- **Scalable Architecture**: Modular design with controllers, routes, and middleware

## Tech Stack

- **Backend**: Node.js, Express.js
- **Database**: PostgreSQL
- **ORM**: Sequelize
- **Validation**: Joi
- **Security**: Helmet, CORS
- **Logging**: Winston, Morgan
- **Testing**: Jest, Supertest
- **Development**: Nodemon, ESLint, Sequelize CLI

## Project Structure

```
market-data-service/
├── src/
│   ├── config/
│   │   └── database.js          # Database configuration
│   ├── controllers/
│   │   ├── symbolController.js      # Symbol CRUD operations
│   │   ├── candleController.js      # Candle data operations
│   │   └── livePriceController.js   # Live price operations
│   ├── middleware/
│   │   ├── errorHandler.js          # Global error handling
│   │   └── validation.js            # Request validation
│   ├── models/
│   │   ├── Symbol.js                # Symbol model
│   │   ├── Candle.js                # Multi-timeframe candle model
│   │   ├── LivePrice.js             # Live price model
│   │   └── index.js                 # Model associations
│   ├── routes/
│   │   ├── symbols.js               # Symbol routes
│   │   ├── candles.js               # Candle routes
│   │   └── livePrices.js            # Live price routes
│   ├── utils/
│   │   └── logger.js                # Logging utility
│   ├── app.js                       # Express app configuration
│   └── server.js                    # Server startup
├── tests/                           # Test files
├── schema.sql                       # Database schema
├── .env                             # Environment variables
├── .gitignore                       # Git ignore rules
├── package.json                     # Dependencies and scripts
└── README.md                        # This file
```

## Technical Specifications  

**Database Constraints**  
```sql
ALTER TABLE candles_1h 
ADD CONSTRAINT unique_symbol_open_time 
UNIQUE (symbol_id, open_time);
```

**Precision Requirements**  
```javascript
// All numeric fields require 15 decimal precision
Joi.number().precision(15)
```

## 🚀 Quick Start

### Local Development

```bash
# Clone repository
git clone <your-repo-url>
cd market-data-service

# Start Docker services
docker-compose up -d

# Access API at http://market-data.local
```

📖 **For detailed local development setup, see [docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)**

### Production Deployment

```bash
# Set up environment variables
cp .env.example .env
# Edit .env with your values

# Start production services
docker-compose -f docker-compose.prod.yml up -d
```

📖 **For complete production deployment guide, see [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**

---

## 📚 Documentation

- **[Local Development Setup](./docs/LOCAL_DEV_SETUP.md)** - Complete local development guide
- **[Production Deployment](./docs/DEPLOYMENT.md)** - Production deployment with SSL certificates
- **[Docker Guide](./DOCKER.md)** - Docker containerization details
- **[Configuration Guide](./docs/CONFIGURATION.md)** - Environment variables and configuration
- **[API Contract](./docs/API_CONTRACT.md)** - API endpoint specifications
- **[MT5 Operation](./docs/MT5_OPERATION.md)** - MT5 Expert Advisor guide

---

## 🆕 FRESH SERVER DEPLOYMENT (Legacy - See DEPLOYMENT.md)

### Step 1: Server Preparation
```bash
# Update system packages
sudo apt update && sudo apt upgrade -y

# Install required tools
sudo apt install -y curl wget git htop nano ufw
```

### Step 2: Install Node.js (v18.x LTS)
```bash
# Using NodeSource repository
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs

# Verify installation
node --version  # Should show v18.x.x
npm --version   # Should show latest version
```

### Step 3: Install PostgreSQL
```bash
# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib

# Start and enable PostgreSQL
sudo systemctl enable postgresql
sudo systemctl start postgresql

# Verify installation
psql --version  # Should show 15.x or higher
```

### Step 4: Deploy Application
```bash
# Create application directory
sudo mkdir -p /root/market-data-service
sudo chown $USER:$USER /root/market-data-service
cd /root/market-data-service

# Clone repository (replace with your actual repo URL)
git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git .
# OR copy your project files to this directory

# Install dependencies
npm install --production

# Set up environment variables
cat > .env << EOF
DB_TYPE=postgres
DB_HOST=localhost
DB_PORT=5432
DB_NAME=financial_data
DB_USER=postgres
DB_PASSWORD=your_secure_password_here

PORT=3001
NODE_ENV=production

JWT_SECRET=your_secure_jwt_secret_key_here
CORS_ORIGIN=*

LOG_LEVEL=info
EOF
```

### Step 5: Database Setup
```bash
# Switch to postgres user
sudo -u postgres psql

# In PostgreSQL shell, run:
CREATE DATABASE financial_data;
ALTER USER postgres PASSWORD 'your_secure_password_here';
GRANT ALL PRIVILEGES ON DATABASE financial_data TO postgres;
\q

# Exit and run migrations
npx sequelize-cli db:migrate

# Verify tables were created
sudo -u postgres psql -d financial_data -c "\dt"
```

### Step 6: Deploy with Docker (Recommended)

> **⚠️ Outdated Instructions:** This section contains legacy deployment steps. For current production deployment with containerized SSL certificates, see **[docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**

```bash
# Navigate to project directory
cd /root/market-data-service

# Copy environment template
cp .env.example .env

# Edit .env file with production values
nano .env

# Build and start production containers
docker-compose -f docker-compose.prod.yml up -d --build
```

📖 **For detailed Docker deployment instructions, see [DOCKER.md](./DOCKER.md)**  
📖 **For production deployment with SSL, see [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**

---

## 💻 Local Development

> **Quick Overview:** For complete setup instructions, troubleshooting, and MT5 EA configuration, see **[docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)**.

### Prerequisites
- **Docker** and **Docker Compose** installed
- **Git** installed
- **Administrator access** (script will prompt for password automatically on macOS)

### Quick Start

#### 1. Clone the Repository
```bash
# Clone from GitHub
git clone https://git.mqldevelopment.com/muhammad.uzair/market-data-service.git
cd market-data-service
```

#### 2. Setup Local Domain (One-time, Required for MT5 EA)
```bash
# Run automated setup script (fully automated on macOS - will prompt for password)
# On Linux, you may need to run with: sudo bash scripts/setup-local-dev.sh
bash scripts/setup-local-dev.sh
```

**Note:** On macOS, the script automatically prompts for your administrator password. On Linux, you may need to run with `sudo`.

**Alternative (Manual):**
```bash
# Add market-data.local to hosts file manually
echo "127.0.0.1 market-data.local" | sudo tee -a /etc/hosts
```

#### 3. Start Docker Services
```bash
# Start all services (database, API, Nginx)
docker-compose up -d

# Verify services are running
docker-compose ps
```

#### 4. Verify Setup
```bash
# Test health endpoint
curl http://market-data.local/health

# Test API endpoints
curl http://market-data.local/api/symbols
curl http://market-data.local/api/health
```

**✅ Setup Complete!** Your API is now accessible at `http://market-data.local`

### Why market-data.local?

MT5 Expert Advisors require valid domain URLs and **do not allow** `localhost` or `127.0.0.1`. The `market-data.local` domain resolves to your local Docker setup, enabling MT5 EA connectivity.

### MT5 EA Configuration

1. **Add URL to MT5 Allowed List:**
   - Open MT5 → Tools → Options → Expert Advisors
   - Click "Allowed URLs" tab → Add → Enter: `http://market-data.local`

2. **Configure EA:**
   - Attach `MarketDataSender.mq5` to any chart
   - Set `ApiBaseUrl = "http://market-data.local"`
   - Other settings as needed

3. **Verify Connection:**
   - Check MT5 Experts tab for EA logs
   - Look for: `✅ Symbols initialized`

### Development Workflow

**View Logs:**
```bash
# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api
docker-compose logs -f nginx
```

**Code Changes:**
- Code in `src/` is automatically reloaded (no container restart needed)
- View API logs to see changes: `docker-compose logs -f api`

**Stop Services:**
```bash
docker-compose down
```

**Restart Services:**
```bash
docker-compose restart
```

### Need More Help?

📖 **For comprehensive setup instructions, troubleshooting, and advanced configuration, see:**
- **[docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)** - Complete local development guide with detailed steps, troubleshooting, and MT5 EA configuration

### Alternative: Non-Docker Local Setup

If you prefer to run without Docker (not recommended for MT5 EA):

<details>
<summary>Click to expand non-Docker setup instructions</summary>

#### Prerequisites
- **Node.js**: v18.x LTS or higher
- **PostgreSQL**: v13.x or higher

#### Setup Steps

1. **Install Dependencies:**
```bash
npm install
```

2. **Set Up Database:**
```bash
createdb financial_data

# Create .env file
cat > .env << EOF
DB_TYPE=postgres
DB_HOST=localhost
DB_PORT=5432
DB_NAME=financial_data
DB_USER=your_local_username
DB_PASSWORD=your_local_password
PORT=3001
NODE_ENV=development
JWT_SECRET=your_development_jwt_secret_key
CORS_ORIGIN=http://localhost:3000
LOG_LEVEL=debug
EOF
```

3. **Run Migrations:**
```bash
npx sequelize-cli db:migrate
```

4. **Start Server:**
```bash
npm run dev
```

5. **Test:**
```bash
curl http://localhost:3001/health
```

**Note:** This setup won't work with MT5 EA (requires domain URL). Use Docker setup for MT5 integration.

</details>


## 🔄 UPDATING EXISTING DEPLOYMENT

### Quick Update (When Code Changes on GitHub)

```bash
# Navigate to project directory
cd /root/market-data-service

# Backup current deployment (optional but recommended)
cp .env .env.backup

# Pull latest changes
git pull origin master

# Rebuild and restart containers (migrations run automatically)
docker-compose -f docker-compose.prod.yml up -d --build

# Verify everything is working
docker-compose -f docker-compose.prod.yml ps
curl https://your-domain.com/health
```

### Full Update (Major Changes)

```bash
# Navigate to project directory
cd /root/market-data-service

# Create backup
sudo cp -r /root/market-data-service /root/market-data-service-backup-$(date +%Y%m%d_%H%M%S)

# Stop containers
docker-compose -f docker-compose.prod.yml down

# Update code
git fetch origin
git reset --hard origin/master

# Update environment if needed
nano .env

# Rebuild and start containers (migrations run automatically)
docker-compose -f docker-compose.prod.yml up -d --build

# Verify deployment
docker-compose -f docker-compose.prod.yml ps
curl https://your-domain.com/health
```

---

## 🛠️ Troubleshooting Guide

### Common Issues & Solutions

#### 1. PostgreSQL User Already Exists
**Error:** `ERROR: role "postgres" already exists`

**Solution:**
```bash
# Just set password for existing user
sudo -u postgres psql
ALTER USER postgres PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE financial_data TO postgres;
\q
```

#### 2. SSL Certificate Not Found
**Error:** `cannot load certificate "/etc/letsencrypt/live/.../fullchain.pem"`

**Solution:**
```bash
# Generate certificate first
sudo certbot --nginx -d your-domain.com

# Check if certificates exist
ls -la /etc/letsencrypt/live/your-domain.com/

# Test nginx configuration
sudo nginx -t
```

#### 3. Docker Container Issues
**Problem:** Container won't start or keeps restarting

**Solution:**
```bash
# Check container logs
docker-compose -f docker-compose.prod.yml logs api

# Check container status
docker-compose -f docker-compose.prod.yml ps

# Restart specific service
docker-compose -f docker-compose.prod.yml restart api
```

## 📊 Monitoring & Maintenance

```bash
# Check service status
docker-compose -f docker-compose.prod.yml ps
sudo systemctl status nginx

# View logs
docker-compose -f docker-compose.prod.yml logs -f api
docker-compose -f docker-compose.prod.yml logs -f db
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

```

### SSL Certificate Auto-Renewal

> **Note:** SSL certificates are now automatically managed via Docker containers. See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for details.

---

## 🔧 Management Commands

### Docker Management
```bash
# Check container status
docker-compose -f docker-compose.prod.yml ps

# View logs
docker-compose -f docker-compose.prod.yml logs -f api
docker-compose -f docker-compose.prod.yml logs -f db

# Restart services
docker-compose -f docker-compose.prod.yml restart

# Stop all services
docker-compose -f docker-compose.prod.yml down

# Start services
docker-compose -f docker-compose.prod.yml up -d

# Rebuild and restart
docker-compose -f docker-compose.prod.yml up -d --build
```

### Nginx Management
```bash
sudo systemctl status nginx      # Check status
sudo systemctl restart nginx     # Restart nginx
sudo nginx -t                    # Test configuration
sudo tail -f /var/log/nginx/error.log  # View error logs
```

### Database Management
```bash
# Connect to database
sudo -u postgres psql -d financial_data

# View tables
\dt

# View table structure
\d table_name

# Backup database
pg_dump financial_data > backup.sql

# Restore database
psql financial_data < backup.sql
```

---

## Environment Variables

See **[docs/CONFIGURATION.md](./docs/CONFIGURATION.md)** for complete environment variable documentation.

**Quick Reference:**
- `DB_PASSWORD` - Database password (required)
- `JWT_SECRET` - JWT secret key (required)
- `DOMAIN_NAME` - Domain for SSL certificates (production)
- `SSL_EMAIL` - Email for Let's Encrypt (production)
- `CORS_ORIGIN` - Allowed CORS origins

## API Endpoints

See **[docs/API_CONTRACT.md](./docs/API_CONTRACT.md)** for complete API documentation.

**Quick Reference:**
- `GET /health` - Health check
- `GET /api/symbols` - Get all symbols
- `GET /api/candles` - Get candle data
- `POST /api/candles/bulk` - Bulk create candles
- `GET /api/live-prices` - Get live prices
- `POST /api/live-prices/bulk` - Bulk update live prices

## Development

### Available Scripts

- `npm start` - Start production server
- `npm run dev` - Start development server with auto-reload
- `npm test` - Run Jest test suite
- `npm run test:watch` - Run tests in watch mode
- `npm run migrate` - Run database migrations
- `npm run migrate:undo` - Revert last migration
- `npm run lint` - Run ESLint
- `npm run lint:fix` - Fix ESLint issues


### Database Migrations

We use Sequelize CLI for database migrations:

```bash
# Create new migration
npx sequelize-cli migration:generate --name your-migration-name

# Run pending migrations
npx sequelize-cli db:migrate

# Revert last migration
npx sequelize-cli db:migrate:undo
```

## Deployment

The project is fully containerized using Docker with automatic SSL certificate management.

- **Development**: See [docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)
- **Production**: See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)
- **Docker Details**: See [DOCKER.md](./DOCKER.md)