# Market Data Service

A high-performance, enterprise-grade financial data API that provides comprehensive market information for 2013+ financial instruments through both RESTful endpoints and real-time WebSocket connections.

## Overview

Market Data Service is a robust Node.js-based platform designed for financial applications requiring reliable, low-latency market data. The service delivers historical candlestick data with H1 (1-hour) timeframe granularity and enables real-time price streaming triggered by database updates.

## Key Features

### 🚀 **Core Functionality**
- **Historical Data API**: RESTful endpoints for H1 candlestick data retrieval
- **Real-time Streaming**: WebSocket service for live price updates
- **Multi-Instrument Support**: Handles 2013+ financial symbols across various markets
- **Database Flexibility**: Support for MySQL, PostgreSQL, SQLite, and Redis

### 🔒 **Security & Performance**
- **Rate Limiting**: Configurable request throttling with express-rate-limit
- **CORS Protection**: Cross-origin resource sharing controls
- **Security Headers**: Helmet.js integration for secure HTTP headers
- **Data Compression**: Response compression for optimized bandwidth

### 📊 **Data Management**
- **Multi-Database Support**: MySQL, PostgreSQL, SQLite, and Redis integration
- **ORM Integration**: Sequelize for database operations
- **Caching Layer**: Redis for high-performance data caching
- **Data Validation**: Comprehensive input validation and sanitization

### 🔍 **Monitoring & Logging**
- **Winston Logging**: Structured logging with multiple transports
- **Error Tracking**: Comprehensive error handling and reporting
- **Request Logging**: Morgan HTTP request logging
- **Log Rotation**: Automated log management

## Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   REST API      │    │   WebSocket     │    │   Database      │
│   Endpoints     │◄──►│   Service       │◄──►│   Layer         │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Request       │    │   Real-time     │    │   Data          │
│   Validation    │    │   Updates       │    │   Persistence   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
```

## Technology Stack

- **Runtime**: Node.js (>=16.0.0)
- **Framework**: Express.js
- **WebSocket**: Socket.IO
- **Databases**: MySQL, PostgreSQL, SQLite, Redis
- **ORM**: Sequelize
- **Security**: Helmet, CORS, Rate Limiting
- **Logging**: Winston, Morgan
- **Testing**: Jest, Supertest, Mocha, Chai

## Installation

### Prerequisites
- Node.js (>=16.0.0)
- MySQL/PostgreSQL/SQLite database
- Redis (optional, for caching)

### Setup

1. **Clone the repository**
   ```bash
   git clone <repository-url>
   cd market-data-service
   ```

2. **Install dependencies**
   ```bash
   npm install
   ```

3. **Environment Configuration**
   ```bash
   cp .env.example .env
   ```

   Configure your `.env` file:
   ```env
   # Database Configuration
   DB_HOST=localhost
   DB_PORT=3306
   DB_NAME=market_data
   DB_USER=your_username
   DB_PASSWORD=your_password

   # Redis Configuration (Optional)
   REDIS_HOST=localhost
   REDIS_PORT=6379

   # Server Configuration
   PORT=3000
   NODE_ENV=development

   # Security
   JWT_SECRET=your_jwt_secret
   API_RATE_LIMIT_WINDOW=15
   API_RATE_LIMIT_MAX=100
   ```

4. **Database Setup**
   ```bash
   # Initialize database
   npm run db:init

   # Or run migrations
   npm run db:migrate
   ```

## Usage

### Start the Server

**Development Mode**
```bash
npm run dev
```

**Production Mode**
```bash
npm start
```

The server will start on `http://localhost:3000`

### API Endpoints

#### Historical Data
```http
GET /api/prices/history/:symbol?timeframe=H1&limit=100
GET /api/symbols
GET /api/prices/:symbol/latest
```

#### Real-time WebSocket Events
```javascript
// Connect to WebSocket
const socket = io('http://localhost:3000');

// Subscribe to price updates
socket.emit('subscribe', { symbol: 'AAPL' });

// Listen for price updates
socket.on('price_update', (data) => {
  console.log('Price update:', data);
});

// Subscribe to candlestick updates
socket.emit('subscribe_candles', { symbol: 'AAPL', timeframe: 'H1' });

// Listen for candlestick updates
socket.on('candlestick_update', (data) => {
  console.log('Candlestick update:', data);
});
```

## Configuration

### Database Configuration

The service supports multiple database configurations through Sequelize ORM:

```javascript
// config/database.js
module.exports = {
  development: {
    dialect: 'mysql', // or 'postgres', 'sqlite'
    host: process.env.DB_HOST,
    port: process.env.DB_PORT,
    database: process.env.DB_NAME,
    username: process.env.DB_USER,
    password: process.env.DB_PASSWORD
  }
};
```

### Rate Limiting

Configure API rate limiting in your environment:

```env
API_RATE_LIMIT_WINDOW=15  # Time window in minutes
API_RATE_LIMIT_MAX=100    # Max requests per window
```

## Testing

```bash
# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run specific test file
npm test candleValidation.test.js
```

## API Documentation

### REST API Endpoints

#### Get Historical Prices
```http
GET /api/prices/history/:symbol
```

**Parameters:**
- `symbol` (path): Financial instrument symbol
- `timeframe` (query, optional): Timeframe (H1, H4, D1) - defaults to H1
- `limit` (query, optional): Number of records - defaults to 100
- `from` (query, optional): Start date (ISO 8601)
- `to` (query, optional): End date (ISO 8601)

**Response:**
```json
{
  "symbol": "AAPL",
  "timeframe": "H1",
  "data": [
    {
      "timestamp": "2023-10-15T10:00:00Z",
      "open": 150.25,
      "high": 151.80,
      "low": 149.90,
      "close": 151.20,
      "volume": 1250000
    }
  ]
}
```

#### Get Available Symbols
```http
GET /api/symbols
```

**Response:**
```json
{
  "symbols": [
    {
      "symbol": "AAPL",
      "name": "Apple Inc.",
      "market": "NASDAQ",
      "sector": "Technology"
    }
  ]
}
```

## Development

### Project Structure
```
├── config/           # Database and configuration files
├── logs/            # Application logs
├── middleware/      # Express middleware
├── models/          # Database models
├── public/          # Static files
├── routes/          # API routes
├── services/        # Business logic services
├── src/            # Main application entry point
├── tests/          # Test files
└── utils/          # Utility functions
```

### Adding New Features

1. **Create Service**: Add business logic in `services/`
2. **Add Routes**: Define API endpoints in `routes/`
3. **Update Models**: Modify database models in `models/`
4. **Add Tests**: Create tests in `tests/`
5. **Update Documentation**: Document new endpoints

## Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## License

This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## Support

For support and questions:
- Create an issue in the repository
- Contact the development team

## Changelog

### Version 1.0.0
- Initial release
- Historical H1 candlestick data API
- Real-time WebSocket price streaming
- Multi-database support
- Comprehensive logging and monitoring