market-data-service

Market Data Service

A high-performance, enterprise-grade financial data API that provides comprehensive market information for 2013+ financial instruments through both RESTful endpoints and real-time WebSocket connections.

Overview

Market Data Service is a robust Node.js-based platform designed for financial applications requiring reliable, low-latency market data. The service delivers historical candlestick data with H1 (1-hour) timeframe granularity and enables real-time price streaming triggered by database updates.

Key Features

🚀 Core Functionality

• Historical Data API: RESTful endpoints for H1 candlestick data retrieval
• Real-time Streaming: WebSocket service for live price updates
• Multi-Instrument Support: Handles 2013+ financial symbols across various markets
• Database Flexibility: Support for MySQL, PostgreSQL, SQLite, and Redis

🔒 Security & Performance

• Rate Limiting: Configurable request throttling with express-rate-limit
• CORS Protection: Cross-origin resource sharing controls
• Security Headers: Helmet.js integration for secure HTTP headers
• Data Compression: Response compression for optimized bandwidth

📊 Data Management

• Multi-Database Support: MySQL, PostgreSQL, SQLite, and Redis integration
• ORM Integration: Sequelize for database operations
• Caching Layer: Redis for high-performance data caching
• Data Validation: Comprehensive input validation and sanitization

🔍 Monitoring & Logging

• Winston Logging: Structured logging with multiple transports
• Error Tracking: Comprehensive error handling and reporting
• Request Logging: Morgan HTTP request logging
• Log Rotation: Automated log management

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   REST API      │    │   WebSocket     │    │   Database      │
│   Endpoints     │◄──►│   Service       │◄──►│   Layer         │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Request       │    │   Real-time     │    │   Data          │
│   Validation    │    │   Updates       │    │   Persistence   │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Technology Stack

• Runtime: Node.js (>=16.0.0)
• Framework: Express.js
• WebSocket: Socket.IO
• Databases: MySQL, PostgreSQL, SQLite, Redis
• ORM: Sequelize
• Security: Helmet, CORS, Rate Limiting
• Logging: Winston, Morgan
• Testing: Jest, Supertest, Mocha, Chai

Installation

Prerequisites

• Node.js (>=16.0.0)
• MySQL/PostgreSQL/SQLite database
• Redis (optional, for caching)

Setup

1. Clone the repository

   git clone <repository-url>
   cd market-data-service
   

2. Install dependencies

   npm install
   

3. Environment Configuration

   cp .env.example .env
   

Configure your .env file:

   # Database Configuration
   DB_HOST=localhost
   DB_PORT=3306
   DB_NAME=market_data
   DB_USER=your_username
   DB_PASSWORD=your_password

   # Redis Configuration (Optional)
   REDIS_HOST=localhost
   REDIS_PORT=6379

   # Server Configuration
   PORT=3000
   NODE_ENV=development

   # Security
   JWT_SECRET=your_jwt_secret
   API_RATE_LIMIT_WINDOW=15
   API_RATE_LIMIT_MAX=100

1. Database Setup

   # Initialize database
   npm run db:init
   
   # Or run migrations
   npm run db:migrate
   

Usage

Start the Server

Development Mode

npm run dev

Production Mode

npm start

The server will start on http://localhost:3000

API Endpoints

Historical Data

GET /api/prices/history/:symbol?timeframe=H1&limit=100
GET /api/symbols
GET /api/prices/:symbol/latest

Real-time WebSocket Events

// Connect to WebSocket
const socket = io('http://localhost:3000');

// Subscribe to price updates
socket.emit('subscribe', { symbol: 'AAPL' });

// Listen for price updates
socket.on('price_update', (data) => {
  console.log('Price update:', data);
});

// Subscribe to candlestick updates
socket.emit('subscribe_candles', { symbol: 'AAPL', timeframe: 'H1' });

// Listen for candlestick updates
socket.on('candlestick_update', (data) => {
  console.log('Candlestick update:', data);
});

Configuration

Database Configuration

The service supports multiple database configurations through Sequelize ORM:

// config/database.js
module.exports = {
  development: {
    dialect: 'mysql', // or 'postgres', 'sqlite'
    host: process.env.DB_HOST,
    port: process.env.DB_PORT,
    database: process.env.DB_NAME,
    username: process.env.DB_USER,
    password: process.env.DB_PASSWORD
  }
};

Rate Limiting

Configure API rate limiting in your environment:

API_RATE_LIMIT_WINDOW=15  # Time window in minutes
API_RATE_LIMIT_MAX=100    # Max requests per window

Testing

# Run all tests
npm test

# Run tests in watch mode
npm run test:watch

# Run specific test file
npm test candleValidation.test.js

API Documentation

REST API Endpoints

Get Historical Prices

GET /api/prices/history/:symbol

Parameters:

• symbol (path): Financial instrument symbol
• timeframe (query, optional): Timeframe (H1, H4, D1) - defaults to H1
• limit (query, optional): Number of records - defaults to 100
• from (query, optional): Start date (ISO 8601)
• to (query, optional): End date (ISO 8601)

Response:

{
  "symbol": "AAPL",
  "timeframe": "H1",
  "data": [
    {
      "timestamp": "2023-10-15T10:00:00Z",
      "open": 150.25,
      "high": 151.80,
      "low": 149.90,
      "close": 151.20,
      "volume": 1250000
    }
  ]
}

Get Available Symbols

GET /api/symbols

Response:

{
  "symbols": [
    {
      "symbol": "AAPL",
      "name": "Apple Inc.",
      "market": "NASDAQ",
      "sector": "Technology"
    }
  ]
}

Development

Project Structure

├── config/           # Database and configuration files
├── logs/            # Application logs
├── middleware/      # Express middleware
├── models/          # Database models
├── public/          # Static files
├── routes/          # API routes
├── services/        # Business logic services
├── src/            # Main application entry point
├── tests/          # Test files
└── utils/          # Utility functions

Adding New Features

1. Create Service: Add business logic in services/
2. Add Routes: Define API endpoints in routes/
3. Update Models: Modify database models in models/
4. Add Tests: Create tests in tests/
5. Update Documentation: Document new endpoints

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Support

For support and questions:

• Create an issue in the repository
• Contact the development team

Changelog

Version 1.0.0

• Initial release
• Historical H1 candlestick data API
• Real-time WebSocket price streaming
• Multi-database support
• Comprehensive logging and monitoring