Browse Source

feat: Add production deployment with automated SSL certificate management

This commit introduces a complete production deployment setup with fully
automated SSL certificate management using Let's Encrypt and Certbot.

## Key Features

### SSL Certificate Management
- Automatic SSL certificate obtainment via Certbot
- Automatic certificate renewal every 12 hours
- Automatic nginx reload after certificate renewal
- Support for Let's Encrypt staging server for testing
- Fully containerized certificate management

### Infrastructure Improvements
- Updated docker-compose.prod.yml with Certbot services
- Dynamic nginx configuration that adapts to certificate availability
- Shared Docker volumes for certificate persistence
- Health checks for all services

### New Scripts
- docker/nginx-entrypoint.sh: Dynamic nginx config processing with SSL support
- docker/certbot-init.sh: SSL certificate initialization wrapper
- docker/init-ssl.sh: Initial SSL certificate obtainment script
- docker/renew-ssl.sh: SSL certificate renewal script

### Configuration Updates
- nginx/nginx.prod.conf: Updated for Let's Encrypt certificate paths
- .env.example: Added production environment variables (DOMAIN_NAME, SSL_EMAIL)
- docker-compose.prod.yml: Integrated Certbot services and volumes

### Documentation
- docs/DEPLOYMENT.md: Complete production deployment guide
- docs/CONFIGURATION.md: Environment variables and configuration guide
- docs/README.md: Documentation index
- Updated DOCKER.md and README.md with deployment references

### Security
- All sensitive files (.env) remain in .gitignore
- No hardcoded secrets or passwords
- Secure certificate storage in Docker volumes

## Deployment Process

1. Set environment variables in .env file
2. Start services: docker-compose -f docker-compose.prod.yml up -d db api nginx
3. Obtain SSL certificate: docker-compose run --rm certbot-init
4. Services automatically handle certificate renewal

## Testing

- HTTP endpoint working: http://market-price.insightbull.io/health
- SSL certificates obtained and configured
- Automatic renewal configured
- All services healthy and running

This deployment has been tested and verified on production server.
Hussain Afzal 2 months ago
parent
commit
c69e943a3b
14 changed files with 1231 additions and 231 deletions
  1. 20 14
      .env.example
  2. 1 0
      .gitignore
  3. 12 17
      DOCKER.md
  4. 65 192
      README.md
  5. 52 5
      docker-compose.prod.yml
  6. 90 0
      docker/certbot-init.sh
  7. 50 0
      docker/init-ssl.sh
  8. 49 0
      docker/nginx-entrypoint.sh
  9. 25 0
      docker/renew-ssl.sh
  10. 260 0
      docs/CONFIGURATION.md
  11. 369 0
      docs/DEPLOYMENT.md
  12. 69 0
      docs/README.md
  13. 4 3
      nginx/nginx.prod.conf
  14. 165 0
      nginx/nginx.prod.conf.template

+ 20 - 14
.env.example

@@ -1,20 +1,26 @@
1
+# ============================================
2
+# Market Data Service - Environment Variables
3
+# ============================================
4
+# Copy this file to .env and update with your values
5
+# cp .env.example .env
6
+
1 7
 # Database Configuration
2
-DB_TYPE=postgres
3
-DB_HOST=localhost
4
-DB_PORT=5432
5
-DB_NAME=financial_data
6 8
 DB_USER=postgres
7 9
 DB_PASSWORD=your_secure_password_here
10
+DB_NAME=financial_data
8 11
 
9
-# Server Configuration
10
-PORT=3001
11
-NODE_ENV=development
12
-
13
-# JWT Configuration (if needed for authentication)
14
-JWT_SECRET=your_secure_jwt_secret_key_here
12
+# Application Configuration
13
+NODE_ENV=production
14
+PORT=3000
15
+CORS_ORIGIN=https://market-price.insightbull.io
16
+JWT_SECRET=your_secure_jwt_secret_here
17
+LOG_LEVEL=info
15 18
 
16
-# CORS Configuration
17
-CORS_ORIGIN=*
19
+# SSL Certificate Configuration
20
+# Domain name for SSL certificate (must match DNS)
21
+DOMAIN_NAME=market-price.insightbull.io
22
+# Email for Let's Encrypt notifications
23
+SSL_EMAIL=your-email@example.com
18 24
 
19
-# Logging
20
-LOG_LEVEL=debug
25
+# Optional: Use staging server for testing (set to 1 for testing)
26
+# SSL_STAGING=0

+ 1 - 0
.gitignore

@@ -119,3 +119,4 @@ backups/
119 119
 *.sql.gz
120 120
 *.sql
121 121
 backup_*.sql
122
+nginx/nginx.prod.no-ssl.conf

+ 12 - 17
DOCKER.md

@@ -166,23 +166,16 @@ Production environment uses optimized settings:
166 166
    DB_PASSWORD=<strong_password>
167 167
    JWT_SECRET=<strong_random_secret>
168 168
    CORS_ORIGIN=https://your-domain.com
169
+   DOMAIN_NAME=your-domain.com
170
+   SSL_EMAIL=your-email@example.com
169 171
    ```
170
-
171
-2. **SSL Certificates** (for HTTPS)
172
-   ```bash
173
-   # Option 1: Use Let's Encrypt (recommended)
174
-   # Certificates will be in /etc/letsencrypt/live/your-domain.com/
175 172
    
176
-   # Option 2: Place certificates manually
177
-   mkdir -p ssl/certs ssl/private
178
-   # Copy fullchain.pem to ssl/certs/
179
-   # Copy privkey.pem to ssl/private/
180
-   ```
173
+   See [CONFIGURATION.md](./docs/CONFIGURATION.md) for complete environment variable documentation.
181 174
 
182
-3. **Update Nginx Configuration**
183
-   - Edit `nginx/nginx.prod.conf`
184
-   - Update SSL certificate paths
185
-   - Update `server_name` if using domain names
175
+2. **SSL Certificates**
176
+   SSL certificates are automatically managed via Docker containers. No manual setup required.
177
+   
178
+   See [DEPLOYMENT.md](./docs/DEPLOYMENT.md) for production deployment with SSL certificate setup.
186 179
 
187 180
 ### Starting Production Environment
188 181
 
@@ -469,14 +462,16 @@ find $BACKUP_DIR -name "backup_*.sql.gz" -mtime +30 -delete
469 462
 ### Development
470 463
 
471 464
 Update MT5 EA settings:
472
-- `ApiBaseUrl`: `http://localhost` or `http://localhost:3000`
465
+- `ApiBaseUrl`: `http://market-data.local`
466
+- See [LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md) for local development setup
473 467
 
474 468
 ### Production
475 469
 
476 470
 Update MT5 EA settings:
477
-- `ApiBaseUrl`: `https://your-domain.com`
478
-- Ensure SSL certificate is valid
471
+- `ApiBaseUrl`: `https://market-price.insightbull.io` (or your production domain)
472
+- SSL certificates are automatically managed
479 473
 - WebSocket connections use `wss://` protocol
474
+- See [DEPLOYMENT.md](./docs/DEPLOYMENT.md) for production deployment
480 475
 
481 476
 ## Performance Tuning
482 477
 

+ 65 - 192
README.md

@@ -120,20 +120,50 @@ UNIQUE (symbol_id, open_time);
120 120
 Joi.number().precision(15)
121 121
 ```
122 122
 
123
-## 🚀 Complete Deployment Guide
123
+## 🚀 Quick Start
124 124
 
125
-## 📋 Prerequisites (All Versions)
125
+### Local Development
126 126
 
127
-### Required Software (Latest Stable Versions)
128
-- **Node.js**: v18.x LTS or higher
129
-- **PostgreSQL**: v13.x or higher
130
-- **Nginx**: v1.24.x or higher
131
-- **Git**: Latest version
132
-- **Certbot** (for SSL certificates)
127
+```bash
128
+# Clone repository
129
+git clone <your-repo-url>
130
+cd market-data-service
131
+
132
+# Start Docker services
133
+docker-compose up -d
134
+
135
+# Access API at http://market-data.local
136
+```
137
+
138
+📖 **For detailed local development setup, see [docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)**
139
+
140
+### Production Deployment
141
+
142
+```bash
143
+# Set up environment variables
144
+cp .env.example .env
145
+# Edit .env with your values
146
+
147
+# Start production services
148
+docker-compose -f docker-compose.prod.yml up -d
149
+```
150
+
151
+📖 **For complete production deployment guide, see [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**
152
+
153
+---
154
+
155
+## 📚 Documentation
156
+
157
+- **[Local Development Setup](./docs/LOCAL_DEV_SETUP.md)** - Complete local development guide
158
+- **[Production Deployment](./docs/DEPLOYMENT.md)** - Production deployment with SSL certificates
159
+- **[Docker Guide](./DOCKER.md)** - Docker containerization details
160
+- **[Configuration Guide](./docs/CONFIGURATION.md)** - Environment variables and configuration
161
+- **[API Contract](./docs/API_CONTRACT.md)** - API endpoint specifications
162
+- **[MT5 Operation](./docs/MT5_OPERATION.md)** - MT5 Expert Advisor guide
133 163
 
134 164
 ---
135 165
 
136
-## 🆕 FRESH SERVER DEPLOYMENT (Step-by-Step)
166
+## 🆕 FRESH SERVER DEPLOYMENT (Legacy - See DEPLOYMENT.md)
137 167
 
138 168
 ### Step 1: Server Preparation
139 169
 ```bash
@@ -220,6 +250,9 @@ sudo -u postgres psql -d financial_data -c "\dt"
220 250
 ```
221 251
 
222 252
 ### Step 6: Deploy with Docker (Recommended)
253
+
254
+> **⚠️ Outdated Instructions:** This section contains legacy deployment steps. For current production deployment with containerized SSL certificates, see **[docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**
255
+
223 256
 ```bash
224 257
 # Navigate to project directory
225 258
 cd /root/market-data-service
@@ -232,135 +265,16 @@ nano .env
232 265
 
233 266
 # Build and start production containers
234 267
 docker-compose -f docker-compose.prod.yml up -d --build
235
-
236
-# Verify containers are running
237
-docker-compose -f docker-compose.prod.yml ps
238
-
239
-# View logs
240
-docker-compose -f docker-compose.prod.yml logs -f api
241 268
 ```
242 269
 
243
-For detailed Docker deployment instructions, see [DOCKER.md](./DOCKER.md).
244
-
245
-### Step 7: Configure Nginx (WITHOUT SSL first)
246
-```bash
247
-# Copy nginx configuration
248
-sudo cp nginx-1.24.0/conf/nginx.conf /etc/nginx/nginx.conf
249
-
250
-# Test configuration
251
-sudo nginx -t
252
-
253
-# Start Nginx
254
-sudo systemctl enable nginx
255
-sudo systemctl start nginx
256
-```
257
-
258
-### Step 8: Install SSL Certificate (Certbot)
259
-```bash
260
-# Install certbot
261
-sudo apt install -y certbot python3-certbot-nginx
262
-
263
-# Generate SSL certificate
264
-sudo certbot --nginx -d your-domain.com
265
-
266
-# Test certificate
267
-sudo certbot certificates
268
-```
269
-
270
-### Step 9: Update Nginx for SSL (After Certificate Generation)
271
-```bash
272
-# Update nginx configuration with SSL
273
-sudo tee /etc/nginx/sites-available/market-data-api << EOF
274
-server {
275
-    listen 80;
276
-    server_name your-domain.com;
277
-
278
-    # Redirect HTTP to HTTPS
279
-    return 301 https://\$server_name\$request_uri;
280
-}
281
-
282
-server {
283
-    listen 443 ssl http2;
284
-    server_name your-domain.com;
285
-
286
-    # SSL Configuration
287
-    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
288
-    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
289
-    ssl_protocols TLSv1.2 TLSv1.3;
290
-    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384;
291
-    ssl_prefer_server_ciphers off;
292
-
293
-    # Security Headers
294
-    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
295
-    add_header X-Frame-Options DENY always;
296
-    add_header X-Content-Type-Options nosniff always;
297
-    add_header X-XSS-Protection "1; mode=block" always;
298
-
299
-    # API Proxy Settings
300
-    location /api/ {
301
-        proxy_pass http://localhost:3001;
302
-        proxy_http_version 1.1;
303
-        proxy_set_header Upgrade \$http_upgrade;
304
-        proxy_set_header Connection 'upgrade';
305
-        proxy_set_header Host \$host;
306
-        proxy_set_header X-Real-IP \$remote_addr;
307
-        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
308
-        proxy_set_header X-Forwarded-Proto \$scheme;
309
-
310
-        # Timeout and buffer settings
311
-        proxy_connect_timeout 60s;
312
-        proxy_send_timeout 60s;
313
-        proxy_read_timeout 60s;
314
-        proxy_buffering on;
315
-        proxy_buffer_size 4k;
316
-        proxy_buffers 8 4k;
317
-        client_max_body_size 50M;
318
-    }
319
-
320
-    location /health {
321
-        proxy_pass http://localhost:3001/health;
322
-        proxy_http_version 1.1;
323
-        proxy_set_header Host \$host;
324
-        proxy_set_header X-Real-IP \$remote_addr;
325
-        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
326
-        proxy_set_header X-Forwarded-Proto \$scheme;
327
-    }
328
-
329
-    location / {
330
-        return 200 "Market Data API is running. Use /api/ for endpoints\\n";
331
-        add_header Content-Type text/plain;
332
-    }
333
-}
334
-EOF
335
-
336
-# Enable site and restart nginx
337
-sudo ln -s /etc/nginx/sites-available/market-data-api /etc/nginx/sites-enabled/
338
-sudo rm -f /etc/nginx/sites-enabled/default
339
-sudo nginx -t
340
-sudo systemctl restart nginx
341
-```
342
-
343
-### Step 10: Final Testing
344
-```bash
345
-# Test HTTP to HTTPS redirect
346
-curl -I http://your-domain.com
347
-
348
-# Test HTTPS API endpoints
349
-curl https://your-domain.com/health
350
-curl https://your-domain.com/api/health
351
-curl https://your-domain.com/api/symbols
352
-
353
-# Test bulk endpoint
354
-curl -X POST https://your-domain.com/api/live-prices/bulk \
355
-  -H "Content-Type: application/json" \
356
-  -d '{"prices": [{"symbolId": 1, "price": 123.45}]}'
357
-```
270
+📖 **For detailed Docker deployment instructions, see [DOCKER.md](./DOCKER.md)**  
271
+📖 **For production deployment with SSL, see [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)**
358 272
 
359 273
 ---
360 274
 
361
-## 💻 LOCAL DEVELOPMENT SETUP
275
+## 💻 Local Development
362 276
 
363
-This section provides a quick overview. For detailed instructions, troubleshooting, and advanced configuration, see **[docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)**.
277
+> **Quick Overview:** For complete setup instructions, troubleshooting, and MT5 EA configuration, see **[docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)**.
364 278
 
365 279
 ### Prerequisites
366 280
 - **Docker** and **Docker Compose** installed
@@ -633,14 +547,8 @@ sudo tail -f /var/log/nginx/error.log
633 547
 ```
634 548
 
635 549
 ### SSL Certificate Auto-Renewal
636
-```bash
637
-# Check renewal status
638
-sudo certbot certificates
639
-sudo systemctl status certbot.timer
640 550
 
641
-# Manual renewal test
642
-sudo certbot renew --dry-run
643
-```
551
+> **Note:** SSL certificates are now automatically managed via Docker containers. See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md) for details.
644 552
 
645 553
 ---
646 554
 
@@ -698,60 +606,26 @@ psql financial_data < backup.sql
698 606
 
699 607
 ## Environment Variables
700 608
 
701
-Create a `.env` file in the root directory with the following variables:
702
-
703
-```env
704
-# Database Configuration
705
-DB_HOST=localhost
706
-DB_PORT=5432
707
-DB_NAME=market_data
708
-DB_USER=your_username
709
-DB_PASSWORD=your_password
710
-
711
-# Server Configuration
712
-PORT=3001
713
-NODE_ENV=development
714
-
715
-# JWT Configuration (if needed for authentication)
716
-JWT_SECRET=your_jwt_secret_key
717
-
718
-# API Keys (if needed for external services)
719
-# BINANCE_API_KEY=your_api_key
720
-# BINANCE_API_SECRET=your_api_secret
609
+See **[docs/CONFIGURATION.md](./docs/CONFIGURATION.md)** for complete environment variable documentation.
721 610
 
722
-# CORS Configuration
723
-CORS_ORIGIN=*
724
-```
611
+**Quick Reference:**
612
+- `DB_PASSWORD` - Database password (required)
613
+- `JWT_SECRET` - JWT secret key (required)
614
+- `DOMAIN_NAME` - Domain for SSL certificates (production)
615
+- `SSL_EMAIL` - Email for Let's Encrypt (production)
616
+- `CORS_ORIGIN` - Allowed CORS origins
725 617
 
726 618
 ## API Endpoints
727 619
 
728
-### Health Check
729
-- `GET /health` - Check service health
730
-
731
-### Symbols
732
-- `GET /api/symbols` - Get all symbols (with filtering)
733
-- `GET /api/symbols/search?q=EURUSD` - Search symbols by name
734
-- `GET /api/symbols/:id` - Get symbol by ID
735
-- `POST /api/symbols` - Create new symbol
736
-- `PUT /api/symbols/:id` - Update symbol
737
-- `DELETE /api/symbols/:id` - Delete symbol (soft delete)
738
-
739
-### Candles
740
-- `GET /api/candles?symbolId=1&startTime=2025-01-01T00:00:00Z&endTime=2025-01-02T00:00:00Z&limit=100&offset=0` - Get candles with filtering
741
-- `GET /api/candles/ohlc?symbolId=1&period=1h&limit=100` - Get OHLC data
742
-- `GET /api/candles/:symbolId/latest` - Get latest candle for symbol
743
-- `POST /api/candles` - Create new candle
620
+See **[docs/API_CONTRACT.md](./docs/API_CONTRACT.md)** for complete API documentation.
621
+
622
+**Quick Reference:**
623
+- `GET /health` - Health check
624
+- `GET /api/symbols` - Get all symbols
625
+- `GET /api/candles` - Get candle data
744 626
 - `POST /api/candles/bulk` - Bulk create candles
745
-- `DELETE /api/candles/cleanup/:symbolId?keep=1000` - Clean up old candles, keep latest N (default 1000)
746
-
747
-### Live Prices
748
-- `GET /api/live-prices` - Get all live prices
749
-- `GET /api/live-prices/exchange/:exchange` - Get live prices by exchange
750
-- `GET /api/live-prices/type/:type` - Get live prices by instrument type
751
-- `GET /api/live-prices/:symbolId` - Get live price for symbol
752
-- `POST /api/live-prices` - Create/update live price
627
+- `GET /api/live-prices` - Get live prices
753 628
 - `POST /api/live-prices/bulk` - Bulk update live prices
754
-- `DELETE /api/live-prices/:symbolId` - Delete live price
755 629
 
756 630
 ## Development
757 631
 
@@ -784,9 +658,8 @@ npx sequelize-cli db:migrate:undo
784 658
 
785 659
 ## Deployment
786 660
 
787
-The project is fully containerized using Docker. For deployment:
788
-
789
-1. **Development**: Use `docker-compose up -d`
790
-2. **Production**: Use `docker-compose -f docker-compose.prod.yml up -d`
661
+The project is fully containerized using Docker with automatic SSL certificate management.
791 662
 
792
-See [DOCKER.md](./DOCKER.md) for complete deployment instructions.
663
+- **Development**: See [docs/LOCAL_DEV_SETUP.md](./docs/LOCAL_DEV_SETUP.md)
664
+- **Production**: See [docs/DEPLOYMENT.md](./docs/DEPLOYMENT.md)
665
+- **Docker Details**: See [DOCKER.md](./DOCKER.md)

+ 52 - 5
docker-compose.prod.yml

@@ -70,26 +70,73 @@ services:
70 70
       - "443:443"
71 71
     volumes:
72 72
       - ./nginx/nginx.prod.conf:/etc/nginx/nginx.conf:ro
73
-      # SSL certificates (mount your certbot certificates here)
74
-      - ${SSL_CERT_PATH:-./ssl/certs}:/etc/nginx/ssl/certs:ro
75
-      - ${SSL_KEY_PATH:-./ssl/private}:/etc/nginx/ssl/private:ro
73
+      - ./docker/nginx-entrypoint.sh:/docker-entrypoint-nginx.sh:ro
74
+      # SSL certificates from certbot (shared volume)
75
+      - certbot_etc:/etc/letsencrypt:ro
76
+      - certbot_www:/var/www/certbot:ro
77
+      - certbot_reload:/var/run/certbot-reload:rw
78
+    environment:
79
+      - DOMAIN_NAME=${DOMAIN_NAME:-default}
76 80
     depends_on:
77
-      api:
78
-        condition: service_healthy
81
+      - api
82
+      - certbot
79 83
     networks:
80 84
       - market-data-network
81 85
     restart: unless-stopped
86
+    entrypoint: ["/bin/sh", "/docker-entrypoint-nginx.sh"]
82 87
     healthcheck:
83 88
       test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/health"]
84 89
       interval: 30s
85 90
       timeout: 10s
86 91
       retries: 3
87 92
 
93
+  # Certbot for SSL Certificate Management
94
+  certbot:
95
+    image: certbot/certbot:latest
96
+    container_name: market-data-certbot
97
+    volumes:
98
+      - certbot_etc:/etc/letsencrypt
99
+      - certbot_www:/var/www/certbot
100
+      - certbot_logs:/var/log/letsencrypt
101
+      - certbot_reload:/var/run/certbot-reload
102
+    networks:
103
+      - market-data-network
104
+    restart: unless-stopped
105
+    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew --quiet && touch /var/run/certbot-reload/reload 2>/dev/null || true; sleep 12h & wait $${!}; done;'"
106
+    # Note: Initial certificate must be obtained using certbot-init service
107
+    # Run: DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com docker-compose -f docker-compose.prod.yml run --rm certbot-init
108
+
109
+  # Certbot Initialization Service (one-time use)
110
+  certbot-init:
111
+    image: certbot/certbot:latest
112
+    container_name: market-data-certbot-init
113
+    volumes:
114
+      - certbot_etc:/etc/letsencrypt
115
+      - certbot_www:/var/www/certbot
116
+      - certbot_logs:/var/log/letsencrypt
117
+      - ./docker/init-ssl.sh:/docker/init-ssl.sh:ro
118
+    networks:
119
+      - market-data-network
120
+    entrypoint: /bin/sh
121
+    command: /docker/init-ssl.sh
122
+    environment:
123
+      - DOMAIN_NAME=${DOMAIN_NAME}
124
+      - SSL_EMAIL=${SSL_EMAIL}
125
+      - SSL_STAGING=${SSL_STAGING:-0}
126
+
88 127
 volumes:
89 128
   market_data_db_data_prod:
90 129
     driver: local
91 130
   market_data_logs_prod:
92 131
     driver: local
132
+  certbot_etc:
133
+    driver: local
134
+  certbot_www:
135
+    driver: local
136
+  certbot_logs:
137
+    driver: local
138
+  certbot_reload:
139
+    driver: local
93 140
 
94 141
 networks:
95 142
   market-data-network:

+ 90 - 0
docker/certbot-init.sh

@@ -0,0 +1,90 @@
1
+#!/bin/sh
2
+# SSL Certificate Initialization Script for Docker
3
+# This script is used to obtain the initial SSL certificate
4
+
5
+set -e
6
+
7
+DOMAIN_NAME="${DOMAIN_NAME:-}"
8
+SSL_EMAIL="${SSL_EMAIL:-}"
9
+STAGING="${SSL_STAGING:-0}"
10
+
11
+if [ -z "$DOMAIN_NAME" ]; then
12
+    echo "ERROR: DOMAIN_NAME environment variable is required"
13
+    echo ""
14
+    echo "Usage:"
15
+    echo "  DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com docker-compose -f docker-compose.prod.yml run --rm certbot-init"
16
+    echo ""
17
+    echo "For testing (staging server):"
18
+    echo "  DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com SSL_STAGING=1 docker-compose -f docker-compose.prod.yml run --rm certbot-init"
19
+    exit 1
20
+fi
21
+
22
+if [ -z "$SSL_EMAIL" ]; then
23
+    echo "ERROR: SSL_EMAIL environment variable is required"
24
+    echo ""
25
+    echo "Usage:"
26
+    echo "  DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com docker-compose -f docker-compose.prod.yml run --rm certbot-init"
27
+    exit 1
28
+fi
29
+
30
+echo "=========================================="
31
+echo "SSL Certificate Initialization"
32
+echo "=========================================="
33
+echo "Domain: $DOMAIN_NAME"
34
+echo "Email: $SSL_EMAIL"
35
+echo ""
36
+
37
+# Use staging server if SSL_STAGING=1 (for testing)
38
+STAGING_FLAG=""
39
+if [ "$STAGING" = "1" ]; then
40
+    echo "WARNING: Using Let's Encrypt staging server (for testing only)"
41
+    STAGING_FLAG="--staging"
42
+fi
43
+
44
+# Wait for nginx to be ready (it needs to serve the challenge)
45
+echo "Waiting for nginx to be ready..."
46
+sleep 5
47
+
48
+# Obtain certificate using webroot method
49
+echo "Requesting SSL certificate from Let's Encrypt..."
50
+certbot certonly \
51
+    --webroot \
52
+    --webroot-path=/var/www/certbot \
53
+    --email "$SSL_EMAIL" \
54
+    --agree-tos \
55
+    --no-eff-email \
56
+    --force-renewal \
57
+    $STAGING_FLAG \
58
+    -d "$DOMAIN_NAME"
59
+
60
+if [ $? -eq 0 ]; then
61
+    echo ""
62
+    echo "=========================================="
63
+    echo "✅ SSL certificate obtained successfully!"
64
+    echo "=========================================="
65
+    echo "Certificate location: /etc/letsencrypt/live/$DOMAIN_NAME/"
66
+    echo ""
67
+    echo "Next steps:"
68
+    echo "1. Update nginx.prod.conf with your domain name:"
69
+    echo "   Replace \${DOMAIN_NAME:-default} with $DOMAIN_NAME"
70
+    echo "   Or set DOMAIN_NAME environment variable in docker-compose.prod.yml"
71
+    echo ""
72
+    echo "2. Restart nginx container:"
73
+    echo "   docker-compose -f docker-compose.prod.yml restart nginx"
74
+    echo ""
75
+    echo "3. Verify HTTPS is working:"
76
+    echo "   curl https://$DOMAIN_NAME/health"
77
+    echo ""
78
+    echo "4. Certificates will auto-renew every 12 hours"
79
+else
80
+    echo ""
81
+    echo "=========================================="
82
+    echo "❌ SSL certificate obtainment failed"
83
+    echo "=========================================="
84
+    echo "Please check:"
85
+    echo "1. Domain DNS is pointing to this server"
86
+    echo "2. Port 80 is accessible from the internet"
87
+    echo "3. Nginx container is running and can serve /.well-known/acme-challenge/"
88
+    exit 1
89
+fi
90
+

+ 50 - 0
docker/init-ssl.sh

@@ -0,0 +1,50 @@
1
+#!/bin/sh
2
+# SSL Certificate Initialization Script
3
+# This script obtains the initial SSL certificate using certbot
4
+
5
+set -e
6
+
7
+DOMAIN_NAME="${DOMAIN_NAME:-}"
8
+SSL_EMAIL="${SSL_EMAIL:-}"
9
+STAGING="${SSL_STAGING:-0}"
10
+
11
+if [ -z "$DOMAIN_NAME" ]; then
12
+    echo "ERROR: DOMAIN_NAME environment variable is required"
13
+    echo "Usage: DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com docker-compose -f docker-compose.prod.yml run --rm certbot-init"
14
+    exit 1
15
+fi
16
+
17
+if [ -z "$SSL_EMAIL" ]; then
18
+    echo "ERROR: SSL_EMAIL environment variable is required"
19
+    echo "Usage: DOMAIN_NAME=yourdomain.com SSL_EMAIL=your@email.com docker-compose -f docker-compose.prod.yml run --rm certbot-init"
20
+    exit 1
21
+fi
22
+
23
+echo "Obtaining SSL certificate for domain: $DOMAIN_NAME"
24
+echo "Email: $SSL_EMAIL"
25
+
26
+# Use staging server if SSL_STAGING=1 (for testing)
27
+STAGING_FLAG=""
28
+if [ "$STAGING" = "1" ]; then
29
+    echo "WARNING: Using Let's Encrypt staging server (for testing only)"
30
+    STAGING_FLAG="--staging"
31
+fi
32
+
33
+# Obtain certificate using webroot method
34
+certbot certonly \
35
+    --webroot \
36
+    --webroot-path=/var/www/certbot \
37
+    --email "$SSL_EMAIL" \
38
+    --agree-tos \
39
+    --no-eff-email \
40
+    --force-renewal \
41
+    $STAGING_FLAG \
42
+    -d "$DOMAIN_NAME"
43
+
44
+echo "SSL certificate obtained successfully!"
45
+echo "Certificate location: /etc/letsencrypt/live/$DOMAIN_NAME/"
46
+echo ""
47
+echo "Next steps:"
48
+echo "1. Restart nginx container: docker-compose -f docker-compose.prod.yml restart nginx"
49
+echo "2. Verify HTTPS is working: curl https://$DOMAIN_NAME/health"
50
+

+ 49 - 0
docker/nginx-entrypoint.sh

@@ -0,0 +1,49 @@
1
+#!/bin/sh
2
+# Nginx entrypoint script that processes environment variables in config
3
+
4
+set -e
5
+
6
+# Default domain name (can be overridden)
7
+DOMAIN_NAME="${DOMAIN_NAME:-default}"
8
+
9
+# Process the template config file (which is read-only mounted)
10
+# and create a processed version in a writable location
11
+PROCESSED_CONFIG="/tmp/nginx.conf"
12
+
13
+# Check if SSL certificates exist
14
+CERT_PATH="/etc/letsencrypt/live/${DOMAIN_NAME}/fullchain.pem"
15
+KEY_PATH="/etc/letsencrypt/live/${DOMAIN_NAME}/privkey.pem"
16
+
17
+# Replace DOMAIN_NAME_PLACEHOLDER in the config file (if it exists)
18
+# If no placeholder, just copy the config
19
+if grep -q "DOMAIN_NAME_PLACEHOLDER" /etc/nginx/nginx.conf 2>/dev/null; then
20
+    sed "s|DOMAIN_NAME_PLACEHOLDER|${DOMAIN_NAME}|g" /etc/nginx/nginx.conf > "$PROCESSED_CONFIG"
21
+else
22
+    cp /etc/nginx/nginx.conf "$PROCESSED_CONFIG"
23
+fi
24
+
25
+# If certificates don't exist, just use the config as-is (should be no-ssl config)
26
+if [ ! -f "$CERT_PATH" ] || [ ! -f "$KEY_PATH" ]; then
27
+    echo "WARNING: SSL certificates not found at $CERT_PATH"
28
+    echo "Using HTTP-only configuration for ACME challenges."
29
+fi
30
+
31
+# Start nginx with auto-reload for certificate updates
32
+# Reload every 6 hours OR when certbot signals a renewal
33
+# Use the processed config file
34
+exec /bin/sh -c "
35
+  # Watch for certbot reload signal
36
+  (while :; do 
37
+    if [ -f /var/run/certbot-reload/reload ]; then
38
+      echo 'Certificate renewed, reloading nginx...'
39
+      nginx -s reload -c $PROCESSED_CONFIG
40
+      rm -f /var/run/certbot-reload/reload
41
+    fi
42
+    sleep 60
43
+  done) &
44
+  # Periodic reload every 6 hours
45
+  (while :; do sleep 6h & wait \${!}; nginx -s reload -c $PROCESSED_CONFIG; done) &
46
+  # Start nginx with processed config
47
+  nginx -c $PROCESSED_CONFIG -g 'daemon off;'
48
+"
49
+

+ 25 - 0
docker/renew-ssl.sh

@@ -0,0 +1,25 @@
1
+#!/bin/sh
2
+# SSL Certificate Renewal Script
3
+# This script renews SSL certificates and reloads nginx
4
+
5
+set -e
6
+
7
+echo "Checking for SSL certificate renewal..."
8
+
9
+# Renew certificates
10
+certbot renew --quiet
11
+
12
+# Check if renewal was successful and certificates were updated
13
+if [ $? -eq 0 ]; then
14
+    echo "SSL certificates renewed successfully"
15
+    
16
+    # Reload nginx to use new certificates
17
+    # Send SIGHUP to nginx process (graceful reload)
18
+    echo "Reloading nginx..."
19
+    nginx -s reload || {
20
+        echo "Warning: Could not reload nginx. You may need to restart the nginx container manually."
21
+    }
22
+else
23
+    echo "No renewal needed or renewal failed"
24
+fi
25
+

+ 260 - 0
docs/CONFIGURATION.md

@@ -0,0 +1,260 @@
1
+# Configuration Guide
2
+
3
+Complete guide for configuring the Market Data Service environment variables and settings.
4
+
5
+## Environment Variables
6
+
7
+### Required Variables
8
+
9
+| Variable | Description | Example |
10
+|----------|-------------|---------|
11
+| `DB_PASSWORD` | PostgreSQL database password | `your_secure_password` |
12
+| `JWT_SECRET` | JWT secret key for authentication | `generated_secret_key` |
13
+| `DOMAIN_NAME` | Domain name for SSL certificates | `market-price.insightbull.io` |
14
+| `SSL_EMAIL` | Email for Let's Encrypt notifications | `admin@insightbull.io` |
15
+
16
+### Optional Variables
17
+
18
+| Variable | Description | Default |
19
+|----------|-------------|---------|
20
+| `DB_USER` | PostgreSQL database user | `postgres` |
21
+| `DB_NAME` | Database name | `financial_data` |
22
+| `NODE_ENV` | Environment mode | `production` |
23
+| `PORT` | API port (internal) | `3000` |
24
+| `CORS_ORIGIN` | Allowed CORS origins | `*` |
25
+| `LOG_LEVEL` | Logging level | `info` |
26
+| `SSL_STAGING` | Use Let's Encrypt staging server | `0` |
27
+
28
+## .env File Setup
29
+
30
+### Production Configuration
31
+
32
+Create a `.env` file in the project root:
33
+
34
+```bash
35
+# Database Configuration
36
+DB_USER=postgres
37
+DB_PASSWORD=your_secure_password_here
38
+DB_NAME=financial_data
39
+
40
+# Application Configuration
41
+NODE_ENV=production
42
+PORT=3000
43
+CORS_ORIGIN=https://market-price.insightbull.io
44
+JWT_SECRET=your_secure_jwt_secret_here
45
+LOG_LEVEL=info
46
+
47
+# SSL Certificate Configuration
48
+DOMAIN_NAME=market-price.insightbull.io
49
+SSL_EMAIL=admin@insightbull.io
50
+```
51
+
52
+### Development Configuration
53
+
54
+For local development:
55
+
56
+```bash
57
+# Database Configuration
58
+DB_USER=postgres
59
+DB_PASSWORD=postgres
60
+DB_NAME=financial_data
61
+
62
+# Application Configuration
63
+NODE_ENV=development
64
+PORT=3000
65
+CORS_ORIGIN=*
66
+JWT_SECRET=dev_secret_key
67
+LOG_LEVEL=debug
68
+
69
+# SSL not needed for local development
70
+```
71
+
72
+## Generating Secure Secrets
73
+
74
+### JWT Secret
75
+
76
+Generate a secure JWT secret:
77
+
78
+```bash
79
+# Using OpenSSL
80
+openssl rand -base64 32
81
+
82
+# Using Node.js
83
+node -e "console.log(require('crypto').randomBytes(32).toString('base64'))"
84
+```
85
+
86
+Then update in `.env`:
87
+```bash
88
+JWT_SECRET=<generated_secret>
89
+```
90
+
91
+### Database Password
92
+
93
+Use a strong, unique password for production:
94
+- Minimum 16 characters
95
+- Mix of uppercase, lowercase, numbers, and symbols
96
+- Don't reuse passwords
97
+
98
+## Domain Configuration
99
+
100
+### Production Domain
101
+
102
+The production domain is: `market-price.insightbull.io`
103
+
104
+**Important:**
105
+- Domain must be set in `.env` as `DOMAIN_NAME=market-price.insightbull.io`
106
+- DNS must point to your server's IP before obtaining SSL certificates
107
+- CORS_ORIGIN should match: `https://market-price.insightbull.io`
108
+
109
+### Local Development Domain
110
+
111
+For local development, use: `market-data.local`
112
+
113
+See [LOCAL_DEV_SETUP.md](./LOCAL_DEV_SETUP.md) for local domain setup.
114
+
115
+## SSL Certificate Configuration
116
+
117
+### Automatic SSL (Production)
118
+
119
+SSL certificates are automatically managed via Docker containers. No manual configuration needed.
120
+
121
+**Required variables:**
122
+- `DOMAIN_NAME` - Your domain name
123
+- `SSL_EMAIL` - Email for Let's Encrypt notifications
124
+
125
+**For testing (staging server):**
126
+```bash
127
+SSL_STAGING=1
128
+```
129
+
130
+See [DEPLOYMENT.md](./DEPLOYMENT.md) for SSL certificate setup instructions.
131
+
132
+## CORS Configuration
133
+
134
+### Production
135
+
136
+```bash
137
+CORS_ORIGIN=https://market-price.insightbull.io
138
+```
139
+
140
+### Development
141
+
142
+```bash
143
+CORS_ORIGIN=*
144
+```
145
+
146
+### Multiple Origins
147
+
148
+For multiple allowed origins, use comma-separated values (check your CORS middleware implementation).
149
+
150
+## Logging Configuration
151
+
152
+### Log Levels
153
+
154
+- `debug` - Detailed debug information (development)
155
+- `info` - General information (production)
156
+- `warn` - Warning messages
157
+- `error` - Error messages only
158
+
159
+### Production
160
+
161
+```bash
162
+LOG_LEVEL=info
163
+```
164
+
165
+### Development
166
+
167
+```bash
168
+LOG_LEVEL=debug
169
+```
170
+
171
+## Database Configuration
172
+
173
+### Docker (Recommended)
174
+
175
+When using Docker, database connection is automatic:
176
+- `DB_HOST=db` (service name in docker-compose)
177
+- `DB_PORT=5432` (internal port)
178
+- Database credentials from `.env`
179
+
180
+### Non-Docker Setup
181
+
182
+For non-Docker setups:
183
+```bash
184
+DB_HOST=localhost
185
+DB_PORT=5432
186
+DB_USER=postgres
187
+DB_PASSWORD=your_password
188
+DB_NAME=financial_data
189
+```
190
+
191
+## Verification
192
+
193
+### Check Configuration
194
+
195
+```bash
196
+# Verify .env file exists
197
+ls -la .env
198
+
199
+# Check environment variables (in Docker)
200
+docker-compose exec api env | grep -E 'DB_|JWT_|DOMAIN_|SSL_'
201
+```
202
+
203
+### Test Configuration
204
+
205
+```bash
206
+# Test database connection
207
+docker-compose exec db psql -U postgres -d financial_data -c "SELECT 1;"
208
+
209
+# Test API health
210
+curl http://localhost/health
211
+```
212
+
213
+## Security Best Practices
214
+
215
+1. **Never commit `.env` file** - Already in `.gitignore`
216
+2. **Use strong secrets** - Generate secure random values
217
+3. **Rotate secrets regularly** - Especially in production
218
+4. **Limit CORS origins** - Don't use `*` in production
219
+5. **Use different secrets** - Different values for dev/staging/prod
220
+
221
+## Troubleshooting
222
+
223
+### Environment Variables Not Loading
224
+
225
+**Problem:** Variables not being read
226
+
227
+**Solution:**
228
+```bash
229
+# Check .env file exists
230
+ls -la .env
231
+
232
+# Verify Docker is reading .env
233
+docker-compose config | grep -A 5 environment
234
+```
235
+
236
+### SSL Certificate Issues
237
+
238
+**Problem:** SSL certificate not working
239
+
240
+**Solution:**
241
+1. Verify `DOMAIN_NAME` matches your actual domain
242
+2. Check DNS points to your server
243
+3. Ensure ports 80 and 443 are open
244
+4. See [DEPLOYMENT.md](./DEPLOYMENT.md) troubleshooting section
245
+
246
+### Database Connection Issues
247
+
248
+**Problem:** Can't connect to database
249
+
250
+**Solution:**
251
+1. Verify `DB_PASSWORD` is correct
252
+2. Check database container is running: `docker-compose ps db`
253
+3. Test connection: `docker-compose exec db pg_isready -U postgres`
254
+
255
+## References
256
+
257
+- **Production Deployment**: See [DEPLOYMENT.md](./DEPLOYMENT.md)
258
+- **Local Development**: See [LOCAL_DEV_SETUP.md](./LOCAL_DEV_SETUP.md)
259
+- **Docker Setup**: See [DOCKER.md](../DOCKER.md)
260
+

+ 369 - 0
docs/DEPLOYMENT.md

@@ -0,0 +1,369 @@
1
+# Production Deployment Guide
2
+
3
+> **Note:** This guide covers production deployment with fully containerized SSL certificate management. For local development, see [LOCAL_DEV_SETUP.md](./LOCAL_DEV_SETUP.md). For Docker usage details, see [DOCKER.md](../DOCKER.md).
4
+
5
+Complete guide for deploying Market Data Service to production with fully containerized SSL certificate management.
6
+
7
+## 🎯 Overview
8
+
9
+This deployment is **fully containerized**, including:
10
+- ✅ Application (Node.js API)
11
+- ✅ Database (PostgreSQL)
12
+- ✅ Reverse Proxy (Nginx)
13
+- ✅ SSL Certificate Obtain & Renewal (Certbot)
14
+- ✅ Automatic certificate renewal every 12 hours
15
+- ✅ Automatic nginx reload after certificate renewal
16
+
17
+## 📋 Prerequisites
18
+
19
+1. **Docker** and **Docker Compose** installed
20
+2. **Domain name** pointing to your server's IP address
21
+3. **Ports 80 and 443** open in firewall
22
+4. **Email address** for Let's Encrypt notifications
23
+
24
+## 🚀 Deployment Steps
25
+
26
+### Step 1: Clone and Prepare
27
+
28
+```bash
29
+# Clone repository
30
+git clone <your-repo-url>
31
+cd market-data-service
32
+
33
+# Create .env file for production
34
+cat > .env << EOF
35
+# Database Configuration
36
+DB_USER=postgres
37
+DB_PASSWORD=your_secure_password_here
38
+DB_NAME=financial_data
39
+
40
+# Application Configuration
41
+NODE_ENV=production
42
+PORT=3000
43
+CORS_ORIGIN=https://yourdomain.com
44
+JWT_SECRET=your_secure_jwt_secret_here
45
+LOG_LEVEL=info
46
+
47
+# SSL Configuration
48
+DOMAIN_NAME=yourdomain.com
49
+SSL_EMAIL=your-email@example.com
50
+EOF
51
+```
52
+
53
+**Important:** Replace all placeholder values with your actual values.
54
+
55
+### Step 2: Start Services (Without SSL First)
56
+
57
+```bash
58
+# Start all services except certbot (we'll get certificates first)
59
+docker-compose -f docker-compose.prod.yml up -d db api nginx
60
+
61
+# Verify services are running
62
+docker-compose -f docker-compose.prod.yml ps
63
+
64
+# Check logs
65
+docker-compose -f docker-compose.prod.yml logs -f
66
+```
67
+
68
+### Step 3: Obtain SSL Certificate
69
+
70
+**Option A: Production Certificate (Recommended)**
71
+
72
+```bash
73
+# Obtain production SSL certificate
74
+DOMAIN_NAME=yourdomain.com \
75
+SSL_EMAIL=your-email@example.com \
76
+docker-compose -f docker-compose.prod.yml run --rm certbot-init
77
+```
78
+
79
+**Option B: Staging Certificate (For Testing)**
80
+
81
+```bash
82
+# Test with Let's Encrypt staging server (doesn't count against rate limits)
83
+DOMAIN_NAME=yourdomain.com \
84
+SSL_EMAIL=your-email@example.com \
85
+SSL_STAGING=1 \
86
+docker-compose -f docker-compose.prod.yml run --rm certbot-init
87
+```
88
+
89
+**What this does:**
90
+- Requests SSL certificate from Let's Encrypt
91
+- Uses webroot method (nginx serves the challenge)
92
+- Stores certificates in Docker volume `certbot_etc`
93
+- Certificates are automatically shared with nginx container
94
+
95
+### Step 4: Restart Nginx with SSL
96
+
97
+```bash
98
+# Restart nginx to load SSL certificates
99
+docker-compose -f docker-compose.prod.yml restart nginx
100
+
101
+# Start certbot for automatic renewal
102
+docker-compose -f docker-compose.prod.yml up -d certbot
103
+```
104
+
105
+### Step 5: Verify Deployment
106
+
107
+```bash
108
+# Check all services are running
109
+docker-compose -f docker-compose.prod.yml ps
110
+
111
+# Test HTTP (should redirect to HTTPS)
112
+curl -I http://yourdomain.com
113
+
114
+# Test HTTPS
115
+curl https://yourdomain.com/health
116
+
117
+# Test API endpoint
118
+curl https://yourdomain.com/api/health
119
+```
120
+
121
+## 🔄 Automatic Certificate Renewal
122
+
123
+Certificates are **automatically renewed** every 12 hours by the certbot container. The renewal process:
124
+
125
+1. Certbot checks if certificates need renewal (30 days before expiry)
126
+2. If renewal is needed, certbot obtains new certificates
127
+3. Certbot signals nginx to reload
128
+4. Nginx reloads within 60 seconds to use new certificates
129
+
130
+**No manual intervention required!**
131
+
132
+### Manual Renewal (If Needed)
133
+
134
+```bash
135
+# Force certificate renewal
136
+docker-compose -f docker-compose.prod.yml exec certbot certbot renew --force-renewal
137
+
138
+# Restart nginx to load new certificates
139
+docker-compose -f docker-compose.prod.yml restart nginx
140
+```
141
+
142
+### Check Certificate Status
143
+
144
+```bash
145
+# View certificate information
146
+docker-compose -f docker-compose.prod.yml exec certbot certbot certificates
147
+
148
+# Check certificate expiry
149
+docker-compose -f docker-compose.prod.yml exec certbot openssl x509 -in /etc/letsencrypt/live/yourdomain.com/cert.pem -noout -dates
150
+```
151
+
152
+## 📊 Monitoring
153
+
154
+### View Logs
155
+
156
+```bash
157
+# All services
158
+docker-compose -f docker-compose.prod.yml logs -f
159
+
160
+# Specific service
161
+docker-compose -f docker-compose.prod.yml logs -f api
162
+docker-compose -f docker-compose.prod.yml logs -f nginx
163
+docker-compose -f docker-compose.prod.yml logs -f certbot
164
+docker-compose -f docker-compose.prod.yml logs -f db
165
+```
166
+
167
+### Health Checks
168
+
169
+```bash
170
+# API health
171
+curl https://yourdomain.com/health
172
+
173
+# Container health status
174
+docker-compose -f docker-compose.prod.yml ps
175
+```
176
+
177
+## 🔧 Configuration
178
+
179
+### Environment Variables
180
+
181
+Key environment variables in `.env`:
182
+
183
+| Variable | Description | Required |
184
+|----------|-------------|----------|
185
+| `DB_PASSWORD` | PostgreSQL password | Yes |
186
+| `JWT_SECRET` | JWT secret key | Yes |
187
+| `DOMAIN_NAME` | Your domain name | Yes |
188
+| `SSL_EMAIL` | Email for Let's Encrypt | Yes |
189
+| `CORS_ORIGIN` | Allowed CORS origins | Recommended |
190
+
191
+### Nginx Configuration
192
+
193
+Nginx configuration is in `nginx/nginx.prod.conf`. The domain name is automatically substituted at runtime from the `DOMAIN_NAME` environment variable.
194
+
195
+### SSL Certificate Paths
196
+
197
+Certificates are stored in Docker volumes:
198
+- **Volume**: `certbot_etc` → `/etc/letsencrypt` in containers
199
+- **Certificate path**: `/etc/letsencrypt/live/DOMAIN_NAME/fullchain.pem`
200
+- **Key path**: `/etc/letsencrypt/live/DOMAIN_NAME/privkey.pem`
201
+
202
+## 🛠️ Troubleshooting
203
+
204
+### Certificate Obtain Fails
205
+
206
+**Problem**: `certbot-init` fails with "Connection refused" or "Challenge failed"
207
+
208
+**Solutions**:
209
+1. Verify domain DNS points to your server:
210
+   ```bash
211
+   dig yourdomain.com
212
+   nslookup yourdomain.com
213
+   ```
214
+
215
+2. Verify ports 80 and 443 are open:
216
+   ```bash
217
+   sudo ufw status
218
+   # If needed: sudo ufw allow 80 && sudo ufw allow 443
219
+   ```
220
+
221
+3. Check nginx is running and accessible:
222
+   ```bash
223
+   curl http://yourdomain.com/.well-known/acme-challenge/test
224
+   ```
225
+
226
+4. Use staging server first to test:
227
+   ```bash
228
+   SSL_STAGING=1 docker-compose -f docker-compose.prod.yml run --rm certbot-init
229
+   ```
230
+
231
+### Nginx Can't Find Certificates
232
+
233
+**Problem**: Nginx fails to start with "SSL certificate not found"
234
+
235
+**Solutions**:
236
+1. Verify certificates exist:
237
+   ```bash
238
+   docker-compose -f docker-compose.prod.yml exec nginx ls -la /etc/letsencrypt/live/
239
+   ```
240
+
241
+2. Check DOMAIN_NAME matches certificate domain:
242
+   ```bash
243
+   # In .env file
244
+   DOMAIN_NAME=yourdomain.com  # Must match exactly
245
+   ```
246
+
247
+3. Restart nginx:
248
+   ```bash
249
+   docker-compose -f docker-compose.prod.yml restart nginx
250
+   ```
251
+
252
+### Certificate Renewal Not Working
253
+
254
+**Problem**: Certificates expire without renewal
255
+
256
+**Solutions**:
257
+1. Check certbot container is running:
258
+   ```bash
259
+   docker-compose -f docker-compose.prod.yml ps certbot
260
+   ```
261
+
262
+2. Check certbot logs:
263
+   ```bash
264
+   docker-compose -f docker-compose.prod.yml logs certbot
265
+   ```
266
+
267
+3. Manually test renewal:
268
+   ```bash
269
+   docker-compose -f docker-compose.prod.yml exec certbot certbot renew --dry-run
270
+   ```
271
+
272
+### Nginx Not Reloading After Renewal
273
+
274
+**Problem**: New certificates not picked up by nginx
275
+
276
+**Solutions**:
277
+1. Check reload signal file:
278
+   ```bash
279
+   docker-compose -f docker-compose.prod.yml exec nginx ls -la /var/run/certbot-reload/
280
+   ```
281
+
282
+2. Manually reload nginx:
283
+   ```bash
284
+   docker-compose -f docker-compose.prod.yml exec nginx nginx -s reload
285
+   ```
286
+
287
+3. Restart nginx:
288
+   ```bash
289
+   docker-compose -f docker-compose.prod.yml restart nginx
290
+   ```
291
+
292
+## 🔐 Security Best Practices
293
+
294
+1. **Strong Passwords**: Use strong, unique passwords for `DB_PASSWORD` and `JWT_SECRET`
295
+2. **Firewall**: Only expose ports 80 and 443 to the internet
296
+3. **Regular Updates**: Keep Docker images updated:
297
+   ```bash
298
+   docker-compose -f docker-compose.prod.yml pull
299
+   docker-compose -f docker-compose.prod.yml up -d
300
+   ```
301
+4. **Backup**: Regularly backup database and SSL certificates:
302
+   ```bash
303
+   # Backup database
304
+   docker-compose -f docker-compose.prod.yml exec db pg_dump -U postgres financial_data > backup.sql
305
+   
306
+   # Backup certificates (optional, they auto-renew)
307
+   docker run --rm -v market-data-service_certbot_etc:/data -v $(pwd):/backup alpine tar czf /backup/certs_backup.tar.gz -C /data .
308
+   ```
309
+5. **Monitor Logs**: Set up log monitoring and alerting
310
+6. **Rate Limiting**: Adjust rate limits in `nginx/nginx.prod.conf` as needed
311
+
312
+## 📝 Maintenance
313
+
314
+### Update Application
315
+
316
+```bash
317
+# Pull latest code
318
+git pull
319
+
320
+# Rebuild and restart
321
+docker-compose -f docker-compose.prod.yml up -d --build
322
+
323
+# Verify
324
+docker-compose -f docker-compose.prod.yml ps
325
+curl https://yourdomain.com/health
326
+```
327
+
328
+### Database Backup
329
+
330
+```bash
331
+# Create backup
332
+docker-compose -f docker-compose.prod.yml exec db pg_dump -U postgres financial_data > backup_$(date +%Y%m%d).sql
333
+
334
+# Restore from backup
335
+docker-compose -f docker-compose.prod.yml exec -T db psql -U postgres financial_data < backup_20250101.sql
336
+```
337
+
338
+### View Certificate Expiry
339
+
340
+```bash
341
+docker-compose -f docker-compose.prod.yml exec certbot certbot certificates
342
+```
343
+
344
+## 🎉 Success Checklist
345
+
346
+- [ ] All services running (`docker-compose ps`)
347
+- [ ] HTTP redirects to HTTPS
348
+- [ ] HTTPS endpoint accessible
349
+- [ ] SSL certificate valid (check browser or `curl -v`)
350
+- [ ] API endpoints responding
351
+- [ ] Health check passing
352
+- [ ] Certbot container running
353
+- [ ] Automatic renewal working (check logs after 12 hours)
354
+
355
+## 📚 Additional Resources
356
+
357
+- [Docker Documentation](https://docs.docker.com/)
358
+- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
359
+- [Nginx Documentation](https://nginx.org/en/docs/)
360
+- [Certbot Documentation](https://eff-certbot.readthedocs.io/)
361
+
362
+## 🆘 Support
363
+
364
+If you encounter issues:
365
+1. Check logs: `docker-compose -f docker-compose.prod.yml logs`
366
+2. Verify configuration: `docker-compose -f docker-compose.prod.yml config`
367
+3. Check service status: `docker-compose -f docker-compose.prod.yml ps`
368
+4. Review this guide's troubleshooting section
369
+

+ 69 - 0
docs/README.md

@@ -0,0 +1,69 @@
1
+# Documentation Index
2
+
3
+Welcome to the Market Data Service documentation. This index helps you find the right guide for your needs.
4
+
5
+## 📖 Documentation Guide
6
+
7
+### Getting Started
8
+
9
+- **[Main README](../README.md)** - Project overview, features, and quick start
10
+- **[Local Development Setup](./LOCAL_DEV_SETUP.md)** - Complete guide for local development with MT5 EA
11
+- **[Production Deployment](./DEPLOYMENT.md)** - Production deployment with containerized SSL certificates
12
+
13
+### Configuration & Setup
14
+
15
+- **[Configuration Guide](./CONFIGURATION.md)** - Environment variables and configuration options
16
+- **[Docker Guide](../DOCKER.md)** - Docker containerization details and usage
17
+
18
+### API & Integration
19
+
20
+- **[API Contract](./API_CONTRACT.md)** - Complete API endpoint specifications
21
+- **[MT5 Operation](./MT5_OPERATION.md)** - MT5 Expert Advisor configuration and operation
22
+
23
+### Reference
24
+
25
+- **[Archive](./ARCHIVE/)** - Historical documentation and reports
26
+
27
+## 🚀 Quick Navigation
28
+
29
+### I want to...
30
+
31
+**Set up local development:**
32
+→ [LOCAL_DEV_SETUP.md](./LOCAL_DEV_SETUP.md)
33
+
34
+**Deploy to production:**
35
+→ [DEPLOYMENT.md](./DEPLOYMENT.md)
36
+
37
+**Configure environment variables:**
38
+→ [CONFIGURATION.md](./CONFIGURATION.md)
39
+
40
+**Use Docker:**
41
+→ [DOCKER.md](../DOCKER.md)
42
+
43
+**Understand the API:**
44
+→ [API_CONTRACT.md](./API_CONTRACT.md)
45
+
46
+**Configure MT5 EA:**
47
+→ [MT5_OPERATION.md](./MT5_OPERATION.md)
48
+
49
+## 📁 Documentation Structure
50
+
51
+```
52
+docs/
53
+├── README.md              # This file - documentation index
54
+├── LOCAL_DEV_SETUP.md     # Local development guide
55
+├── DEPLOYMENT.md          # Production deployment guide
56
+├── CONFIGURATION.md       # Configuration guide
57
+├── API_CONTRACT.md        # API specifications
58
+├── MT5_OPERATION.md      # MT5 EA guide
59
+└── ARCHIVE/              # Historical documentation
60
+    └── PRODUCTION_READINESS_REPORT.md
61
+```
62
+
63
+## 🔗 External Links
64
+
65
+- [Docker Documentation](https://docs.docker.com/)
66
+- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
67
+- [Nginx Documentation](https://nginx.org/en/docs/)
68
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
69
+

+ 4 - 3
nginx/nginx.prod.conf

@@ -70,9 +70,10 @@ http {
70 70
         server_name _;
71 71
 
72 72
         # SSL Configuration
73
-        # Update these paths to match your SSL certificate locations
74
-        ssl_certificate /etc/nginx/ssl/certs/fullchain.pem;
75
-        ssl_certificate_key /etc/nginx/ssl/private/privkey.pem;
73
+        # Using Let's Encrypt certificates from certbot container
74
+        # DOMAIN_NAME_PLACEHOLDER will be replaced by nginx entrypoint script
75
+        ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME_PLACEHOLDER/fullchain.pem;
76
+        ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME_PLACEHOLDER/privkey.pem;
76 77
 
77 78
         # SSL Security Settings
78 79
         ssl_protocols TLSv1.2 TLSv1.3;

+ 165 - 0
nginx/nginx.prod.conf.template

@@ -0,0 +1,165 @@
1
+# Nginx configuration for production environment
2
+# HTTP to HTTPS redirect, SSL/TLS, WebSocket support
3
+# This is a template - DOMAIN_NAME will be replaced at runtime
4
+
5
+user nginx;
6
+worker_processes auto;
7
+error_log /var/log/nginx/error.log warn;
8
+pid /var/run/nginx.pid;
9
+
10
+events {
11
+    worker_connections 2048;
12
+    use epoll;
13
+}
14
+
15
+http {
16
+    include /etc/nginx/mime.types;
17
+    default_type application/octet-stream;
18
+
19
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
20
+                    '$status $body_size_sent "$http_referer" '
21
+                    '"$http_user_agent" "$http_x_forwarded_for"';
22
+
23
+    access_log /var/log/nginx/access.log main;
24
+
25
+    sendfile on;
26
+    tcp_nopush on;
27
+    tcp_nodelay on;
28
+    keepalive_timeout 65;
29
+    types_hash_max_size 2048;
30
+    client_max_body_size 50M;
31
+
32
+    # Hide nginx version
33
+    server_tokens off;
34
+
35
+    # Gzip compression
36
+    gzip on;
37
+    gzip_vary on;
38
+    gzip_proxied any;
39
+    gzip_comp_level 6;
40
+    gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml+rss;
41
+
42
+    # Rate limiting (optional, adjust as needed)
43
+    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/m;
44
+    limit_req_zone $binary_remote_addr zone=ws_limit:10m rate=10r/s;
45
+
46
+    # Upstream API server
47
+    upstream api {
48
+        server api:3000;
49
+        keepalive 32;
50
+    }
51
+
52
+    # HTTP server - redirect to HTTPS
53
+    server {
54
+        listen 80;
55
+        server_name _;
56
+
57
+        # Allow Let's Encrypt challenges
58
+        location /.well-known/acme-challenge/ {
59
+            root /var/www/certbot;
60
+        }
61
+
62
+        # Redirect all other traffic to HTTPS
63
+        location / {
64
+            return 301 https://$host$request_uri;
65
+        }
66
+    }
67
+
68
+    # HTTPS server
69
+    server {
70
+        listen 443 ssl http2;
71
+        server_name _;
72
+
73
+        # SSL Configuration
74
+        # Using Let's Encrypt certificates from certbot container
75
+        ssl_certificate /etc/letsencrypt/live/DOMAIN_NAME_PLACEHOLDER/fullchain.pem;
76
+        ssl_certificate_key /etc/letsencrypt/live/DOMAIN_NAME_PLACEHOLDER/privkey.pem;
77
+
78
+        # SSL Security Settings
79
+        ssl_protocols TLSv1.2 TLSv1.3;
80
+        ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
81
+        ssl_prefer_server_ciphers off;
82
+        ssl_session_cache shared:SSL:10m;
83
+        ssl_session_timeout 10m;
84
+        ssl_session_tickets off;
85
+
86
+        # Security Headers
87
+        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
88
+        add_header X-Frame-Options "DENY" always;
89
+        add_header X-Content-Type-Options "nosniff" always;
90
+        add_header X-XSS-Protection "1; mode=block" always;
91
+        add_header Referrer-Policy "strict-origin-when-cross-origin" always;
92
+
93
+        # Health check endpoint (no rate limiting)
94
+        location /health {
95
+            proxy_pass http://api/health;
96
+            proxy_http_version 1.1;
97
+            proxy_set_header Host $host;
98
+            proxy_set_header X-Real-IP $remote_addr;
99
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
100
+            proxy_set_header X-Forwarded-Proto $scheme;
101
+            
102
+            access_log off;
103
+        }
104
+
105
+        # WebSocket support for Socket.io (with rate limiting)
106
+        location /socket.io/ {
107
+            limit_req zone=ws_limit burst=20 nodelay;
108
+            
109
+            proxy_pass http://api;
110
+            proxy_http_version 1.1;
111
+            
112
+            # WebSocket upgrade headers
113
+            proxy_set_header Upgrade $http_upgrade;
114
+            proxy_set_header Connection "upgrade";
115
+            
116
+            # Standard proxy headers
117
+            proxy_set_header Host $host;
118
+            proxy_set_header X-Real-IP $remote_addr;
119
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
120
+            proxy_set_header X-Forwarded-Proto $scheme;
121
+            
122
+            # WebSocket timeouts (Socket.io uses long-lived connections)
123
+            proxy_read_timeout 86400s;
124
+            proxy_send_timeout 86400s;
125
+            proxy_connect_timeout 60s;
126
+            
127
+            # Disable buffering for real-time data
128
+            proxy_buffering off;
129
+            proxy_cache off;
130
+        }
131
+
132
+        # API endpoints (with rate limiting)
133
+        location /api/ {
134
+            limit_req zone=api_limit burst=50 nodelay;
135
+            
136
+            proxy_pass http://api;
137
+            proxy_http_version 1.1;
138
+            proxy_set_header Host $host;
139
+            proxy_set_header X-Real-IP $remote_addr;
140
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
141
+            proxy_set_header X-Forwarded-Proto $scheme;
142
+            
143
+            # Timeout settings
144
+            proxy_connect_timeout 60s;
145
+            proxy_send_timeout 60s;
146
+            proxy_read_timeout 60s;
147
+            
148
+            # Buffer settings
149
+            proxy_buffering on;
150
+            proxy_buffer_size 4k;
151
+            proxy_buffers 8 4k;
152
+        }
153
+
154
+        # Root endpoint
155
+        location / {
156
+            proxy_pass http://api;
157
+            proxy_http_version 1.1;
158
+            proxy_set_header Host $host;
159
+            proxy_set_header X-Real-IP $remote_addr;
160
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
161
+            proxy_set_header X-Forwarded-Proto $scheme;
162
+        }
163
+    }
164
+}
165
+