Indian NSE Stock Insights

Runs on /home/ubuntu/nse-public-mcp/ — completely isolated from /home/ubuntu/swingtrader/.

1. Clone

cd /home/ubuntu
git clone https://github.com/alokbarnwal/nse-public-mcp.git
cd nse-public-mcp

2. Create database and user

sudo mysql <<'SQL'
CREATE DATABASE IF NOT EXISTS nse_public CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'nse_writer'@'localhost' IDENTIFIED BY 'CHANGE_ME_STRONG_PASSWORD';
GRANT ALL PRIVILEGES ON nse_public.* TO 'nse_writer'@'localhost';
FLUSH PRIVILEGES;
SQL

3. Apply schema

mysql -u nse_writer -p nse_public < db/schema.sql
mysql -u nse_writer -p nse_public < db/schema_indicators.sql
mysql -u nse_writer -p nse_public < db/schema_price_action.sql

4. Install dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

5. Configure environment

cp .env.example .env
chmod 600 .env
nano .env                 # set DB_PASSWORD to match step 2

6. (Optional) Refresh full Nifty 500 list

The repo ships with the top ~100 stocks hardcoded. To expand to all 500:

python -m config.stocks --refresh        # fetches NSE archives, rewrites config/stocks.py

7. Load stock universe into DB

python -m config.stocks --load-to-db

8. Backfill historical data (long-running, use screen/tmux)

screen -S backfill
source .venv/bin/activate
python -m data.backfill --all
# Ctrl-A D to detach. Reattach: screen -r backfill

Expected runtime:

• daily 5y × 500: ~30 min
• 15min 2y × 500: ~2 h
• 5min 6mo × 500: ~1 h

If anything fails, the failure list is written to backfill_failures_<timeframe>.json. Re-run:

python -m data.backfill --timeframe 15min --resume

8b. Bulk-compute indicators (long-running, use screen/tmux)

After the candle backfill completes, populate the indicators table for all 500 stocks across all 3 timeframes:

screen -S indicators
cd /home/ubuntu/nse-public-mcp
python3 -m indicators.runner --all
# Ctrl-A D to detach

Expected runtime: ~30–45 min. Idempotent — safe to re-run if interrupted.

8c. Bulk-compute price action (long-running, use screen/tmux)

After indicators are populated, compute price-action features (zones, S/R, patterns, breakouts, fibs, gaps, volume profile) for all 500 stocks across all 3 timeframes:

screen -S price_action
cd /home/ubuntu/nse-public-mcp
python3 -m price_action.runner --timeframe daily --all-features
python3 -m price_action.runner --timeframe 15min --all-features
python3 -m price_action.runner --timeframe 5min  --all-features
# Ctrl-A D to detach

Expected runtime: ~60–90 min total. Idempotent — safe to re-run if interrupted.

9. Install crontab

crontab -e

Add:

# nse-public-mcp candle updaters
*/5  9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_5min  >> logs/cron_5min.log 2>&1
*/15 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_15min >> logs/cron_15min.log 2>&1
0    18   * * 1-5 cd /home/ubuntu/nse-public-mcp && /home/ubuntu/nse-public-mcp/.venv/bin/python -m jobs.cron_daily >> logs/cron_daily.log 2>&1

# nse-public-mcp indicator updaters (each runs after the matching candle fetch)
30       18   * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_daily >> logs/indicators_daily.log 2>&1
20,35,50 9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_15min >> logs/indicators_15min.log 2>&1
5        16   * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_15min >> logs/indicators_15min.log 2>&1
2-57/5   9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_indicators_5min  >> logs/indicators_5min.log 2>&1

# nse-public-mcp price-action updaters (each runs after the matching indicator cron)
35       18   * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_daily  >> logs/price_action_daily.log 2>&1
25,55    9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_15min >> logs/price_action_15min.log 2>&1
9-54/15  9-15 * * 1-5 cd /home/ubuntu/nse-public-mcp && /usr/bin/python3 -m jobs.cron_price_action_5min  >> logs/price_action_5min.log 2>&1

Cron times are server-local. Verify the server is on IST (timedatectl) — if not, shift the hour ranges. The 5-min indicator cron is offset to 2-57/5 so it runs ~2 minutes after cron_5min and never reads stale candles. The price-action 5-min cron is offset to 9-54/15 (minutes 9, 24, 39, 54) so it runs ~2 minutes after each cron_indicators_5min slot.

10. Verification

mysql -u nse_writer -p nse_public <<'SQL'
SELECT 'daily' AS tf, COUNT(*) AS n FROM candles_daily
UNION ALL SELECT '15min', COUNT(*) FROM candles_15min
UNION ALL SELECT '5min',  COUNT(*) FROM candles_5min;

-- Per-stock coverage on daily — flag any with < 1000 rows (~4 trading years)
SELECT symbol, COUNT(*) AS n
FROM candles_daily
GROUP BY symbol
HAVING n < 1000
ORDER BY n;

-- Latest data freshness
SELECT MIN(candle_date) AS earliest, MAX(candle_date) AS latest
FROM candles_daily;

-- Recent cron health
SELECT job_type, timeframe, symbols_success, symbols_failed, started_at, duration_seconds
FROM fetch_log
ORDER BY started_at DESC
LIMIT 20;
SQL

DB size on disk:

sudo du -sh /var/lib/mysql/nse_public/