🗂️ Data Scraper Monitor
Track data ingestion, scraping schedules, and system health
System Status
ACTIVE
All scrapers operational
Total Indicators
10,342
Across all sources
Last Scrape
12:00
May 29, 2025
Success Rate
99.8%
Last 24 hours
Avg Speed
2.3s
Per data source
S3 Storage
1.2GB
macro-data-lake
⏰ Scraper Schedule
Macro Data Scraper
AWS Lambda
Scheduled
Function
scrapeMacroData
Schedule
Daily @ 12:00 UTC
Next Run
In 18h 24m
Last Success
12:00:42 UTC
ML Predictor
AWS Lambda
Scheduled
Function
MLPredictor
Schedule
Daily @ 12:15 UTC
Next Run
In 18h 39m
Last Success
12:15:18 UTC
Universal Index
AWS Lambda
Manual
Function
createUniversalIndex
Schedule
On-demand
Indicators
33 indexed
Last Update
2 hours ago
📡 Data Sources
| Source | Provider | Indicators | Last Update | Status |
|---|---|---|---|---|
| FRED Economic Data | fred_direct | 8,234 | 12:00:15 UTC | Active |
| NY Fed Reference Rates | ny_fed_direct | 15 | 12:01:32 UTC | Active |
| OFR Financial Research | ofr_direct | 127 | 12:04:21 UTC | Active |
| US Treasury Direct | treasury_direct | 289 | 12:03:55 UTC | Active |
| Polygon Market Data | polygon_direct | 1,221 | Real-time | Active |
| ECB Statistics | ecb_direct | 456 | 12:02:48 UTC | Active |
📊 Scraping Activity (24h)
📜 Recent Activity
12:15:18
ML Predictor Completed
Generated predictions for 3 assets, confidence: 82%
12:15:00
ML Predictor Started
Loading data from S3 bucket macro-data-lake
12:04:21
OFR Data Scraped
Successfully scraped 127 indicators in 4.2s
12:03:55
Treasury Data Scraped
Updated 289 treasury indicators including auction data
12:02:48
ECB Data Scraped
Retrieved 456 European indicators including CISS stress indices
12:01:32
NY Fed Rates Updated
SOFR: 5.31%, EFFR: 5.33%, OBFR: 5.32%
12:00:42
Macro Data Scraper Completed
Successfully processed 8,234 FRED indicators
12:00:00
Daily Scrape Initiated
EventBridge triggered scrapeMacroData Lambda function