Scraper Staging Portal
System Connected

Distributed Scraper Infrastructure

Welcome to the staging dashboard. This portal lets you review active engines, monitor real-time Celery task queues via Flower, browse structured database contents via Mongo Express, and check complete documentation. The API service is fully functional and running in production environment mode.

Seeded Crawlers 11 Active platforms configured
Registered Searches 7 User search setups active
Scraped Listings 270 Total database index count
Celery Scrape Runs 8 S: 7 | F: 1 | R: 0

Developer & Staging Shortcuts

Bypasses front-end to explore backend services
📖 System Documentation Beautiful rendering of the system design, parser guides, and architecture specs. 🛠 Interactive API Swagger Direct access to standard REST endpoints (sites, results, JWT auth, WebSocket status). ⚡ Celery Task Monitor (Flower) Real-time status of worker pools, queue length, and task execution graphs. 📊 Mongo Database GUI Mongo Express client to view collection schemas, indices, and raw data documents.

Interactive Task Console

Define and launch a crawler job directly to the Celery workers

🔑 Administrator Authentication Required

Enter admin credentials to unlock the task launcher.

📡 Real-time Event Stream

Disconnected
Authenticate to establish WebSocket event connection...

Seeded Scraping Engines

Multi-tier parsing modules registered in system
Marktplaats Netherlands
HTTP Client
Tier 1 Max 10 workers
Active & Ready
eBay Global
HTTP/API
Tier 1 Max 10 workers
Active & Ready
Vinted Global
HTTP/API
Tier 1 Max 10 workers
Active & Ready
Leboncoin France
Browser/Stealth
Tier 1 Max 3 workers
Active & Ready
Subito Italy
HTTP Client
Tier 2 Max 10 workers
Active & Ready
Willhaben Austria
HTTP Client
Tier 2 Max 10 workers
Active & Ready
Allegro Poland
HTTP Client
Tier 1 Max 10 workers
Active & Ready
Wallapop Spain/Global
HTTP/API
Tier 2 Max 10 workers
Active & Ready
Blocket Sweden
HTTP Client
Tier 2 Max 10 workers
Active & Ready
Delcampe Global
HTTP Client
Tier 3 Max 10 workers
Active & Ready
Catawiki Global
Browser/Stealth
Tier 3 Max 3 workers
Active & Ready

Recent Scrape Executions

Last 5 Celery tasks
Started At Status New/Total Duration Target / Errors
2026-05-23 11:45:08 SUCCESS +27 (252) 4.34s
Site: e806716b
2026-05-23 11:43:08 FAILED +0 (0) 5.34s
Site: 17f7f6ab
Catawiki fetch failed: Browser navigation failed: Playwright got HTTP 403 Forbidden for https://www.catawiki.com/en/l?q=macbook pro+m1
2026-05-23 11:41:38 SUCCESS +87 (576) 6.10s
Site: e806716b
2026-05-23 11:40:40 SUCCESS +81 (540) 5.23s
Site: e806716b
2026-05-23 11:40:20 SUCCESS +0 (0) 0.49s
Site: e806716b

Recent System Activity Logs

Last 5 server events
Timestamp Level Source Log Message
2026-05-23 11:45:08 INFO scheduler Manual search trigger dispatched for search a2db28d7-85eb-4671-9b1f-750c85a17eff
View details
{'job_name': 'manual_trigger', 'searches_dispatched': 1, 'timestamp': '2026-05-23T11:45:08.621444', 'search_id': 'a2db28d7-85eb-4671-9b1f-750c85a17eff', 'celery_task_id': '88858d64-fa76-4cfb-9362-c358cc2105c0'}
2026-05-23 11:43:13 ERROR worker Scraper execution failed for site 'Catawiki' on search '7e558346-1510-42eb-925c-ee30c3e91571': Catawiki fetch failed: Browser navigation failed: Playwright got HTTP 403 Forbidden for https://www.catawiki.com/en/l?q=macbook pro+m1
View details
{'traceback': 'Traceback (most recent call last):\n  File "/app/scraper/scrapers/base.py", line 247, in fetch_browser\n    raise ScraperBlockedError(f"Playwright got HTTP 403 Forbidden for {url}")\nscraper.scrapers.exceptions.ScraperBlockedError: Playwright got HTTP 403 Forbidden for https://www.catawiki.com/en/l?q=macbook pro+m1\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/app/scraper/scrapers/catawiki.py", line 33, in search\n    html = await self.fetch_browser(url)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/app/scraper/scrapers/base.py", line 253, in fetch_browser\n    raise ScraperError(f"Browser navigation failed: {str(e)}") from e\nscraper.scrapers.exceptions.ScraperError: Browser navigation failed: Playwright got HTTP 403 Forbidden for https://www.catawiki.com/en/l?q=macbook pro+m1\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/app/scraper/workers/tasks.py", line 74, in run_async\n    results = await scraper.search(keywords)\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/app/scraper/scrapers/catawiki.py", line 36, in search\n    raise ScraperError(f"Catawiki fetch failed: {str(e)}") from e\nscraper.scrapers.exceptions.ScraperError: Catawiki fetch failed: Browser navigation failed: Playwright got HTTP 403 Forbidden for https://www.catawiki.com/en/l?q=macbook pro+m1\n'}
2026-05-23 11:43:08 INFO scheduler Manual search trigger dispatched for search 7e558346-1510-42eb-925c-ee30c3e91571
View details
{'job_name': 'manual_trigger', 'searches_dispatched': 1, 'timestamp': '2026-05-23T11:43:08.397091', 'search_id': '7e558346-1510-42eb-925c-ee30c3e91571', 'celery_task_id': '3ea5c1fe-66bf-4f7a-a664-a1e6523624da'}
2026-05-23 11:41:38 INFO scheduler Manual search trigger dispatched for search d1a8b296-d0c4-4778-8e20-05ef55485e64
View details
{'job_name': 'manual_trigger', 'searches_dispatched': 1, 'timestamp': '2026-05-23T11:41:38.477518', 'search_id': 'd1a8b296-d0c4-4778-8e20-05ef55485e64', 'celery_task_id': '67ff3897-1b01-4e1d-831d-b4f74cd19da0'}
2026-05-23 11:40:40 INFO scheduler Manual search trigger dispatched for search 2595a075-97f1-46d7-8cfb-a969e7efa940
View details
{'job_name': 'manual_trigger', 'searches_dispatched': 1, 'timestamp': '2026-05-23T11:40:40.007281', 'search_id': '2595a075-97f1-46d7-8cfb-a969e7efa940', 'celery_task_id': 'c922b9e6-1562-4872-96a8-3eb8fbfadc17'}