DEV Community

Jaison Hernando Velasco Camacho
Jaison Hernando Velasco Camacho

Posted on

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

#python#asyncio#tutorial#webdev

Building Real-Time Asynchronous Web Crawlers with Python Asyncio, Playwright, and Webhooks

Web scraping has evolved from fetching static HTML using requests and BeautifulSoup to handling dynamic JavaScript-rendered Single Page Applications (SPAs). Modern web platforms rely heavily on client-side rendering, WebSocket connections, and complex DOM manipulations.

In this comprehensive tutorial, we will build a production-ready, asynchronous web crawler using Python's asyncio, Microsoft's Playwright (async API), and an automated Webhook dispatch pipeline to stream data in real-time.

🏗️ Architecture Overview

Our crawler follows an asynchronous Producer-Consumer design pattern:

  1. URL Queue (Producer): Manages target URLs with concurrency limits and deduplication.
  2. Browser Pool (Worker): Spawns headless chromium instances via Playwright async context managers.
  3. Data Extractor: Extracts structured metadata, dynamic DOM elements, and network metrics.
  4. Webhook Dispatcher (Consumer): Posts JSON payloads asynchronously to a designated webhook endpoint as soon as a page is scraped. 
 +--------------------+
| Target URLs |
+---------+----------+
|
v
+--------------------+
| asyncio.Queue |
+---------+----------+
|
+--------+--------+
| |
v v
+--------------+ +--------------+
| Worker 1 | | Worker 2 | (Playwright Async Pages)
+------+-------+ +------+-------+
| |
+--------+--------+
|
v
+--------------------+
| Webhook Dispatch | (HTTPX / Async POST)
+--------------------+
Enter fullscreen modeExit fullscreen mode

🛠️ Prerequisites & Setup

Ensure you have Python 3.9+ installed. Install the required dependencies:

pip install playwright httpx pydantic
playwright install chromium
Enter fullscreen modeExit fullscreen mode

🐍 Complete Implementation Code (crawler.py)

Here is the complete, runnable asynchronous web crawler engine:

importasyncioimportloggingimporttimefromtypingimportDict,Any,List,OptionalfrompydanticimportBaseModel,HttpUrlimporthttpxfromplaywright.async_apiimportasync_playwright,BrowserContext,Page# Configure Logging
logging.basicConfig(level=logging.INFO,format="%(asctime)s [%(levelname)s] %(message)s",handlers=[logging.StreamHandler()])# Models
classScrapeTarget(BaseModel):url:strdepth:int=0max_depth:int=1classScrapedPayload(BaseModel):url:strtitle:strstatus_code:intcontent_length:intscraped_at:floatlinks:List[str]classAsyncCrawlerEngine:def__init__(self,webhook_url:Optional[str]=None,concurrency:int=3):self.webhook_url=webhook_urlself.concurrency=concurrencyself.queue:asyncio.Queue[ScrapeTarget]=asyncio.Queue()self.visited:set=set()self.http_client=httpx.AsyncClient(timeout=10.0)asyncdefsend_webhook(self,payload:ScrapedPayload):"""Dispatches real-time scraped data via Webhook POST request."""ifnotself.webhook_url:logging.info(f"[Payload Processed] {payload.url} -> Title: '{payload.title}'")returntry:response=awaitself.http_client.post(self.webhook_url,json=payload.model_dump())logging.info(f"[Webhook Delivered] {payload.url} -> HTTP {response.status_code}")exceptExceptionase:logging.error(f"[Webhook Failed] {payload.url}: {e}")asyncdefscrape_page(self,context:BrowserContext,target:ScrapeTarget):"""Scrapes a single page using Playwright async page API."""page:Page=awaitcontext.new_page()try:logging.info(f"[Scraping] {target.url}")response=awaitpage.goto(target.url,wait_until="domcontentloaded",timeout=30000)title=awaitpage.title()content=awaitpage.content()status=response.statusifresponseelse200# Extract internal/external links
href_elements=awaitpage.query_selector_all("a[href]")links=[]foreleminhref_elements:href=awaitelem.get_attribute("href")ifhrefandhref.startswith("http"):links.append(href)payload=ScrapedPayload(url=target.url,title=title,status_code=status,content_length=len(content),scraped_at=time.time(),links=links[:10]# Cap top 10 links
)awaitself.send_webhook(payload)# Enqueue child links if depth allows
iftarget.depth<target.max_depth:forlinkinlinks[:5]:iflinknotinself.visited:self.visited.add(link)awaitself.queue.put(ScrapeTarget(url=link,depth=target.depth+1,max_depth=target.max_depth))exceptExceptionaserr:logging.error(f"[Error Scraping] {target.url}: {err}")finally:awaitpage.close()asyncdefworker(self,context:BrowserContext):"""Worker task consuming URLs from the asyncio queue."""whileTrue:try:target=awaitasyncio.wait_for(self.queue.get(),timeout=3.0)awaitself.scrape_page(context,target)self.queue.task_done()exceptasyncio.TimeoutError:breakexceptExceptionase:logging.error(f"[Worker Exception] {e}")self.queue.task_done()asyncdefrun(self,seed_urls:List[str]):"""Main engine loop establishing Playwright browser pool."""forurlinseed_urls:self.visited.add(url)awaitself.queue.put(ScrapeTarget(url=url,depth=0,max_depth=1))asyncwithasync_playwright()asp:browser=awaitp.chromium.launch(headless=True)context=awaitbrowser.new_context(user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")workers=[asyncio.create_task(self.worker(context))for_inrange(self.concurrency)]awaitself.queue.join()awaitasyncio.gather(*workers)awaitcontext.close()awaitbrowser.close()awaitself.http_client.aclose()if__name__=="__main__":seeds=["https://news.ycombinator.com","https://httpbin.org/html"]# Replace with your webhook receiver URL (e.g. webhook.site)
webhook_endpoint="https://httpbin.org/post"engine=AsyncCrawlerEngine(webhook_url=webhook_endpoint,concurrency=2)asyncio.run(engine.run(seeds))
Enter fullscreen modeExit fullscreen mode

⚡ Key Technical Highlights

  1. Non-blocking I/O: Leverages Python asyncio loop with Playwright's native asynchronous driver to handle high concurrency without blocking CPU cycles.
  2. Headless Execution & Anti-Detection: Uses custom User-Agents and browser contexts to render dynamic JavaScript SPA content cleanly.
  3. Real-time Webhook Streaming: Uses httpx.AsyncClient to fire non-blocking JSON payloads instantly as pages complete scraping, enabling downstream message brokers (e.g. Kafka, RabbitMQ) to consume crawl data without delay.
  4. Queue-driven Scaling: Scalable worker pool managing concurrency and deduplication via asyncio.Queue and thread-safe sets.

🛡️ Production Best Practices

  • Rate Limiting & Delays: Introduce random asyncio.sleep(random.uniform(1.0, 3.0)) between navigation requests to respect target servers.
  • Proxy Rotation: Pass proxy={"server": "http://user:pass@proxy_host:port"} inside browser.new_context().
  • Resource Cleanup: Always use try/finally blocks or async context managers to close browser pages, avoiding memory leaks.

🎯 Conclusion

By combining Python's asyncio, Playwright's headless browser control, and real-time Webhook streaming, you can build modern web scrapers capable of extracting dynamic JavaScript content at scale.

Top comments (0)