Crawl a Website

Crawl a website using Webᵀ Crawl

Not Python? API Reference

Ensure you have set your Web Transpose API Key.

Crawl a website in just 3 lines.

import webtranspose as webt

crawl = webt.Crawl(
  "https://www.example.com",
  max_pages=100,
  render_js=True,
)
await crawl.crawl() # crawl.queue_crawl() for async

Saving and Re-accessing Crawl

Creating a Crawl will create a unique crawl.crawl_id that can be used in the future to get the crawl again.

crawl_id = crawl.crawl_id

crawl = webt.get_crawl(crawl_id)

Crawl Parameters

url · string · required

max_pages · int · default: 100

render_js · bool · default: false

allowed_urls · string

A list of allowed URL paths.

Exmaple: ["https://webtranspose/*"]

banned_urls · string

A list of banned URL paths.

Exmaple: ["https://webtranspose/blogs/*"]

verbose · bool

n_workers · int · default: 1

api_key · string