POST/v1/squids

Create Squid

Create a new squid container for a specific crawler to organize and run scraping tasks

This endpoint creates a new squid for a specified crawler by providing the crawler's hash ID. A squid is a container that groups together related tasks and configurations for a specific scraping operation.

What is a Squid?

A squid acts as a project workspace for your scraping tasks:

  • Groups tasks: All URLs or items you want to scrape with the same crawler
  • Stores configuration: Crawler parameters, concurrency settings, delivery options
  • Manages runs: Tracks execution history and results
  • Enables scheduling: Set up automated recurring scrapes

Headers

KeyValueRequired
AuthorizationToken YOUR_API_KEYYes
Content-Typeapplication/jsonYes

Request Body

crawlerstring
Required
The unique ID (hash) of the crawler to use for this squid
Example: 4734d096159ef05210e0e1677e8be823
namestring
Optional
Custom name for the squid. If not provided, an auto-generated name will be used.
Example: My Google Maps Scraper

Response Field Explanations

id
string
Unique squid identifier
Example: c106a44a98044ef18acc59986ae10967
name
string
Squid name (auto-generated as "Crawler Name (N)" if not provided)
Example: Google Maps (1)
crawler
string
ID of the associated crawler
Example: 4734d096159ef05210e0e1677e8be823
is_active
boolean
Whether the squid is active and can run
Example: true
concurrency
integer
Number of concurrent tasks (default: 1)
Example: 1
params
object
Squid-level parameters with default values from crawler
Example: {}
schedule
objectnull
Cron schedule configuration (null if not scheduled)
Example: null
to_complete
boolean
Whether to stop after all tasks complete (false = run continuously)
Example: false
Pro Tip

Pro Tip

After creating a squid, you can update its parameters and settings using the Update Squid endpoint before adding tasks.
Note

Note

The squid automatically inherits default parameters from the crawler. Check the response 'params' field to see what defaults were applied.
Pro Tip

Pro Tip

Custom names help organize multiple squids using the same crawler. Without a custom name, squids are numbered sequentially.
Warning

Warning

Make sure to get the crawler ID from the List Crawlers endpoint first. Using an invalid crawler ID will result in an error.