Skip to main content
POST
/
v1
/
squids
Create Squid
curl --request POST \
  --url https://api.lobstr.io/v1/squids \
  --header 'Authorization: <authorization>' \
  --header 'Content-Type: <content-type>' \
  --data '
{
  "crawler": "<string>",
  "name": "<string>"
}
'
{
  "id": "<string>",
  "name": "<string>",
  "crawler": "<string>",
  "is_active": true,
  "concurrency": 123,
  "params": {},
  "schedule": {},
  "to_complete": true
}
This endpoint creates a new squid for a specified crawler by providing the crawler’s hash ID. A squid is a container that groups together related tasks and configurations for a specific scraping operation.

​
What is a Squid?

A squid acts as a project workspace for your scraping tasks:
  • Groups tasks: All URLs or items you want to scrape with the same crawler
  • Stores configuration: Crawler parameters, concurrency settings, delivery options
  • Manages runs: Tracks execution history and results
  • Enables scheduling: Set up automated recurring scrapes

​
Headers

​
Authorization
string
required
Your API authentication token. Value: Token YOUR_API_KEY
​
Content-Type
string
required
Must be application/json. Value: application/json

​
Request Body

​
crawler
string
required
The unique ID (hash) of the crawler to use for this squid. Example: "4734d096159ef05210e0e1677e8be823"
​
name
string
Custom name for the squid. If not provided, an auto-generated name will be used. Example: "My Google Maps Scraper"

​
Response Field Explanations

​
id
string
Unique squid identifier. Example: "c106a44a98044ef18acc59986ae10967"
​
name
string
Squid name (auto-generated as “Crawler Name (N)” if not provided). Example: "Google Maps (1)"
​
crawler
string
ID of the associated crawler. Example: "4734d096159ef05210e0e1677e8be823"
​
is_active
boolean
Whether the squid is active and can run. Example: true
​
concurrency
integer
Number of concurrent tasks (default: 1). Example: 1
​
params
object
Squid-level parameters with default values from crawler. Example: {}
​
schedule
object | null
Cron schedule configuration (null if not scheduled). Example: null
​
to_complete
boolean
Whether to stop after all tasks complete (false = run continuously). Example: false
After creating a squid, you can update its parameters and settings using the Update Squid endpoint before adding tasks.
The squid automatically inherits default parameters from the crawler. Check the response ‘params’ field to see what defaults were applied.
Custom names help organize multiple squids using the same crawler. Without a custom name, squids are numbered sequentially.
Make sure to get the crawler ID from the List Crawlers endpoint first. Using an invalid crawler ID will result in an error.

​
Code Examples

curl -X POST "https://api.lobstr.io/v1/squids" \
  -H "Authorization: Token YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "crawler": "4734d096159ef05210e0e1677e8be823",
    "name": "My Google Maps Scraper"
  }'

​
Response

200
{
  "id": "e86b29c032024b66aff529e1d43c2bd7",
  "account": [],
  "concurrency": 1,
  "crawler": "4734d096159ef05210e0e1677e8be823",
  "created_at": "2025-02-03T14:24:23Z",
  "is_active": true,
  "name": "My Google Maps Scraper",
  "params": {
    "max_results": 200,
    "ratings": "Any rating",
    "country": "United States",
    "language": "English (United States)"
  },
  "schedule": null,
  "to_complete": false
}