List Squids

curl --request GET \
  --url https://api.lobstr.io/v1/squids \
  --header 'Authorization: <authorization>'

{
  "total_results": 123,
  "limit": 123,
  "page": 123,
  "total_pages": 123,
  "result_from": 123,
  "result_to": 123,
  "data": [
    {}
  ],
  "next": {},
  "previous": {},
  "data[].id": "<string>",
  "data[].name": "<string>",
  "data[].created_at": "<string>",
  "data[].is_active": true,
  "data[].is_ready": true,
  "data[].crawler": "<string>",
  "data[].crawler_name": "<string>",
  "data[].icon": "<string>",
  "data[].is_public": true,
  "data[].concurrency": 123,
  "data[].params": {},
  "data[].to_complete": true,
  "data[].export_unique_results": true,
  "data[].no_line_breaks": true,
  "data[].last_run_status": "<string>",
  "data[].last_run_at": "<string>",
  "data[].total_runs": 123,
  "data[].next_launch_at": {},
  "data[].google_sheet_fields": {},
  "data[].ftp_fields": {},
  "data[].s3_fields": {},
  "data[].webhook_fields": {},
  "data[].schedule": {},
  "data[].schedule_time": "<string>",
  "data[].download_url": {}
}

GET

squids

List Squids

curl --request GET \
  --url https://api.lobstr.io/v1/squids \
  --header 'Authorization: <authorization>'

{
  "total_results": 123,
  "limit": 123,
  "page": 123,
  "total_pages": 123,
  "result_from": 123,
  "result_to": 123,
  "data": [
    {}
  ],
  "next": {},
  "previous": {},
  "data[].id": "<string>",
  "data[].name": "<string>",
  "data[].created_at": "<string>",
  "data[].is_active": true,
  "data[].is_ready": true,
  "data[].crawler": "<string>",
  "data[].crawler_name": "<string>",
  "data[].icon": "<string>",
  "data[].is_public": true,
  "data[].concurrency": 123,
  "data[].params": {},
  "data[].to_complete": true,
  "data[].export_unique_results": true,
  "data[].no_line_breaks": true,
  "data[].last_run_status": "<string>",
  "data[].last_run_at": "<string>",
  "data[].total_runs": 123,
  "data[].next_launch_at": {},
  "data[].google_sheet_fields": {},
  "data[].ftp_fields": {},
  "data[].s3_fields": {},
  "data[].webhook_fields": {},
  "data[].schedule": {},
  "data[].schedule_time": "<string>",
  "data[].download_url": {}
}

This endpoint returns a paginated list of all squids belonging to the authenticated user. You can search by name and control pagination with query parameters.

Headers

Authorization

string

required

Your API authentication token. Value: Token YOUR_API_KEY

Query Parameters

name

string

Filter squids by name (case-insensitive partial match). Example: Google

limit

integer

Number of results per page. Example: 10

page

integer

Page number for pagination. Example: 2

Response Structure

total_results

integer

Total number of squids matching the query. Example: 15

limit

integer

Number of results per page. Example: 50

page

integer

Current page number. Example: 1

total_pages

integer

Total number of pages available. Example: 1

result_from

integer

Starting index of results in current page. Example: 1

result_to

integer

Ending index of results in current page. Example: 15

data

array

Array of squid objects with comprehensive details

string | null

Full URL to the next page (null if on last page). Example: null

string | null

Full URL to the previous page (null if on first page). Example: null

Squid Object Fields

data[].id

string

Unique squid identifier. Example: "c106a44a98044ef18acc59986ae10967"

data[].name

string

Display name of the squid. Example: "Google Maps (1)"

data[].created_at

string

Timestamp when the squid was created. Example: "2024-01-15T10:30:00Z"

data[].is_active

boolean

Whether the squid is active and can run. Example: true

data[].is_ready

boolean

Whether the squid has tasks and is ready to run. Example: true

data[].crawler

string

Unique identifier of the associated crawler. Example: "4734d096159ef05210e0e1677e8be823"

data[].crawler_name

string

Display name of the crawler. Example: "Google Maps Search Export"

data[].icon

string

URL to the crawler’s icon image. Example: "https://cdn.lobstr.io/icons/google-maps.png"

data[].is_public

boolean

Whether the crawler is publicly available. Example: true

data[].concurrency

integer

Number of concurrent tasks the squid runs. Example: 2

data[].params

object

Crawler-specific parameter configuration object. Example: {"language": "en"}

data[].to_complete

boolean

Whether to stop after completing all tasks. Example: false

data[].export_unique_results

boolean

Whether to export only unique results. Example: true

data[].no_line_breaks

boolean

Whether to remove line breaks from exported data. Example: false

data[].last_run_status

string

Status of the most recent run (completed, running, error, etc.). Example: "DONE"

data[].last_run_at

string

Timestamp of the most recent run. Example: "2024-01-20T15:45:00Z"

data[].total_runs

integer

Total number of times the squid has been executed. Example: 5

data[].next_launch_at

string | null

Scheduled time for the next automated run (null if not scheduled). Example: "2024-01-21T10:00:00Z"

data[].google_sheet_fields

object | null

Google Sheets integration configuration (null if not configured). Example: null

data[].ftp_fields

object | null

FTP integration configuration (null if not configured). Example: null

data[].s3_fields

object | null

AWS S3 integration configuration (null if not configured). Example: null

data[].webhook_fields

object | null

Webhook integration configuration (null if not configured). Example: null

data[].schedule

string | null

Cron expression for scheduled runs (null if not scheduled). Example: "0 10 * * *"

data[].schedule_time

string

Human-readable schedule description. Example: "Daily at 10:00 AM"

data[].download_url

string | null

Temporary signed URL to download results (null if no results available). Example: "https://storage.lobstr.io/downloads/xyz123..."

Use the ‘name’ query parameter to quickly find squids. It performs a case-insensitive partial match on squid names.

The ‘next’ and ‘previous’ fields contain full URLs for easy pagination. Use them directly to navigate through pages.

Check ‘last_run_status’ to see if a squid has completed, is running, or encountered errors. Useful for monitoring multiple squids.

Squids with ‘is_ready: false’ haven’t been configured with tasks yet. Add tasks before starting a run.

The ‘download_url’ field provides a temporary signed URL to download results. URLs expire after a certain time period.

Code Examples

curl -X GET "https://api.lobstr.io/v1/squids?name=Google&limit=10&page=1" \
  -H "Authorization: Token YOUR_API_KEY"

Response

200

{
  "total_results": 34,
  "limit": 10,
  "page": 1,
  "total_pages": 4,
  "result_from": 1,
  "result_to": 10,
  "data": [
    {
      "id": "e86b29c032024b66aff529e1d43c2bd7",
      "created_at": "2025-02-03T14:24:23Z",
      "name": "My Google Maps Scraper",
      "is_ready": true,
      "is_active": true,
      "concurrency": 2,
      "params": {
        "country": "United States",
        "ratings": "4.0+",
        "language": "English (United States)",
        "max_results": 150
      },
      "crawler": "4734d096159ef05210e0e1677e8be823",
      "crawler_name": "Google Maps Search Export",
      "last_run_status": "DONE",
      "total_runs": 5,
      "last_run_at": "2025-03-05T09:27:17.012423",
      "next_launch_at": null,
      "is_public": true
    }
  ],
  "next": "https://api.lobstr.io/v1/squids?limit=10&page=2",
  "previous": null
}

Update Squid Get Squid Details

​Headers

​Query Parameters

​Response Structure

​Squid Object Fields

​Code Examples

​Response

Headers

Query Parameters

Response Structure

Squid Object Fields

Code Examples

Response