Skip to main content
GET
/
v1
/
crawlers
List Crawlers
curl --request GET \
  --url https://api.lobstr.io/v1/crawlers \
  --header 'Authorization: <authorization>'
{
  "total_results": 123,
  "limit": 123,
  "page": 123,
  "total_pages": 123,
  "result_from": 123,
  "result_to": 123,
  "data": [
    {}
  ],
  "next": {},
  "previous": {},
  "data[].id": "<string>",
  "data[].name": "<string>",
  "data[].slug": "<string>",
  "data[].is_public": true,
  "data[].is_available": true,
  "data[].is_premium": true,
  "data[].has_issues": true,
  "data[].max_concurrency": 123,
  "data[].credits_per_row": 123,
  "data[].account": {},
  "data[].icon": "<string>"
}
This endpoint returns a paginated list of all available crawlers on the lobstr.io platform. Each crawler is designed to extract specific types of data from different sources. Use this endpoint to discover which crawlers are available for your data collection needs before creating squids or adding tasks.

​
Headers

​
Authorization
string
required
Your API authentication token. Value: Token YOUR_API_KEY
​
Content-Type
string
default:"application/json"
Content type of the request. Value: application/json

​
Response Structure

​
total_results
integer
Total number of crawlers available. Example: 47
​
limit
integer
Number of results per page. Example: 50
​
page
integer
Current page number. Example: 1
​
total_pages
integer
Total number of pages available. Example: 1
​
result_from
integer
Starting index of results in current page. Example: 1
​
result_to
integer
Ending index of results in current page. Example: 47
​
data
array
Array of crawler objects with comprehensive details
​
next
string | null
Full URL to the next page (null if on last page). Example: null
​
previous
string | null
Full URL to the previous page (null if on first page). Example: null

​
Crawler Object Fields

​
data[].id
string
Unique identifier (hash) for the crawler. Example: "4734d096159ef05210e0e1677e8be823"
​
data[].name
string
Human-readable name of the crawler. Example: "Google Maps Search Export"
​
data[].slug
string
URL-friendly identifier. Example: "google-maps-search-export"
​
data[].is_public
boolean
Whether the crawler is publicly available. Example: true
​
data[].is_available
boolean
Whether the crawler is currently available for use (false if paused). Example: true
​
data[].is_premium
boolean
Requires premium subscription or extra credits. Example: false
​
data[].has_issues
boolean
Currently experiencing issues (runs paused until fixed). Example: false
​
data[].max_concurrency
integer
Maximum number of concurrent tasks allowed for this crawler. Example: 20
​
data[].credits_per_row
number
Credits charged per result row. Example: 0.5
​
data[].account
object | null
Required account sync details (null if no authentication needed). Example: null
​
data[].icon
string
Base64-encoded SVG icon for UI display. Example: "data:image/svg+xml;base64,PHN2ZyB4bWxucz..."
Cache the list of crawlers locally as it doesn’t change frequently. This reduces unnecessary API calls and improves your application’s performance.
Each crawler has a unique hash identifier that you’ll use when creating squids or configuring data collection tasks.
Check the crawler parameters endpoint to see what configuration options are available for each crawler before starting your scraping operations.

​
Code Examples

curl -X GET "https://api.lobstr.io/v1/crawlers" \
  -H "Authorization: Token YOUR_API_KEY"

​
Response

200
{
  "total_results": 47,
  "limit": 50,
  "page": 1,
  "total_pages": 1,
  "result_from": 1,
  "result_to": 47,
  "data": [
    {
      "id": "6c5227261fbbd3430f926d409f4c44bf",
      "object": "crawler",
      "account": null,
      "name": "Twitter Profile Scraper",
      "slug": "twitter-profile-scraper",
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 20,
      "credits_per_row": 2
    },
    {
      "id": "c47dfc2beba9fdc2472d3b3321b06b66",
      "object": "crawler",
      "account": {
        "type": "facebook-sync",
        "baseurl": "https://www.facebook.com",
        "cookies": [
          {
            "name": "c_user",
            "required": true
          },
          {
            "name": "xs",
            "required": true
          }
        ],
        "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+"
      },
      "name": "Facebook Pages Search Export",
      "slug": "facebook-pages-search-export",
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 20,
      "credits_per_row": 20
    },
    {
      "id": "a1d243d3824a1ac773414416eb80362d",
      "object": "crawler",
      "account": {
        "type": "sales-nav-sync",
        "baseurl": "https://www.linkedin.com",
        "cookies": [
          {
            "name": "li_at",
            "required": true
          },
          {
            "name": "JSESSIONID",
            "required": true
          }
        ],
        "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+"
      },
      "name": "Sales Navigator Leads Scraper",
      "slug": "sales-navigator-leads-scraper",
      "icon": "PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCI+PC9zdmc+",
      "is_public": true,
      "is_available": true,
      "is_premium": false,
      "has_issues": false,
      "max_concurrency": 10,
      "credits_per_row": 3
    }
  ],
  "next": null,
  "previous": null
}