How to scrape data and send to a webhook

​

Goal

In this how to guide, we will go over how to scrape data from a website and send it to a webhook. For this example, we will use https://books.toscrape.com/ as the data source. The goal is to scrape data from categories Poetry and Travel and send them to a webhook. We want to be able to do this on demand or on a schedule.

​

Step by step

​

1. Create a project and API

• Create new project.

• Create new API (books) with the following code. This API receives a param category and scrapes the books in this category. Notice that we are using Optimized extractors here. Notice also that the API returns an array of of the scraped books data.

• Run it and make sure everything is working well. If you need a test param.

{
  "category": "Novels"
}

• Deploy the project.

import { BrowserContext, Page } from "@intuned/playwright-core";
import { extendPlaywrightPage } from "@intuned/sdk/playwright";

interface Params {
  // Add your params here
  category: string;
}

export default async function handler(
  params: Params,
  _playwrightPage: Page,
  context: BrowserContext
) {
  const page = extendPlaywrightPage(_playwrightPage);

  await page.goto("https://books.toscrape.com/");

  // playwright logic
  await page.getByRole("link", { name: params.category }).click();

  // @intuned/sdk helper!
  const result = await page.extractArrayOptimized({
    itemEntityName: "book",
    label: "books-scraper",
    itemEntitySchema: {
      type: "object",
      properties: {
        name: {
          type: "string",
          description: "name of the book",
          primary: true,
        },
        price: {
          type: "string",
          description: "price of the book. An example is £26.80",
        },
      },
      required: ["name", "price"],
    },
  });

  return result;
}

​

2. Create a job with webhook sink

Jobs are a way to schedule recurring or batched/grouped executions. For more info about Jobs, checkout Jobs.

In this example, we know that we need to call the books api twice (one with category Poetry and another with category Travel) and send the result to a webhook. We will create a job that does this on demand.

Creating jobs can be done via UI or API. For this example, we will use the UI. Checkout Jobs API overview for more info.

• Get a webhook url, for testing, you can use https://webhook.site/ to get a temp url. In a real scenario, you will use your own webhook url and persist the data to store.

• Go to the Jobs tab in the UX and create a new job with the following config. Don’t forget to replace <YOUR_WEBHOOK_URL> with your webhook url.

{
  "id": "books-webhook",
  "configuration": {
    "runMode": "Order-Irrelevant",
    "maxConcurrentRequests": 5,
    "retry": {
      "maximumAttempts": 5
    }
  },
  "sink": {
    "type": "webhook",
    "url": "<YOUR_WEBHOOK_URL>"
  },
  "payload": [
    {
      "apiName": "books",
      "parameters": {
        "category": "Poetry" 
      }
    },
    {
      "apiName": "books",
      "parameters": {
        "category": "Travel"
      }
    }
  ]
}

​

3. Trigger the job

• Now that you have created a job, you can trigger it manually. This will run the job immediately and send the result API data to the webhook.

• You will have 1 webhook call for each API in the payload.

• You can look at the webhook payload below. It includes API name, parameters, runId, result and more.

{
  "apiInfo": {
    "name": "books",
    "parameters": { "category": "Poetry" },
    "runId": "vLCJ-j5ymLoi4DM",
    "result": {
      "status": "completed",
      "result": [
        { "name": "A Light in the Attic", "price": "£51.77" },
        { "name": "The Black Maria", "price": "£52.15" },
        { "name": "Shakespeare's Sonnets", "price": "£20.66" },
        { "name": "Olio", "price": "£23.88" },
        { "name": "You can't bury them all: Poems", "price": "£33.63" },
        { "name": "Slow States of Collapse: Poems", "price": "£57.31" },
        {
          "name": "Untitled Collection: Sabbath Poems 2014",
          "price": "£14.27"
        },
        { "name": "Poems That Make Grown Women Cry", "price": "£14.19" },
        { "name": "Night Sky with Exit Wounds", "price": "£41.05" },
        { "name": "salt.", "price": "£46.78" },
        {
          "name": "Quarter Life Poetry: Poems for the Young, Broke and Hangry",
          "price": "£50.89"
        },
        {
          "name": "Out of Print: City Lights Spotlight No. 14",
          "price": "£53.64"
        },
        { "name": "Les Fleurs du Mal", "price": "£29.04" },
        { "name": "Howl and Other Poems", "price": "£40.45" },
        {
          "name": "Leave This Song Behind: Teen Poetry at Its Best",
          "price": "£51.17"
        },
        {
          "name": "The Collected Poems of W.B. Yeats (The Collected Works of W.B. Yeats #1)",
          "price": "£15.42"
        },
        { "name": "The Crossover", "price": "£38.77" },
        { "name": "Booked", "price": "£17.49" },
        { "name": "Twenty Love Poems and a Song of Despair", "price": "£30.95" }
      ],
      "statusCode": 200
    }
  },
  "workspaceId": "9c67b0f1-6698-480d-9eca-0dd26ccd93f2",
  "project": {
    "name": "new-project",
    "id": "ea4d13e8-c434-4bf8-9723-3b97f03a1c8b"
  },
  "projectJob": { "id": "books-webhook" },
  "projectJobRun": { "id": "f3b3eef1-c201-4bad-9f32-8eb8b04fbae9" }
}

​

4. Create a scheduled job

Lets assume that you need a job to be ran every day so you can keep your internal store updated with the latest books in some categories.

To do this, you will need to create a new job with a schedule configuration. Here is an example job config:

{
  "id": "books-webhook",
  "configuration": {
    "runMode": "Order-Irrelevant",
    "maxConcurrentRequests": 5,
    "retry": {
      "maximumAttempts": 5
    }
  },
  "schedule": {
    "intervals": [
      {
        "every": "1d"
      }
    ]
  },
  "sink": {
    "type": "webhook",
    "url": "<YOUR_WEBHOOK_URL>"
  },
  "payload": [
    {
      "apiName": "books",
      "parameters": {
        "category": "Poetry" 
      }
    },
    {
      "apiName": "books",
      "parameters": {
        "category": "Travel"
      }
    }
  ]
}

​

Related