Goal
Lets assume that we want to scrape the books data from https://books.toscrape.com/, however, we are interested in more than the book name and we want to get the book name, UPC and the number of reviews on each book. If you go to the website, you will see that the info on the main page doesn’t contain this extra info (UPC as an example) and that we would need to navigate to each book details page to get this info.

Step by step
1. Create a project and required APIs
- Create new project.
-
Create
book-details.ts
API. This API take a parambookFullUrl
and then navigates to this url, scrapes the needed data (name, upc, numberOfReviews) from that page and returns it.
`book-details.ts` code
`book-details.ts` code
Run it and make sure everything is working well.
book-details.ts
Testing parameters
- Create
books-all.ts
API. This API scrapes the name and url for all the books on https://books.toscrape.com/. For each url, it calls theextendPayload
function. This function extends the payload of job run and add a new payload item to it. This new payload item will run as part of the same job run.
`books-all.ts` code
`books-all.ts` code
The API doesn’t take any parameters and gets the data on the first page. Run it and make sure everything is working well.
- Deploy the project.
2. Create a job
Jobs are a way to schedule recurring or batched/grouped executions. For more info about Jobs, checkout Jobs. In this scenario, we will schedule a job that runs daily with one payload item (api:books-all
). When this API run, it will extend the job run payload and include 1 payload item (api: book-details
) for each book on the main page.
Creating jobs can be done via UI or API. For this example, we will use the UI. Checkout Jobs API overview for more info.
-
Get a webhook url, for testing, you can use
https://webhook.site/
to get a temp url. In a real scenario, you will use your own webhook url and persist the data to store. -
Go to the Jobs tab in the UX and create a new job with the following config. Don’t forget to replace
<YOUR_WEBHOOK_URL>
with your webhook url.

3. Trigger the job
- Now that you have created a job, you can trigger it manually. This will run the job immediately and send the result APIs data to the webhook.
-
You will have 1 webhook call for each API in the payload. In this case, even though you have 1 payload item in the job payload config, you will see that the job run had 20 more - one for each payload item that was added by the
books-all
API. - You can look at the webhook payload below. It includes API name, parameters, runId, result and more.

Webhook payload
Webhook payload
