Nested scheduling

A best practice with Intuned APIs is to keep them as small as possible. This approach ensures more efficient retries and results in increased reliability and flexibility.

However, there are scenarios where this practice has limitations. Consider the case of scraping information about all new books added to https://books.toscrape.com/. For each book, you might need details such as UPC, Price, Availability, Description, Image, and Category. Since this information is not available on the main page, scraping each book’s page individually is necessary.

If you were to write a single API to perform all these tasks — going to the list, and for every book, going to the subpage to retrieve information — the execution time would be lengthy, and any failure would require restarting the entire process.

This is where nested scheduling becomes important. Nested scheduling allows you to schedule one API that can append more APIs to the job/queue payload. In the example mentioned, you can schedule a job with one API to scrape the list of books, and for each book, append another API to scrape the specific book page. In the event of an API failure, you only need to restart the failed APIs, not the entire job. To implement this, a helper function is provided, facilitating the appending of more APIs to the job/queue payload.

Here is an example:

import { extendPayload } from "@intuned/sdk/runtime";

export default async function handler(
    ....
) {
    //
    extendPayload({
        api: "api-to-extned",
        parameters: {
            param1: "val1"
        }
    });
}

This way a Job/Queue’s payload that contains one API, can result in executing multiple APIs.