How to Use Dumpling AI’s Scrape URL Endpoint in Make.com

The Scrape URL module in Dumpling AI enables you to extract data from any webpage in a clean, structured, and customizable format. Whether you need plain text, HTML, or screenshots, this module simplifies web data collection and ensures efficiency. In this guide, we’ll explore its configuration, inputs, outputs, and potential use cases, and provide a detailed setup process in Make.com using the HTTP Request module.

Understanding the API

Endpoint

• URL: POST /api/v1/scrape

Headers

• Content-Type: application/json
• Authorization: Bearer <API_KEY>

Request Body Parameters

1. url (Required): The URL of the webpage to scrape.
2. format (Optional): Specifies the output format (markdown, html, screenshot).
3. cleaned (Optional): Boolean value to clean unnecessary HTML or ads.
4. renderJs (Optional): Enables or disables JavaScript rendering (default: true).

Step-by-Step Guide to Set Up in Make.com

Step 1: Add the HTTP Request Module

1. Open your Make.com scenario.
2. Add the HTTP Request module to your workflow.

Step 2: Configure the HTTP Request Module

1. Method: POST.
2. URL: Enter the API endpoint:
   https://app.dumplingai.com/api/v1/scrape.
3. Headers:
   • Content-Type: application/json
   • Authorization: Bearer <Your_API_Key>
4. Request Body:
   Use JSON to specify the scraping options. 

Step 3: Test the Module

1. Click Run Once to test the configuration.
2. If successful, the response will include:
   • title: Title of the webpage.
   • metadata: Metadata of the page (e.g., description, keywords).
   • content: Scraped data in the specified format.

Outputs and Applications

Outputs

1. title: The webpage’s title.
2. metadata: Metadata information, such as meta tags and descriptions.
3. content: Scraped data in the chosen format (markdown, html, or raw content).

Applications

1. Content Extraction
   • Extract blog posts, articles, or product descriptions for analysis.
2. SEO Analysis
   • Gather metadata from competitor websites to improve your SEO strategy.
3. Web Data Research
   • Scrape structured content for reports or data-driven decisions.

Conclusion

Dumpling AI’s Scrape URL module is a versatile tool for extracting clean and structured data from web pages. By integrating this feature with Make.com’s HTTP Request module, you can automate complex web scraping tasks with ease. Use it for content analysis, SEO monitoring, or research, and take advantage of its flexibility in handling different formats and rendering options. Start scraping smarter today!