How To Automate Competitor Store Scraping with Dumpling AI , Monday.com, Chatgpt and Make.com.

Keeping track of your competitors’ product offerings, pricing, and descriptions can give your business a strategic advantage. Manually visiting each competitor’s website and extracting product details can be time-consuming and inefficient. This tutorial will walk you through an automated solution using Make.com that scrapes competitor store pages, extracts key product details, and organizes them in Monday.com for easy analysis.

Benefits of This Automation:

Saves time: Automates the data extraction process, eliminating the need for manual work.
Improves accuracy: Reduces human errors and ensures up-to-date data.
Enhances decision-making: Helps businesses stay competitive with real-time insights.

Overview

The automation begins by monitoring a board in Monday.com. When an item (e.g., a competitor’s website URL) is added or updated, the workflow is triggered.

1. Set Up the Monday.com Trigger

Module: monday: WatchBoardItems

Add a “Watch Board Items” module in Make.com.
Connect it to your Monday.com account.
Choose the board where competitor store URLs are stored.
Set the module to watch for “New and Updated Items.”
Limit the results to 1 to process each update individually.

Step 2: Crawling the Competitor’s Store with Dumpling AI

Module: DumplingAI: crawl website

Once a competitor’s website URL is added to Monday.com, the automation uses Dumpling AI to crawl the page and extract content.

Steps:

Add a “Crawl Website” module from Dumpling AI.
Connect your Dumpling AI account.
Map the URL field from the Monday.com module to this step.
Set crawl depth to 2 (to capture product pages linked from the homepage).
Select Markdown as the output format for easy parsing.

Step 3: Extracting Product URLs Using OpenAI

Module: openai-gpt-3:CreateCompletion

After crawling the competitor’s store, the automation extracts product links using OpenAI’s GPT-4o model.

Steps:

Add a “Create Completion” module from OpenAI.
Connect your OpenAI API.
Use a prompt that instructs the AI to extract 10 product URLs from the crawled content.(optional, you can extract all the URLs scraped by the previous Dumpling AI module)
Set the response format to JSON for structured output.
Parse the AI response to extract the list of product URLs.

Example prompt:”Extract 10 product URLs from this page: {{4.results[].content}}. Output in JSON format: { ‘product’: [‘url1’, ‘url2’] }”

4. Scrape Individual Product Pages

Now that we have product URLs, the automation loops through them and scrapes product details.

Modules: BasicRepeater + DumplingAI:scrape URL

Repeater: Runs 3 times (adjustable) to process multiple URLs.
Scraper: For each URL, extract product details in markdown.
Mapper:
- URL: {{6.result.product[15.i]}} (loops through GPT-4’s extracted URLs).
- Render Javascript: Enable to load dynamic content (e.g., prices loaded via JS).

Pro Tip: Increase the repeater’s repeats value if you need to scrape more than 3 products.

Repeater step:

Dumpling AI step:

5. Aggregate Data for Summarization

Module: TextAggregator
Combine all scraped product content into a single text block.

Input: {{14.content}} (scraped markdown from all product pages).

6. Generate Product Summaries with GPT-4

Module: openai-gpt-4:CreateCompletion
GPT-4 creates concise summaries with prices, features, and unique selling points.

Sample Prompt:”Summarize each product. Include:

– Product name

– Short summary

– Key features

– Price/range

Example: XYZ Wireless Headphones ($149-$179)…”

7. Update Monday.com with Competitor Data

Module: monday: ChangeMultipleColumnValues

Push the AI-generated summaries back to your Monday.com board.

Parameters:
- Item ID: Map to the Monday.com item (e.g., 1800150412).
- Column: Update the “Products info” column (long_text_mkmncwbq) with {{18.result}}.

Pro Tips for Success

Test Crawling Depth: Ensure the competitor’s site structure aligns with your depth setting.
Refine Prompts: Adjust GPT-4 prompts to match your industry’s terminology.
Error Handling: Use Make.com’s error workflows to retry failed scrapes.

Conclusion

With this Make.com automation, you can effortlessly track competitor products, analyze trends, and make informed business decisions. This workflow ensures you have real-time insights without spending hours manually collecting data.