---
title: Custom Web Scraper — Competitive Analysis Tool
type: article
created: '2026-04-05'
updated: '2026-04-05'
source_docs:
- raw/2026-03-25-asymmetric-marketing-call-132669649.md
tags:
- web-scraping
- competitive-analysis
- ai-tooling
- automation
- api
- hubspot
- salesforce
layer: 2
client_source: null
industry_context: null
transferable: true
---

# Custom Web Scraper — Competitive Analysis Tool

## Overview

Mark built a custom web scraper capable of extracting structured data from any website, including those that deploy anti-scraping protections. The tool is part of a broader shift toward an API-first, automation-heavy workflow that also encompasses HubSpot and Salesforce integrations.

This tool enables competitive intelligence gathering at scale — pulling pricing, product catalogs, and other structured data from competitor sites into a usable database format — without manual copy-paste work.

## Capabilities

- **Universal site access:** Circumvents common anti-scraping measures, so the tool is not limited to sites that expose a public API or permit crawling.
- **Structured output:** Returns scraped data in database form, ready for analysis or import into other systems.
- **Google Places integration:** Can query the Google Places API to enumerate businesses by category and geography (e.g., "every HVAC company in Milwaukee"), pulling all associated metadata.
- **Competitor product/pricing data:** Point the scraper at a competitor's e-commerce or SaaS site to extract product listings, pricing tiers, and related data.

## Primary Use Cases

| Use Case | Example |
|---|---|
| Competitor pricing analysis | Scrape a SaaS competitor's pricing page and track changes over time |
| Market mapping | Pull all businesses of a given type in a target geography via Google Places API |
| Product catalog extraction | Extract a competitor's full product/SKU list into a structured database |

## API-First Approach (Related Context)

The scraper is part of a broader tooling philosophy Mark described in the [[meetings/2026-04-05-asymmetric-marketing-call|April 2026 marketing call]]:

> "Everything I'm doing now in HubSpot, I'm doing through the API, and I'm not doing anything on the front end anymore. I'm finding that I can get a lot more done a lot faster, and also things that I could never have even done before."

This API-first approach applies across:
- **HubSpot** — automations and data operations previously requiring hours of UI work are now scripted
- **Salesforce** — fixes and updates that would have required specialist contractors are handled via API calls
- **Web scraping** — data extraction that previously required manual research or paid third-party tools

The reported productivity gain is approximately 3× output per working day compared to six months prior.

## Limitations & Considerations

- **Legal/ethical scope:** The intended use is data extraction (pricing, product info), not copying creative assets or layouts. Usage should remain within applicable terms of service and legal boundaries for the target site.
- **Maintenance:** Anti-scraping measures evolve; the circumvention logic may require updates as target sites change their defenses.
- **No UI:** This is a developer-operated tool, not a self-service interface for non-technical team members.

## Related Tools & Articles

- [[knowledge/integrations/hubspot-api-workflows|HubSpot API-First Workflows]]
- [[knowledge/integrations/salesforce-api-workflows|Salesforce API Workflows]]
- [[knowledge/tools/pomelli-google-labs|Pomelli — Google Labs Campaign Image Tool]]
- [[meetings/2026-04-05-asymmetric-marketing-call|Asymmetric Marketing Call — Website Rebuild & Campaign Status]]