Web scraping services turn public pages into structured records your business can store, compare and trust: prices, product specs, public directories, listings, news metadata or regulatory notices. The valuable part is not a one-off spreadsheet. It is a collector that keeps running, notices when a site changes, and refuses to deliver a row it cannot justify.
Contents
At CodeWiz we build that collector as software: scheduled crawls, parsers, validation and a delivery path into your database, sheet or API. We also draw a hard line around what may be collected. If a site’s robots.txt, terms or access controls say no — or if the data is personal and you have no legal basis — we do not build a workaround. We use the official API, ask for permission, or we stop.
The service is for retailers watching public prices and assortments, for marketplaces that need a catalogue they do not control, and for teams that keep pasting from the web into a CRM. How scraped intelligence supports commercial decisions is a separate article. This one is about the pipeline, the parsing and the rules.
For scraping, open-source tools are the default when they fit the job.
Web Scraping Services for Automated Collection
Web scraping automation is a pipeline with a schedule, a queue and a memory. A list of start URLs goes in. Fetchers download pages at a polite rate. Parsers turn HTML or embedded data into records. Validators accept or quarantine each record. A sink writes the accepted rows to PostgreSQL, a CSV drop, a webhook or your existing app. The next run updates changed records instead of appending an endless duplicate history.
What a production collector includes
- A source registry: start URLs, allowed paths, the parser version and who asked for the data.
- A queue, so a slow site cannot block every other source, and a retry policy for timeouts and transient HTTP errors.
- Raw storage of the fetched page for a limited time, so a parser bug can be replayed without downloading again.
- Change detection, so you hear about a new price or a removed listing rather than re-reading the whole archive.
- An identified user agent with a contact URL, a documented schedule and a concurrency cap per host.
JavaScript-rendered pages are fetched with a real browser only when the content is public and the rules allow it. The same identity, rate limit and robots.txt check apply. A headless browser is not a disguise. Where the site offers an API or a product feed, we take that instead.
Structured Data Parsing
Structured data parsing is the step that turns a page into fields with types: a decimal price, a currency, an availability enum, a timestamp, a stable external id. We prefer sources that are already structured. JSON-LD and microdata on a product page, a public JSON response the page itself calls, an XML or CSV feed. HTML is the fallback, read with CSS selectors or XPath anchored to stable markup, not to the colour of a button.
| Source | When we use it | Why it wins or loses |
|---|---|---|
| Official API | Whenever the publisher offers one you are allowed to use | Stable fields and documented rate limits; may be paid or incomplete |
| Feeds (CSV, XML, JSON) | Shops and partners who publish a catalogue | Easy to validate; you depend on their update frequency |
| JSON-LD and microdata | Pages that already describe a product, article or organisation | Cleaner than scraping layout; coverage is often partial |
| HTML parsing | Public pages with no API, when the rules allow collection | Breaks when the layout changes; needs monitoring |
Selectors are versioned with the parser. Kept raw pages let us fix a parser after a redesign and reprocess, instead of discovering a week later that every price became null. A field is extracted by a rule we can test, or the record goes to review.
Content Extraction from Messy Pages
Content extraction pulls the article body, the specification block, the branch list or the table out of the navigation and the footer. For articles we keep the title, date and canonical URL. For specification tables we keep the row labels, because a bare “12” is useless without “warranty, months”.
PDFs show up often: price lists, public tenders, manuals. We extract text and tables, and we OCR scanned pages only when the document is public and in scope. A low-confidence line is a flagged cell, not a silent number in the database.
We store the source URL, the fetch time and the fields you asked for. We do not republish full articles, images or product copy. A knowledge base over documents you own is a different build, covered in our article on RAG and prompt engineering. Scraping collects public structured inputs, and only where collection is allowed.
Data Validation Before Anyone Trusts the Numbers
Every record is checked against a schema before it goes downstream: types, required fields, currencies, URL shape and ranges you define. A price of zero, or a price that jumped by an absurd factor overnight, is quarantined rather than treated as a new market fact.
Checks we put on every source
- Schema validation, so a missing name or a price with a currency symbol still attached never reaches the warehouse table.
- Cross-field rules: the discount price is not above the list price, the end date is after the start date, the sum of a table matches its footer.
- Identity and deduplication: the same product seen on two URLs collapses to one record, and a rerun updates it.
- Freshness: each row carries the fetch time, and a source that has gone quiet raises an alert instead of serving last month’s prices as if they were current.
- Golden samples: a handful of URLs with known-good values, fetched on a schedule, so a layout change fails a test rather than failing your pricing meeting.
Quarantined rows stay visible, with the raw fragment and the rule that failed, so someone can fix the parser or accept an exception. The same explicit limits show up in our guide to API rate limiting: whether you are protecting your own API or calling someone else’s, the ceiling has to be real.
The Limits of Web Scraping Services
Legal and ethical limits are settled before the first request. The test is simple: would you send the site owner a description of the bot, the pages it reads and the rate it runs at? If not, we do not build it.
robots.txt and technical access rules
We fetch and honour robots.txt. A path marked Disallow for our user agent is not fetched, crawled sideways or loaded in a browser to get around the file. Crawl-delay, where the operator publishes one, is a floor on our schedule, not a suggestion. Google’s introduction to robots.txt is the plain-language reference for what the file means. Where robots.txt and a more specific written permission disagree, we follow the stricter of the two unless the site owner has told us otherwise in writing.
A polite collector identifies itself and stays slow enough that it is not a traffic incident:
from urllib.robotparser import RobotFileParser
import httpx
AGENT = "CodeWizBot/1.0 (+https://codewizdev.com/contact/)"
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
url = "https://example.com/catalog/page/3"
if not rp.can_fetch(AGENT, url):
raise SystemExit("blocked by robots.txt")
with httpx.Client(headers={"User-Agent": AGENT}, timeout=20.0) as client:
response = client.get(url)
response.raise_for_status()On top of that check, the worker queue enforces a per-host rate, a concurrency cap and a backoff when the site returns 429 or 503. We do not rotate identities, solve CAPTCHAs, bypass logins or walk around paywalls. Those are access controls. Hitting them means we stop and use an API, a licence or nothing.
Terms of use
If the terms forbid automated collection, we do not scrape that site, even when the page is technically reachable. “It is public” does not replace permission the publisher withheld. A feed or partner API is usually easier to keep alive, and we record that decision next to the source.
GDPR and personal data
A name, email, phone number or profile is personal data under the GDPR even on a public page. Collecting it needs a legal basis, a purpose and a retention limit. We avoid personal data unless your counsel already accepts a basis. Public business records and product facts are the usual scope. Private accounts, inboxes and anything behind a login are out. The European Commission’s overview of EU data protection is why “it was on a website” is not a processing ground.
Rate limits and load
We cap requests per host, honour Retry-After, and run large backfills outside peak hours when we know them. A collector that gets blocked has usually already been a bad guest. We would rather finish overnight at a gentle pace than pull a catalogue in minutes. The same ceiling applies in reverse to your own APIs, which is why the rate-limiting guide above sits next to this service.
Running a Scraper After Launch
Layouts change, fields move and feeds gain a column. Monitoring should distinguish “the site is down” from “the parser is wrong” from “the source asked us to slow down.”
- Alerts on fetch failures, on a sudden drop in extracted records, and on golden-sample mismatches.
- A parser version on every row, so you can see which code produced a number.
- Separate configuration per source, so fixing one site does not risk the others.
- A documented off switch. Stopping collection of a source should be a config change, not a code hunt.
- Retention for raw pages that is long enough to debug and short enough to justify.
The commercial side of this work is covered in how web data becomes something a business can act on. The pipeline here is what makes that data reliable enough for operations to trust.
Common Scraping Mistakes We Fix
- A script on a laptop, with no queue, no identity and no record of which pages were skipped.
- Ignoring robots.txt because the homepage is allowed and a subdirectory “probably” is too.
- Treating a blocked response as a puzzle to evade, instead of as a stop sign.
- Parsing prices with a regular expression that also matches the recommended-products carousel.
- No schema check, so a template change writes empty prices into a live pricing tool.
- Collecting personal profiles “just in case,” with no purpose, no basis and no deletion path.
- No source URL and no timestamp, so a disputed figure cannot be traced.
What Changes the Scope and Cost
We estimate after looking at the sources, not from a per-page rate card. The cost follows the difficulty of doing this properly:
- How many sites, how often, and whether each one needs its own parser.
- Whether an API or feed exists, which is usually cheaper to maintain than HTML.
- JavaScript rendering, logins you are authorised to use, or PDFs and OCR.
- How strict validation must be, and whether a person reviews quarantined rows.
- Delivery: a database table, an API, a shop import, or a report.
- Legal review of the sources, which we would rather do at the start than after a complaint.
How We Work
- Discovery: we list the decisions the data must support, the fields you need and the sources you have in mind.
- Rules check: we read robots.txt and the terms, flag personal data, and drop or replace any source we should not collect.
- Plan: we choose API, feed or HTML per source, plus the schedule, rate limits, schema and delivery.
- Build: we implement fetch, parse and validation, with golden samples from real pages.
- Test: we compare extracted records to the pages, try a rerun for duplicates, and confirm quarantines behave.
- Launch and support: we turn on monitoring, hand over the source list and the off switch, and fix parsers when layouts change.
Why CodeWiz
- Engineering, not a folder of scripts: queues, schemas, retries and monitoring are part of the delivery, because a scrape that only ran once is a demo.
- The legal line is explicit: robots.txt, terms, GDPR and rate limits are designed in. We will decline a source rather than “make it work.”
- Open-source first: Python, PostgreSQL and a queue you can run on your own server, without a per-row licence.
- It connects to the rest of your stack: shops, APIs, databases and the internal tools that will consume the data, built by the same team.
- Fair pricing: premium development without premium prices, from Banja Luka, working remotely with clients locally and internationally. More about the studio is on the about CodeWiz page.
Frequently Asked Questions
How much do web scraping services cost?
It depends on the number of sources, whether they offer an API or feed, how often you need fresh data, and how hard the pages are to parse. One stable catalogue with a feed is a small project. Many HTML sources, with validation and monitoring, cost more because each site is its own maintenance obligation. We quote after reviewing the actual sources.
Is web scraping legal?
It can be, and it can also be a breach of contract, a privacy problem or a technical abuse, depending on the site and the data. We only collect publicly available information where robots.txt and the terms allow it, we do not bypass access controls, and we avoid personal data unless there is a proper legal basis. If a source fails that test, we do not scrape it.
Do you ignore CAPTCHAs or blocks?
No. A CAPTCHA, a login wall, a paywall or an IP block is a signal to stop. We will not rotate identities or disguise the client to get past it. The alternatives are an official API, written permission, or dropping that source.
How do you keep the data accurate?
With a schema, cross-field checks, deduplication, fetch timestamps and a small set of known-good pages that are tested on a schedule. Rows that fail are quarantined with the reason. A layout change shows up as a failed sample or a sudden drop in record count, not as a week of silent bad prices.
How often can you refresh the data?
As often as the source can reasonably bear, and as often as your decision needs. A daily or nightly run is typical for prices and catalogues. We set the rate per site from its size, its robots.txt and how it responds, and we would rather be a few hours slower than get the collector blocked.
Can you deliver into our database or shop?
Yes. The usual sinks are PostgreSQL, a CSV or JSON drop, a webhook, or an import into the application you already run. We map fields to your schema and keep the source URL on each row so a value can be traced back to the page it came from.
Ready for Data You Can Actually Use?
Tell us which decisions the data should support and which sites you have in mind. We will say what can be collected, what should go through an API instead, and what we will not touch. Contact us to start, or see our web scraping services.