With all due respect: Spoken like someone who's never actually tried to do scraping on a large scale. This line just cracks me up:
> If people can still access the same data just by going to Amazon with their browser, I don't see any fundamental reason why they wouldn't be able to get the data if they really wanted to.
Just so naive, like a little kid saying they can build a rocket to the moon because they have a cardboard tube and some petrol.
In its own right web-scraping is difficult, and when the adversary is intentionally fighting against it it can be almost impossible. Modern web-scrapers often have to be smart enough to incorporate almost a full web-stack of technology, since if you lack JS or CSS rendering it is trivial to hide content from you or worse give you bad content the users won't even see (and also keep in mind the scraper has to "understand" CSS using its own logic, as it cannot "view" the result).
They can also dynamically alter how the page is rendered (e.g. if you're treating DOM like a tree and going down to a specific branch, it is trivial to re-order the HTML to break that). CSS effectively removes the relationship between HTML tag order and location on the page.
If I had a team of engineers working full time, I could likely maintain a 90%+ good Amazon web-scraper after about a year of pure development on it (likely using something like WebKit as a starting point). But nobody puts in this type of effect because sooner or later Amazon would sue you for something obscure and effectively win via lawyers fees alone.
An easier solution is just to hire Mechanical Turk-like people to look at pages, note down the price, and pay them 1c/price. It would cost an absolute ton of money, but you're able to start operating almost immediately (no prior development overhead). Plus you could limit it to Amazon's top 100,000 items and nothing more.
> If people can still access the same data just by going to Amazon with their browser, I don't see any fundamental reason why they wouldn't be able to get the data if they really wanted to.
Just so naive, like a little kid saying they can build a rocket to the moon because they have a cardboard tube and some petrol.
In its own right web-scraping is difficult, and when the adversary is intentionally fighting against it it can be almost impossible. Modern web-scrapers often have to be smart enough to incorporate almost a full web-stack of technology, since if you lack JS or CSS rendering it is trivial to hide content from you or worse give you bad content the users won't even see (and also keep in mind the scraper has to "understand" CSS using its own logic, as it cannot "view" the result).
They can also dynamically alter how the page is rendered (e.g. if you're treating DOM like a tree and going down to a specific branch, it is trivial to re-order the HTML to break that). CSS effectively removes the relationship between HTML tag order and location on the page.
If I had a team of engineers working full time, I could likely maintain a 90%+ good Amazon web-scraper after about a year of pure development on it (likely using something like WebKit as a starting point). But nobody puts in this type of effect because sooner or later Amazon would sue you for something obscure and effectively win via lawyers fees alone.
An easier solution is just to hire Mechanical Turk-like people to look at pages, note down the price, and pay them 1c/price. It would cost an absolute ton of money, but you're able to start operating almost immediately (no prior development overhead). Plus you could limit it to Amazon's top 100,000 items and nothing more.