Can anyone recommend scraping adapters (businesses or tech) that are robust to this sort of thing? I'm talking about something higher level than, say, Beautiful Soup -- something you can configure to point at an endpoint, essentially request a sql row subscription from it, and not have to mind it too much.
Both the traversal/retrieval and data-interpretation parts seem to have interesting aspects when you consider current website design. Some websites make themselves hard even for humans to read (consider why safari reader mode exists).
This seems like a potentially valuable service, in the sense of being a schlep. I wonder how many places have home-grown scraping efforts as part of their business and how annoying it is for them to maintain.
At work we use Mozenda[0] with success, though I can't tell you much beyond that because I'm not involved with that project. I've also heard of Agenty[1].
Both the traversal/retrieval and data-interpretation parts seem to have interesting aspects when you consider current website design. Some websites make themselves hard even for humans to read (consider why safari reader mode exists).
This seems like a potentially valuable service, in the sense of being a schlep. I wonder how many places have home-grown scraping efforts as part of their business and how annoying it is for them to maintain.