Question about gathering search results for a phrase with Python. I wrote a simple script for my personal SEO purposes that would go out and collect the top 30 search results on Google and then return a list of those URL's. The trouble is everytime I try to run my script Google starts returning an error about the search not being allowed. The URL I'm using works fine if entered manually but Google seems to know I'm trying to get access through a script. Is there a way around this or an API that I should be using? My searches into the subject haven't given me much.
betatype 0 Newbie Poster
mn_kthompson 3 Junior Poster
Google does not allow searches from any devices that does not provide a commonly used agent string. When you gather data using python the agent string is something like "python 2.6". Google doesn't allow that because they don't want people using scripts to gather their search results.
The answer is to spoof your agent string. I recently wrote up a script that does something similar to what you're trying to do. I wrote a blog entry about it and included sample code that you might want to check out.
http://blackfistsecurity.blogspot.com/2009/01/improving-your-google-hacking-with.html
Be a part of the DaniWeb community
We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.