Search google and download results (Sentiment Analysis)

[Hooovahh]

---

@James.M wrote:

Google has implemented lots of safeguards to ensure that it's search engine can't be scraped.

---

Well sorta, I needed to do some basic stuff once so I wrote LabVIEW code that would do the search and return the results.  After  a couple dozen searches Google forces you to prove you are a human by entering a captcha.  So I had my LabVIEW code detect the captcha, and show it to me, so I could enter it, and have the code continue.

I don't feel great about it, but I only needed it for a few hundred searches one day.  I'm sure Google has bigger script kiddies to worry about.

[James.M]

Did you use the Vision OCR? Wouldn't you need to train a bunch of different character sets in order to catch captchas?

[MilanR]

There does seem to be an approach for full web searches provided by Google's Custom Search feature if you did not want to deal with Captchas, etc: http://stackoverflow.com/questions/4082966/what-are-the-alternatives-now-that-the-google-web-search-...

I haven't attempted it myself, but according to that stackoverflow thread you can do 100 queries per day for free on Google. Also other search engines appear to have APIs with different limits. For example Bing currently offers 5000 queries a month for free: http://datamarket.azure.com/dataset/bing/search

[Hooovahh]

---

@James.M wrote:

Did you use the Vision OCR? Wouldn't you need to train a bunch of different character sets in order to catch captchas?

---

I didn't have it answer the OCR, just prompt me to answer it.  

[James.M]

Oh, I gotcha. Then you shouldn't feel bad. You weren't bypassing their captcha system you just weren't typing in every single search by hand.

[MilanR]

I was curious so I put together an example using the Bing Search API available here: https://datamarket.azure.com/dataset/bing/search

After signing up I clicked the "EXPLORE THIS DATASET button" and at the top of the Service Explorer window clicked 'Show' next to Primary Account Key to get my account key.

  

After Getting the account key I was able to use the following VI to make search queries:

The following blogspot described the authentication format used which wasn't too bad but not easy to find either: https://coderwall.com/p/d5a8cg/bing-search-api-authentication

[Hooovahh]

Oh this is so cool, thanks for this.  Now I can sleep with a clean conscience knowing I'm not cheating a search engine out of page views, if even that is what I was doing.  I'll have to do some testing to see if my Google results and Bing results return the same first result, which is what I was using in my previously mentioned code.