php - How to fetch the amount of results Google returns when you query "site: example.com"?

2

I know this question is asked a lot, but all the answers are outdated because the Search API is disabled by Google.

Google used to have a Search API and with that API it was possible to search in Google using an API. Most answers on StackOverflow link to the deprecated documentation of that API, but because Google turned off that API, I can't solve the issue using the API.

I tried to fix the issue by crawling the google result page and fetching the count from the "about x results found in y seconds" text. But as far as I know, this is restricted because of Google Policy. https://support.google.com/webmasters/answer/66357?hl=en

Anybody has an idea how to fix this issue without breaking the Google Policy? I work with the Symfony Framework in PHP, but I won't mind it if I need to load in a javascript library or a command line tool?

221

Answer

Solution:

Now that the Search API is gone, it is no longer possible to lawfully scrape Google's general search results.

From the title to your question, it appears that you're looking to scrape a specific site. That can still be done lawfully using the Custom Search JSON/Atom API at https://developers.google.com/custom-search/json-api/v1/overview.

People are also looking for solutions to the problem: PHP Fatal error in laravel DatabaseSeeder

Source

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.

Ask a Question

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

Similar questions

Find the answer in similar questions on our website.