Bulk Index and Crawl URLs to Google Search Engine Console

Today we will learn how to bulk Index URL to google search engine.

We will see manual and automated Indexing or Crawling techniques for helping your Website’s SEO optimization strategies.

You may find the need for the automation of this activity due to multiple reasons.

For example, the article is updated or article enhancement, etc.

You often try to index URLs to Google Search Engine using manual ways for your WordPress website or any custom websites.

Bulk Indexing will be a great SEO strategy you can adopt overall.

We shall cover the below aspects in today’s article,

Manual URL Indexing and Crawling

It is always difficult to perform indexing manually like using the below search console window option.

• https://search.google.com/search-console

The issue with Manual indexing

• To submit URLs, you must log in each time.

• Only authorized users, primarily Admin, can do indexing after logging in.

• It takes time, and if you have numerous URLs to send, you’ll have to submit each one manually.

• Crawling or indexing by Google can take some time ( delay may be longer than days or weeks)

• It’s conceivable that you forgot to index URLs (including modified or newly added URLs)

SEO Strategies – Automation for Url Index or crawling using Google Indexing API

Using an automation method, all of the above-mentioned challenges can be resolved.

We’ll develop a script to do authentication and authorization in this section.

Once authorized, the script can be updated with new URLs or existing modified URLs to submit or resubmit as posts are updated or resubmitted.

With this approach, you can submit multiple URLs to google as per the quota limit.

Quota limit is 200 URls as default.

Getting started – Bulk Index URLs to Google Search Engine

We shall cover the below set of steps needed to be performed,

• Create a service account
• Site Ownership- Add your service account as a site owner
• Writing a script and using an access token
• Submit URL Programmatically

A typical manual step involves below,

• Login to Google Search Console
• Submit each URL individually
• Wait for indexing to be effective

Before we start using the Indexing API, there are a few things that need to be set up,

Authorize Requests to Google Indexing API

Application requests must be authorized by an authenticated user who has access to that data.

• Request applications sent to the Indexing API must include an authorization token.

• Make sure to Enable Indexing API using the below link,
  • https://console.developers.google.com/flows/enableapi?apiid=indexing.googleapis.com&credential=client_key

Below we will cover high-level steps for achieving the Authorization.

Create a Project

Before you can send requests to the Indexing API, you need to tell Google about the client and activate access to the API.

You do this by using the Google Cloud Console to create a project, which is a named collection of settings and API access information, and register your application.

To get started using Indexing API, visit use the setup tool, which guides you through creating a project in the Google API Console, enabling the API, and creating credentials.

Create a service account

• Go to  Service accounts and Select your project else create a new project as explained above.

• Click Add Create Service Account, and enter a name and description for the service account. You can use the default service account ID, or choose a different, unique one. When done click Create.

• The Service account permissions (optional) section that follow is not required. Click Continue.

• On the Grant users access to this service account. You can add a specific role or keep the role optional.

• Once done you shall see your service account create as below,

• In the side panel, click on Action or: symbol. Please select Manage keys.

Select the format for your private key: JSON is recommended.

• Click Create. A new public/private key pair will be generated and downloaded to your machine;

Site Ownership – Add your service account as a site owner

For site ownership, please perform the below steps,

Prove as the Owner of the site 

The verification process of proving that you own the property needs to be done by you.

Please note that verified owners will have access to sensitive Search data for a site. Please click here for more details

Grant owner status to your service account 

The next step is to make your Service account(created in the above step) a site owner.

This step is an important step toward getting actual access to the Indexing API.

• Please Open Webmaster Central

• Click on the verified website.

• In the Verified owner list, click Add an owner.

• Provide your service account email as the delegated owner. You can find your service account email address in two places:
  • client_email – This is a field in the JSON private key in the JSON service account file which we downloaded in one of the above steps.

An email has the below pattern,

<your-service-account>@<your-project-name>.google.com.iam.gserviceaccount.com

Final Step- Bulk Submit URLs Programmatically

That’s all, you are now all set to set up the automation.

With required prerequisites to start invoking the Google Indexing API.

As a next step, please use any of the scripts of your choice(you are comfortable with) to call the Google Indexing API.

Please see the below article for the final steps where we will use a script to send the URL for Google indexing programmatically.

• Python script for Indexing the URL

• Using JAVA for indexing/Crawling the URL

• Node js example for indexing API

Please choose the language of your choice to script the indexing or crawling logic using automation techniques.

Do you have any comments or ideas or any better suggestions to share?

Please sound off your comments below.

Happy Coding !!

---

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published(2024) best practices and guidelines for software design and development.

---