Crawling budget. The concept and how to calculate

Crawling budget is a topic that is relatively rare to see in publications about SEO and marketing. So let's understand what it is and how to calculate it in more detail. 

Crawling budget. The concept and how to calculate

To understand the concept of a "crawling budget," you first need to understand what the term "crawling" in SEO even means.

Crawling is the process of crawling pages of sites by a crawler (a search robot) and their indexing to form search results.

The concept of "crawling budget" was introduced by Google, which does not mean the amount of money you invest in promotion. Instead, it's a limit of pages that a search bot crawls in a certain amount of time on your site. 

A crawling budget is an essential factor in promoting your site. The more pages on the site, the more attention should be paid to it because the search robot can spend it on copies of pages, erroneous pages, and important pages. However, it simply will not be enough.

Causes of lack of crawling budget

We distinguish three such reasons:

1. Not updated and irrelevant content.

   When indexed, search bots (as well as Google in general) pay attention to the relevance of information on the site. The less your site is updated, the less often it is visited by bots. The important point is that the number of pages on the site also depends on the amount of crawling budget allocated. On large sites, its shortage is felt much more often.

2. Errors of a technical part.

   If the site has technical errors, it will definitely affect all aspects of its promotion.

3. Краулер тратит краулинговый бюджет на неважные страницы.

   If too much budget is spent on such pages, the main ones may not get into the output.

We decided to take a closer look at the stages of determining a crawling budget, so you can use them to determine if there are errors in this direction and correct them. 

Stages of determining a crowding budget

Step 1: Number of pages in the index

Answer how many pages should be in the index. For this, you can use popular services such as Screaming Frog or Netpeak Spider (we recommend you to read our article about free analogs of these resources).

Step 2: Number of times the robot crawled your site

At this stage, you need to find out how many times a search robot crawled your site.

It can be done in one of two ways, which is more suitable for your destination:

• Method 1. Use the Google Console service.

  

  Quite an uncomplicated method of collecting this kind of information, but we should note right away that it does not have a high level of accuracy, and there are some disadvantages in its use.

  Advantages of this method:

  • Not difficult to use.
  • Good for sites with fewer pages than 50 thousand (most of them have a clear structure so that the bots index all).

  Cons:

  • Not accurate enough for large sites (where the number of pages is more than 50k)
  • The information is only submitted for 90 days.

  How to use this service:

  1. Go to the "Settings" section.
  2. Scan statistics
  3. Open the report

  Here, the indicator we need is the "total scan requests." But in addition, you can find out information about specific pages that the bot scans, the percentage of responses, and the number of correct and incorrect pages.

• Method 2: Analyzing server logs

  This method is more complicated but, in contrast to the previous, accurate and more in-depth.

  Logs are files with information on the operation of a PC or server that collects data on the IP address, GET-request, etc. You can use LogViewer or Screaming Frog Log Analyzer for such analysis. 

  Advantages of this method:

  • Suitable for large sites
  • It is accurate, unlike the previous, and helps to identify problems such as faulty site structure, as well as errors with code 404, 300, 500
  • In addition to the calculation of the crawling budget, this analysis shows by one or different paths "walks" the crawler, the time of arrival at one page or another, the number of visits to the same page, etc.
    Cons:

  Cons:

  • More difficult to apply.
  • Requires special knowledge.

Step 3: Calculating the Crawling Budget Directly

Before calculating the crawling budget, you should first find out the average number of robot visits per day. To do this, you should use the following formula:

RA/day = number of robot appeals / period for which the appeals rate was taken, where:

• RA/day is the average number of robot requests per day
• the number of robot referrals is the total number of times the robot accessed your site for crawling
• the period for which the conversion rate is taken is the analyzed period in days.

For example, if you used Google Console, the data is shown for 90 days. Suppose the crawlers accessed your site 5000 times. Thus, according to the formula, the calculation will look like this:

5000 / 90 = 55,56

Now, based on this data, you can calculate the crawling budget:

Crawling budget = average number of pages in the index / average robot accesses per day, where

• Number of pages in the index - the number of pages on the site that should be seen and scanned by the robot
• The average number of robot referrals per day - the sum of robot referrals for the period of crawling 

Suppose we have found out with the help of a crawling service that the number of such pages should be 150:

150 / 55,56 = 2,7

Evaluation of the data:

• If the figure is less than or equal to 3 - the crawling budget on your site is enough
• 4-10 - in this case, it is worth analyzing the site for errors in the distribution of the crawling budget
• More than 10 - the crawling budget on your site is not enough. Also, such a result shows a high probability of technical errors or the content itself.

If you have realized that the crawling budget is not enough, we recommend you contact usto conduct a comprehensive audit of your site and work on eliminating errors.