Warning Cookies are used on this site to provide the best user experience. If you continue, we assume that you agree to receive cookies from this site. OK

Crawling budget. The concept and how to calculate

today 09/09/2022
timer 15

Crawling budget is a topic that is relatively rare to see in publications about SEO and marketing. So let's understand what it is and how to calculate it in more detail. 

Crawling budget. The concept and how to calculate


To understand the concept of a "crawling budget," you first need to understand what the term "crawling" in SEO even means.

Crawling is the process of crawling pages of sites by a crawler (a search robot) and their indexing to form search results.

The concept of "crawling budget" was introduced by Google, which does not mean the amount of money you invest in promotion. Instead, it's a limit of pages that a search bot crawls in a certain amount of time on your site. 

A crawling budget is an essential factor in promoting your site. The more pages on the site, the more attention should be paid to it because the search robot can spend it on copies of pages, erroneous pages, and important pages. However, it simply will not be enough.

Causes of lack of crawling budget

We distinguish three such reasons:

  1. Not updated and irrelevant content.

    When indexed, search bots (as well as Google in general) pay attention to the relevance of information on the site. The less your site is updated, the less often it is visited by bots. The important point is that the number of pages on the site also depends on the amount of crawling budget allocated. On large sites, its shortage is felt much more often.

  2. Errors of a technical part.

    If the site has technical errors, it will definitely affect all aspects of its promotion.

  3. Краулер тратит краулинговый бюджет на неважные страницы.

    If too much budget is spent on such pages, the main ones may not get into the output.

We decided to take a closer look at the stages of determining a crawling budget, so you can use them to determine if there are errors in this direction and correct them. 

Stages of determining a crowding budget

Step 1: Number of pages in the index

Answer how many pages should be in the index. For this, you can use popular services such as Screaming Frog or Netpeak Spider (we recommend you to read our article about free analogs of these resources).

Step 2: Number of times the robot crawled your site

At this stage, you need to find out how many times a search robot crawled your site.

It can be done in one of two ways, which is more suitable for your destination:

  • Method 1. Use the Google Console service.


    Quite an uncomplicated method of collecting this kind of information, but we should note right away that it does not have a high level of accuracy, and there are some disadvantages in its use.

    Advantages of this method:

    • Not difficult to use.
    • Good for sites with fewer pages than 50 thousand (most of them have a clear structure so that the bots index all).


    • Not accurate enough for large sites (where the number of pages is more than 50k)
    • The information is only submitted for 90 days.

    How to use this service:

    1. Go to the "Settings" section.
    2. Scan statistics
    3. Open the report

    Here, the indicator we need is the "total scan requests." But in addition, you can find out information about specific pages that the bot scans, the percentage of responses, and the number of correct and incorrect pages.

  • Method 2: Analyzing server logs

    This method is more complicated but, in contrast to the previous, accurate and more in-depth.

    Logs are files with information on the operation of a PC or server that collects data on the IP address, GET-request, etc. You can use LogViewer or Screaming Frog Log Analyzer for such analysis. 

    Advantages of this method:

    • Suitable for large sites
    • It is accurate, unlike the previous, and helps to identify problems such as faulty site structure, as well as errors with code 404, 300, 500
    • In addition to the calculation of the crawling budget, this analysis shows by one or different paths "walks" the crawler, the time of arrival at one page or another, the number of visits to the same page, etc.


    • More difficult to apply.
    • Requires special knowledge.

Step 3: Calculating the Crawling Budget Directly

Before calculating the crawling budget, you should first find out the average number of robot visits per day. To do this, you should use the following formula:

RA/day = number of robot appeals / period for which the appeals rate was taken, where:

  • RA/day is the average number of robot requests per day
  • the number of robot referrals is the total number of times the robot accessed your site for crawling
  • the period for which the conversion rate is taken is the analyzed period in days.

For example, if you used Google Console, the data is shown for 90 days. Suppose the crawlers accessed your site 5000 times. Thus, according to the formula, the calculation will look like this:

5000 / 90 = 55,56

Now, based on this data, you can calculate the crawling budget:

Crawling budget = average number of pages in the index / average robot accesses per day, where

  • Number of pages in the index - the number of pages on the site that should be seen and scanned by the robot
  • The average number of robot referrals per day - the sum of robot referrals for the period of crawling 

Suppose we have found out with the help of a crawling service that the number of such pages should be 150:

150 / 55,56 = 2,7

Evaluation of the data:

  • If the figure is less than or equal to 3 - the crawling budget on your site is enough
  • 4-10 - in this case, it is worth analyzing the site for errors in the distribution of the crawling budget
  • More than 10 - the crawling budget on your site is not enough. Also, such a result shows a high probability of technical errors or the content itself.

If you have realized that the crawling budget is not enough, we recommend you contact usto conduct a comprehensive audit of your site and work on eliminating errors.

Структура страницы
Stay Connected
Подпишитесь на нашу рассылку и получайте информацию о выходе новых статей, информацию об эксклюзивных скидках и другое
Или же подпишитесь на наш Telegram, чтоб всегда быть в курсе наших новостей.

Recent posts

timer 8

How to work in the absence of electricity

Today, due to missile attacks, each of us faces daily blackouts or anxiety due to possible power outages. How to work in such conditions? Consider this article.

timer 10

Ошибки в оформлении интернет-магазина - работа с корзиной

More and more online stores are opening every day, and everyone wants to do better, more conveniently, and stand out from the competition. We begin to analyze the main mistakes made in the development and design of an online store. And let's start with such an element as a basket.

The main mistakes of your online store - working with the basket

According to the Baymand Institute, 69.99% of users abandon carts without making a purchase.

The reasons for this may be as follows:

  • Analysis of goods and prices on other sites
  • Readiness to buy later
  • Studying information about the company

These factors only depend a little on your site and how the basket is designed. They will always be. But there are several mistakes allowing you to lose your customers additionally. Let's talk about them in more detail.

The main mistakes in the process of choosing a product and directly adding it to the cart

  1. Shopping cart in an unexpected place

    Every website owner (or web designer) wants to make their website unique and creative. But we must not forget that the user already has a certain stereotype about the site's overall structure, in particular, the location of the basket and its appearance.

    You should not place the basket where it will not be looked for (upper right corner). It is also inappropriate to experiment with variations of the basket (replacing it with other icons or leaving only the word).

  2. Register to add to the cart

    Of course, if a user comes to your site knowing he would buy goods from you because there are no analogs, he will go through such registration. But, since we know that the competition in the online trading market is huge, unnecessary actions can push a potential buyer to choose competitors.

    It is much more logical to leave the registration procedure directly to the ordering procedure.

  3. The button does not work or does not work correctly

    This is a fairly obvious point, but it is worth taking care not only that the tool is present but also that it performs its function. When you click the "Add to Cart" button, the buyer should see changes in the button's color, and the corresponding window should also open.

    We recommend adding a "Back to shopping" button or similar to the cart, as it is likely that the user, after going to the cart, will remember that he forgot to add something else or want to change his choice, or check if something needs to be ordered. If there is no such button, a person may have an opinion that his choice will not be saved if he goes to the main menu, or he simply does not want to perform unnecessary actions.
  4. Product information

    Be sure to indicate whether the product is in stock and whether it is possible to order it now. At a time like this, users especially want to check if the store is open. So show the user that the product is "in stock."

    However, do not oversaturate this block. Here it is quite possible to limit yourself to the name, price, quantity of goods, and a link to it.
  5. Delivery and payment information

    Immediately show the terms of delivery and payment that you offer. This information should not come as a surprise to the buyer at the time of ordering.

    6. Unwanted functionality

    Some online stores add a feature where double-clicking the "Buy" or "Add to Cart" button removes an item from it. This is inconvenient and incomprehensible to the buyer.

The main mistakes in the process of editing the selection and checkout

  1. Inflexible editing functionality

    Perhaps, when choosing goods, a person accidentally chose the wrong number of goods or made a mistake with the color/size. And change the decision, looking at the final cost.

    In any case, it will be convenient if you can adjust your choice right in the basket. For example, the "+" and "-" buttons are available, which will edit the quantity, and so on.

  2. The unsuccessful appearance of the list of goods

    It would be a mistake to create a shopping cart with only a text description of the products. Add a photo of the order, so the person will remember precisely what the order looked like and understand if everything went well.

    We also recommend adding a link to the product card so that the buyer can once again check all the necessary characteristics.

  3. Removing items from the shopping cart

    The "Empty Cart" button is too similar to "Checkout" or is too close to no. Generally, it's always better to visually distinguish between actions like "delete" and "done."

    The well-thought-out functionality for returning accidentally deleted goods will also be user-friendly. For example, if the user accidentally clicks the cross, don't delete the product immediately; show it gray and inactive first, and add a "Restore" button for some period. It certainly won't be redundant.

    In addition, please note that if, after emptying the cart, the user sees emptiness, this does not incline him to subsequent purchases. Phrases such as "It's empty" and "There is nothing in the basket" have a somewhat negative attitude. It is better to replace them with CTA phrases like "Select products to appear in the basket" or add a button "Start shopping."

  4. No total

    Most people, when adding products to the cart, expect to see the total amount of the goods (preferably at least tentatively, but also indicate the cost of delivery)

  5. Additional products and promotions

    Of course, offering promotional or similar/complementary products to order is a good idea. But they should be provided only on similar topics and unobtrusively.

    Before placing an order, remind you to enter a promotional code or show the opportunity to choose a promotional offer (for example, how it works in Dominos pizzeria, before placing an order, you can select a promotion from the list, "every second pizza 50% discount", "sauce for a large pizza as a gift" and so on).


It is not in vain that we always mention that there are no trifles in working with sites. However, such a seemingly insignificant detail can affect the User experience of your potential customers.

The main tips can be summarized as follows:

  • User-friendliness over creative solutions
  • Do not add unnecessary information to the cart, overload the user with registration when adding products, etc.
  • Make sure that all buttons work correctly. Then, you can continue shopping and edit your choice directly from the basket.

For this and other elements of your online store to bring the desired conversion, you can always contact us for its creation and promotion.

timer 15

The best IP-telephony services and CRM. How to increase the efficiency of sales

Any business owner understands how important it is to optimize and automate work processes as much as possible. This is especially important for the sales department because it is there that it is important to fulfill the client's needs promptly. Chatbots, CRM, and other services play a huge role here, making it easier and, importantly, faster to work with the client and close the sale.

timer 15

Correct indexing of pagination pages

Incorrect settings or ignoring the environment of indexing pagination pages harm the promotion of your site. This article will look at how to set these metrics most efficiently.

All high scores, don't forget to follow the updates of our articles, we always publish high-quality and relevant content for you