Bot Directive Efficiency

Bot Directive Efficiency is a metric (scored 0-100) that assesses the clarity, consistency, and effectiveness of a website's robots.txt file and on-page meta robots tags.

These directives are the primary mechanism for instructing search engine crawlers and other bots on how to access, crawl, and index content. Inefficient or conflicting directives can lead to critical issues, such as blocking important resources (CSS/JS), preventing the indexing of valuable pages, or causing pages to appear in search results without descriptions. For GEO, clear directives ensure that an AI's crawlers can efficiently access and process all necessary content without wasting resources.

Calculation Methodology

The score is calculated based on a comprehensive audit of the robots.txt file and the on-page meta robots tags. The score starts at 100 and is penalized for each issue found.

Robots.txt Analysis (60% weight)

• Accessibility: Check if the robots.txt file is accessible and returns a 200 OK status code.
• Syntax Validity: Parse the file for syntax errors, such as typos in directives (e.g., `Disalow` instead of `Disallow`).
• Resource Blocking: Scan for directives that block critical resources like CSS or JavaScript files, which can prevent proper rendering and understanding of the page.
• Sitemap Presence: Verify that the robots.txt file includes a link to the XML sitemap.

On-Page Meta Robots Tag Analysis (40% weight)

• Conflict with robots.txt: Check for pages that are disallowed in robots.txt but do not have a `noindex` meta tag. This sends conflicting signals to crawlers.
• Multiple Meta Tags: Ensure there is only one meta robots tag on the page. Multiple tags can cause confusion.
• Valid Directives: Verify that the directives used in the meta robots tag are valid (e.g., `index`, `nofollow`, `noarchive`).

Calculating The Bot Directive Efficiency Score

The calculation of the Bot Directive Efficiency Score is a multi-step process that involves analyzing the robots.txt file and the on-page meta robots tags. Here is a simplified pseudo-code representation of how this score is calculated.

BEGIN
 INITIALIZE score to 100 and create an empty issues_list.
 FETCH and PARSE robots.txt file for the domain.
 CHECK robots.txt for accessibility, syntax errors, blocking of critical resources (CSS/JS), and sitemap presence.
 FOR EACH problem found, DEDUCT points from score and ADD description to issues_list.

 FETCH and PARSE the target webpage.
 SCAN <head> for <meta name="robots"> tags.
 CHECK for conflicts (e.g., page disallowed in robots.txt but not marked 'noindex').
 CHECK for multiple meta robots tags on the page.
 FOR EACH conflict or error, DEDUCT points from score and ADD description to issues_list.

 RETURN final score and issues_list.
END

Conclusion

Bot Directive Efficiency is a technical but crucial aspect of GEO. By ensuring that your robots.txt file and meta robots tags are clear, consistent, and correct, you can guide AI crawlers to efficiently access and index your content, which is a prerequisite for being included in AI-generated answers.