Google Revamps Open Source Robots.txt Parser Code on GitHub

The robots.txt parser released by Google is an open source C++ library that Google's production systems use for parsing and matching rules in robots.txt files. This library has been in use for over 20 years and includes code dating back to the 1990s. It has evolved over time to address the various ways webmasters write robots.txt files and the corner cases that need to be covered.

To use the updated robots.txt parser from Google, you can access the code on GitHub where it has been made available. The update introduces new capabilities in the parser class, allowing the export of parsing information about the robots.txt body, and adds a new library to access that information. If you encounter any issues while using it, you are encouraged to file an issue on the GitHub repository.

Google updated the robots.txt parser on GitHub recently, with the push to the repository made by Gary Illyes from Google yesterday morning.

The updated robots.txt parser includes new capabilities such as:

• The ability to export parsing information about the robots.txt body.
• A new library that provides access to the parsing information.

This new library has been used by Google Search Console for a significant period ('many moons') without encountering issues.

Google open sourced the robots.txt parser to share the evolved library with the broader community. The parser reflects over 20 years of learning about how webmasters write robots.txt files, including handling corner cases. By open sourcing it, Google aims to standardize the way robots.txt files are interpreted and to contribute to the internet draft with the knowledge they've gained.