BingBot Now Supports Brotli, Set to Test Zstd Compression Soon

Brotli compression is a lossless data compression algorithm that is designed to be efficient and fast for web content compression. Bingbot, Microsoft's web crawler, now fully supports Brotli compression for all URLs, meaning it can handle web content compressed with Brotli algorithm. This support was achieved after a successful ramp-up to 100%, indicating that Bingbot can now interact with web servers that use Brotli compression to serve web pages more efficiently.

Bingbot is testing zstd Zstandard compression, which is another lossless data compression algorithm. The reason for testing this new compression standard is to further optimize the crawling process by potentially reducing the amount of data that needs to be transferred during the crawling of websites. This could lead to faster load times and more efficient data usage.

Microsoft plans to begin testing Zstd compression for Bingbot in a phased rollout that will follow in the weeks after the announcement made by Fabrice Canel on April 18, 2024. This suggests that the testing phase will start shortly after this date, although a specific timeline for full implementation was not provided.

Bingbot's support for Brotli compression is similar to Googlebot, which started using Brotli compression back in 2020. However, as far as is known, Googlebot does not use Zstd compression. This means that Bingbot could have an advantage in terms of compression technology if it successfully implements Zstd compression, as it would be supporting a compression standard that Googlebot does not currently use.