Crawl-delay

Crawl-delay was meant to throttle aggressive bots on fragile servers, but Google ignores it entirely — you control Googlebot's rate through Search Console settings, not this line. Bing and some other crawlers do honor it, so it retains niche value for taming lesser bots that would otherwise hammer a small site. If Googlebot is genuinely overloading your server the better fixes are faster responses, caching, or a temporary 503, since a large crawl-delay elsewhere can starve your pages of crawling instead.