Here is how GoogleBot behaves if it can’t access your robots.txt file
As you can see, it is better to be quickly informed if the robots.txt file is not crawlable.
How to monitor the robots.txt file for free?
This is a feature offered for free on Oseox Monitoring.
Just take 1 minute to create an account.
