Your website is slow?
Your website has a lot of pages?
Worse than that, your website is large and slow?
Don’t panic, we have a solution to optimize the crawl and have a more effective monitoring.
Go to the configuration page of your website by clicking on the « configure » button.
Take a look at the crawler options.
You have some possibilities.
Increase the number of crawlers
Rob, our bot has a lot of brothers. Increase the number of crawlers will significantly increase the crawl speed.
It’s a good solution if you’re notified that we don’t crawled all your URL.
Of course, your server must be able to support the charge.
Change the pending time
Crawl a slow website is more expensive cause our server resources are longer involved. By default, we wait until 30 seconds to have a response from your server.
To accelerate the crawl, you can reduce the pending time. Be careful, it can increase the number of errors if a crawl is planned when your website is overloaded.
If you want to reduce the number of notifications that you receive because of temporary errors or because your website is very slow, you can increase the pending time.
However, if a website gives a response after 30 seconds… SEO is a minor issue, right?
Number of retry
If you receive notifications for temporary problems because your website is slow or overloaded, you can increase the number of retry.
By default, when we detect a change, before to notify you, we crawl the same URL after 5 minutes, but some problems can persist more than 5 minutes.
To avoid this kind of errors, you can increase the pending time for 10 or 15 minutes.
Be careful, you’ll be notified less often, and the crawl time will be longer.
HTML page weight
If the HTML source code is really heavy (several MB), it is quite possible that our crawler decides or fails to scrap the content of the URL to be monitored.
If your pages weigh several MB (without counting images and external resources of course), there are certainly other priorities than SEO to deal with 🙂
Conclusion
As we’ve seen, a lot of solutions can help you to optimize the crawl of large and/or slow websites.
Oseox Monitoring and Oseox Sitemap can be configured with a dedicated configuration for each website.
