This is how to avoid GreenGeeks restricting Googlebot’s access to your website. If Googlebot attempts too many connections to GreenGeeks servers, even the mighty Google will face restrictions so as to not overload it.
If you configure your own websites, starting with an empty root directory or are migrating from another server without these restrictions to GreenGeeks, this post will help you avoid indexing restrictions with Google.The below happened with websites I migrated from servers that were working perfectly with Googlebot to index the websites.
Unreachable: robots.txt
The problem I came across with GreenGeeks hosting was that the robots.txt file on several websites hosted there was noted as being unreachable by Googlebot. This was according to Google Search Console’s URL Inspection tool, so straight from the horse’s mouth so to speak.
Because Greengeeks are so secure, they can block Google from accessing your site under certain circumstances. During indexing of a well-optimised website, Google will have no problem finding relevant pages thanks to the well-planned structure. The problem is that a well structured website encourages Google to keep returning and visit more of the pages each time. After a set number of attempts to view pages on your website, Greengeeks servers will restrict access to Google’s robot.
It was a busy time for me helping software brands grow their customer base, so I put in a support request ‘wondering if there were site specific things like maybe Google tried to index a lot of content from those sites quickly and got blocked.’ For reference, I included the help page Google provided, which explicitly states ‘your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txt file.’
Finding A Solution
Finding a solution to the issue wasn’t immediately obvious. This was this is the first time Google had this issue indexing sites including with other hosting providers I used personally or with clients. The reply to my support request was that ‘Google IPs are globally whitelisted on all [GreenGeeks] shared servers.’
I was told that to pursue a solution, I’d need to provide ‘a step-by-step instruction to reproduce the issue on [their] end, including the involved details like login information and URLs, for [GreenGeeks] to be able to test it live and advise [me] further.’ To reproduce this error, one would need to log into my Google Search Console account (or be granted access), then inspect the website’s pages again.
As an aside, I won’t be giving out passwords for Search Console since that would grant a person I don’t even know access to Gmail, Analytics, Google Drive, Google Sheets and on and on. Given a support request, hosts are able to assist and access relevant parts of your account to solve issues. No passwords should ever be given out to any support staff.
Of course the error would be reproduced if the login credentials were granted, but one would only be referred to the Google Search Console help file referenced above. We’d still not have a solution because the answer would once again be ‘Google IPs are globally whitelisted on all [their] shared servers.’
In the end, Greengeeks support told me to get back to Google and tell them it’s a false positive. So tell Google what? Who tells Google “hey Google you made a mistake”? And also when does Google make a mistake when trying to index websites? They don’t because their whole business is indexing the website so they can sell adverts on the pages that they index on their search engine.
The problem was definitely on their end since I’d never had this issue before with any other host and Google have no reason to give false information about their ability to index websites. I had to give this more thought and come up with a solution myself.
Troubleshooting Googlebot Restriction
Troubleshooting issues is a big part of marketing consultancy and web development is no different. You might even hire a Web Producer to deal with website related issues like this. It is something I’m very used to from my time working in the SaaS industry. The key is approaching the problem with an open mind, looking at things logically to come up with a solution. Start with something you know (a known known if you will), and go from there.
Restricted Server Access
Although Google IPs may be whitelisted by GreenGeeks, I remembered their servers do restrict repeated connection attempts. While testing things like how well Google Analytics 4 is working, I’d use a VPN to access sites hosted on their servers. I found such visits could result in the need to solve a reCAPTCHA (or similar).
That got me thinking along the lines of ‘okay, that explains the 5xx error reported by Google’. 5xx server errors are given when a server fails to fulfil a request, is unable to, or refuses to fulfil a request as I’d say in this case.
Here are some possible errors Google was receiving:
500 Internal Server Error
This is a generic error message, given when an unexpected condition is encountered and no more specific message is suitable. It’s less likely Google was given this very generic error in this case.
503 Service Unavailable
Given when the server cannot handle a request because it is overloaded or down for maintenance. It’s highly likely that this was the error given to Google when trying to access the robots.txt files.
509 Bandwidth Limit Exceeded
Very unlikely Google was given this message since the account in question did not have bandwidth limitations.
Too Many Server Requests
Looking at Search Console’s failure percentage helped me work out the error was related to the number of pages Search Console saw Googlebot was trying to check in a short period of time. I could correlate the increase in failures reported by search console with my activity migrating websites then requesting Google inspect pages and add them to the index.
Since the hosting in question was shared, it’s logical to assume other users on the same server would also have visits from Googlebot which adds to the load.
Confirmed Errors
Looking at the server log files produced by their own server, I saw it had delivered over six hundred 403 errors in a few weeks, confirming to me there was indeed something restricting Googlebot on their end. But why am I rooting through server logs? Not everyone would think to do that and it’s something their support team should look into on behalf of their customer. But First line support will not have the technical knowledge and probably also not the access required to do a slightly more than basic investigation into support ticket and that can spell bad news for customers.
This is one way I increase businesses efficiency through business process optimisation. Creating quick access documents that are easily searchable would provide answers to customers with these kinds of questions, increasing customer satisfaction and retention. How? By decreasing their frustration with support not being able to solve their problem.
Default Files
My next task was to work out what could be different with sites hosted on GreenGeeks as opposed to other website hosting providers. That’s where my ‘site specific things’ support ticket question was coming from. What specifically was different with the sites that had the robots.txt access error compared to others hosted in the same account that did not have that error?
The errors occurred on websites that all worked without issue before being transferred to GreenGeeks’ servers. So I took a look at the root directory of sites that had been created specifically on GreenGeeks’ servers to see what, if any differences there were.
I found GreenGeeks included a default robots.txt and that file included a crawl delay. So there we have it, the GreenGeeks server is set up in a way that despite ‘whitelisting’ Google IPs, it could still restrict access if too many requests were being made in a given period of time. The crawl delay added to the default robots.txt confimed this, instructing robots to observe a delay of 60 seconds.
What I then discovered after more research was that Google does not (or might not) support the crawl-delay command, so I’ll have to use it as a possible solution for now and monitor the situation. Have a look at a video that explains how Google adjusts its behaviour according to the server’s ability to handle requests.
Unreachable: robots.txt Solution
To avoid an Unreachable: robots.txt error in Google Search Console, the best solution is to include the following lines in your robots.txt:
user-agent: *
disallow:
crawl-delay: 60
The key line is ‘crawl-delay: 60’, which means Google, other search engines and spiders etc will pause for 60 seconds between server requests so as to not overload it.
Unfortunately, I can’t guarantee this will solve the problem, but it’s the only option I have for now. Hopefully a note has been made of this thanks to GreenGeeks’ standard operating procedures so the next customer who asks about an Unreachable: robots.txt error can resolve their issue in a swift manner.
My conclusion is GreenGeeks run their servers to be very secure. This is evidenced by the way they handle customer log-in, small things they restrict their users from doing and by the way their servers restrict multiple visits, which should help prevent things such as denial of service (DDoS) attacks.
This is the kind of problem solving you need from a technical partner, to keep your focus on delivering for your customers.