.Google.com's John Mueller answered an inquiry regarding why Google.com indexes pages that are forbidden from crawling by robots.txt as well as why the it's safe to neglect the similar Search Console files about those creeps.Robot Website Traffic To Inquiry Criterion URLs.The individual talking to the question chronicled that bots were producing links to non-existent query guideline URLs (? q= xyz) to webpages along with noindex meta tags that are actually also blocked in robots.txt. What motivated the inquiry is actually that Google.com is crawling the web links to those pages, obtaining blocked by robots.txt (without seeing a noindex robotics meta tag) after that getting turned up in Google Browse Console as "Indexed, though blocked by robots.txt.".The person asked the adhering to inquiry:." However below's the big inquiry: why would Google.com index pages when they can't even view the web content? What's the benefit in that?".Google.com's John Mueller validated that if they can't creep the webpage they can not see the noindex meta tag. He also creates an intriguing reference of the web site: search driver, recommending to ignore the results considering that the "common" individuals will not view those results.He created:." Yes, you're correct: if our experts can not crawl the web page, our team can not find the noindex. That mentioned, if we can't crawl the web pages, at that point there's not a whole lot for us to index. Therefore while you may view some of those web pages along with a targeted web site:- concern, the average customer won't see them, so I wouldn't fuss over it. Noindex is also fine (without robots.txt disallow), it merely indicates the Links will definitely find yourself being crawled (as well as end up in the Search Console file for crawled/not recorded-- neither of these standings result in concerns to the remainder of the site). The vital part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in using the Website: hunt evolved hunt operator for diagnostic reasons. Some of those main reasons is given that it is actually not attached to the normal hunt index, it's a distinct factor entirely.Google's John Mueller commented on the site hunt driver in 2021:." The quick response is actually that a site: concern is not implied to become complete, nor used for diagnostics objectives.A site concern is actually a certain kind of hunt that limits the outcomes to a specific site. It is actually primarily only the word internet site, a digestive tract, and then the site's domain name.This question restricts the results to a particular web site. It's not implied to be a complete assortment of all the pages from that internet site.".2. Noindex tag without using a robots.txt is actually fine for these kinds of situations where a robot is linking to non-existent webpages that are receiving found out through Googlebot.3. Links along with the noindex tag will produce a "crawled/not recorded" entry in Browse Console and that those won't possess an adverse effect on the rest of the web site.Read the concern as well as respond to on LinkedIn:.Why will Google mark pages when they can't also see the web content?Included Picture by Shutterstock/Krakenimages. com.