by Anonymous Coward writes:
on Wednesday June 09, 2010 @09:47AM (#32509612)
Google has pulled my site robots.txt file 32 times this month and it is only the 9th - about 4 times a day. I'm showing almost 2000 web pages pulled by Google indexers in this same time period. My site is tiny, private, not very large.
By bandwidth, Google is only 2.4% of the total site traffic, so far, this month.
I agree Google is "fresher" than they used to be. OTOH, my non-commercial site has approximately doubled readers in each of the last 6 months by publishing 1 new posting about every other day.
I suspect other, more use sites are hit hourly or even more often by google.
MSN-Bot appears to visit 10 times a day, but is much more selective about which pages it indexes. Since my site is date organized, this seems smarter than what google does. Some times, I do edit older stories with new knowledge or corrections which google will see, eventually and MSN will not. Zero referrals from any microsoft searches seen.
Yahoo! slurp barely touches my site. Only 1 referral has been seen.
Google sends about 30% of the total traffic, but most is from social networking with "hey, check this out" type referrals. Not bad for a technical article site.
You do know many spam/exploit bots use your robots file to look for admin logins or sensitive info. Just because the browser agent was the same as Google doesn't mean it really was, you have to check the agent's IP to be reasonably sure it's legit.
Considering that Google even says they have previously only indexed sites every 10 days, it's much more likely you have 3 Google indexes and 29 exploit scans.
Google has pulled my site robots.txt file 32 times this month and it is only the 9th - about 4 times a day.
Maybe your "Expires" HTTP header tells it to? Well, for robots.txt it's not that important, but I'm often frustrated how few people know about the expires header and how much traffic they could save.
32 Google indexer visits this month (Score:5, Interesting)
Google has pulled my site robots.txt file 32 times this month and it is only the 9th - about 4 times a day. I'm showing almost 2000 web pages pulled by Google indexers in this same time period. My site is tiny, private, not very large.
By bandwidth, Google is only 2.4% of the total site traffic, so far, this month.
I agree Google is "fresher" than they used to be. OTOH, my non-commercial site has approximately doubled readers in each of the last 6 months by publishing 1 new posting about every other day.
I suspect other, more use sites are hit hourly or even more often by google.
MSN-Bot appears to visit 10 times a day, but is much more selective about which pages it indexes. Since my site is date organized, this seems smarter than what google does. Some times, I do edit older stories with new knowledge or corrections which google will see, eventually and MSN will not. Zero referrals from any microsoft searches seen.
Yahoo! slurp barely touches my site. Only 1 referral has been seen.
Google sends about 30% of the total traffic, but most is from social networking with "hey, check this out" type referrals. Not bad for a technical article site.
Re: (Score:2)
I can't belive you didn't post a link.
I mean this is slashdot...getting your site slashdotted is part of the fun.
-Keith
It probably wasn't really Google than indexed you (Score:2, Insightful)
Re: (Score:2)
Google has pulled my site robots.txt file 32 times this month and it is only the 9th - about 4 times a day.
Maybe your "Expires" HTTP header tells it to? Well, for robots.txt it's not that important, but I'm often frustrated how few people know about the expires header and how much traffic they could save.