從谷歌機器人獲得我的網絡服務器點擊

Question

我不明白爲什麼我看到這些日誌到我的nginx服務器。

66.249.79.115 - - [06/Oct/2015:18:50:17 +0000] "GET /profile/?Rohatgi.Nikhil HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.115 - - [06/Oct/2015:18:50:49 +0000] "GET /profile/?Mukherjee.PankajKumar HTTP/1.1" 404 1038 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.115 - - [06/Oct/2015:18:51:21 +0000] "GET /profile/?Khorana.Ashish HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:51:52 +0000] "GET /profile/?Mittal.AshokKumar HTTP/1.1" 404 1034 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:52:24 +0000] "GET /profile/?Suri.Divya HTTP/1.1" 404 1029 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 
66.249.79.117 - - [06/Oct/2015:18:52:56 +0000] "GET /profile/?gupta.member) HTTP/1.1" 404 1030 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 

我最近做了我的網站直播。

從日誌我可以理解他們來自谷歌機器人一點點，但我想明白爲什麼我得到這個，我怎麼能阻止他們？ 如果我阻止，我的網站將不會受到任何傷害。

Answer 1

這種行爲是正常的，如果你把你的網站最近生活。 Google Bots正在抓取網頁併爲新網頁編制索引，因此可以使用Google搜索找到它們。正如你所看到的，谷歌機器人假裝是iPhone（iPhone;像iPhone OS X的CPU iPhone OS 8_3）。

如果要阻止googlebot，您可以按照以下指南進行操作：Nginx + CDN + GoogleBot or how to avoid many useless Googlebot hits請注意，如果您屏蔽了谷歌爬蟲機器人，則無法使用谷歌搜索找到您的網頁。 如果你想阻止更多的蜘蛛/爬行機器人，請參考post用戶GD-hussle。請參閱Crawling, indexing & ranking。