FeedHall

137M

Hosts


Crawler toplist

Crawler Mentions
* 13.9M
Mediapartners-Google 5.3M
ia_archiver 959K
Mediapartners-Google* 907K
Googlebot 559K
HaosouSpider 483K
Yandex 428K
Googlebot-Image 426K
AdsBot-Google 400K
Twitterbot 274K
Mediapartners (Googlebot) 226K
Googlebot-Mobile 221K
Adsbot-Google 183K
Slurp 158K
GoogleBot 113K
msnbot 106K
YandexDirect 104K
bingbot 100K
MSNBot 97K
YadirectBot 96K

Crawler shamelist

Crawler Disallow count
* 2.4M
MJ12bot 1.9M
AhrefsBot 1.5M
Baiduspider 1.2M
Yandex 1.1M
NerdyBot 921K
BLEXBot 875K
SemrushBot 845K
ia_archiver 825K
Nutch 764K
psbot 689K
dotbot 613K
WebCopier 559K
WebStripper 523K
Teleport 517K
spbot 511K
Offline Explorer 503K
008 498K
SiteSnagger 493K
TeleportPro 492K

Disallowed paths

Path Disallow count
/ 95.5M
/wp-admin/ 18.4M
15.7M
/api/ 15.4M
/search 12.4M
/checkout/ 12.4M
/cart/ 12.2M
/admin/ 9M
/search/ 8M
/account/ 7.5M
/static/ 7.4M
/config 7.3M
/cart$ 7.3M
/commerce/ 7.2M
/*?tag=* 7.2M
/account$ 7.2M
/*?view=* 7.2M
/checkout$ 7.2M
/*&view=* 7.2M
/*&tag=* 7.2M

Server toplist

Server Count
Apache 19.6M
nginx 12.4M
8M
cloudflare 4.3M
GSE 2.4M
Microsoft-IIS/8.5 2.1M
Microsoft-IIS/7.5 1.9M
LiteSpeed 1.3M
nginx/1.14.0 1.2M
openresty 1M
nginx/1.10.2 1M
Apache/2 913K
Microsoft-IIS/10.0 669K
Apache/2.4.18 (Ubuntu) 595K
nginx/1.12.2 588K
nginx/1.10.1 583K
Apache/2.4.10 (Debian) 477K
Microsoft-IIS/6.0 463K
Apache/2.2.15 (CentOS) 462K
Apache/2.4.7 (Ubuntu) 447K