FeedHall


83.4M

Hosts


Crawler toplist

Crawler Mentions
* 7.9M
Mediapartners-Google 3.6M
ia_archiver 865K
Mediapartners-Google* 596K
HaosouSpider 374K
AdsBot-Google 286K
Googlebot 272K
Yandex 243K
Mediapartners (Googlebot) 197K
Googlebot-Image 170K
Twitterbot 91K
GoogleBot 71K
Slurp 70K
Googlebot-Mobile 68K
Adsbot-Google 60K
MSNBot 51K
bingbot 46K
googlebot 43K
YandexDirect 42K
msnbot 40K

Crawler shamelist

Crawler Disallow count
* 1.8M
MJ12bot 1.1M
AhrefsBot 795K
Baiduspider 664K
Yandex 645K
BLEXBot 532K
NerdyBot 492K
ia_archiver 458K
SemrushBot 400K
psbot 363K
ZyBorg 357K
bingbot 336K
dotbot 320K
Nutch 312K
spbot 301K
Sogou 295K
Yahoo! Slurp 286K
Googlebot 243K
008 237K
twiceler 226K

Disallowed paths

Path Disallow count
/ 50.2M
/wp-admin/ 9.7M
/api/ 8.1M
/search 7.1M
/checkout/ 6M
/cart/ 6M
/static/ 4.3M
/account/ 4.3M
/config 4.2M
/cart$ 4.2M
/commerce/ 4.2M
/*&view=* 4.2M
/account$ 4.2M
/*?view=* 4.2M
/checkout$ 4.2M
/*?tag=* 4.2M
/*?author=* 4.2M
/*?month=* 4.2M
/*&tag=* 4.2M
/*&month=* 4.2M

Server toplist

Server Count
Apache 22.4M
nginx 12.7M
cloudflare 3.1M
GSE 2.9M
Microsoft-IIS/8.5 2.5M
Microsoft-IIS/7.5 2.4M
nginx/1.10.2 1.7M
nginx/1.14.0 1.5M
LiteSpeed 1.4M
openresty 1.4M
Apache/2 1.1M
nginx/1.10.1 1M
Microsoft-IIS/6.0 669K
Apache/2.4.18 (Ubuntu) 664K
nginx/1.12.2 652K
Microsoft-IIS/10.0 646K
nginx/1.10.0 576K
Apache/2.4.37 (Unix) 518K
Apache/2.4.10 (Debian) 501K
Apache/2.2.15 (CentOS) 489K