FeedHall

34.6M

Hosts


Crawler toplist

Crawler Mentions
* 975K
Mediapartners-Google 407K
ia_archiver 80K
Mediapartners-Google* 68K
Googlebot 36K
HaosouSpider 34K
AdsBot-Google 31K
Yandex 28K
Googlebot-Image 27K
Mediapartners (Googlebot) 19K
Twitterbot 13K
Googlebot-Mobile 11K
Adsbot-Google 10K
Slurp 9.5K
GoogleBot 8.5K
bingbot 6.6K
YandexDirect 6.5K
MSNBot 6.5K
msnbot 5.9K
YadirectBot 5.9K

Crawler shamelist

Crawler Disallow count
* 172K
MJ12bot 128K
AhrefsBot 102K
Baiduspider 82K
Yandex 76K
NerdyBot 67K
BLEXBot 60K
ia_archiver 55K
SemrushBot 52K
Nutch 51K
psbot 43K
ZyBorg 38K
dotbot 37K
spbot 36K
WebCopier 32K
bingbot 32K
WebStripper 30K
008 30K
Teleport 29K
Offline Explorer 29K

Disallowed paths

Path Disallow count
/ 21.6M
/wp-admin/ 4.5M
/api/ 3.9M
/search 3.3M
/checkout/ 3M
/cart/ 3M
/static/ 2.1M
/account/ 2.1M
/config 2M
/cart$ 2M
/commerce/ 2M
/*&view=* 2M
/*?view=* 2M
/checkout$ 2M
/account$ 2M
/*?tag=* 2M
/*?author=* 2M
/*?month=* 2M
/*&tag=* 2M
/*&month=* 2M

Server toplist

Server Count
APACHE 2.5M
NGINX 1.5M
CLOUDFLARE 434K
GSE 326K
MICROSOFT-IIS/8.5 271K
MICROSOFT-IIS/7.5 258K
LITESPEED 168K
NGINX/1.14.0 161K
NGINX/1.10.2 155K
OPENRESTY 141K
APACHE/2 122K
NGINX/1.10.1 87K
MICROSOFT-IIS/10.0 77K
APACHE/2.4.18 (UBUNTU) 72K
NGINX/1.12.2 71K
MICROSOFT-IIS/6.0 68K
APACHE/2.4.10 (DEBIAN) 58K
APACHE/2.2.15 (CENTOS) 56K
APACHE/2.4.7 (UBUNTU) 53K
NGINX/1.10.0 48K