This reports gives information about content duplication condition in a way close to search engines algorithms. Google, Bing, Baidu, Yandex - all actual and commercial successful search engines uses ngrams analysis in their duplicate content detection systems, authorship detection, categorization and classification, web antispam, sentiment analysis, etc. You can check wikipedia.org for detailed information about ngrams.
Our reports are ready to use by seo specialists and copywriters teams. With this reports you can estimate text uniqueness ratio and find pieces of content with high level of plagiarism in Internet. Just pay attention at red marked ngrams in reports — change it, drop it and get profit. Your content will become more valuable from search engines point of view. It will increase number of indexed pages in search engines, forcing them to give more traffic to your site.
Full domain report, supplemented with consolidated url report and internal unique report, provided with detailed comments, available by request.
Public dataset overview
Crawl stats |
Started at |
2017-04-20 03:02:12 |
Finished at |
2017-04-20 14:32:20 |
Processed urls |
10,000,000 |
Ngrams dataset statistics |
Ngrams found |
5,231,405,510 |
Ngrams in p found |
2,129,807,944 |
Unique ngrams found |
3,939,488,551 |
URL average statistics (200 ok) |
cnt |
Average number of ngrams on page |
710 |
Average number of external links on page |
22 |
Average number of internal links on page |
121 |
Response codes |
cnt |
% |
200 ok |
7851691 |
78.52% |
301 (second) |
536721 |
5.37% |
302 (second) |
446410 |
4.46% |
dns failed |
425463 |
4.25% |
request timeout |
273021 |
2.73% |
403 |
103368 |
1.03% |
charset fail |
89216 |
0.89% |
wrong ssl |
79054 |
0.79% |
404 |
58902 |
0.59% |
303 |
29319 |
0.29% |
307 |
19050 |
0.19% |
0 |
17849 |
0.18% |
500 |
16117 |
0.16% |
400 |
13794 |
0.14% |
Other |
17716 |
0.18% |
Top 15 TLD |
cnt |
% |
.com |
5142528 |
51.43% |
.org |
489993 |
4.90% |
.net |
414328 |
4.14% |
.de |
354742 |
3.55% |
.co.uk |
346023 |
3.46% |
.ru |
265105 |
2.65% |
.fr |
177909 |
1.78% |
.nl |
175284 |
1.75% |
.com.br |
152617 |
1.53% |
.com.au |
132118 |
1.32% |
.it |
122253 |
1.22% |
.pl |
115906 |
1.16% |
.es |
98545 |
0.99% |
.ca |
81732 |
0.82% |
.info |
75287 |
0.75% |
Top 15 server locations |
cnt |
% |
United States |
4392386 |
43.92% |
Germany |
855132 |
8.55% |
United Kingdom |
461483 |
4.61% |
France |
410436 |
4.10% |
Netherlands |
278531 |
2.79% |
Russian Federation |
233706 |
2.34% |
Japan |
230433 |
2.30% |
Canada |
187776 |
1.88% |
Italy |
169329 |
1.69% |
Spain |
140580 |
1.41% |
Poland |
127710 |
1.28% |
Australia |
111173 |
1.11% |
Switzerland |
105057 |
1.05% |
Romania |
92669 |
0.93% |
Brazil |
92044 |
0.92% |