Ngram text analysis

We provide services on drawing up content evaluation reports on a commercial basis

This reports gives information about content duplication condition in a way close to search engines algorithms. Google, Bing, Baidu, Yandex - all actual and commercial successful search engines uses ngrams analysis in their duplicate content detection systems, authorship detection, categorization and classification, web antispam, sentiment analysis, etc. You can check wikipedia.org for detailed information about ngrams.

Our reports are ready to use by seo specialists and copywriters teams. With this reports you can estimate text uniqueness ratio and find pieces of content with high level of plagiarism in Internet. Just pay attention at red marked ngrams in reports — change it, drop it and get profit. Your content will become more valuable from search engines point of view. It will increase number of indexed pages in search engines, forcing them to give more traffic to your site.

Commercial ngram dataset collects individually and starts from 5 billion entities. Full domain report, supplemented with consolidated url report and internal unique report, provided with detailed comments, available by request.

Public dataset overview

Crawl stats
Started at 2017-04-20 03:02:12
Finished at 2017-04-20 14:32:20
Processed urls 10,000,000
Ngrams dataset statistics
Ngrams found 5,231,405,510
Ngrams in p found 2,129,807,944
Unique ngrams found 3,939,488,551
URL average statistics (200 ok) cnt
Average number of ngrams on page 710
Average number of external links on page 22
Average number of internal links on page 121
Response codes cnt %
200 ok 7851691 78.52%
301 (second) 536721 5.37%
302 (second) 446410 4.46%
dns failed 425463 4.25%
request timeout 273021 2.73%
403 103368 1.03%
charset fail 89216 0.89%
wrong ssl 79054 0.79%
404 58902 0.59%
303 29319 0.29%
307 19050 0.19%
0 17849 0.18%
500 16117 0.16%
400 13794 0.14%
Other 17716 0.18%
Top 15 TLD cnt %
.com 5142528 51.43%
.org 489993 4.90%
.net 414328 4.14%
.de 354742 3.55%
.co.uk 346023 3.46%
.ru 265105 2.65%
.fr 177909 1.78%
.nl 175284 1.75%
.com.br 152617 1.53%
.com.au 132118 1.32%
.it 122253 1.22%
.pl 115906 1.16%
.es 98545 0.99%
.ca 81732 0.82%
.info 75287 0.75%
Top 15 server locations cnt %
United States 4392386 43.92%
Germany 855132 8.55%
United Kingdom 461483 4.61%
France 410436 4.10%
Netherlands 278531 2.79%
Russian Federation 233706 2.34%
Japan 230433 2.30%
Canada 187776 1.88%
Italy 169329 1.69%
Spain 140580 1.41%
Poland 127710 1.28%
Australia 111173 1.11%
Switzerland 105057 1.05%
Romania 92669 0.93%
Brazil 92044 0.92%