About

This is an experimental search engine built from the Common Crawl corpus.

How big is the index?

The index contains over 4 billion pages across 40.5 million hosts and 33.5 million domains. For comparison, Google reported 400 billion pages during antitrust trial. My goal was to reach 1% of that figure, although a lot has happened since 2020.

Where does the data come from?

Instead of building my own crawler, I decided to use Common Crawl for demo purposes. It's a free and widely used repository of web crawl data.

How does the search work?

The engine uses Tantivy, a classic full-text search library written in Rust. Candidates are retrieved using BM25 (a text-relevance signal) and then reranked with PageRank (a host-level signal).

Is the result count exact?

No, the displayed count is an upper bound estimate from term frequencies.

Can I search in a particular language?

Yes, you can use any language. Search results can be narrowed down using language filter. Labels come from the crawl metadata. In total, the index contains more than 150 languages.

Next steps

The project is still in an early stage. If you'd like to sponsor it or simply say hi, you can reach me at evgeny@xorsoft.dev