About

The goal of the project is to build an alternative web search engine.

How big is the index?

The index contains over 4 billion pages across 33.5 million domains. For comparison, Google reported 400 billion pages in the latest antitrust trial. My goal was to reach 1% of that figure, although a lot has happened since 2020.

Where does the data come from?

The index is built from the Common Crawl corpus, a free and widely used repository of web crawl data.

How does the search work?

The engine uses tantivy, a full-text search library written in Rust. Candidates are retrieved using BM25 (a text-relevance signal) and then reranked with PageRank (a host-level signal). The displayed count is an upper-bound estimate from term frequencies.

Can I search in a particular language?

Yes, you can use any language. Search results can be narrowed down using a language filter. Labels come from the crawl metadata. In total, the index contains more than 150 languages.

Next steps

I decided to start with classic full-text search and reproduce earlier studies. The logical next steps are:

• Introducing semantic search.
• Improving content extraction and scoring.
• Crawler: building a distributed crawler and its infrastructure.
• Expanding sources: OpenWebSearch, Wikipedia, third-party integrations, own crawled data.

Monetization

There are currently no plans to monetize the project.

Funding

I believe it's possible to scale the project with a small team, but the biggest limitation is hardware capacity. Deploying an army of crawlers and processing terabytes of data requires dozens of servers - well beyond my hobby budget. If you'd like to sponsor the project or simply say hi, you can reach me at evgeny@xorsoft.dev.