Indexing a billion pages
blog.mwmbl.org
It’s two years since we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021. A good time to take stock of where we are and where we’re going. We’ve indexed over 100 million pages Thanks to our volunteers, who crawl the web using the Firefox extension and command line script, we’re crawling up to a million pages a day, as you can see on our stats page. There are around 50-60 users crawling on an average day.
    • @Nima@leminal.spaceEnglish
      11•24 days ago

      it always has to be something that sounds like it was thought up tiredly at 4am by the college freshman who coded it.

      • Tywèle [she|her]English
        8•24 days ago

        Or like something that got typed by that same college freshman by falling asleep and hitting their head on the keyboard.