
A crawler for linguistics, made at Masaryk University’s Natural Language Processing Centre in Brno and used by Lexical Computing, the Czech company behind Sketch Engine. It collects text from public web pages to build corpora: huge collections of real language that dictionary makers and language researchers search. Sketch Engine says it isn’t for search engines or generative AI.
Worth knowing: The corpora built with it from 2017 to 2020 add up to about 200 billion tokens (words and punctuation marks), mostly English; it is made to get the most text out of every megabyte it downloads (corpus.tools).
How it makes money: Lexical Computing sells subscriptions to search the corpora built with it: from €7.09 a month for an academic, €12.03 for freelancers such as translators and marketers, and by quote for companies and dictionary makers. (sketchengine.eu)
On this site: It read the rules and the homepage and went away: perhaps it found too few words for a corpus. A site of pictures, films and a 3D head isn’t much use to a dictionary maker.
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →