seen from China
seen from United Kingdom
seen from United States
seen from United Kingdom
seen from United States
seen from United States
seen from United States
seen from United States
seen from United States
seen from South Korea
seen from Germany
seen from Germany
seen from Australia
seen from United States
seen from Malaysia
seen from United States
seen from United States
seen from United States
seen from United States
seen from France
The four most common words in New Who titles are “the” (133), “of” (54), “doctor” (10), and “time” (9).
Which means that, by sheer statistical title criteria, “The Time of the Doctor” — the 241st story — is the most New Who episode of New Who.
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
Free to watch • No registration required • HD streaming
Even the languages we haven't deciphered yet, such as the one in the Voynich Manuscript, appear to follow this rule.
four-letter words
Using this tool to determine unique amount of words in a text (and generating a vocabulary list). Since it can’t handle a huge amount of input I’m just going to do maybe a chapter or half a chapter of each story and see the result: http://www.zhtoolkit.com/apps/wordlist/create-list.cgi
This tool is useful for figuring out how difficult a text is compared to something else you’ve read, and it generates word lists! So you have something to study from (if you aren’t using Pleco, want like a Graded Reader experience with intensive reading word lists, etc). You can think of meanU as the “easiness rating” - the higher it is the easier the text is to read.
**This list gets long below the cut, but will hopefully help serve as a reference. For a rough estimate I’d say anything from 1.9+ seems to be manageable for someone who knows HSK 4 words and is using a click-dictionary (like Pleco, Zhongwen Chrome Extension, Mandarinspot.com Annotator Bookmarklet, eReader dictionary, etc). Some ~1.9 ones are easier or harder depending on genre familiarity. For example I think 论如何错误地套路一个魔教教主 is extremely easy if you’re used to wuxia genre as its very simple for the genre, versus something like 魔道祖师, also if wuxia vocab isn’t an issue for you then priest novels like 天涯客 tend to be easier than 镇魂). For ease’s sake, ones 2+ seem to be the most ‘doable’ for actual extensive reading though. Again, this depends on which genres you’re more familiar with - but in general the ones scored 2+ below are novels I had more success reading without dictionary lookup in general.
Anything marked *** I would recommend as ‘easier,’ and anything as ** I would recommend as possibly easier depending on your genre familiarity.
Update: I have gone through this list and now most text samples analyzed were generally between 1900-2100 words, so now the scores should be more comparable. While a 1.9 may or may not be easy based on your level, in general if something is scored lower it will still be harder and if scored higher will still be ‘easier’ than a 1.9. Your genre familiarity will also affect things. So when looking for “similar difficulty” material and “slightly” harder or easier, these scores should hopefully be a bit more useful now.
* Unique unknown is the count of the Chinese words not in the public common word filter, nor in your user known word list * meanU is the average frequency of all words. Here, it is the average of the log(10) frequencies. It is a very rough measure of text difficulty. A value of ~1.9 is somewhat difficult, and ~2.6 is probably easier. (Ref: [http://www.soc.cornell.edu/hayes-lexical-analysis/])
First, novels I’ve heard recommended as ‘easier’ to read:
***小王子
Characters:3196 Word Count:2192 Unique Words:744 (33.9%) Unique unknown*:608 (81.7%) meanU(log10)*:2.004
**So it’s ease rating is 2, fairly easy! That makes sense, at least based on my experience reading it right now.
***地图 by 倪匡
Characters:3005 Word Count:2094 Unique Words:681 (32.5%) Unique unknown*:551 (80.9%) meanU(log10)*:2.072
**This author was recommended as very approachable on chinese learning forums, sci-fi short stories (around 100 pages per story) for people who know HSK 4+. (Also again shout out to this text analyzer tool because the vocabulary lists it generates are super useful for looking through ahead of a reading to help prepare).
***他们的故事 by 一根黄瓜丝儿
Characters:3085 Word Count:2172 Unique Words:730 (33.6%) Unique unknown*:613 (84.0%) meanU(log10)*:2.008
**Score of 2 makes sense, it was the first webnovel I was able to read (with the help of Pleco Reader click-definitions). It’s definitely on the easier side. If there’s unknown words in this, a huge portion of them are very common daily life words or simple novel description words so they were worth learning for me.
***论如何错误地套路一个魔教教主
Characters:2403 Word Count:1766 Unique Words:691 (39.1%) Unique unknown*:565 (81.8%) meanU(log10)*:1.962
**This is The Wrong Way To A Demon Sect Leader and I recommend it hands down as an intro wuxia or bl novel. The reading is actually fairly easy, and if its not then learning any of the words here are pretty basic wuxia genre words you will keep using. (Also its a great listening reading method novel to do since its audiobook matches perfectly to the text, and it’s english translation is pretty literal).
Wordclouds
Did a crash course in python and picked Alpha Centauri’s quotes as the base text for a wordcloud. Used stopwords to strip out common English words and also got rid of “will”, “us” and “one”. I included the lines where it says where the quote is from.
Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.
Free to watch • No registration required • HD streaming
Boredom Project
Sooooo. I was thinking a few days ago that how frequent was each word in the lyrics of Poets of the Fall. I found a really good Word macro, that I used and it beautifully listed out everything. I didn’t change the excluded words and I didn’t modify lots of things. All in all I was just curious and I like to see data. Then I took the easy way out and used an online word cloud generator to visualize this data. The result is this word cloud. It was a fun thing to do and while it’s not in the least representative, it’s kind of good to look at. :) And while I’m at it, here’s the most frequent 20 words:
I (515)
You (494)
Your (280)
My (259)
In (257)
Me (224)
It (169)
Like (169)
We (160)
All (158)
I’m (155)
Love (145)
So (140)
With (139)
When (130)
On (129)
Will (119)
It’s (115)
This (111)
That (99)
Please note that this is just a fun project, it’s not perfect but I worked with it. If anyone finds it fascinating, please don’t repost, but reblog. Thanks. :)
Literary analyst J. Chaix reveals some of the extraordinary ways that author James Hogg manipulated word frequencies in his classic, 1824 book, "The Private ...
A brief intro about how James Hogg used word frequencies to hide significance and meaning in his original, 1824 Gothic classic, "The Private Memoirs and Confessions of a Justified Sinner"...
wordfreq - Access a database of word frequencies, in various natural languages.
wordfreq is a Python library for looking up the frequencies of words in many languages, based on many sources of data.
this is neat, there’s support for CJK languages too:
warning though this package/library which includes the databases is sort of massive!:
Sources and supported languages
This data comes from a Luminoso project called Exquisite Corpus, whose goal is to download good, varied, multilingual corpus data, process it appropriately, and combine it into unified resources such as wordfreq.
Exquisite Corpus compiles 8 different domains of text, some of which themselves come from multiple sources:
Wikipedia, representing encyclopedic text
Subtitles, from OPUS OpenSubtitles 2016 and SUBTLEX
News, from NewsCrawl 2014 and GlobalVoices
Books, from Google Books Ngrams 2012
Web text, from the Leeds Internet Corpus and the MOKK Hungarian Webcorpus
Twitter, representing short-form social media
Reddit, representing potentially longer Internet comments
Miscellaneous word frequencies: in Chinese, we import a free wordlist that comes with the Jieba word segmenter, whose provenance we don't really know
The following languages are supported, with reasonable tokenization and at least 3 different sources of word frequencies:
Language Code # Large? WP Subs News Books Web Twit. Redd. Misc. ──────────────────────────────┼──────────────────────────────────────────────── Arabic ar 5 Yes │ Yes Yes Yes - Yes Yes - - Bosnian bs [1] 3 │ Yes Yes - - - Yes - - Bulgarian bg 3 - │ Yes Yes - - - Yes - - Catalan ca 4 - │ Yes Yes Yes - - Yes - - Czech cs 3 - │ Yes Yes - - - Yes - - Danish da 3 - │ Yes Yes - - - Yes - - German de 7 Yes │ Yes Yes Yes Yes Yes Yes Yes - Greek el 3 - │ Yes Yes - - Yes - - - English en 7 Yes │ Yes Yes Yes Yes Yes Yes Yes - Spanish es 7 Yes │ Yes Yes Yes Yes Yes Yes Yes - Persian fa 3 - │ Yes Yes - - - Yes - - Finnish fi 5 Yes │ Yes Yes Yes - - Yes Yes - French fr 7 Yes │ Yes Yes Yes Yes Yes Yes Yes - Hebrew he 4 - │ Yes Yes - Yes - Yes - - Hindi hi 3 - │ Yes - - - - Yes Yes - Croatian hr [1] 3 │ Yes Yes - - - Yes - - Hungarian hu 3 - │ Yes Yes - - Yes - - - Indonesian id 3 - │ Yes Yes - - - Yes - - Italian it 7 Yes │ Yes Yes Yes Yes Yes Yes Yes - Japanese ja 5 Yes │ Yes Yes - - Yes Yes Yes - Korean ko 4 - │ Yes Yes - - - Yes Yes - Malay ms 3 - │ Yes Yes - - - Yes - - Norwegian nb [2] 4 - │ Yes Yes - - - Yes Yes - Dutch nl 4 Yes │ Yes Yes Yes - - Yes - - Polish pl 5 Yes │ Yes Yes Yes - - Yes Yes - Portuguese pt 5 Yes │ Yes Yes Yes - Yes Yes - - Romanian ro 3 - │ Yes Yes - - - Yes - - Russian ru 6 Yes │ Yes Yes Yes Yes Yes Yes - - Serbian sr [1] 3 - │ Yes Yes - - - Yes - - Swedish sv 4 - │ Yes Yes - - - Yes Yes - Turkish tr 3 - │ Yes Yes - - - Yes - - Ukrainian uk 4 - │ Yes Yes - - - Yes Yes - Chinese zh [3] 6 Yes │ Yes - Yes Yes Yes Yes - Jieba
