Text Analysis with Term Frequency for Mark Twain’s Novels

September 12, 2017
By

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Introduction

Samuel Langhorne Clemens, otherwise known as Mark Twain, is one of the most important American writers.”The Adventures of Tom Sawyer” is probably one of my most favorite books in all English literature. Happy to see that Twain’s river novels remain required reading for young students, he is read more widely now than ever!

Project Gutenberg offers over 53,000 free books. I will use four of Twain’s best novels for this analysis:

• Roughing It
• Life on the Mississippi
• The Adventures of Tom Sawyer

We will be using the following packages for the analysis:

Data preprocessing

We’ll retrieve these four books using the gutenbergr package:

An important preprocessing step is tokenization. This is the process of splitting a text into individual words or sequences of words. The unnest_tokens function is a way to do just that. The result is converting the text column to be one-token-per-row like so:

After removing stop words, we can find the most common words in all the four books as a whole.

Sentiment

Sentiment analysis is not the focus today, but since we are here already, why not have a quick look?

We did not spot anomaly in the sentiment analysis results except word “miss’ is identified as a negative word, actually, it is used as a title for the tough old spinster Miss Watson in “Adventures of Huckleberry Finn”.

tf-idf

To blatantly quote the Wikipedia article on tf-idf:

In text analysis, tf-idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in information retrieval and text mining.

For our purpose, we want to know the most important words(highest tf-idf) in Mark Twain’s four books overall, and most important words(highest tf-idf) in each of these four books. Let’s find out.

Terms with the highest tf-idf across all the four novels

Each novel has its own highest tf-idf words. However, the language he used across these four novels are pretty similar, such as term “city” has high tf-idf in “Roughing it” and “Life on the Mississippi”.

Term frequency

Just for the kicks, let’s compare Mark Twain’s works with those of Charles Dicken’s. Let’s get “A Tale of Two Cities”, “Great Expectations”, “A Christmas Carol in Prose; Being a Ghost Story of Christmas”, “Oliver Twist” and “Hard Times”.

What are the most common words in these novels of Charles Dickens?

The top term for both author is the same – “time”. Other than that, their language are very different.

Text Analysis with Term Frequency for Mark Twain’s Novels was originally published by Susan Li at Susan Li | Data Ninja on September 13, 2017.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.