Our Story Begins

One Friday my colleagues were discussing Wordle in work chat. Strategies were discussed, with a focus on what is the best word to start out with. ‘HEATS’ was proposed, also ‘SURLY’, ‘DEVIL’, ‘JUDGE’, and ‘AEROS’, with supporting arguments about frequency of letter usage, letters most commonly appearing at different positions in words, etc. One colleague thought it would be interesting if somebody did some looking into the problem from a probability and statistics point of view to, as another co-worker put it ‘Monty Hall this for me’.

Now while I have long been a fan of word games with a special love for Scrabble, I would see people sharing images of green and yellow tiles and talk about Wordle on social media, and I would keep scrolling. At one point I looked it up in the App Store thinking it was an app, and faced with some unappealing obvious knock-offs from developers attempting to latch on to the fad, I closed out the App Store and resumed making trouble on Reddit. But now I was intrigued. Surely I could grab some dictionaries and other supporting data (frequency of usage, past winning words) and whip up some winning strategy in no time. And so our story begins.

The Data

As we all know the key to any data project is getting decent data, and when you are doing stuff with R generally it’s not a problem of not having a feature or dataset, but having to decide which of the available options to go with.

Words

On this one I got off to a shaky start. Grady Ward’s English words english.words (CELEX database) in the vwr package seemed the way to go. Along my journey I discovered a suggested ‘best first’ word (ROATE – it’s some legal term whose definition started leaving my head before I finished reading it) wasn’t in there, and to add insult to injury a recent WORDLE answer (humor) wasn’t in there because CELEX spelled it the British way. This is unacceptable behaviour.

Via this SQL person’s post I learned about the Scrabble dictionary SOWPODS which the author helpfully directed readers to on github.

Via the r/wordle subreddit I discovered to some disappointment that the dictionary they use and more secrets are rather easily obtainable via checking out the site’s Javascript. Though I suspected that might be true, I opted not to look for fear of ruining my own fun of ruining the game too much. I still haven’t peeked behind that curtain.

Frequencies

Next up, we consider word usage frequency in English. Again, there are many options. Some costing money, so forget that. I ended up using the dataset from Kaggle: ‘1/3 Million Most Frequent English Words on the Web‘ (yes, it’s a bit skewed, for example ‘click’ is way up there in the ranking for obvious reasons. But this is just a for fun project, so we’ll take it).

Past Winning Words

Finally there’s a list of ‘past answers’ that has the first 200 answers which I’ve been updating. You can find it here with my version in my github repo.

Literature Research

I usually like to have a look at what other people have done when faced with a problem I’m working on. Re-inventing all the wheels is never a quick path to success. A quick Google revealed multiple R-bloggers posts as well as other math-y takes on Wordle. It occurred to me I might ruin the fun if I read these, and decided to go it alone and come back to this to compare notes. I’ll come back to those at the end.

Coding Time

Although I have a full blown R Studio install on my MacBook and we use R Studio Workbench on a nice beefy server at work, I often use R Studio Cloud for my side projects for many reasons including it’s free, it doesn’t take a long time to start up, and it’s pretty nicely integrated with shinyapps.io for hosting Shiny apps (this is also free with certain restrictions, but dealing with small data like this is no computational heavy load.) I set up a project in R Studio Cloud, and started hacking away.

Some Scoring Systems

The initial naive approach I used was: for all the 5 letter words, get stats on how often the letter at the nth position appears at the nth position in other 5 letter words. This was weighted by frequency of usage because…I had the data, so use it man. I multiplied the 5 numbers together and took the log (because by then they are tiny tiny numbers). ‘Top scorers’ are words like sores, shies, tores(?), shoes. S is as we know an ultra popular letter appearing often at the beginning or end of words including my name. As an added bonus we can insist on 5 unique letters in the word, then we get tores, cores, tries, pores, etc.

This is a ‘naive’ approach b/c obviously letters appearances in a word aren’t independent, some runs of letters (n-grams) are much more common than others. To quote the late MF DOOM: there’s rules to this shit. An observed trick(?) of the Wordle crew is choosing a word from a list of rhyming words, e.g. light, might, fight, right. This suggests letters appearing early in the word should be weighted more heavily, because guessing the last few letters sometimes does you no good.

After Michael from the sub-title told me about Absurdle, which takes an adversarial approach (when you make a guess, it changes the target to ‘the farthest word from your guess’), I looked at ‘if all letters in this word are eliminated, how many words remain?’ With this system, top words are stoae(?), toeas(?), aloe, and aeons.

Assistive Technology

OK that’s all good, but what to do with this? I put together a Shiny Dashboard and used it for a couple weeks on both Wordle and Absurdle. This human-in-the-loop (my favorite Data Science buzzphrase) system performed really nicely, greatly reducing time spent solving Wordles, freeing me up for productive activities like refining code to help me with Wordles, and write Substack posts about it. The best thing about it is it is very responsive (good performance IS good user experience) and it lends itself well to trying out different things. Enhance people, don’t replace people.

What’s Next?

Now that I’ve gone it alone I plan to have a look at things other folks have done (but not to look at the code for Wordle itself, come on.) There are even competitions springing up – may the best algorithm win! An obvious next step would be setting up some framework for running lots of simulations to test alternate strategies.

Bibliographic Section

There’s a lot of chatter on Twitter from R people about Wordle.

https://explainextended.com/2022/01/27/a-good-first-word-for-wordle/ – a guy does this in SQL, and reminds us why, no matter how awesome SQL is, this is a terrible idea. Use the right tool for the job.

https://www.r-bloggers.com/2022/01/shinywordle-a-shiny-app-to-solve-the-game-worldle-and-the-power-of-regular-expressions/ – this guy also wrote a Shiny app, and also used Regular Expressions, as I did.

https://github.com/Kinkelin/WordleCompetition – ‘Implement and test your own AI to find the optimal Wordle Strategy’. There’s also a Discord where you can hang out and compare notes with your fellow nerds.