# Handwriting recognition using R

**Yixuan's Blog - R**, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

This title is a bit exaggerating since handwriting recognition is an advanced topic

in machine learning involving complex techniques and algorithms. In this blog I’ll

show you a simple demo illustrating how to recognize a single number (0 ~ 9) using R.

The overall process is that, you draw a number in a graphics device in R using your mouse,

and then the program will “guess” what you have input. It is just for **FUN**.

There are two major problems in this number recognition problem, that

is, how to describe the trace of your handwriting, and how to classify

this trace to the give classes (0 ~ 9).

For the first question, we could first detect the motion of your mouse

in the graphics device, and then record the coordinates of you mouse

cursor at a sequence of time points. This could be done via the

`getGraphicsEvent()`

function in **grDevices** package. For example, after I

drew a number 2 in the graphics window like below, the coordinates of

each point in the trace were assigned to a pair of variables `px`

and `py`

.

The scatterplot of `px`

and `py`

versus their orders in the trace is

shown below.

To be comparable among different traces, we normalize the Order to be

within (0, 1] (that is, transform 1, 2, …, n to 1/n, 2/n, …, 1).

Also, since this recording is discrete but the real trace should be

continuous, we use the `spline()`

function to interpolate at unknown

points, resulting in the following figure.

The dots in the figure have normalized orders of 0.02, 0.04,

0.06, …, 1, at which the x and y coordinates are obtained by

interpolation. Therefore, we could use $r = (x, y)$ where

$x = (x_1, x_2, …, x_{50})’$ and $y = (y_1, y_2, …, y_{50})’$ to

represent the information of the number 2 I have drawn. Somewhat

confused by the operations above? Well, the idea behind this

normalization and interpolation is simple: use 50 “uniformly

ordered” points (I call them “recording points”) to represent the trace.

So it comes to the second question – given a trace, how to classify

it? Obviously we first need a training set, the recording points of

number 0 to number 9 generated as above. Then we’ll compare the

given trace with each one in the training set and find out which

number resembles it most.

Several criteria could be used to measure the similarity, but some

important rules should be considered. We still use $r = (x, y)$ to

represent the recording points of a trace, and use $Sim(r_1, r_2)$ to

stand for the similarity between two traces. Notice that this

similarity should not be sensitive to the scale and location of

traces. That is, if I draw a number in another location in the

window, or in a larger or smaller size, the recognition should not be

influenced. In mathematics, this could be expressed by

where $k_1 > 0$, $k_2 > 0$, $b_1$, $b_2$ are real numbers.

In my code, I simply define the similarity as the sum of Pearson

correlation coefficients of x and y, that is,

The whole source code is (note that I use 500 recording points

instead of 50):

```
<span class="n">library</span><span class="p">(</span><span class="n">grid</span><span class="p">);</span>
<span class="n">getData</span> <span class="o">=</span> <span class="k">function</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">.Platform</span><span class="o">$</span><span class="n">OS.type</span> <span class="o">==</span> <span class="s1">'windows'</span><span class="p">)</span> <span class="n">x11</span><span class="p">()</span> <span class="k">else</span> <span class="n">x11</span><span class="p">(</span><span class="n">type</span> <span class="o">=</span> <span class="s1">'Xlib'</span><span class="p">);</span>
<span class="n">pushViewport</span><span class="p">(</span><span class="n">viewport</span><span class="p">());</span>
<span class="n">grid.rect</span><span class="p">();</span>
<span class="n">px</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
<span class="n">py</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
<span class="n">mousedown</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">length</span><span class="p">(</span><span class="n">buttons</span><span class="p">)</span> <span class="o">></span> <span class="m">1</span> <span class="o">||</span> <span class="n">identical</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="m">2L</span><span class="p">))</span>
<span class="k">return</span><span class="p">(</span><span class="n">invisible</span><span class="p">(</span><span class="m">1</span><span class="p">));</span>
<span class="n">eventEnv</span><span class="o">$</span><span class="n">onMouseMove</span> <span class="o">=</span> <span class="n">mousemove</span><span class="p">;</span>
<span class="n">NULL</span>
<span class="p">}</span>
<span class="n">mousemove</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">px</span> <span class="o"><<-</span> <span class="n">c</span><span class="p">(</span><span class="n">px</span><span class="p">,</span> <span class="n">x</span><span class="p">);</span>
<span class="n">py</span> <span class="o"><<-</span> <span class="n">c</span><span class="p">(</span><span class="n">py</span><span class="p">,</span> <span class="n">y</span><span class="p">);</span>
<span class="n">grid.points</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">);</span>
<span class="n">NULL</span>
<span class="p">}</span>
<span class="n">mouseup</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span> <span class="p">{</span>
<span class="n">eventEnv</span><span class="o">$</span><span class="n">onMouseMove</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
<span class="n">NULL</span>
<span class="p">}</span>
<span class="n">setGraphicsEventHandlers</span><span class="p">(</span><span class="n">onMouseDown</span> <span class="o">=</span> <span class="n">mousedown</span><span class="p">,</span>
<span class="n">onMouseUp</span> <span class="o">=</span> <span class="n">mouseup</span><span class="p">);</span>
<span class="n">eventEnv</span> <span class="o">=</span> <span class="n">getGraphicsEventEnv</span><span class="p">();</span>
<span class="n">cat</span><span class="p">(</span><span class="s2">"Click down left mouse button and drag to draw the number,
right click to finish.n"</span><span class="p">);</span>
<span class="n">getGraphicsEvent</span><span class="p">();</span>
<span class="n">dev.off</span><span class="p">();</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">seq</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="n">length.out</span> <span class="o">=</span> <span class="n">length</span><span class="p">(</span><span class="n">px</span><span class="p">));</span>
<span class="n">spx</span> <span class="o">=</span> <span class="n">spline</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">px</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="m">500</span><span class="p">)</span><span class="o">$</span><span class="n">y</span><span class="p">;</span>
<span class="n">spy</span> <span class="o">=</span> <span class="n">spline</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">py</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="m">500</span><span class="p">)</span><span class="o">$</span><span class="n">y</span><span class="p">;</span>
<span class="k">return</span><span class="p">(</span><span class="n">cbind</span><span class="p">(</span><span class="n">spx</span><span class="p">,</span> <span class="n">spy</span><span class="p">));</span>
<span class="p">}</span>
<span class="n">traceCorr</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">dat1</span><span class="p">,</span> <span class="n">dat2</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">cor</span><span class="p">(</span><span class="n">dat1</span><span class="p">[,</span> <span class="m">1</span><span class="p">],</span> <span class="n">dat2</span><span class="p">[,</span> <span class="m">1</span><span class="p">])</span> <span class="o">+</span> <span class="n">cor</span><span class="p">(</span><span class="n">dat1</span><span class="p">[,</span> <span class="m">2</span><span class="p">],</span> <span class="n">dat2</span><span class="p">[,</span> <span class="m">2</span><span class="p">]);</span>
<span class="p">}</span>
<span class="c1"># Please set the proper path of this file.
</span><span class="n">load</span><span class="p">(</span><span class="s2">"train.RData"</span><span class="p">);</span>
<span class="n">guess</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">verbose</span> <span class="o">=</span> <span class="n">FALSE</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">test</span> <span class="o">=</span> <span class="n">getData</span><span class="p">();</span>
<span class="n">coefs</span> <span class="o">=</span> <span class="n">sapply</span><span class="p">(</span><span class="n">recogTrain</span><span class="p">,</span> <span class="n">traceCorr</span><span class="p">,</span> <span class="n">dat2</span> <span class="o">=</span> <span class="n">test</span><span class="p">);</span>
<span class="n">num</span> <span class="o">=</span> <span class="n">which.max</span><span class="p">(</span><span class="n">coefs</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">num</span> <span class="o">==</span> <span class="m">10</span><span class="p">)</span> <span class="n">num</span> <span class="o">=</span> <span class="m">0</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">verbose</span><span class="p">)</span> <span class="n">print</span><span class="p">(</span><span class="n">coefs</span><span class="p">);</span>
<span class="n">cat</span><span class="p">(</span><span class="s2">"I guess what you have input is "</span><span class="p">,</span> <span class="n">num</span><span class="p">,</span> <span class="s2">".n"</span><span class="p">,</span> <span class="n">sep</span> <span class="o">=</span> <span class="s2">""</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">guess</span><span class="p">();</span>
```

To run the code, you must load the “training set”, the file

`train.RData`

, into R using the `load()`

function, and then call

`guess()`

to play with it.

Have fun!

Download: Source code and training dataset

**leave a comment**for the author, please follow the link and comment on their blog:

**Yixuan's Blog - R**.

R-bloggers.com offers

**daily e-mail updates**about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.

Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.