Handwriting recognition using R

[This article was first published on Yixuan's Blog - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

This title is a bit exaggerating since handwriting recognition is an advanced topic
in machine learning involving complex techniques and algorithms. In this blog I’ll
show you a simple demo illustrating how to recognize a single number (0 ~ 9) using R.
The overall process is that, you draw a number in a graphics device in R using your mouse,
and then the program will “guess” what you have input. It is just for FUN.

There are two major problems in this number recognition problem, that
is, how to describe the trace of your handwriting, and how to classify
this trace to the give classes (0 ~ 9).

For the first question, we could first detect the motion of your mouse
in the graphics device, and then record the coordinates of you mouse
cursor at a sequence of time points. This could be done via the
getGraphicsEvent() function in grDevices package. For example, after I
drew a number 2 in the graphics window like below, the coordinates of
each point in the trace were assigned to a pair of variables px and py.

Record trace

The scatterplot of px and py versus their orders in the trace is
shown below.

Record points

To be comparable among different traces, we normalize the Order to be
within (0, 1] (that is, transform 1, 2, …, n to 1/n, 2/n, …, 1).
Also, since this recording is discrete but the real trace should be
continuous, we use the spline() function to interpolate at unknown
points, resulting in the following figure.

Record splines

The dots in the figure have normalized orders of 0.02, 0.04,
0.06, …, 1, at which the x and y coordinates are obtained by
interpolation. Therefore, we could use $r = (x, y)$ where
$x = (x_1, x_2, …, x_{50})’$ and $y = (y_1, y_2, …, y_{50})’$ to
represent the information of the number 2 I have drawn. Somewhat
confused by the operations above? Well, the idea behind this
normalization and interpolation is simple: use 50 “uniformly
ordered” points (I call them “recording points”) to represent the trace.

So it comes to the second question – given a trace, how to classify
it? Obviously we first need a training set, the recording points of
number 0 to number 9 generated as above. Then we’ll compare the
given trace with each one in the training set and find out which
number resembles it most.

Several criteria could be used to measure the similarity, but some
important rules should be considered. We still use $r = (x, y)$ to
represent the recording points of a trace, and use $Sim(r_1, r_2)$ to
stand for the similarity between two traces. Notice that this
similarity should not be sensitive to the scale and location of
traces. That is, if I draw a number in another location in the
window, or in a larger or smaller size, the recognition should not be
influenced. In mathematics, this could be expressed by

where $k_1 > 0$, $k_2 > 0$, $b_1$, $b_2$ are real numbers.

In my code, I simply define the similarity as the sum of Pearson
correlation coefficients of x and y, that is,

The whole source code is (note that I use 500 recording points
instead of 50):

<span class="n">library</span><span class="p">(</span><span class="n">grid</span><span class="p">);</span>
<span class="n">getData</span> <span class="o">=</span> <span class="k">function</span><span class="p">()</span>
<span class="p">{</span>
    <span class="k">if</span><span class="p">(</span><span class="n">.Platform</span><span class="o">$</span><span class="n">OS.type</span> <span class="o">==</span> <span class="s1">'windows'</span><span class="p">)</span> <span class="n">x11</span><span class="p">()</span> <span class="k">else</span> <span class="n">x11</span><span class="p">(</span><span class="n">type</span> <span class="o">=</span> <span class="s1">'Xlib'</span><span class="p">);</span>
    <span class="n">pushViewport</span><span class="p">(</span><span class="n">viewport</span><span class="p">());</span>
    <span class="n">grid.rect</span><span class="p">();</span>
    <span class="n">px</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
    <span class="n">py</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
    <span class="n">mousedown</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
    <span class="p">{</span>
        <span class="k">if</span><span class="p">(</span><span class="n">length</span><span class="p">(</span><span class="n">buttons</span><span class="p">)</span> <span class="o">></span> <span class="m">1</span> <span class="o">||</span> <span class="n">identical</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="m">2L</span><span class="p">))</span>
            <span class="k">return</span><span class="p">(</span><span class="n">invisible</span><span class="p">(</span><span class="m">1</span><span class="p">));</span>
        <span class="n">eventEnv</span><span class="o">$</span><span class="n">onMouseMove</span> <span class="o">=</span> <span class="n">mousemove</span><span class="p">;</span>
        <span class="n">NULL</span>
    <span class="p">}</span>
    <span class="n">mousemove</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
    <span class="p">{</span>
        <span class="n">px</span> <span class="o"><<-</span> <span class="n">c</span><span class="p">(</span><span class="n">px</span><span class="p">,</span> <span class="n">x</span><span class="p">);</span>
        <span class="n">py</span> <span class="o"><<-</span> <span class="n">c</span><span class="p">(</span><span class="n">py</span><span class="p">,</span> <span class="n">y</span><span class="p">);</span>
        <span class="n">grid.points</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">);</span>
        <span class="n">NULL</span>
    <span class="p">}</span>
    <span class="n">mouseup</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">buttons</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">eventEnv</span><span class="o">$</span><span class="n">onMouseMove</span> <span class="o">=</span> <span class="n">NULL</span><span class="p">;</span>
        <span class="n">NULL</span>
    <span class="p">}</span>
    <span class="n">setGraphicsEventHandlers</span><span class="p">(</span><span class="n">onMouseDown</span> <span class="o">=</span> <span class="n">mousedown</span><span class="p">,</span>
                             <span class="n">onMouseUp</span> <span class="o">=</span> <span class="n">mouseup</span><span class="p">);</span>
    <span class="n">eventEnv</span> <span class="o">=</span> <span class="n">getGraphicsEventEnv</span><span class="p">();</span>
    <span class="n">cat</span><span class="p">(</span><span class="s2">"Click down left mouse button and drag to draw the number,
        right click to finish.n"</span><span class="p">);</span>
    <span class="n">getGraphicsEvent</span><span class="p">();</span>
    <span class="n">dev.off</span><span class="p">();</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">seq</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="m">1</span><span class="p">,</span> <span class="n">length.out</span> <span class="o">=</span> <span class="n">length</span><span class="p">(</span><span class="n">px</span><span class="p">));</span>
    <span class="n">spx</span> <span class="o">=</span> <span class="n">spline</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">px</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="m">500</span><span class="p">)</span><span class="o">$</span><span class="n">y</span><span class="p">;</span>
    <span class="n">spy</span> <span class="o">=</span> <span class="n">spline</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">py</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="m">500</span><span class="p">)</span><span class="o">$</span><span class="n">y</span><span class="p">;</span>
    <span class="k">return</span><span class="p">(</span><span class="n">cbind</span><span class="p">(</span><span class="n">spx</span><span class="p">,</span> <span class="n">spy</span><span class="p">));</span>
<span class="p">}</span>
<span class="n">traceCorr</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">dat1</span><span class="p">,</span> <span class="n">dat2</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">cor</span><span class="p">(</span><span class="n">dat1</span><span class="p">[,</span> <span class="m">1</span><span class="p">],</span> <span class="n">dat2</span><span class="p">[,</span> <span class="m">1</span><span class="p">])</span> <span class="o">+</span> <span class="n">cor</span><span class="p">(</span><span class="n">dat1</span><span class="p">[,</span> <span class="m">2</span><span class="p">],</span> <span class="n">dat2</span><span class="p">[,</span> <span class="m">2</span><span class="p">]);</span>
<span class="p">}</span>

<span class="c1"># Please set the proper path of this file.
</span><span class="n">load</span><span class="p">(</span><span class="s2">"train.RData"</span><span class="p">);</span>
<span class="n">guess</span> <span class="o">=</span> <span class="k">function</span><span class="p">(</span><span class="n">verbose</span> <span class="o">=</span> <span class="n">FALSE</span><span class="p">)</span>
<span class="p">{</span>
    <span class="n">test</span> <span class="o">=</span> <span class="n">getData</span><span class="p">();</span>
    <span class="n">coefs</span> <span class="o">=</span> <span class="n">sapply</span><span class="p">(</span><span class="n">recogTrain</span><span class="p">,</span> <span class="n">traceCorr</span><span class="p">,</span> <span class="n">dat2</span> <span class="o">=</span> <span class="n">test</span><span class="p">);</span>
    <span class="n">num</span> <span class="o">=</span> <span class="n">which.max</span><span class="p">(</span><span class="n">coefs</span><span class="p">);</span>
    <span class="k">if</span><span class="p">(</span><span class="n">num</span> <span class="o">==</span> <span class="m">10</span><span class="p">)</span> <span class="n">num</span> <span class="o">=</span> <span class="m">0</span><span class="p">;</span>
    <span class="k">if</span><span class="p">(</span><span class="n">verbose</span><span class="p">)</span> <span class="n">print</span><span class="p">(</span><span class="n">coefs</span><span class="p">);</span>
    <span class="n">cat</span><span class="p">(</span><span class="s2">"I guess what you have input is "</span><span class="p">,</span> <span class="n">num</span><span class="p">,</span> <span class="s2">".n"</span><span class="p">,</span> <span class="n">sep</span> <span class="o">=</span> <span class="s2">""</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">guess</span><span class="p">();</span>

To run the code, you must load the “training set”, the file
train.RData, into R using the load() function, and then call
guess() to play with it.

Have fun!

Download: Source code and training dataset

To leave a comment for the author, please follow the link and comment on their blog: Yixuan's Blog - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)