Ron Pearson (aka TheNoodleDoodler)

David Olive’s median confidence interval

April 21, 2012

As I have discussed in a number of previous posts, the median represents a well-known and widely-used estimate of the “center” of a data sequence. Relative to the better-known mean, the primary advantage of the median is its much reduced outlier sensitivity. This post briefly describes a simple confidence interval for the median that is discussed in a paper...

Gastwirth’s location estimator

March 3, 2012

By Ron Pearson (aka TheNoodleDoodler)

The problem of outliers – data points that are substantially inconsistent with the majority of the other points in a dataset – arises frequently in the analysis of numerical data. The practical importance of outliers lies in the fact that even a few of these points can badly distort the results of an otherwise reasonable data analysis. This outlier-sensitivity...

Measuring associations between non-numeric variables

February 4, 2012

By Ron Pearson (aka TheNoodleDoodler)

Measuring associations between non-numeric variables

It is often useful to know how strongly or weakly two variables are associated: do they vary together or are they essentially unrelated? In the case of numerical variables, the best-known measure of association is the product-moment correlation coefficient introduced by Karl Pearson at the end of the nineteenth century. For variables that are ordered but not necessarily numeric...

Moving window filters and the pracma package

January 14, 2012

By Ron Pearson (aka TheNoodleDoodler)

Moving window filters and the pracma package

In my last post, I discussed the Hampel filter, a useful moving window nonlinear data cleaning filter that is available in the R package pracma. In this post, I briefly discuss this moving window filter in a little more detail, focusing on two important practical points: the choice of the filter’s local outlier detection threshold, and the question of...

Cleaning time-series and other data streams

November 27, 2011

By Ron Pearson (aka TheNoodleDoodler)

Cleaning time-series and other data streams

The need to analyze time-series or other forms of streaming data arises frequently in many different application areas. Examples include economic time-series like stock prices, exchange rates, or unemployment figures, biomedical data sequences like electrocardiograms or electroencephalograms, or industrial process operating data sequences like temperatures, pressures or concentrations. As a specific example, the figure below shows four data sequences:...

Harmonic means, reciprocals, and ratios of random variables

November 11, 2011

By Ron Pearson (aka TheNoodleDoodler)

Harmonic means, reciprocals, and ratios of random variables

In my last few posts, I have considered “long-tailed” distributions whose probability density decays much more slowly than standard distributions like the Gaussian. For these slowly-decaying distributions, the harmonic mean often turns out to be a much better (i.e., less variable) characterization than the arithmetic mean, which is generally not even well-defined theoretically for these distributions. Since the harmonic...

The Zipf and Zipf-Mandelbrot distributions

October 23, 2011

By Ron Pearson (aka TheNoodleDoodler)

The Zipf and Zipf-Mandelbrot distributions

In my last few posts, I have been discussing some of the consequences of the slow decay rate of the tail of the Pareto type I distribution, along with some other, closely related notions, all in the context of continuously distributed data. Today’s post considers the Zipf distribution for discrete data, which has come to be extremely popular as...

Is the “Long Tail” a Useless Concept?

September 28, 2011

By Ron Pearson (aka TheNoodleDoodler)

In response to my last post, “The Long Tail of the Pareto Distribution,” Neil Gunther had the following comment:

“Unfortunately, you've fallen into the trap of using the ‘long tail’ misnomer. If you think about it, it can't possibly be the length of the tail that sets distributions like Pareto and Zipf apart; even the negative exponential and Gaussian...
Read more »

The Long Tail of the Pareto Distribution

September 17, 2011

By Ron Pearson (aka TheNoodleDoodler)

The Long Tail of the Pareto Distribution

In my last two posts, I have discussed cases where the mean is of little or no use as a data characterization. One of the specific examples I discussed last time was the case of the Pareto type I distribution, for which the density is given by:

p(x) = ak^a/x^a+1

defined for all x > k, where k and a...

Some Additional Thoughts on Useless Averages

August 27, 2011

By Ron Pearson (aka TheNoodleDoodler)

Some Additional Thoughts on Useless Averages

In my last post, I described three situations where the average of a sequence of numbers is not representative enough to be useful: in the presence of severe outliers, in the face of multimodal data distributions, and in the face of infinite-variance distributions. The post generated three interesting comments that I want to respond to here.

First and foremost, I...