Detecting outliers and fraud with R and SQL Server on my bank account data – Part 1

[This article was first published on R – TomazTsql, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Detecting outliers and fraudulent behaviour (transactions, purchases, events, actions, triggers, etc.) takes a large amount of experiences and statistical/mathetmatical background.

One of the samples Microsoft provided with release of new SQL Server 2016 was using simple logic of Benford’s law. This law works great with naturally occurring numbers and can be applied across any kind of problem. By naturally occurring, it is meant a number that is not generated generically such as a page number in a book, incremented number in your SQL Table, sequence number of any kind, but numbers that are occurring irrespective from each other, in nature (length or width of trees, mountains, rivers), length of the roads in the cities, addresses in your home town, city/country populations, etc. The law calculates the log distribution of numbers from 1 to 9 and stipulates that number one will occur 30% of times, number two will occur 17% of time, number three will occur 12% of the time and so on. Randomly generated numbers will most certainly generate distribution for each number from 1 to 9 with probability of 1/9. It might also not work with restrictions; for example height expressed in inches will surely not produce Benford function. My height is 188 which is 74 inches or 6ft2. All three numbers will not generate correct distribution, even though height is natural phenomena.

So Probability of number starting with number n equals to log(n+1) – log(n) with base 10. Keeping in mind this formula. So what is probability that a number starts with 29, is log(30) – log(29) = 1.4771 – 1.4623 = 0.0148. But keep in mind that this law applies only to numbers where number 1 appears approx. 30% of the time.

So just a quick example to support this theory, we will take

benfordL <- function(d){return(log(1+1/(1:9))/log(10))}
#distribution of numbers according to formula
#plot with numbers

Scatterplot with  log distribution of numbers from 1 to 9:


Code for Benford’s Law is available at RosettaCode for your favorite programming language.

Now I want to check this rule agains the data in WideWorldImporters. I will use the following query to test Benford’s Law:

SELECT TransactionAmount
FROM [WideWorldImporters].[Sales].[CustomerTransactions]
    TransactionAmount > 0

For this manner I will execute R Script:

SET @RScript = N'
            WWWTrans <- InputDataSet
            get_first_num <- function(number){return(as.numeric(substr(number,1,1)))}
            pbenford <- function(d){return(log10(1+(1/d)))}
            lead_dig <- mapply(get_first_num,WWWTrans$TransactionAmount)
            obs_freq_WWWTrans <- table(lead_dig)/1000
            OutputDataSet <- data.frame(obs_freq_WWWTrans)'

SET @SQLScript = N'
        FROM [WideWorldImporters].[Sales].[CustomerTransactions]
            TransactionAmount > 0'

 EXECUTE sp_execute_external_script
          @language = N'R'
        , @script = @RScript
        , @input_data_1 = @SQLScript
WITH result SETS ((
                     lead_dig INT
                    ,Freq DECIMAL(10,2)


By comparing distribution of general Benford’s Law and TransactionAmount there are some discrepancies but in general they can follow same log distribution.


In Part 2 I will cover outlier detection with GLM and Random forest using my bank account dataset.

For Benford’s Law there are couple of R packages available Benford.Analysis and BenfordTests.

To leave a comment for the author, please follow the link and comment on their blog: R – TomazTsql. offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)