13,247,804 members (42,688 online)
Tip/Trick
alternative version

#### Stats

57.7K views
4 bookmarked
Posted 21 Jun 2011

# Statistical Outliers detection: worksheet solution

, 16 Feb 2015
 Rate this:
Statistical Outliers detection in Microsoft Excel worksheet using Median() and array formula

### Theoretical background

"Outliers" are defined as numeric values in any random data set, which have an unusually high deviation from either the statistical mean (average) or the median value. In other words, these numbers are either relatively very small or too big. Detecting the outliers in a data set represents a complex statistical problem, with a corresponding variety of different methodologies and computational techniques as described, for example, in the NIST publication [1]. In general, finding the "Outliers" in a data set could be done by calculating the deviation for each number, expressed as either a "Z-score" or "modified Z-score" and testing it against certain predefined threshold. Z-score typically refers to number of standard deviation relative to the statistical average (in other words, it's measured in "Sigmas"). Modified Z-score applies the median computation technique to measure the deviation and in many cases provides more robust statistical detection of outliers. Mathematically the Modified Z-score could be written (as suggested by Iglewicz and Hoaglin [1]) as:

`Mi=0.6745 * (Xi -Median(Xi)) / MAD,`

where MAD stands for Median Absolute Deviation. Any number in a data set with the absolute value of modified Z-score exceeding 3.5 is considered an "Outlier". Modified Z-score could be used to detect outliers in Microsoft Excel worksheet as described below.

### Sample computation of outliers in Excel worksheet using Media/MAD

Step 1. Open Microsoft Excel worksheet and enter a sample set of 10 randomly selected numbers in column A, starting with the first row: 3, 1, -23, 7, 0, 12, -2, 7, 2, 1 (note: don’t enter commas)

Step 2. In the first row of column C (in other words, C1) enter the formula:

`=MEDIAN(A1:A10)`

The value in this cell corresponds to the median calculated on a data set entered at step 1.

Step 3. In the second row of column C (in other words, C2) enter the array formula:

`{=MEDIAN(ABS(MEDIAN(A1:A10)-A1:A10))}`

The value in this cell (C2) corresponds to MAD [1].

Step 4. Enter the formula:

`=IF((ABS(C\$1-A1)>3.5*C\$2), "OUTLIER", "NORMAL")`

in the first row of column B and extend it down to the 10th row. Final result of “outlier’s detection” should appear in column B, indicating two "outliers" numbers (-23 and 12) as shown below:

```3 NORMAL
1 NORMAL
-23 OUTLIER
7 NORMAL
0 NORMAL
12 OUTLIER
-2 NORMAL
7 NORMAL
2 NORMAL
1 NORMAL```

### Sample computation of outliers using STD/Mean values

The first method described above in based on Modified Z-score and underlying Median/MAD computation. In many cases (particularly for small sample sizes) it provides more robust statistical outliers detection than a "traditional" z-score, which implies the detection based on Standard Deviation and Mean (average). Just for reference purposes, the sample computation below describes the outliers detection based on "2-Sigma" z-score; in other words, any numeric value in a data set considered an "outlier" if its absolute value of deviation from the statistical average exceeds 2*STD. Note: as mentioned above, the first method is typically more robust and recommended for small sample sizes:

Step 1. Use the same Excel Worksheet with sample set of 10 sample numbers in column A as described in previous example.

Step 2. Enter the formula: `=AVERAGE(A1:A10)` into the cell F1. This corresponds to the statistical mean calculated on a data set in column A.

Step 3. enter the formula: `=STDEV(A1:A10)` into the cell F2. This corresponds to the Standard Deviation.

Step 4. Enter the formula: `=IF(ABS(A1-F\$1)>2*F\$2,"OUTLIER", "NORMAL")`in the first row of column E and extend it down to the 10th row. Final result of the “outlier’s detection” using statistical STD/AVERAGE on z-score level of 2 should appear in column E, indicating only one "outlier" number: -23 (see below); this method is missing a second outliers, which is 12, as found in the previous method.

```NORMAL
NORMAL
OUTLIER
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL
NORMAL```

### Notes

Both methods of statistical detection of Outliers discussed above could be easily implemented within .NET framework. Suggested MS Excel implementation based upon worksheet functions was chosen mostly to serve the didactic/demo purposes, providing the maximum level of simplicity and clarity. For more details on statistical outliers detection methodology and computation technique you could refer to the reference [1]. Also, various statistical calculations could be performed online by using free Statistics Calculator [2], which is actually implemented as ASP.NET/C# solution.

## Share

 President Infosoft International Inc United States
Dr. Alexander Bell is a seasoned full-stack Software Engineer (Win/Web/Mobile). He holds PhD in EE/IT, authored 37 inventions and published 300+ technical articles. Currently focused on HTML5/CSS3, Javascript, .NET/WPF/C#, Angular.js, SQL, 'Big Data', Machine Learning, AI, IoT. Alex participated in App Innovation Contests (AIC 2102/2013) with multiple winning submissions. Portfolio samples:

## You may also be interested in...

 Pro Pro

 First Prev Next
 Small error walobu18-Feb-15 15:37 walobu 18-Feb-15 15:37
 Re: Thanks a bunch, ASP-DOT-NET-DEV-U-THE-BEST! :) Actually, I t... DrABELL30-Jun-11 14:37 DrABELL 30-Jun-11 14:37
 Reason for my vote of 5 A robust, simple and efficient metho... Alex Tuma28-Jun-11 10:22 Alex Tuma 28-Jun-11 10:22
 Re: Hi Alex, You are absolutely right: this very robust yet simp... DrABELL28-Jun-11 15:03 DrABELL 28-Jun-11 15:03
 Hi Walt, Thanks for your note. Let's me clarify couple thing... DrABELL28-Jun-11 4:20 DrABELL 28-Jun-11 4:20
 It appears that you are assuming a normal distribution, with... Walt Fair, Jr.25-Jun-11 17:22 Walt Fair, Jr. 25-Jun-11 17:22
 Re: FYI, Walt, DrABELL posted a comment but did not post it as a... AspDotNetDev30-Jun-11 9:58 AspDotNetDev 30-Jun-11 9:58
 Last Visit: 31-Dec-99 19:00     Last Update: 17-Nov-17 21:32 Refresh 1