I visited R-Bloggers and they redirected me today to http://blog.yhathq.com/posts/rodeo-native.html
need to check it out.
wtorek, 22 grudnia 2015
poniedziałek, 21 grudnia 2015
Politico: Sober look at Polish politics
http://www.politico.eu/article/give-pis-a-chance-poland-kaczynski-andrzej-duda-szydlo/
CoffeeBreak: Outliers2, what about skewness
My new version of outlier's highlighting VBA
Sub outliers_IQR()
Dim Rng As Range, rTest As Range: Set rTest = Selection
Dim rQ1 As Double: rQ1 = WorksheetFunction.Quartile(rTest, 1)
Dim rQ3 As Double: rQ3 = WorksheetFunction.Quartile(rTest, 3)
Dim IQR As Double: IQR = Q3 - Q1
Dim Factor As Double: Factor = 2.2
For Each Rng In rTest
If Rng > (rQ3 + Factor * IQR) Or Rng < (rQ1 - Factor * IQR) Then
Rng.Interior.Color = RGB(255, 0, 0) '.Value = "Outlier" 'or delete the data with Rng.clearcontents
Else
Rng.Interior.Color = xlNone
End If
Next
End Sub
What about skewness? The proposal to deal with it can be found here: https://wis.kuleuven.be/stat/robust/papers/2008/outlierdetectionskeweddata-revision.pdf
I need to decipher this math and implement it above.
Some related reading:
http://www.real-statistics.com/students-t-distribution/identifying-outliers-using-t-distribution/grubbs-test/
http://datapigtechnologies.com/blog/index.php/highlighting-outliers-in-your-data-with-the-tukey-method/
http://stats.stackexchange.com/questions/60235/how-accurate-is-iqr-for-detecting-outliers
Sub outliers_IQR()
Dim Rng As Range, rTest As Range: Set rTest = Selection
Dim rQ1 As Double: rQ1 = WorksheetFunction.Quartile(rTest, 1)
Dim rQ3 As Double: rQ3 = WorksheetFunction.Quartile(rTest, 3)
Dim IQR As Double: IQR = Q3 - Q1
Dim Factor As Double: Factor = 2.2
For Each Rng In rTest
If Rng > (rQ3 + Factor * IQR) Or Rng < (rQ1 - Factor * IQR) Then
Rng.Interior.Color = RGB(255, 0, 0) '.Value = "Outlier" 'or delete the data with Rng.clearcontents
Else
Rng.Interior.Color = xlNone
End If
Next
End Sub
What about skewness? The proposal to deal with it can be found here: https://wis.kuleuven.be/stat/robust/papers/2008/outlierdetectionskeweddata-revision.pdf
I need to decipher this math and implement it above.
Some related reading:
http://www.real-statistics.com/students-t-distribution/identifying-outliers-using-t-distribution/grubbs-test/
http://datapigtechnologies.com/blog/index.php/highlighting-outliers-in-your-data-with-the-tukey-method/
http://stats.stackexchange.com/questions/60235/how-accurate-is-iqr-for-detecting-outliers
Excel mark duplicates.
Simple and interesting:
- in one helper column concatenate the columns you are checking for duplicates.
- in another =COUNTIF([concatenated_colum_range];checked_concatenated_value)>1
It returns TRUE when there are more than 1.
Found here: http://www.ozgrid.com/forum/showthread.php?t=59302 and http://www.ozgrid.com/Excel/highlight-duplicates.htm and http://blog.contextures.com/archives/2013/04/11/highlight-duplicate-records-in-an-excel-list/
- in one helper column concatenate the columns you are checking for duplicates.
- in another =COUNTIF([concatenated_colum_range];checked_concatenated_value)>1
It returns TRUE when there are more than 1.
Found here: http://www.ozgrid.com/forum/showthread.php?t=59302 and http://www.ozgrid.com/Excel/highlight-duplicates.htm and http://blog.contextures.com/archives/2013/04/11/highlight-duplicate-records-in-an-excel-list/
czwartek, 17 grudnia 2015
CoffeeBreak: Outliers, Kernel Density
Reading on using IQR to identify outliers in Excel:
http://datapigtechnologies.com/blog/index.php/highlighting-outliers-in-your-data-with-the-tukey-method/
http://brownmath.com/stat/nchkxl.htm
esp. see worksheet: http://brownmath.com/stat/prog/normalitycheck.xlsm
Using SD, not a good idea but a good macro to start with:
http://www.mrexcel.com/forum/excel-questions/732424-how-remove-outliers-data-set-2.html
Cheatsheets for R: distributions: http://www.r-bloggers.com/ggplot2-cheatsheet-for-visualizing-distributions/
Good Wikipedia entry: https://en.wikipedia.org/wiki/Outlier
Advanced article: http://d-scholarship.pitt.edu/7948/1/Seo.pdf
For beginners: https://www.dataz.io/display/Public/2013/03/20/Describing+Data%3A+Why+median+and+IQR+are+often+better+than+mean+and+standard+deviation
====
Simple MAD solution with Excel formulas: http://www.codeproject.com/Tips/214330/Statistical-Outliers-detection
http://datapigtechnologies.com/blog/index.php/highlighting-outliers-in-your-data-with-the-tukey-method/
http://brownmath.com/stat/nchkxl.htm
esp. see worksheet: http://brownmath.com/stat/prog/normalitycheck.xlsm
Using SD, not a good idea but a good macro to start with:
http://www.mrexcel.com/forum/excel-questions/732424-how-remove-outliers-data-set-2.html
Sub outliers_mod2()
Dim dblAverage As Double, dblStdDev As Double
Dim NoStdDevs As Integer
Dim rTest As Range, Rng As Range
'Application.ScreenUpdating = False
NoStdDevs = 3 'adjust to your outlier preference of sigma
Set rTest = Selection 'Application.InputBox("Select a range", "Get Range", Type:=8)
dblAverage = WorksheetFunction.Average(rTest)
dblStdDev = WorksheetFunction.StDev(rTest)
For Each Rng In rTest
If Rng > dblAverage + NoStdDevs * dblStdDev Or Rng < dblAverage - NoStdDevs * dblStdDev Then
Rng.Interior.Color = RGB(255, 0, 0) '.Value = "Outlier" 'or delete the data with Rng.clearcontents
End If
Next
'Application.ScreenUpdating = True
End Sub
====
Normalize(simple linear normalize) data in Excel
http://stats.stackexchange.com/questions/70801/how-to-normalize-data-to-0-1-range?newreg=f530ce192d144b109ec26a077cab00af
Kernel Density/Regression, alternatives to histogram.
====
good article: http://www.stat-d.si/mz/mz4.1/vidmar.pdf http://people.revoledu.com/kardi/tutorial/index.html
2d: http://www.r-bloggers.com/recipe-for-computing-and-sampling-multivariate-kernel-density-estimates-and-plotting-contours-for-2d-kdes/
Density plugin: http://www.prodomosua.eu/ppage02.html
Good VBA example for density plots (incl UDF function).
http://www.iimahd.ernet.in/~jrvarma/software.php
(found in this page: http://www.mathfinance.cn/category/vba/1/5/)
Some R code http://www.wessa.net/rwasp_density.wasp#output
Another article: http://www.rsc.org/images/data-distributions-kernel-density-technical-brief-4_tcm18-214836.pdf
Some plugin/vba to check: http://www.rsc.org/Membership/Networking/InterestGroups/Analytical/AMC/Software/RobustStatistics.asp
Cheatsheets for R: distributions: http://www.r-bloggers.com/ggplot2-cheatsheet-for-visualizing-distributions/
Good Wikipedia entry: https://en.wikipedia.org/wiki/Outlier
Advanced article: http://d-scholarship.pitt.edu/7948/1/Seo.pdf
For beginners: https://www.dataz.io/display/Public/2013/03/20/Describing+Data%3A+Why+median+and+IQR+are+often+better+than+mean+and+standard+deviation
====
Simple MAD solution with Excel formulas: http://www.codeproject.com/Tips/214330/Statistical-Outliers-detection
wtorek, 8 grudnia 2015
Coffee break: MSAccess Transform/Pivot syntax.
http://blogannath.blogspot.com/2010/02/microsoft-access-tips-tricks-crosstab.html
http://stackoverflow.com/questions/27710313/ms-access-sql-transform-aggregate-manipluation-of-values-for-pivot
pivot datepart (aby otrzymać quarters)
https://msdn.microsoft.com/en-us/library/bb208956%28v=office.12%29.aspx
http://weblogs.sqlteam.com/jeffs/archive/2007/09/10/group-by-month-sql.aspx
sum if
http://allenbrowne.com/ser-67.html
http://www.blueclaw-db.com/accessquerysql/crosstab.htm
Crosstabs on two columns
http://stackoverflow.com/questions/24706235/sql-pivot-cross-tab-and-multiple-columns
http://stackoverflow.com/questions/27710313/ms-access-sql-transform-aggregate-manipluation-of-values-for-pivot
pivot datepart (aby otrzymać quarters)
https://msdn.microsoft.com/en-us/library/bb208956%28v=office.12%29.aspx
http://weblogs.sqlteam.com/jeffs/archive/2007/09/10/group-by-month-sql.aspx
sum if
http://allenbrowne.com/ser-67.html
http://www.blueclaw-db.com/accessquerysql/crosstab.htm
Crosstabs on two columns
http://stackoverflow.com/questions/24706235/sql-pivot-cross-tab-and-multiple-columns
Subskrybuj:
Posty (Atom)