tag:blogger.com,1999:blog-87185533328265730262024-03-14T06:07:38.675-07:00Jacek Kotowski's toolbox.Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.comBlogger567125tag:blogger.com,1999:blog-8718553332826573026.post-78681102372252694772018-07-04T05:49:00.000-07:002018-07-04T05:58:43.481-07:00Simple kriging playground. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja6uuRepnTEg11-c0SJEP0xZN28Ec-EvH3QF-vbs_INJZ79l5PvR70w8Lv60EYUtkGNqvcHIKfsZ-l-wvbQ0qiQ2uX0Rih7-Z1_L-1-xlOHi_lTYlA0OhfZKrl-2UeLyIBXdhc0A77zVk/s1600/Rplot.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="534" data-original-width="754" height="452" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja6uuRepnTEg11-c0SJEP0xZN28Ec-EvH3QF-vbs_INJZ79l5PvR70w8Lv60EYUtkGNqvcHIKfsZ-l-wvbQ0qiQ2uX0Rih7-Z1_L-1-xlOHi_lTYlA0OhfZKrl-2UeLyIBXdhc0A77zVk/s640/Rplot.png" width="640" /></a></div>
<br />
<br />
<span id="goog_145201275"></span><span id="goog_145201276"></span><br />
<br />
<br />
Mój playground z Małopolską.
<br />
<br />
<br />
<br />
<pre>
library(tidyverse)
library(gstat)
library(sf)
# ---02. Przerabiam dane wysokościowe
dane_plik <- "dane/malopolskie.txt"
dane_nmt <- read_delim(dane_plik, " ", col_names = c("x", "y", "z"), col_types = "ddd") %>%
sample_n(size = 1000, replace=F) %>%
st_as_sf( coords = c("x", "y"), crs = 2180, agr = "constant", precision = 0.1) %>%
st_transform(crs = 4326) %>%
as_Spatial(.)
#---03. Przerabiam shp na grid
woj_TERYT <- "12"
malopolska_shp <- st_read("dane/województwa.shp") %>%
st_transform(crs = 4326) %>%
filter(jpt_kod_je==woj_TERYT)
# grid
bbox<- st_bbox(malopolska_shp)
lon <- seq(bbox$xmin, bbox$xmax, length.out = 500)
lat <- seq(bbox$ymin, bbox$ymax, length.out = 500)
grd <- expand.grid(lon = lon, lat = lat) %>%
st_as_sf(coords = c("lon", "lat"), crs = 4326, agr = "constant") %>%
st_join(malopolska_shp, left=FALSE) %>%
as_Spatial(.)
malopolska_shp<-malopolska_shp %>% as_Spatial()
#---04. Variogram
#Jak w lm z~(jakaś zmienna np stężenie w glebie ~ odległość od rzeki)
dt.vgm <- variogram(z~1,dane_nmt)
class(dt.vgm)
dt.fit <-
fit.variogram(dt.vgm, model = vgm(1,"Lin",600,1)) # fit model
# vgm() list of models
plot(dt.vgm, dt.fit)
### 05. Kriging
lzn.kriged <- krige((z) ~ 1, dane_nmt , grd, model=dt.fit)
lzn.kriged %>% as.data.frame %>% rename(lon=coords.x1, lat=coords.x2) %>%
ggplot(aes(x=lon, y=lat)) + geom_tile(aes(fill=var1.pred)) + coord_equal() +
scale_fill_gradient2(low="green", mid = "yellow", high="red",midpoint = 0) +
theme_bw()
</pre>
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-69617721500647575682017-09-12T01:15:00.005-07:002017-09-12T02:00:49.353-07:00Dlaczego R? hclusthclust - Próba analizy odpowiedzi w formularzu zapisów na konferencję WhyR. kolumna: "Chcesz.podzielić.się.odpowiedzią.na.to.pytanie..Chętnie.przedstawimy.najciekawsze.odpowiedzi"<br />
- bo lingua franca,<br />
- bo ogromne możliwości,<br />
- bo ggplot2,<br />
- bo nie Python, bo nie SAS,<br />
- bo za free,<br />
- bo Data Science...<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdMaujpAsF-A5iOoESSR442-w7T-FL1QBzMbitvK63cxpl9fk8IHSPTBkefld4RNsj4i5S4BotreJfNno0W8PUnHEm0lT47doU3J7C4-ucfrJOa4lgUPGbUONIRxt0H5J7TSIDIJOERb4/s1600/whyR.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="813" data-original-width="961" height="541" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdMaujpAsF-A5iOoESSR442-w7T-FL1QBzMbitvK63cxpl9fk8IHSPTBkefld4RNsj4i5S4BotreJfNno0W8PUnHEm0lT47doU3J7C4-ucfrJOa4lgUPGbUONIRxt0H5J7TSIDIJOERb4/s640/whyR.png" width="640" /></a></div>
<br />
Interpretacja subiektywna w Paincie :-)<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxVmrHq-cxIGumBjImRwYkc_A1gbVVy224lyjHYAu8hVMZ1_sf00junaJacOvPH6x0GUddgzoo22h-BlERHxfxTf_Fh2YkY1iD2X0s_NCI9z_-iAIsWGVsN6DaI1W1KwPTw_CB99tBNVY/s1600/whyR2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="813" data-original-width="961" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxVmrHq-cxIGumBjImRwYkc_A1gbVVy224lyjHYAu8hVMZ1_sf00junaJacOvPH6x0GUddgzoo22h-BlERHxfxTf_Fh2YkY1iD2X0s_NCI9z_-iAIsWGVsN6DaI1W1KwPTw_CB99tBNVY/s640/whyR2.png" width="640" /></a></div>
R za darmo daje ogromne możliwości i pozwala się rozwijać i komunikować (lingua franca), szczególnie dzięki GGPlot2. Jest za darmo. Jest alternatywą dla SASa i Pythona.<br />
<br />
Edycja w EzGif.com<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW56LksUtOarr55MfJoeV9LgRX-qGbVAfHT3UYaCreSm4OGCJufi2rP1diMD1iJ4EWfJ6s9V_0e7wTzmouL_QCTsOXgOrFuRrPNjh3NVEuqv9q3yISsLtcx2gwDpUftvgQdEp8zNRRmmA/s1600/ezgif.com-optimize.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="813" data-original-width="961" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW56LksUtOarr55MfJoeV9LgRX-qGbVAfHT3UYaCreSm4OGCJufi2rP1diMD1iJ4EWfJ6s9V_0e7wTzmouL_QCTsOXgOrFuRrPNjh3NVEuqv9q3yISsLtcx2gwDpUftvgQdEp8zNRRmmA/s640/ezgif.com-optimize.gif" width="640" /></a></div>
<br /> Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-8149216064824457092017-06-09T02:02:00.002-07:002017-06-09T02:02:43.438-07:00Introductory Python ML, short 2 day course.Conclusions:<br />
1. R is so much easier, portable, supercool. Jupyter Notebooks are far from Rstudio R Notebooks. <br />2. Python can be learned/is similar, just remember, indenting is part of syntax :-)<br />
3. I must learn Pandas! Scikit Learn! Seaborn! maybe by comparison to dplyr, ggplot2.<br /><br />Microsoft, please allow more of R/ExcelVBA/Python interoperability for all.<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-35285664847363817872017-06-09T01:49:00.004-07:002017-06-09T01:56:20.663-07:00Excel: Conditional formatting string numbers...Why I have never discovered that until today!!!:<br />
<br />
[=0]"";rrrr-mm-dd <br />
<br />
"If there is 0, then do not put an erroneous date (a result of some bug in Excel insisting there is a 1900-01-00 date)". Is there more? Like coloring the font dependent on conditions met? Supercool! Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-73688241847469681202017-05-06T14:49:00.000-07:002017-05-06T14:49:04.583-07:00Big Data on a laptop, some options. Data mining on streams.<br />
http://moa.cms.waikato.ac.nz/rmoa-massive-online-data-stream-classifications-with-r-moa/<br />
http://jwijffels.github.io/RMOA/<br />
https://cran.r-project.org/web/packages/stream/stream.pdf <br />
<br />
Database light backend.<br />
https://www.monetdb.org/blog/monetdblite-r<br />
<br />
Data mining<br />
https://rdrr.io/cran/ffbase/man/bigglm.ffdf.html<br />
https://cran.r-project.org/web/packages/speedglm/speedglm.pdf<br />
https://cran.r-project.org/web/packages/randomForest.ddR/randomForest.ddR.pdfDaimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-47035647408529941842017-05-05T07:00:00.001-07:002017-05-05T07:32:52.720-07:00Bayes for beginners videos.How to explain it plain E: <br />
https://www.khanacademy.org/math/statistics-probability/probability-library/conditional-probability-independence/v/calculating-conditional-probability <br />
http://www.watchknowlearn.org/Video.aspx?VideoID=16751&CategoryID=4457<br />
<br />
https://www.khanacademy.org/math/ap-statistics/probability-ap/stats-conditional-probability/v/bayes-theorem-visualized<br />
https://www.khanacademy.org/partner-content/wi-phi/wiphi-critical-thinking/wiphi-fundamentals/v/bayes-theorem<br />
https://www.youtube.com/watch?v=Y-V4rfdl3NI<br />
https://brilliant.org/wiki/bayes-theorem/<br />
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-17742312175721476172017-03-24T04:23:00.003-07:002017-03-24T05:21:14.261-07:00Risk matrix in R (interesting readings)<br />
<br />
1) Risk matrix examples
http://davidmeza1.github.io/2015/12/17/2015-12-17-Creating-a-Risk-Matrix-in-R.html <br />
<br />
2) Use ggrepel instead of jitter <a href="https://cran.r-project.org/web/packages/ggrepel/vignettes/ggrepel.html">https://cran.r-project.org/web/packages/ggrepel/vignettes/ggrepel.html</a><br />
for multiple points.<br />
<br />
3) Is there ggrepel for Excel (?) http://stackoverflow.com/questions/30294041/excel-bubble-chart-overlapping-data-labelDaimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-18958284785695726482017-03-21T10:08:00.002-07:002017-03-21T10:08:43.143-07:00Stats Day2: CovarianceCovariance <br />
Cov(for sample) = sum[(x - śr(x))*(y-śr(y))]/n-1 ... we are interesting in the sign. Does not provide strength. Is not standardized. <br />
<br />
Covariance matrix...diagonal shows variance of each variable, off-diagonal show covariances betw. each variable pair.<br />
<br />
Correlation (Pearson, r)<br />
r =Cov(x,y)/[SD(x)* SD(y)]Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-26536904294516923162017-03-20T05:49:00.002-07:002017-03-20T08:36:50.396-07:00Stats Day1With Spring day 1 I start regular intro statistics study:<br />
<br />
My notes from day 1: <br />
<b>Standard Error of Mean (SD of a sample):</b><br />
<b>SD/ root(n) </b>- the larger sample size, the error decreases,<br />
<br />
<br />
<b>Z-Score</b>: (x-mean)/SD<br />
https://en.wikipedia.org/wiki/Standard_score<br />
<br />
<b>T-Score</b>: <br />
Conversion Score = z * (NewSD)+NewMean<br />
NewSD for T-Score = 10 and NewMean = 50<br />
T-Score = Z-Score*10+50<br />
More: Essentials of Testing and Assessment: A Practical Guide for Counselors ...Autorzy Edward S. Neukrug,R. Charles Fawcett p 133 (fragment available online via Google)<br />
<br />
<br />
<b>Coefficient of Variation</b><br />
CV% = (SD/Mean)*100<br />
<br />
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-41535024871566277632017-03-20T04:48:00.000-07:002017-03-20T04:48:01.510-07:00ExcelVBA: From recorded macro to reusable function.Problem: with each change in slicer, pivot chart formatting changes.<br />
<br />
So I want all my lables vertical (and with each change the damn thing returns to horizontal). My recorded macro did not return anything sensible. Lecture on chart label properties brought me to this solution:<br />
<br />
<span style="font-size: x-small;">Sub Macro1()<br />'<br />' Wersja "Recorder"<br />'<br />Dim mySrs As Series<br /><br />With ActiveSheet.ChartObjects("chValueofOffers").Chart<br /> <br /><br /> .SeriesCollection(1).DataLabels.Orientation = xlUpward<br /> .SeriesCollection(2).DataLabels.Orientation = xlUpward<br /><br /><br />End With </span><br />
<br />
<br />
Next I wanted to iterate through series collection so that I do not need to change macro for more series:<br />
<br />
<span style="font-size: x-small;">Sub Macro2()<br />'<br />' Wersja "Obiektowo pętlowa"<br /><br />'</span><br /><span style="font-size: x-small;">Set seriesCol = ActiveSheet.ChartObjects("chValueofOffers").Chart.SeriesCollection<br /> <br />For Each mySeries In seriesCol<br /> mySeries.DataLabels.Orientation = xlUpward<br />Next<br /><br />Set seriesCol = ActiveSheet.ChartObjects("chValueofOffersTotal").Chart.SeriesCollection</span><br /><br /> <br />End Sub<br />
<br />
Finally... what if I have several charts on the same sheet that need vertical labels. Why not making it a reusable function with chart name as attribute. Now... with each change in the slicer the labels in the two charts get corrected to vertical.<br />
<br />
<span style="font-size: x-small;"><br />Private Function DataLabelsVertical(mySheet As String, chtName As String)<br /><br /> Dim mySeries As Series<br /> Set seriesCol = Worksheets(mySheet).ChartObjects(chtName).Chart.SeriesCollection<br /><br /> For Each mySeries In seriesCol<br /> mySeries.DataLabels.Orientation = xlUpward<br /> Next<br /><br />End Function<br /><br />Private Sub Worksheet_Change(ByVal Target As Range)<br /> DataLabelsVertical "Value_tenders", "chValueofOffers"<br /> DataLabelsVertical "Value_tenders", "chValueofOffersTotal"<br /> DataLabelsVertical "Status", "chStatusValue"<br /> DataLabelsVertical "Status", "chStatusValueTotal"<br />End Sub</span><br /><span style="font-size: x-small;"></span><span style="font-size: x-small;"><br />End Function</span><br />
<br />
Sources:<br />
http://www.java2s.com/Code/VBA-Excel-Access-Word/Excel/Loopthrougheachseriesinchartandaltermarkercolors.htm<br />
http://stackoverflow.com/questions/21165581/vba-looping-through-all-series-within-all-charts<br />
<br />
<br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-76415649642182847682017-02-17T08:19:00.003-08:002017-02-17T08:19:23.574-08:00R: self organizing mapsInteresting topics, see<br />
<br />https://www.r-bloggers.com/self-organising-maps-for-customer-segmentation-using-r/<br />http://www.slideshare.net/shanelynn/2014-0117-dublin-r-selforganising-maps-for-customer-segmentation-shane-lynn<br />
<br />
<br />
Other interesting reading<br />https://www.r-bloggers.com/r-an-integrated-statistical-programming-environment-and-gis/<br /><br />https://www.r-bloggers.com/how-to-perform-pca-with-r/Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-49211878144071726372017-02-16T04:32:00.001-08:002017-02-16T04:32:42.328-08:00SQL playground, remove duplicates and count students that passed exam.<span><span><span data-ft="{"tn":"K"}"><span class="UFICommentBody _1n4g"><span><span><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman","serif";}
</style>
<![endif]-->
</span></span></span></span></span></span><br />
<div class="MsoNormal">
<span lang="PL" style="color: #1f497d; font-family: "Calibri","sans-serif"; font-size: 11.0pt; mso-ansi-language: PL;">Patent na usuwanie duplikatów z tabeli “in
place” tj bez nadpisywania jej, bez usuwania i wstawiania innej:</span></div>
<div class="MsoNormal">
<span lang="PL" style="color: #1f497d; font-family: "Calibri","sans-serif"; font-size: 11.0pt; mso-ansi-language: PL;">Do przetestowania:</span></div>
<div class="MsoNormal">
<span lang="PL" style="color: #1f497d; font-family: "Calibri","sans-serif"; font-size: 11.0pt; mso-ansi-language: PL;">Kolumny ID (nie powtarza się). imie,
nazwisko, dane (może duplikować się).</span></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">
DELETE
*<br />
FROM Tabela1<br />
WHERE [id] NOT IN<br />
(SELECT Max(Tabela1.id) AS id<br />
FROM Tabela1 <br />
GROUP BY Tabela1.imie, Tabela1.nazwisko, Tabela1.dane);</div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">
<br /></div>
<div class="MsoNormal" style="mso-margin-bottom-alt: auto; mso-margin-top-alt: auto;">
<span lang="PL" style="mso-ansi-language: PL;">w linii GROUP BY można określić gdzie
szukamy duplikatów, w jakich kolumnach <br />
Select Max... można użyć First(), albo Min(), żeby określić, które zduplikowane
rekordy zachować.</span></div>
<br />
<br />
<br />
Są trzy tabele<br />
1. Studenci (_indeks_, imie, nazwisko)<br /> 2. Kursy (_id_, tytul, godzin, punktow), gdzie punktow oznacza ile punktow za zaliczenie kursu się dostaje<br />
3. Szkolenia( osoba, kurs, zaliczenie) gdzie osoba to numer indeksu
studenta, zaliczenie to data zaliczenia, a kurs to numer kursu<br />
i pierwsze zadanie z matury brzmi: Ilu studentów zaliczyło w pierwszym terminie? (do 30 czerwca 2016)<span><span><span data-ft="{"tn":"K"}"><span class="UFICommentBody _1n4g"><span><span><pre> </span></span></span></span></span></span><br />
<span><span><span data-ft="{"tn":"K"}"><span class="UFICommentBody _1n4g"><span><span>Select count([_indeks_Stud]) AS ilu_zdalo From </span></span><span><span><br /><span> (Select </span><span>[Studenci$].[_indeks_Stud]</span><br /><span> From </span><span>([Kursy$] </span></span></span></span></span></span></span><br />
<span><span><span data-ft="{"tn":"K"}"><span class="UFICommentBody _1n4g"><span><span><span> Inner Join </span><span>[Szkolenia$] </span><span>On [Kursy$].[_id_kurs] = [Szkolenia$].kurs) </span></span></span></span></span></span></span><br />
<span><span><span data-ft="{"tn":"K"}"><span class="UFICommentBody _1n4g"><span><span><span> Inner Join</span> <span>[Studenci$] </span><span>On [Studenci$].[_indeks_Stud] = [Szkolenia$].osoba</span><br /><span> Where </span><span>[Szkolenia$].zaliczenie <= #2016-06-30#</span><br /><span> Group By</span><span> [Studenci$].[_indeks_Stud]</span><br /><span> Having </span><span>Sum([Kursy$].punktow) >= 15)</span><br /><./pre><br /><span>Kod
powstał w ide Flyspeed SQL Query na bazie stworzonej w zakładkach w
Excelu, stąd znaki dolara.... kod więc powinien działać po wklejeniu do
Microsoft Query.</span></span></span></span></span></span></span>Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-50495612647177643282017-02-16T03:43:00.001-08:002017-02-16T03:46:25.690-08:00Python in RStudioData used:<br />
Titanic data: https://www.kaggle.com/c/titanic/data<br />
and tutorial: http://nbviewer.jupyter.org/github/savarin/pyconuk-introtutorial/blob/master/notebooks/Section%201-0%20-%20First%20Cut.ipynb<br />
<br />
Flights data: http://ucl.ac.uk/~uctqiax/data/flights.csv<br />
<br />
Software used:<br />
Portable scientific winpython (with pandas scikit-learn):<br />
https://sourceforge.net/projects/winpython/?source=typ_redirect<br />
To work it needed windows updates (my OS is windows 7):<br />
https://www.microsoft.com/en-us/download/confirmation.aspx?id=49093<br />
<br />
To install packages from source it needed:<br />
http://landinghub.visualstudio.com/visual-cpp-build-tools<br />
I needed<a href="https://blog.rstudio.org/2016/03/29/feather/"> feather </a>package so I dowloaded it and used python command: pip install<br />
as taught here: https://github.com/winpython/winpython/wiki/Installing-Additional-Packages<br />
and installed from source: https://github.com/wesm/feather/tree/master/python<br />
<br />
To learn how to use other languages in RStudio: http://rmarkdown.rstudio.com/authoring_knitr_engines.html<br />
<br />
I also wanted to try if some portable version of bash would work. No problem:<br />
http://win-bash.sourceforge.net/ <br />
<br />
<br />
<br />
Code for my playground.<br />
<pre>---
title: "R Notebook"
output: html_notebook
---
## Bash
```{bash, engine.path="C:\\Users\\jkotows2\\Desktop\\shell.w32-ix86\\bash.exe"}
cat flights1.csv flights2.csv flights3.csv > flights.csv
```
## Python
http://rmarkdown.rstudio.com/authoring_knitr_engines.html
```{python, engine.path="C:\\Users\\jkotows2\\Desktop\\WinPython\\python-3.6.0.amd64\\python.exe"}
import pandas
import feather
# Read flights data and select flights to O'Hare
flights = pandas.read_csv("C:\\Users\\jkotows2\\Desktop\\_flights\\flights.csv")
flights = flights[flights['dest'] == "ORD"]
# Select carrier and delay columns and drop rows with missing values
flights = flights[['carrier', 'dep_delay', 'arr_delay']]
flights = flights.dropna()
print (flights.head(10))
# Write to feather file for reading from R
feather.write_dataframe(flights, "C:\\Users\\jkotows2\\Desktop\\_flights\\flights.feather")
```
## Back to R
```{r}
library(feather)
library(ggplot2)
# Read from feather and plot
flights <- read_feather("C:\\Users\\jkotows2\\Desktop\\_flights\\flights.feather")
ggplot(flights, aes(carrier, arr_delay)) + geom_point() + geom_jitter()
```
</pre>
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-91046387227537050852017-02-15T03:34:00.003-08:002017-02-15T03:34:42.070-08:00lpsolve - solver in RTo study:<br />
http://flovv.github.io/From_descritpive_to_prescriptive/<br />
https://icyrock.com/blog/2013/12/linear-programming-in-r-using-lpsolve/<br />
http://lpsolve.r-forge.r-project.org/<br />
http://horicky.blogspot.co.uk/2013/01/optimization-in-r.html<br />
http://lpsolve.sourceforge.net/5.5/R.htmDaimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-21015575535135401142017-01-19T04:12:00.001-08:002017-01-20T01:56:10.286-08:00R packages for the lazy.Data entry <br />
Datapasta: Copy data from Excel or HTML to an R file - with keyboard shortcuts. And do it in a nice readable txt "tribble" format. https://github.com/MilesMcBain/datapasta<br />
<br />
Quick overview<br />
Quickly plot data having no time for a nice ggplot2 code. https://github.com/stefan-schroedl/plotluck<br />
<br />
Quick ensembleR predict data: ensembleR package<br />
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-84457050380179035552016-12-16T07:09:00.002-08:002016-12-16T07:31:26.880-08:00R data mining for beginners with Rattle - new versionSupport for Microsoft version of R, new stuff worth checking, Data Mining GUI for those intimidated by R code: <a href="http://togaware.com/rattle-5-0-0-alpha-released/">http://togaware.com/rattle-5-0-0-alpha-released/ <br />Install: install.packages("rattle", repos="http://rattle.togaware.com")
</a><br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-75241267874407038612016-12-13T02:19:00.000-08:002016-12-13T02:19:46.414-08:00Excel: Rank if with sumproduct (and VBA)Look here: <a href="http://www.mrexcel.com/forum/excel-questions/82479-rank-if-2.html">http://www.mrexcel.com/forum/excel-questions/82479-rank-if-2.html</a><br />
eg.<br />
=SUMPRODUCT(--(A$3:A$11=A3)*(B3<B$3:B$11))+1<br />
<br />
Big thanks. Helped me a lot today.Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-27552862383748461432016-11-28T05:15:00.001-08:002016-11-28T05:28:03.070-08:00Music from my dream...<br />Ives: The Unanswered Question / Premil Petrovic / No Borders Orchestra
<iframe width="640" height="360" src="https://www.youtube.com/embed/u-d8JSZE2Bc" frameborder="0" allowfullscreen></iframe>
Similar:
<iframe width="640" height="360" src="https://www.youtube.com/embed/6Jggiq2uXu0" frameborder="0" allowfullscreen></iframe>
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-62526127240993928662016-11-25T07:18:00.001-08:002016-11-25T07:18:39.571-08:00Marketing with R, very interesting articles.My search started with the question, how to reduce the huge amount of information about the history of purchases, contacts of hundred thousands of customers to manageable few variables.<br />
I came across Recency, Frequency, Monetary value (RFM) analysis. <br />
<div>
</div>
<div>
Review articles of these users on R bloggers.<br />
<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://www.r-bloggers.com/author/jack-han/&source=gmail&ust=1480173166440000&usg=AFQjCNGJbL7KEGzNoa2IWgD5MW5VKeTbdg" href="https://www.r-bloggers.com/author/jack-han/" target="_blank">https://www.r-bloggers.com/<wbr></wbr>author/jack-han/</a><br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://www.r-bloggers.com/author/analyze-core-r-language/&source=gmail&ust=1480173166440000&usg=AFQjCNEwnG28Wyw3xhguDa-jsSA8hN7oqg" href="https://www.r-bloggers.com/author/analyze-core-r-language/" target="_blank">https://www.r-bloggers.com/<wbr></wbr>author/analyze-core-r-<wbr></wbr>language/</a></div>
<br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-53417543597886345062016-11-04T06:16:00.000-07:002016-11-04T08:24:49.674-07:00Cleaning data with Regex in Excel (then in R)...I have a dirty column that should contain duration in months as integer, but it contains variants of strings: due to end on dd-mm-yy, duration x days, from dd.mm.yyyy through/to dd.mm.yyyy y.<br />
Headache.<br />
<br />
This is the day I started playing with regex.<br />
<br />
First, how to find a date after a specific word:<br />
http://stackoverflow.com/questions/19193251/regex-to-get-the-words-after-matching-string<br />
<br />
After some reading (eg. http://stackoverflow.com/questions/18139493/finding-first-word-after-regex-match, http://stackoverflow.com/questions/546220/how-to-match-the-first-word-after-an-expression-with-regex<br />
<br />
<br />
modifications and experimentations I reached the following compact solution:<br />
<br />
[\n\r].*[skończy się:|until:|do:|zakończenia:]\s*(\w+.\w+.\w+)<br />
<br />
Tested with: "The contract will last until: dd.mm.yyy or to the [...]" it will return just the date "dd.mm.yyy<br />
<br />
Cute!<br />
<br />
Macros, that work with Excel:<br />
<br />
http://analystcave.com/excel-regex-tutorial/ (I used RegexExecute macro with success) <br />
http://www.vbaexpress.com/kb/getarticle.php?kb_id=841<br />
http://stackoverflow.com/questions/22542834/how-to-use-regular-expressions-regex-in-microsoft-excel-both-in-cell-and-loops<br />
http://www.vbaexpress.com/kb/getarticle.php?kb_id=841<br />
http://superuser.com/questions/978532/how-do-i-use-regex-using-instr-in-vba<br />
http://stackoverflow.com/questions/19481175/general-purpose-udfs-for-using-regular-expressions-in-excel (set of very useful tools)<br />
<br />
<br />
<br />
R code to test:<br />
<br />
http://stackoverflow.com/questions/34804708/matching-a-word-after-another-word-in-r-regex<br />
https://renkun.me/blog/2014/02/20/extract-information-from-texts-with-regular-expressions-in-r.html <br />
<br />
In R the following worked with me:<br />
<br />
library(stringr)<br /><br />str_extract(string=df$duration, pattern= "(?<=(dnia:|zakończenia:|until:)\\s)(\\w+.\\w+.\\w+)" )<br />
<br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-47217068169198577142016-10-24T03:13:00.001-07:002016-10-24T03:36:56.010-07:00ReadingList: Data Bases in R1. Data Table by Matt Dowle.<br />
Cheatsheet: https://s3.amazonaws.com/assets.datacamp.com/img/blog/data+table+cheat+sheet.pdf<br />
https://www.analyticsvidhya.com/blog/2016/05/data-table-data-frame-work-large-data-sets/ <br />
<br />
Basics: https://www.dezyre.com/data-science-in-r-programming-tutorial/r-data-table-tutorial<br />
https://www.r-bloggers.com/a-data-table-r-tutorial-by-datacamp-intro-to-dti-j-by/<br />
https://www.r-bloggers.com/elegant-fast-data-manipulation-with-data-table/ <br />
<br />
Adv: https://www.r-bloggers.com/advanced-tips-and-tricks-with-data-table/<br />
https://www.r-bloggers.com/efficient-aggregation-and-more-using-data-table/<br />
Quick union: https://www.r-bloggers.com/r-quick-tip-collapse-a-lists-of-data-frames-with-data-table/<br />
Aggregation: https://www.r-bloggers.com/two-of-my-favorite-data-table-features/<br />
Binning: https://www.r-bloggers.com/using-data-table-for-binning/<br />
<br />
Joining: http://www.magesblog.com/2014/06/the-joy-of-joining-datatables.html#more<br />
Manipulation the fast way: http://www.magesblog.com/2012/11/datatable-rocks-data-manipulation-fast.html#more <br />
<br />
Solving common problems: https://www.r-bloggers.com/solve-common-r-problems-efficiently-with-data-table/ <br />
<br />
Matt Dowle presentations:<br />
http://datascience.la/matt-dowles-data-table-talk-from-user-2014/<br />
<br />
Roll:<br />
https://www.r-bloggers.com/understanding-data-table-rolling-joins/<br />
<br />
https://www.r-bloggers.com/advanced-tips-and-tricks-with-data-table/ <br />
<br />
All articles: <br />
https://github.com/Rdatatable/data.table/wiki/Articles<br />
<br />
H2O data table?<br />
http://t.co/TYGG6KcMpx<br />
https://www.analyticsvidhya.com/blog/2016/05/h2o-data-table-build-models-large-data-sets/ <br />
<br />
<br />
2. Dplyr by Hadley Wickham... plenty of stuff everywhere (to be elaborated on)<br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-9408627702400818062016-10-21T06:42:00.000-07:002016-10-24T04:04:46.145-07:00Free books from Gitbook... R stuff.<div>
<b>General: </b></div>
<div>
Przewodnik po pakiecie, Przemysław Biecek https://pbiecek.gitbooks.io/przewodnik/content/</div>
R for Data Science, Garrett Grolemund Hadley Wickham http://r4ds.had.co.nz/<br />
<br />
<b>Programming: </b><br />
Efficient R programming, Colin Gillespie, Robin Lovelace https://csgillespie.github.io/efficientR/<br />
<div>
<div>
<a href="http://adv-r.had.co.nz/">Advanced R</a><a href="http://adv-r.had.co.nz/"></a> by Hadley Wickham http://adv-r.had.co.nz/</div>
<div>
<br />
<b>Time series:</b>Analiza i prognozowanie szeregów czasowych – case studies Część I , Adam Zagdański, Artur Suchwałko http://quantup.pl/szeregi-czasowe-case-studies/<br />
<br /></div>
<div>
<b>Visual analysis:</b></div>
<div>
Geostatystyka w R, Jakub Nowosad (see also presentations pazuR https://nowosad.github.io/presentations/)https://bookdown.org/nowosad/Geostatystyka/eksploracyjna-analiza-danych-przestrzennych.html<br />
Not compiled files for Ggplot2 book by Hadley Wickham: https://github.com/hadley/ggplot2-book<br />
<br />
R for Excel users<br />
http://quantup.pl/operacje-w-R-znane-z-Excela/<br />
http://quantup.pl/wykresy-w-R-znane-z-Excela/<br />
<br />
<b>Report writing:</b></div>
<div>
Authoring Books with R Markdown, Yihui Xiehttps://bookdown.org/yihui/bookdown/<br />
<br />
<b>Data mining:</b><br />
<b></b>
The caret Package, Max Kuhn http://topepo.github.io/caret/index.html</div>
</div>
<div>
<br /></div>
<div>
<b>Python</b></div>
<div>
PythonPython Data Science Handbook</div>
<div>
Jake VanderPlas Soon to be published: https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/README.md</div>
Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-77304438028448953112016-09-06T01:45:00.000-07:002016-09-06T01:45:12.164-07:00Extract data from many word documents to table.<br />
Best: <br />
http://stackoverflow.com/questions/35850705/extract-select-data-from-multiple-word-files-to-excel<br />
<br />
Find specific word, extract data<br />
http://stackoverflow.com/questions/3567441/extract-data-from-word-document-to-an-excel-spreadsheet<br />
<br />
http://superuser.com/questions/774734/copy-data-from-multiple-ms-word-files-to-excel-using-vba?noredirect=1&lq=1 <br />
<br />
Perhaps the data is in form fields?<br />
http://www.techrepublic.com/blog/10-things/10-steps-to-transferring-word-form-data-to-an-excel-sheet/<br />
<br />
<br />
Not exactly that:<br />
http://stackoverflow.com/questions/7524064/excel-vba-question-need-to-access-data-from-all-excel-files-in-a-directory-wi<br />
http://stackoverflow.com/questions/7531526/need-to-pull-data-from-all-files-in-a-directory<br />
http://stackoverflow.com/questions/22017966/extract-data-from-text-file-and-put-in-excel-file-using-vba<br />
http://www.excelforum.com/l/777970-asdf.html<br />
http://www.dslreports.com/forum/r24300164-Word-50-word-docs-extracted-to-excel-tablesDaimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0tag:blogger.com,1999:blog-8718553332826573026.post-29786629983731553502016-09-02T05:23:00.001-07:002016-09-02T05:29:57.036-07:00What to do with old smartphone.1. Surveillance:<br />
Salient Eye (will work with old Androids, good job Israel!)<br />
http://www.salient-eye.com/yey-we-got-some-new-old-smartphones/<br />
http://www.makeuseof.com/tag/use-old-smartphone-tablet-security-camera/<br />
http://www.usatoday.com/story/tech/columnist/komando/2015/01/30/old-phone-camera/22532911/<br />
http://www.haaretz.com/israel-news/business/.premium-1.577834<br />
2. Road cam<br />
http://www.benchmark.pl/testy_i_recenzje/polecane-rejestratory-jazdy-na-androida.html/<br />
3. Internet radio<br />
(can also be done with old Nokias with Symbian and WiFi)Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com1tag:blogger.com,1999:blog-8718553332826573026.post-70191490982474601362016-08-12T06:25:00.000-07:002016-08-12T07:05:27.883-07:00R: data exploration nomad's top reading this month, mlRAlternative to caret: <b>mlR</b><br />
To practice: https://www.analyticsvidhya.com/blog/2016/08/practicing-machine-learning-techniques-in-r-with-mlr-package/<br />
Official tutorial: http://mlr-org.github.io/mlr-tutorial/devel/html/<br />
<br />
<br />
Other resolutions:<br />
<b>Learn R control structures</b>: http://www.statmethods.net/management/controlstructures.html<br />
<b>Learn xgboost bare without the convenience of caret</b> (prepare sparse/dense matrix data, finetune): https://github.com/dmlc/xgboost/blob/master/R-package/vignettes/xgboostPresentation.Rmd<br />
http://xgboost.readthedocs.io/en/latest/R-package/xgboostPresentation.html <br />
<b>Use my own model in train in caret</b> (or learn to put in the grid the parameters not originally supported in caret)<br />
https://github.com/dmlc/xgboost/blob/master/R-package/vignettes/xgboostPresentation.Rmd<br />
<b>Cleaning data, operations on text, imputation of NA's etc !!!</b><br />
<br />
Other:<br />
Time series wanderings:<br />
http://swmprats.net/forum/potm-single-variable-exploration/35-august-potm-time-series-decomposition-with-swmpr<br />
<br />
<br />
Excel trivia: learn how to find multiple matches:<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://fiveminutelessons.com/learn-microsoft-excel/use-index-lookup-multiple-values-list&source=gmail&ust=1471089168687000&usg=AFQjCNGfHqS3DVduT7AlfyB5YrkNmnW6LA" href="https://fiveminutelessons.com/learn-microsoft-excel/use-index-lookup-multiple-values-list" target="_blank">https://fiveminutelessons.com/<wbr></wbr>learn-microsoft-excel/use-<wbr></wbr>index-lookup-multiple-values-<wbr></wbr>list</a><br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=http://stackoverflow.com/questions/26424226/excel-return-multiple-matching-values-from-a-column-horizontally-in-one-row&source=gmail&ust=1471089169192000&usg=AFQjCNGYF8oTNI4ioXqg7zRaqr9QPmZ1jg" href="http://stackoverflow.com/questions/26424226/excel-return-multiple-matching-values-from-a-column-horizontally-in-one-row" target="_blank">http://stackoverflow.com/<wbr></wbr>questions/26424226/excel-<wbr></wbr>return-multiple-matching-<wbr></wbr>values-from-a-column-<wbr></wbr>horizontally-in-one-row</a><br />
http://eimagine.com/how-to-return-multiple-match-values-in-excel-using-index-match-or-vlookup/<br />
<br />
<br />Daimyo Shimadahttp://www.blogger.com/profile/01416465855368249776noreply@blogger.com0