#timeseriesanalysis
#rstudio
Hello everyone uh we are very happy to have masi today she wants to talk about H Trend seasonality um cycling and out layers in Time series analyzes in our software so we are very happy to have her and also happy for uh your attendance so stages yours um hello everybody warm welcome
From our Our Ladies to all of you will um f Our Ladies on ladies start um today um yes um today we are are uh going to talk about time series characteristics um called Trend seasonality cycling and outliers in Time series analyzes okay um first of all we have a brief
Definition of Time series A Time series is a sequence of observation uh which are indexed in time order and um some of the examples of um time series in hydrology can be rainfall wind speed discharge temperature Sunny hours or Velo velocity um that can be categorized uh
In different kinds of time interval like hourly daily seasonal six months monthly or yearly but uh please note that uh one important point is that we have to know why it is important to recognize the time service characteristics before doing any operation the reason it’s so important in hydrologi time service
Analyzes is vital for predicting riverflow patterns facilitating efficient Water Resource Management assessing dra and flood risk contributing to sustainable water infrastructure planning and realtime monitoring to changing hydrological conditions so this is so vital for about um identifying the time series characteristics before doing any operation um each time series um has
Four characteristics cont contains Trend outliers Trend seasonality and cycling and we have to after identifying them we have to remove them from the time series okay outliers uh outliers in time service data are values that significantly differ from the patterns and trends of the other value use in the time
Series and this is a a simple graph um how we can recognize them in um our programming language there are two different kinds of methods visualization and numerical methods visualization contains three different kinds of plots box plot histogram and QQ norm and numerical method uh but uh if you want to use the
Numerical method for finding the outliers we have to uh calculate the average and standard Dev deviation and uh please note that all of the methods will be explained in detail after finishing the slides we will go uh through the r and we will explain them in details they they are just an
Introduction okay what is a trend uh the trend represent the long-term change in the level of time Series this change can be either upward or downward for identifying Trend again we have two different kinds of method by visualization or using statistical method we can recognize the trend visualization contain plot and drawing a
Line and statistical method can be um Kendall Kendall is a statistical method seasonality seasonality is the presence of variations that occur at specific regular intervals such as weekly monthly or quarterly seasonality can be caused by factors such as weather or Holly day and consist of periodic and repetitive patterns in a Time
Series time series cycling cycling is the presence of variations that occur at nons space specific regular intervals so so um please note that it must be highlighted that a big difference between seasonality and cycling is a cycle occures when the data exhibits rise and Falls that are not of a fixed
Frequency but seasonality is always of a fixed and noun period uh for analyzing seasonality and cycle in um we can uh use same methods we have uh autocorrelation function method and decomposition uh and the package which we are going to use them in this section
Are read XL for importing the data DP LR for filtering Kendall for Trend analyzes and t-s series for decomposition this is just a brief introduction now we are going in through the um our programming language we have some examples and uh we will explain every characteristics in details and also every
Package okay first of all by using the um read XL package I’ve uh imported our um time series uh here we have a monthly time series of a hyro hyro metric station in Germany called Steinberg and uh okay it contains um 200 60 observation okay now I think it’s better
Okay uh um our time serice contains 26 observation and um the class of our data is a data frame and uh this data frame contains 216 r and one colume okay so this is a monly Time series we are going to find at first finding the outliers the first method for finding the
Outliers is visualization by visualization the data easily we can find which one is um out of the pattern which one is isn’t normal in outtime serious okay so um the first one is a box blood um please please pay attention that um the class of dat the data is is
Data frame so if I use the data frame a data frame time areas I can easily plot a box plot and recognize the outliers here there are four data which are which are not normal and they are out of range so now I know that in this time
I have four outliers okay but just the number of uh of the outliers not the exact place of them okay if I want to know more detail about the outliers about these four outliers I can use this this code and by using this code here I have a section called out means
Outliers and they are the exact amount of these four data the exact amount but not the place just I know the amount I know the amount of them the second uh method is um drawing a histogram but please uh be aware that if you want to um to have a
Histogram of a Time series you have to work on uh you you are not allowed to use data frame so that’s why I I’ve changed the class of dat data frame to Time series by using TS now I have a Time series and I draw the
Histogram okay so you can see that again we have four outliers but again just just know about the number of them four outliers the third graph is QQ Norm again QQ Norm uh is able to apply on time service not okay this is a QQ Norm in a QQ plot
In a QQ plot all of the data should be dense and if some of them are out of range here in this example we have four um so these are the outlier years okay um if we want to if if I want to show all of them in one
Graph uh we you can use uh this code and by using this code here we have three different kinds of uh graphs uh which are able to show the outliers histogram there are four a box blot they are four and in a QQ plot again we have four
Outliers in this example okay now uh so far we know that there are four outliers now I have to remove them from the time series for removing them I have to know the exact place the exact um the exact number of the rows of the outliers so I have to find the the
Number of rows this is this is the code which um we can we can share the our script after section by using um which function you can uh find the exact place of the of the outliers um okay I’ve Nam it I’ve named it not
Liers and if um if I check the lengths of the outliers you can see that this is for for um okay would you please excuse me wait a second because we have excuse me IM we have someone who is just I think um trying to do something so I’m trying to remove
Him or her wait a second okay then I also want to just delete the zoom link uh in this way that no one can join sorry for Interruption go ahead okay good okay um so I’ve called it outliers because we are going to uh find the
Exact place of the outliers and as you can see if I check the lengths of the outliers it shows four it means that we have four um for outliers and now about the exact place of them uh yes and and here uh this code can um can show us the
Exact place of the outliers it means that we have four outlier which are placed on 26 the r number 26 111 133 and 145 so now um okay so now uh we know about the number of outliers and the exact place of them in the Second Step I’m going to
Remove them from the from the time series for removing it removing for removing the outliers we can use filter function and filter for U using the filter function you have to use the dplr um package okay so I’ve uploaded and then by using this code I filter the
Time series and removing these for data and now if we check the the second um time series which I’ve called it Steinberg 2 we have 200 212 rows but at in the in the last step we had 216 it means that here uh we removed the
We removed the outliers and uh yes so this is the explanation of finding and removing the outliers by visualization method the second method is numerical method about outliers for numerical method we have to calculate the average of the data and the standard deviation of the data so I’ve called
Them mean and St D but please be be aware again if you want to use the numerical method you have to use time series not a data frame so that’s why I’m using Steinberg TS okay this is the average and this is the STD okay the next step is that we have
To Define a threshold we have to Define a threshold for finding the outliers it means that we have to Define a threshold if any of the data is in between they are the time series which we wanted and if they are out of the threshold they we called it
Outliers so um I defined t mean and T Max usually usually uh we uh by subtracting the average of three times standard deviation and adding average to three times a standard deviation we defined a threshold this is a uh this is a usual method which scientists us it
But it’s totally depends on you and um and some of the some of the scientists prefer to use four times SD it’s totally depend on the field we are we now we are talking about hydrology time series that’s why uh I uh introduced three times STD okay so uh now I defined the
Threshold okay uh Now by using this code uh we have to find the out lers it means that by using the function which function it means that go through the time series and find the data which are less than t mean or more than T
Marks okay at first I have to find them if I if I run the the code uh you can easily find the outliers but the difference is that uh in the visualization method we had four outliers but in the numerical method we find the three outliers it’s because of the
Threshold it’s because of the threshold um okay now I want to remove the outliers I know them I know the amount of them now I have to remove them um for a removing the outliers by using the numerical method again I use which function it means that go through the
Time series and find the data which are more than t t mean or and less than t- marks so it means that I’m trying to find the data uh which are in between the threshold so if I CH if I check the length of the outline ler I
Have 213 it means that the this method removed the uh removed uh these three uh outliers from the time series okay so this is the numerical method for finding the outliers and visualization method now we are going to talk about the trends the next characteristics uh for finding the
Trend uh we have statistical method for statistical method we have to use a statistical test uh the most famous one in this in this field is kendle so we will U use the Kendall package and U again again be aware that we have to use time series not a data frame so that’s
Why I’ve applied the code on the uh time series um time service data and um okay in this statistical test n hypothesis is there is no pattern in the data and vice versa there is a trend in the data in this example the P values is more than
0.05 it means that n hypothesis is it’s is completely acceptable and there is any pattern in the data um just okay and also we have another example about F about having Trend but in this example there isn’t any pattern so you have to refer to P values and about visualization
Method about the visualization method you can easily plot the Time series by using plot function okay I have a plot and then you have to uh draw a line okay uh if I draw a line and the red line shows a upward Trend I can say
That okay there is an upward Trend or if the red line shows a um downward Trend we can easily say that there is an downward Trend but here in this example as we tested with candle test there isn’t any Trend you can you can see the results also in uh By Us by
Visualization the the data uh the red line is straight so it means that there isn’t any Trend but also we have a we have another example I’ve prepared another example for um finding for um better understanding of the trend if I plot it I’ve called it I’ve called the the new
Example Trend underline data okay I plot it I plot it and uh you can see that yes there is an upward Trend and if I draw a line the red line shows an upward Trend shows an upward Trend it means that there is a uh this time service has
Um characteristics which has to be removed and for the for the second example also if I want to use the statistical method Kendall I’ve applied the Kendall and and and as you see p values is less than 0.05 it means that uh n hypothesis is accept is rejected and there is a trend
In the data okay so for removing the trend for removing the trend from the data uh I’ve introduced two methods the first one and the most famous one is taking logarithm uh I’ve called it one and uh so it just easily apply logar logarithm uh to the time
Series and now if I um draw the the red line again uh uh we can see that uh again we have the upward Trend so so it means that this one it means that this one is not um it’s it’s not a suitable solution for
Our time series so we uh we have to use the second one the second solution the second one is difference it means that every data uh by subtracting every data from the previous one we will have a new Time series and uh if I plot it and a red line you
Can see that the trend now the trend is removed totally removed and the red line shows any Trend okay so this is the this is the second second solution second ARR and um just just for showing the statistical method if I uh run the statistical method for the second one you can see
That key value is more than 0.05 and it means that uh the nle hypothesis is acceptable and there is no pattern in the data so the trend is removed and the time service is ready for um is ready for for the next operation for modeling or or anything
Else okay um now seasonality and cycling for seasonality and cycling we uh use the same method for um for both of them numerical and visualization the most famous way for um finding and removing seasonality is decomposition I’ve used the uh t-s series um TS series package and
Uh one of them is DEC composition and the other one is autocorrelation function if I uh but but please be aware that for um for drawing the autocorrelation function again we have to use the uh use a data Frame data so that’s why I’ve uh came back to the to
The first example not a T Series a data frame uh if I draw the autocorrelation function you can see that there is there is a repetitive pattern in the data this graph is showing a seasonality okay so it means that there is a seasonality and we have to remove it and
The second one is decomposition deom for uh using the decomposition we have to um Define a Time series with data so that’s why I’ve um I defined a Time series with uh with the DAT and now I’ve applied decomposed function okay uh if I uh plot the decomposed
Function here you have the this is the main graph this is the main graph of the observation and this part is the seasonality part of the graph this part so and we have to uh by subtracting the seasonality part of the observation uh we will have a Time series time series without any
Seasonality so I U subtract the Deon the seasonal part from the main time series just subtracting if I applied it now the final time series is the is the one which uh we are able to do modeling or anything or next steps on the on time series okay um
Okay um you can uh for for any further question or uh if you need more explanation or for um having the the rscript uh you can contact me easily also thank you for your attention thank you for the participation and and um please note that our ladies support any
Information sharing uh and you can be in touch with our ladies and these are the uh these are the connections and connect with us okay thank you and now I’m ready for your question thank you so much very nice and informative presentation here mash it okay let’s see we
Have uh one question here I have a question why do you apply the log function in order to calculate at Trend okay umal dear dear osalo if I if I pronounce it correctly uh taking logarithm taking logarithm from the time series is one of the most famous way to removing the TR
Trend from the time series and at first we have to usually we try the logarithm and if if we won’t have the uh best result um we will go to the second step that’s why that’s why I choosen a Time series A Time series which uh if I if we
Take the logarithm again we have Trend so we will go to the but in uh in most of the most of the time series by taking logarithm you will have a a Time series without any Trend okay another question are you you talking about kandall correlation in your speak uh no no no
Kandall correlation is a different kinds of U kinds of statistical methods we are talking about mendal we are talking about manal um statistical method for finding the trend but but please be aware that for finding the trend not removing the trend man Kendall not Kendall correlation
Okay and for the sharing of the slides H slides and also the script will be on our GitHub page if you follow us on Twitter or LinkedIn or also GitHub we will have all the presentations this one and previous ones also we will put the video of today’s
Talk on YouTube you can find it later so yes I we will share the video okay any other question um you’re welcome osalo um in Time series analyzes removing the out layers always not good idea and we have to consider them in analyzes M uh yes we have to consider
Them in analyzes but it depends on the um number of the data if you have 200 in this example we had 216 data in a Time series and by removing four outliers uh we won’t uh we won’t have any um interrupt in the data so so it’s
So it’s okay but again again I emphasize that it depends on your uh on your field in hydrology um for the data in 216 data is okay but if we have in 216 data if we have more than 10 outliers yes this is exactly correct and we are not allowed
To uh remove all of them okay it depends on you you have to make decision about your outliers and sometimes if you want to um uh if you want to modeling especially nonlinear models it’s better to keep them but if you are going to use linear models uh it’s okay usually we remove
Them okay another question in hydrology out layers can sometimes carry some useful information must we removed them always no because some of them maybe it will be like there like extreme values so we should take care of them as well yes no no it’s not a must
It’s not a must it’s totally depends on the on the aim if you are going to um yes I I I’ve explained it and if for nonlinear models it’s better to keep them but for nonlinear for linear models it’s better to remove them and also if
You are working on Extreme values if you are going to uh do flood prediction DRFT prediction so you have to keep them okay any other you can also follow us uh on these websites thank you again for your wonderful presentation Masid Jun okay thank you thank you so much and
Thank you for uh for the attention um feeling there is another question the ACF the autocorrelation function what what you mean by the ACF like yes ACF is the autocorrelation function and by uh plotting the autocorrelation function we can see easily that if there is any um when we have ACF analy if
There is a repeated and repetitive patterns in the data it means that you you have to um prepare the data before anything else so uh and the code this is a simple code ACF and then time series but the the point is that you have to
Work on data frame not time series and um yes if you want I can show it again yes just just simple ACF this is the this is the code ACF and time ser and the data frame and this is the this is the graph and as you see there is a repetitive
Pattern and also also there are a lot of points which we we can identify by using autocorrelation function like um now we we know that this is a short memory Time series and uh this is a nonlinear time series but but here we have to focus on um characteristics at first seasonality
So the first point is seasonality we can get from the stf there is a new package as TS sto for analyzing time series data great thank you for letting us know yes thank you for sharing