This is the third part of a 9 part tutorial series about the R Statistical Programming Language, targeted at data analysts and programmers that are active on Steem.
In this course you will learn the R programming language through practical examples.
The R source code can be found via one of the official mirrors at
This tutorial is part of a series, the text as well as the code is available in my github repo
The first two tutorials introduced the basics of R, the rest of this series will focus on worked examples where we will slowly introduce new concepts and learn some of the extended functionality of the R programming language.
Intermediate
If you're a data junkie like me you will always be on the lookout for new and interesting datasets. Ok data may not be very interesting but exploring it with machine learning models and dynamic visualizations is fun. Getting insights from the noise is a skill and I assure you it will make people speak in hushed tones when you walk by :)
So what data can you scrape and where is the best place to look for it?
The web is a great source of data which can be easily accessed. It is important to note that scraping web data may be against the terms of service of a website so you should check this before attempting to download data. When and if you should download data is outside the scope of this tutorial.
If you find a webpage with data you can of course print it off and manually type the results in a spreadsheet and viola, but for anything more than a few numbers you may be better off automating it in some way.
Tip 1: Check if the website has a download button or API
I don't know how many times I have spent ages pulling together a script for data scraping to realise there is a form or download button which delivers all the data in a nicely formatted spreadsheet at the touch of a button.
Tip 2: Check if the data is available from somewhere else
If someone has done the work already there is no point in reinventing the wheel, e.g. use steemsql or other services.
Tip 3: Copy and paste
This technique is often the quickest. I would recommend this method for individual tables on ad hoc analysis but be careful, formats can sometimes get messed up.
Tip 4: Use Excel
Excel lets you Link to External Data and import the tables for the website. This is a great option for many purposes, you can even write VBA macros to loop thorough sites. It works pretty well for some ues cases.
These first 4 tips are my go to methods for scraping data. See if they get you what you want before building a scraper.
I would recommend using a more advanced method in the following 3 cases
I thought this was a tutorial about R...
R is not the best tool for webscraping, in fact R is not the best tool for many things. If you have some industrial scale scraping to do, Python is king. You can build crawlers with it and there are libraries to handle almost any web page format. R is not particularly fast but the benefit of using R is that it is easy to get data for many useful situations. Today we will download Coinmarket Cap data and plot the historical Steem price using just 6 lines of code.
I use the following method for Webscraping;
You need to install Packages for almost any task besides basic statistics. Before delving into our worked example for today we will take a minute to learn how to search for R packages....
We learned in the last lesson the main repository for R packages is called CRAN. On their website you can see a list of the packages available
There are 12,770 packages available. It can be hard to find the one you want and this is actually a limitation of R. Anyone can submit a package. There is no central coding standard, just basic submission and documentation standards applied. This means that there may be multiple packages that do the similar things. In order to help you find the one you are looking for packages are organised into Task Views, or groupings of related packages.
I will leave it with you to explore the key R packages in each task view, for today's tutorial we will continue with a very easy to use package for querying we data.
As we did in the last lesson we will begin by opening up R Studio and creating a new script for storing our code. The package.
We will next install the scraping package we will be using today, "htmltab".
Type the following code in the script and hit ctrl + enter to run it in the command line.
install.packages(htmltab, dependencies=True)
Next we will load the package htmltab, Type the following code in the script and hit ctrl + enter to run it in the command line.
library(htmltab)
Useful Packages for Data and Plotting...
Before we continue we will next load 2 packages we have seen before and which we will make use of in each tutorial.
library(data.table)
library(ggplot2)
The first stage of web scraping is find a web link for a page that has data to download.
In this tutorial we are going to download historical prices from Coinmarket Cap for Steem. If you head over to Coinmarket Cap you will notice a tab Historical Data. This is where we will start. The "Finding the Weblink" stage generally involves a bit of research till you find the right page.
Examining this page we see this page only goes back for 3 days but you can choose a longer range by selecting the date button.
You will notice the URL at the top now changes on this site. This is what we have been looking for!
We have a suitable url to scrape so we can move on to coding it up in R.
We will next copy and paste the url to our R script and store it in a variable which we will call "url".
In our script type
url <- “https://coinmarketcap.com/currencies/steem/historical-data/?start=20170719&end=20180719”
This makes it easy to change if we want to change the date range for example.
We have put the link in Quotation marks and you may also notice we are now using a symbol <- for assignment. This is similar to the assignment operator we have been using before =, but <- is more generally used in R for assigning variables, it has a slightly different meaning than assignment with an equals sign.
Next in order to retrieve the data you can type into the script the following command and hit ctrl + enter
cmc <- htmltab(url)
Now if you type into the console
cmc
We are going to plot this data but before we do we need to clean it and format it correctly. In the last lesson we introduced the concept of a data.frame, a rectangular data structure. That is what cmc is, many packages use data.frame for historic reasons but we prefer storing data as a newer data.table structure. I would urge you to forget about using data frames and just head straight to using Data Tables.
We will convert our data.frame to a data.table by typing the following command
cmc <- data.table(cmc)
We now have the Coinmarket Steem Prices for the last year stored in our data strcuture cmc.
To see the first row of the data table type
cmc[1]
We mentioned that our dats is stored as characters. Before we can plot our data we need to convert the dates to a date data type and the prices to numeric data types. In the previous lesson that there are two basic data types in R;
Date is not a base data type but a combination of the two, a richer higher level data type.
To convert a character to a number we use the function as.numeric()
A data.table has a neat syntax for performing column wise operations, which we can see in the following example where we create a new column called "LowPrice" from the existing column called "Low"
cmc[ ,LowPrice := as.numeric(Low)]
Our date is a Character variable, which we are going to convert to a date variable.
Can you guess what the function to do this is?
To convert a character to a number we use the function as.Date()
R is case sensitive and there are varying conventions. This can get a bit annoying but it is never hard to find the formula you are looking for thanks to R studio and the interactive console
The as.Date(x, format) function takes two arguments, the character variable to convert (x) and the format of the data (format)
The following table shows the available formats.
Source: https://www.statmethods.net/input/dates.html
In our case the command we wish to type is as follows
cmc[,PriceDate:=as.Date(Date, "%b %d, %Y")]
Type
cmc
ML is a few lessons off but today we can graph the price of Steem for the last year using the ggplot2 package. Type the following command in our script and his ctrl + enter.
ggplot(cmc, aes(x=PriceDate, y=LowPrice))+geom_point()
You will see the Plot in the bottom right tab of R Studio
In this lesson:
url <- "https://coinmarketcap.com/currencies/steem/historical-data/?start=20170719&end=20180719"
cmc <- htmltab(url)
cmc <- data.table(cmc)
cmc[, LowPrice:=as.numeric(Low)]
cmc[,PriceDate:=as.Date(Date, "%b %d, %Y")]
ggplot(cmc, aes(x=PriceDate, y=LowPrice))+geom_point()
This course will cover the basics of R over a series of 9 lessons. We began with some essential techniques (in the first 2 lessons) and I will take you on a tour of some of the more advanced features of R with worked examples that have a Cryptocurrency and Steem flavour.
Data Wrangling occupies a significant portion of any Data Analysts time. The next tutorial will extend this Web Scraping Tutorial to look at more advanced data wrangling with R.
For a complete list of the lessons in this course you can find them on github. Feel free to reuse these tutorials but if you like what you see please don't forget to star me on github and upvote this post.
Thank you for reading. I write on Steemit about Blockchain, Cryptocurrency and Travel.
R logo source: https://www.r-project.org/logo/
