Data Manipulation

In the last session, we learned how to load data of various sources into R.

Today’s first part will be about how to manipulate data in R. Importantly, we will learn how to

• select certain variables from a data
• subset a data set
• recode and rename certain variables

We will work with fictive data set of student grades. We start by downloading the grades.csv file from the training website, setting our working directory and loading the data:

library(this.path)
setwd(here())
data_grades <- read.table("data/grades.csv",
                       header = TRUE, sep = ",", stringsAsFactors = FALSE)

Selecting Variables

We begin with selecting interesting variables from a data set. For our grades data set, we want to preserve information about ID, Name, and Exam_Score, and drop all other information.

Variables can be selected by name, after which we inspect the first and last three rows in the data set:

data <- data_grades[, c("ID", "Name", "Exam_Score")]
head(data, 3)

      ID                Name Exam_Score
1 i40333 Hyden-Terry, Dakota         55
2 i41204     Polson, Destiny         82
3 i41428       al-Azad, Nuha         52

tail(data, 3)

        ID             Name Exam_Score
40 i198051      Nies, Tyler         62
41 i198310 Montano, Marquez         72
42 i198859  Nyberg, Bich Sa         91

Selecting Variables

Or by variable indexes

data1 <- data_grades[, c(1, 2, 7)]

though this is not that convenient unless you know the column numbers of the variables you want to select.

A more convenient alternative is to use the following function:

data2 <- subset(data_grades, select = c(ID, Name, Exam_Score))

The objects data, data1 and data2 are all identical so you can use your preferred way of working!

Subsetting Rows

Next, we want to subset the data set, i.e. preserve interesting rows while removing the others.

For the grades data set, we might be interested in information about students in tutorial group 1:

# select tutorial 1 students only
data_tutorial1 <- data_grades[data_grades$Tutorial == 1, ]

or alternatively:

data_tutorial1 <- subset(data_grades, Tutorial == 1)

Subsetting also works using characters. For instance, to retrieve only information for females:

# select female students only
data_females <- data_grades[data_grades$Gender == 'Female', ]

Inspect your new data sets!

Exercise 4.1

Use the grades data set.

1. Generate a data set that contains information about the student ID, student name, their tutorial group, participation grade and their exam score.

2. Further reduce the data set obtained under 1 to only display information of students in tutorial group 4.

3. Further reduce the data set obtained under 2 to only display information of students with an exam score of more than 80. How many such students are there?

Transforming Variables

Let us continue with further data manipulations. The variable Tutorial is currently an integer:

class(data_grades$Tutorial)

[1] "integer"

but it should be a factor (a categorical variable). This can be easily changed in R:

data_grades$Tutorial <- as.factor(data_grades$Tutorial)

after which you can inspect its new class:

class(data_grades$Tutorial)

[1] "factor"

Transforming Variables

When inspecting the variable itself, R now mentions the different levels of the factors:

data_grades$Tutorial

 [1] 2 3 4 2 1 4 3 1 1 3 4 2 4 4 3 1 3 4 1 2 1 1 3 1 3 3 2 3 2 1 4 4 4 2 3 2 2 4
[39] 4 2 1 3
Levels: 1 2 3 4

which you can also directly retreive via:

levels(data_grades$Tutorial)

[1] "1" "2" "3" "4"

Adding Variables

Sometimes we want to add a variable to an existing data set.

For instance, we want to add the exam score on 10 instead of 100. To add a new variable, use the $ operator and specify a new variable name:

data_grades$Exam_Score_10 <- data_grades$Exam_Score/10

head(data_grades[, c("Exam_Score", "Exam_Score_10")], 3)

  Exam_Score Exam_Score_10
1         55           5.5
2         82           8.2
3         52           5.2

Exercise 4.2

1. What is the class of the variable Tutor? Transform it into a factor. How many tutors are there?

2. Add a new variable to compute the final score of each student, which is the weighted average of their participation grade (20%) and their exam score (80%)

3. Retrieve the final score of the students in tutorial group 2. Obtain summary statistics of their final scores.

4. What is the lowest and the highest final score in tutorial group 2? Retrieve this information from the summary statistics as well as by using a dedicated function.

R Base Graphics

• We will cover R base graphics.
• Other alternatives include `ggplot2’…

To create plots with R’s standard graphics package, there are high-level and low-level plotting functions.

• High-level functions generate a new graphic (and open a device).
• Low-level functions add elements to an existing graphic.

Simple plots

load('data/climate_long.Rdata')
plot(long_data$TEMP)     ## Plotting a single variable

Simple plots

plot(x = long_data$MONTH, y = long_data$TEMP)     ## Scatter plot wrt month

Multiple plots

par(mfrow = c(1,2)) # multiple plots in a row
plot(long_data$TEMP)     ## Plotting a single variable
plot(x = long_data$MONTH, y = long_data$TEMP)     ## Scatter plot wrt month

Functions calling methods

Notice that function plot() calls methods.

It will perform different operations depending on the class of the passed object. (We study the lm() function in detail in the next session!)

ols_result <- lm(TEMP~MONTH, data = long_data)
plot(ols_result)

Exercise 4.3: Load data from Yahoo Finance, and see how plot function behaves.

• Retrieve the AAPL series from Yahoo Finance from January first 2024 until today (see Session 1). You can use

library(quantmod)
getSymbols("AAPL", src = "yahoo", from = "2024-01-01", to = Sys.Date())

• Plot the whole dataset
• Plot a selected column, for example closing prices AAPL$AAPL.Close
• Comment on the x axis of the data and how it is different from the plots we considered earlier.
• The difference is due to the quantmod package.

Creating and saving a graph

• To run this code, create a folder ‘figures’.

load("data/climate_wide.Rdata")
pdf("figures/plot_data_short.pdf")
hist(wide_data$MAASTRICHT, breaks = 20)
dev.off()

Customizing Graphics

• Adding points to an existing plot
• Function `dev.off()’is called after all the plotting, to save the file and return control to the screen.

load("data/climate_wide.Rdata")
plot(wide_data$MAASTRICHT) # temperatures in Maastricht
lines(wide_data$EINDHOVEN) # temperatures in Eindhoven in lines

Customizing Graphics

• The plot() function takes several many arguments that can change the layout of the plots. See ?par for all graphical options; there are many!

• Some examples:

  • col: color of lines / points
  • lty, lwd: Line type and thickness
  • pch: Point type (1-16)
  • main, sub: Title, subtitle
  • xlab, ylab: x and y axis labels
  • log, xlog and ylog for logarithmic scales
  • xlim, ylim: x and y axis limits (for overriding R’s default choices)
  • mfcol, mfrow: Multiple plots in one graphics window (column-wise/row-wise)

Low-Level Graphic Functions

• lines: Draw lines
• abline: Quickly add horizontal, vertical lines, and lines using equation \(y = bx + a\)
• points: Add points
• arrows: Add arrows
• title: Add a title
• legend: Add a legend
• text: Add text at \((x,y)\) coordinates
• mtext: Add text with positional specification like side=1,...,4

Exercise 4.4: Plot temperatures for Maastricht

We want to visualize the daily temperatures in the climate data specifically for Maastricht. First, make a basic plot of temperatures in Maastricht then customise the plot in the following ways:

1. The title of the X-axis should say ‘Month’, the title of the Y-axis ‘Average Temperature’.

2. Make the plot a line plot with a blue line. (Hint: specifying the colour literally as "blue" works)

3. Make the tick marks appear on the inside of the figure rather than the outside.

4. Calculate the average temperature.

5. Add a horizontal line with the average maximum temperature

You will need to consult the help file for this exercise; see this therefore more as an exercise in how to navigate R’s help system, than an exercise in plotting (which we will cover in more detail later).

You may want to ask ChatGPT for help.

Manually saving R plots

• Use the plot functions without creating a graph.
• Use the `plots’ area to save image manually.

Different plot types

You can manually save graphs of several formats.

Best practice is to save a graph through a device such as pdf or similar:

• pdf(): Adobe PDF (easily integrated into LaTeX).
• svg(): Scalable Vector Graphics (commonly used on websites).
• png(), jpeg(), tiff(), bmp(): Various bitmap formats.

jpeg("figures/MaasTemperature.jpeg")
plot(x = wide_data$MAASTICHT, y = wide_data$MONTH)
dev.off()

A more complex example for plotting data over time gradually

• Especially when data is large, a gradual illustration of data over time can be handy.
• In this exercise, we plot the temperatures in Maastricht as if the data are becoming available gradually.
• For this, we will use a loop that iterates over time points (months).
• The concept of a loop was only mentioned in Session 2.

wide_data$MAASTRICHT
     xlab = "Time", ylab = "Value", main = "Adding Data Over Time", type = 'l')

# Gradually plot more and more of the data using a `for loop`
for (i in 4:nrow(wide_data)) {
  plot(1:i, wide_data$MAASTRICHT[1:i], # notice index i is increasing the number of plotted points
     xlab = "Time", ylab = "Value", main = "Adding Data Over Time", type = 'l')
}

Exercise 4.5 Make a continuous plot of temperatures

• Use the last loop example to plot temperatures gradually.
• Start with an initial number of 3 observations, as in the example.
• Make sure that the range of the x and y axes match with the whole dataset in the first plot.
• Within the for loop, add lines to the first plot, instead of plotting the data again.
• Pause the program within the for loop to simulate “gradual” effect.
• You can use ChatGPT or help functions for ?ylim, ?Sys.sleep

Advanced graphics in R

• R has several advanced graphics packages such as ggplot2 (see book by Hadley Wickham), plotly, Rgnuplot,…
• We will focus on ggplot2 as this is widely used.
• The ggplot2 package in R enables to build complex plots from data in a structured and layered manner.
• The plot is based on a specified data frame, aesthetic mappings (like x and y axes), and layers such as points, lines, or bars (geom_* functions).
• Advantages: flexible, consistent syntax, and fancy graphics.
• Disadvantages: Very different syntax e.g. compared to functions.

Typical ggplot help function sections

• Section What it Gives You
• Title Short description
• Description What the function does
• Usage The function’s syntax
• Arguments What each argument means
• Details Extra explanation and special behavior
• Aesthetics Visual mappings (like x, y, color, etc.)
• Examples Example code to learn from

?ggplot2::geom_line

Using ggplot: Grammar of graphics

• Graphics with ggplot2 are built step-by-step, adding new elements as layers
• A plot starts with the function ggplot(). This is the main object we will add layers to.
• Each layer is added with a plus sign (+) between layers. This allows for extensive flexibility and customization of plots.
• Three components need to be specified for the plot:
  • data: data to feed in
  • aesthetics: how you will connect variables (columns) from your data to a visual dimension. Horizontal positioning, size, color etc.
  • geometries: This is a specification of what object will actually be drawn on the plot. This could be a point, a line, a bar, etc.

Using ggplot: Grammar of graphics (cont’d)

• Several optional additional layers customize the graphics and help make flexible graphs.
  • Scales: How a variable is mapped to its aesthetic. Can be linear, in log scale etc.
  • Statistical transformations: Specification of whether and how the data are combined/transformed before being plotted.
  • Coordinate system: Specification of how the position aesthetics (x and y) are depicted, for example cartesian or polar coordinates.
  • Facet: This is a specification of data variables that partition the data into smaller “sub plots”, or panels.

Example ggplot (cont’d)

• Help files of ggplot2 are also slightly different from the standard R help files

?ggplot # a bit complicated help file
help(package = "ggplot2") # a nicer list of all layer functions, see 'geom_line'

Example: Histogram of temperatures in Maastricht

• Notice the syntax difference in parentheses and use of + for layers
• Notice the data wide_data is a data frame with an index

library('ggplot2')
load("data/climate_wide.Rdata") # load data
wide_data$index <- 1:nrow(wide_data) # create data frame
ggplot(wide_data, aes(x = MAASTRICHT)) +
  geom_histogram(bandwidth = 200)

Example: Plot of temperatures in Maastricht

• We will make a similar plot to exercise 4.4: a plot of temperatures in Maastricht
• Aesthetics are defined with line width, color.
• Geometrics is defined by the function geom_line (line plot)

load("data/climate_wide.Rdata") # load data
wide_data$index <- 1:nrow(wide_data) # create data frame
ggplot(wide_data, aes(x = index, y = MAASTRICHT)) +
  geom_line(color = "blue", linewidth = 1) # adds lines #

Example: Possible confusion with aesthetics

• A point that is easy to make a mistake: mapping variables in aesthetics.
• See color is defined in aes below compared to the earlier slide.
• Check the weird legend that appears in the plot, and the line is still red.
• See the help file for ggplot2::geom_line.
• There is a lot of information in the help file as a result of flexibility, but the proper use is explained.

load("data/climate_wide.Rdata") # load data
wide_data$index <- 1:nrow(wide_data) # create data frame
ggplot(wide_data, aes(x = index, y = MAASTRICHT, color = "blue")) +
  geom_line(linewidth = 1) # adds lines #

Example: Adding additional geometrics and aesthetics

• Adding layers, such as a new set of points is simple:

load("data/climate_wide.Rdata") # load data
wide_data$index <- 1:nrow(wide_data) # create data frame
ggplot(wide_data, aes(x = index, y = MAASTRICHT)) +
  geom_line(color = "blue", size = 1, linewidth = 2) +
  geom_line(aes(x = index, y = EINDHOVEN), linewidth = 0.3) # Adds Eindhoven data to the last plot

Example: Adding more layers

• Adding more layers for labels and legends

load("data/climate_wide.Rdata") # load data
wide_data$index <- 1:nrow(wide_data) # create data frame
ggplot(wide_data, aes(x = index, y = MAASTRICHT)) +
  geom_line(color = "blue", size = 1, linewidth = 2) +
  geom_line(aes(x = index, y = EINDHOVEN), linewidth = 0.3) + # Adds Eindhoven data to the last plot
  labs(
    x = "X Axis", y = "Y Axis", color = "Legend Title",   # Axis labels and legend title
    title = "Line Plot with Two Variables"
  ) +
  scale_color_manual(values = c("blue", "red")) +  # Custom colors
  theme_minimal()  # Minimal theme for a clean look

Example ggplot: Color points for Maastricht and Eindhoven temperatures

• For this example, we will use the long dataframe since we will plot values for both cities.
• This is a more complex graph where aesthetics defined in geometrics create point colors according to a variable in the data frame.

library('ggplot2')
load("data/climate_long.Rdata") # load data
long_data$index <- 1:nrow(long_data) # create data frame
ggplot(data = long_data) +
  geom_point(aes(x = index, y = TEMP, color = NAME))

Further references for ggplot2

• R for data science

• Collaborative and Reproducible Data Science in R

Exercise 4.6: Use ggplot2 for more flexible and advanced plots

• Make the same plot as in Exercise 4.4 using package `ggplot2’
• We suggest to use the wide data
• Use help functions for geom_point, geom_hline, labs.
• You can give it a try to use ChatGPT, but the outcome is difficult to understand if you don’t have familiarity with ggplot to begin with.