---
title: "Manipulating Data Sets"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

In this tutorial we will learn how to manipulate data sets in R:

- adding column names
- adding additional variables (columns)
- summarizing the data
- merging and combining two or more data sets

## Getting started 

This tutorial is a R Mardown document that you can use as laboratory notebook and as base script for data manipulation.

- the document contains text/notes (white background) as well as r-code chunks (grey background)
- you can execute/render the file by clicking on the `Knit` button, which will execute the r-code chunks and render the text in a given output format (here: HTML)
- You can also execute the r-chunks one-by-one in the console by clicking on the small green arrow at the upper right hand side of the chunk
- r-chunks may be named (first string after `r` up to the first comma), but names must be unique

See [Introduction to R Markdown](http://rmarkdown.rstudio.com/articles_intro.html) and [Getting Started](http://rmarkdown.rstudio.com/lesson-1.html) for more details on R Markdown documents.

Let's get started

- download [Tutorial_Data-Set_Manipulation.Rmd](Tutorial_Manipulating-data-sets.Rmd)
- save the file in your course directory
- copy the file and rename the copy in `Data-Set_Manipulation.Rmd`
- open the new file in [R-Studio](https://www.rstudio.com/) by double clicking on the file

## Data frames

As examples we will use the data sets that we created in the [Tutorial on Data extraction and Data Formats](Tutorial_Data_Extraction-and-Data-Formats.html) on the distribution of content verbs and their parts-of-speech across registers in different corpora of the Brown corpus family (the BROWN, FROWN, FLOB and LOB corpus).

- [distr_vfull_lemma-pos-reg_brown.txt](data/distr_vfull_lemma-pos-reg_brown.txt)
- [distr_vfull_lemma-pos-reg_frown.txt](data/distr_vfull_lemma-pos-reg_frown.txt)
- [distr_vfull_lemma-pos-reg_flob.txt](data/distr_vfull_lemma-pos-reg_flob.txt)
- [distr_vfull_lemma-pos-reg_lob.txt](data/distr_vfull_lemma-pos-reg_lob.txt)

We will first load the data set from the BROWN corpus and store it in the data frame `d.brown` and add column names representing the features (`verb, pos, register`).

```{r load data}
# load the data and store it in d.brown
d.brown <- read.table("data/distr_vfull_lemma-pos-reg_brown.txt", header=F, fill=T, sep="\t", row.names=NULL, quote="")
# add column names
colnames(d.brown) <- c("verb","pos","register")
```

We can now have a first look at the data frame.

```{r}
head(d.brown)
```

A data frame is an object class in R to store multivariate data sets.
Each column in a data frame can have a different class (e.g. numeric, factor, character). 

- `numeric` for number values
- `factor` for categorical values
- `character` for string values

If not specified explicitlty, R assignes the class of a column automatically based on the values fo the column. 
We will see later that this does not always work as desired.

With `summary` we can get a summarization of the values in the data frame. Depending on the class each column may be treated differently.    
Our example contains only columns of the class `factor`. Thus, the summary for each column lists the most frequent values with their frequency.

```{r}
summary(d.brown)
```

We can access single rows and columns, or groups by the rownames or columnnames (if they exists) or by their number.

```{r accessing rows and columns, echo=TRUE, eval=FALSE}
# row number 3
d.brown[3,]
# rows 3-10
d.brown[3:10,]
# column number 2
d.brown[,2]
# column number 2:3
d.brown[,2:3]
# column numbers 1 and 3
d.brown[,c(1,3)]
# column by columnname
d.brown$verb
d.brown[,"verb"]
# several columns by name
d.brown[,c("verb","register")]
```

## Adding columns to a data frame
The data set does not yet contain meta information about the corpus it was extracted from. Thus, we add the columns

- `corpus` with the value `brown`
- `year` with the value `1961`
- `lgvar` (for language variety) with the value `AE`

If we would simply assign the values to R as follows, the variables would automatically be assinged the class `character`.


```{r add an additional variable1 (column)}
d.brown$corpus <- "brown"
d.brown$year <- "1961"
d.brown$lgvar <- "AE"
```

With a summary we will then simply get the information about the total number of rows in the column the `class` and `mode`

```{r}
summary(d.brown)
```

However, the meta data of the corpus are categorical features. 
We can tell R this by specifying the class explicitly when assigning the values.

```{r add an additional variable (column)}
d.brown$corpus <- factor("brown")
d.brown$year <- factor("1961")
d.brown$lgvar <- factor("AE")
```

A summary shows the difference.

```{r}
summary(d.brown)
```

### Merging data frame to add additional features

What about the `register` references. 
The single characters are not very telling. 
Let's add a column with more speaking names.
The data set [brown_registers.txt](data/brown_registers.txt) contains a mapping of the character codes and the more telling names of the registers. 

```{r load data set registers}
# load data set with register information
d.reg <- read.table("data/brown_registers.txt",header=T,sep="\t",row.names=NULL,quote="")
d.reg
```

We can see that the registers fall into different `types`. 
Thus, we will add a variable register type (`reg.type`) to the data set.

```{r add variable based on other variable}
# categorize registers using character codes
d.reg$reg.type[d.reg$register %in% c("A","B","C")] <- "press"
d.reg$reg.type[d.reg$register %in% c("D","E","F","G","H")] <- "prose"
d.reg$reg.type[d.reg$register %in% "J"] <- "academic"
# categorize registers using regular expression
d.reg$reg.type[grep("fiction",d.reg$reg.long)] <- "fiction"
d.reg$reg.type <- as.factor(d.reg$reg.type)
```

```{r}
d.reg
```

## Merging data frames

Now we would like to add this information to our data set `d.brown`.
As you can see the data frames `d.reg` and `d.brown` both contain a column with the name `register`, which includes the same set of values.
We will use this column to `merge` the two data sets based on this column and overwrite `d.brown`

```{r add variables based on variables in another data set}
# merge information of two data sets
d.brown <- merge(d.brown,d.reg)
```

As a result `d.brown` now additionally contains the columns `reg.long` and `reg.type` from the `register` data set.

```{r}
head(d.brown)
```


```{r general summary}
summary(d.brown)
```

## Another data set

Now we have to load and modify the data set from the FROWN corpus in the same way.
This time we combine all commands in one chunk and save the new data set as `d.frown`.

```{r all-in-one: frown}
# load the data set
d.frown <- read.table("data/distr_vfull_lemma-pos-reg_frown.txt",header=F,sep="\t",row.names = NULL,quote="")
# add column names
colnames(d.frown) <- c("verb","pos","register")
# add corpus information
d.frown$corpus <- factor("frown")
d.frown$year <- factor("1991")
d.frown$lgvar <- factor("AE")
# add register information
d.frown <- merge(d.frown,d.reg)
```

## Combining two data frames

We can now combine the two data sets (`d.brown` and `d.frown`) by concatenating them.

```{r combine two data sets}
d.all <- rbind(d.brown,d.frown)
```

```{r echo=FALSE}
#DT::datatable(d.all)
```


```{r}
summary(d.all)
```

## Saving data sets in a file

For further use we will save our newly created data set in the data directory as `distr_vfull_lemma-pos-reg_brown-all.txt`

```{r}
write.table(d.all,file = "data/distr_vfull_lemma-pos-reg_brown-all.txt",quote=FALSE,sep="\t",row.names = FALSE)
```


## Exercise

- Extract corresponding data sets from the [FLOB](https://corpora.clarin-d.uni-saarland.de/cqpweb/flob/) and the [LOB](https://corpora.clarin-d.uni-saarland.de/cqpweb/lob/) corpus.
- Manipulate the data sets in the same way as `d.brown` and `d.frown`
- Add the data sets to `d.all` using `rbind`
- don't forget to save the newly created data sei in the data directory overwriting `distr_vfull_lemma-pos-reg_brown-all.txt`
<!-- - TIP
    - copy the `all-in-one: frown` chunk below; ATTENTION: you have to rename the chunk as chunks have to have unique names!
    - don't forget to rbind the two additional data frames to d.all
    - and don't forget to SAVE the new d.all
-->
<br>

|                                                                       |     |         |
|:--------------------------------------------------------              |:---:|-----------------------------------------------:|
[ << Back: Data Extraction and Data Formats <<](Tutorial_Data-Extraction-and-Data-Formats.html) |   | [>> Next: Data Analysis I (Frequency distribution, normalization)>>](Tutorial_Data_analysis_norm-and-freq.html) |

<br>
