Reference

OpenRefine is a powerful and free, open source tool that can be used for data cleaning.

OpenRefine will automatically track any steps you take in working with your data, and will leave your original data intact.

Faceting can identify errors or outliers in data.

Clustering can identify outliers in data and help us fix errors in bulk.

GREL (General Refine Expression Language) is a powerful tool for transforming data.

OpenRefine provides various ways to sort and filter data without affecting the raw data.

All changes are being tracked in OpenRefine (apart from individual cell changes and sorting!), and this information can be used for scripts for future analyses or reproducing an analysis.

Scripts can (and should) be published together with the dataset as part of the digital appendix of the research output.

Cleaned data or entire projects can be exported from OpenRefine.

Projects can be shared with collaborators, enabling them to see, reproduce and check all data cleaning steps you performed.

Other examples and resources online are good for learning more about OpenRefine

Start the python interpreter by typing python in the shell.

Variables are named memory locations, they are used to access data.

A list is an ordered collection of items of any type.

Values in the list can be accessed using their index in square brackets e.g. my_list[ix]

Lists can be manipulated in place using attribute functions e.g. my_list.reverse()

Ranges of values in a list can be obtained via slicing e.g. mylist[start:stop]

We can use the ‘for in’ syntax to loop over collections or generators.

The python function open lets us read r or write w to files by creating a file handler.

We can use string operations such as line.split(',') to process data in files.

We can use logical operations to change the behavior of our code when it meets certain conditions.

Using if, elif, and else we can check conditions and add a branch that runs if none of the conditions are met.

We can combine conditions using and and or to make more complicated logical statements.

A function is created using the def keyword.

Functions take variables that are specified in the function definition and use the return keyword to specify their output.

We can use a module to keep our functions separate to the main body of our code to improve code readability.

Python has built in error names that give a hint to the type of problem you are looking for.

We can use the traceback to find which bit of the code threw an error.

Python uses the sys library to acess command line arguments. sys.argv is a list of command line arguments.

Python program outputs can be used in a pipeline, however, due to the way python works we need to use the signal library to make sure it handles piping output correctly.

Python has many libraries that add to the core language to improve functionality in specific use cases.

Numpy is a numerical python library that makes working with vectors, matricies, or large data tables easier.

Numpy can be used to load datasets directly from CSV files bypassing Pythons built in file systems.

We can use matplotlib to create and manipulate a wide variety of plots in Python.

Once a plot has been made we can use matplotlib’s function savefig to output it in formats appropriate for publication.

Pep8 provides a guide for styling your python code.

Although R has a steeper learning curve than some other data analysis software, R has many advantages - R is interdisciplinary, extensible, great for data wrangling and reproducibility, and produces high quality graphics.

Values can be assigned to objects, which have a number of attributes. Objects can then be used in arithmetic operations (and more).

Functions automate sets of commands, many are predefined but it’s also possible to write your own. Functions usually take one or more inputs (called arguments) and often return a value.

A vector is the most common and basic data structure in R. A vector is composed of a series of values, which can be either numbers or characters.

Vectors can be subset by providing one or several indices in square brackets or by using a logical vector (often the output of a logical test).

Missing data are represented in vectors as NA. You can add the argument na.rm = TRUE to calculate the result while ignoring the missing values. - CSV files can be read in using read.csv().

Data frames are a data structure for most tabular data, and what we use for statistics and plotting.

It is possible to subset dataframes by specifying the coordinates in square brackets. Row numbers come first, followed by column numbers.

Factors represent categorical data. They are stored as integers associated with labels and they can be ordered or unordered. Factors can only contain a pre-defined set of values, known as levels.

dplyr is a package for making tabular data manipulation easier and tidyr reshapes data so that it is in a convenient format for plotting or analysis. They are both part of the tidyverse package.

A subset of columns from a dataframe can be selected using select().

To choose rows based on a specific criterion, use filter().

Pipes let you take the output of one function and send it directly to the next, which is useful when you need to do many things to the same dataset.

To create new columns based on the values in existing columns, use mutate().

Many data analysis tasks can be approached using the split-apply-combine paradigm: split the data into groups, apply some analysis to each group, and then combine the results. This can be achieved using the group_by() and summarize() functions.

Dates can be formatted using the package ‘lubridate’.

To reshape data between wide and long formats, use pivot_wider() and pivot_longer() from the tidyr package.

Export data from a dataframe to a csv file using write_csv().

ggplot2 is a plotting package that makes it simple to create complex plots from data in a data frame.

Define an aesthetic mapping (using the aes function), by selecting the variables to be plotted and specifying how to present them in the graph.

Add ‘geoms’ – graphical representations of the data in the plot using geom_point() for a scatter plot, geom_boxplot() for a boxplot, and geom_line() for a line plot.

Faceting splits one plot into multiple plots based on a factor from the dataset.

Every single component of a ggplot graph can be customized using the generic theme() function. However, there are pre-loaded themes available that change the overall appearance of the graph without much effort.

The gridExtra package allows us to combine separate ggplots into a single figure using grid.arrange().

Use ggsave() to save a plot and edit the arguments (height, width, dpi) to change the dimension and resolution.

Good data organisation is the foundation of any research project.

Never modify your raw data. Always make a copy before making any changes.

Keep track of all of the steps you take to clean your data in a plain text file.

Organise your data according to tidy data principles.

Record metadata in a separate plain text file (such as README.txt) in your project root folder or folder with data.

Include only one piece of information in a cell.

Avoid using multiple tables or spreading data about multiple tabs within one spreadsheet.

Record zeros as zeros.

Avoid spaces, numbers and special characters in column headers.

Avoid special characters in your data.

Use an appropriate null value to record missing data.

Record units in column headers.

Place comments in a separate column.

Do not use formatting to convey information.

Use extreme caution when working with date data.

Splitting dates into their component values can make them easier to handle.

Always copy your original spreadsheet file and work with a copy so you do not affect the raw data.

Use data validation to prevent accidentally entering invalid data.

Data stored in common spreadsheet formats will often not be read correctly into data analysis software, introducing errors into your data.

Exporting data from spreadsheets to formats like CSV or TSV puts it in a format that can be used consistently by most programs.