Setup for Introduction to Research Computing

Best Practices in Data Organisation Using Spreadsheets

Introduction to the Data for this Lesson

The data used in this lesson comes from a project observing a small mammal community in southern Arizona, US. This is part of a project studying the effects of rodents and ants on the plant community that has been running for almost 40 years. The rodents are sampled on a series of 24 plots, with different experimental manipulations controlling which rodents are allowed to access which plots. This is a real dataset that has been used in over 100 publications. It is published at Ecological Archives and can be found on Portal Project Database. This data is open and free to use for research purposes.

For Interest Only: Portal Project Teaching Dataset

The Portal Project Teaching Database is a simplified version of the Portal Project Database designed for teaching. It provides a real world example of life-history, population, and ecological data, with sufficient complexity to teach many aspects of data analysis and management, but with many complexities removed to allow students to focus on the core ideas and skills being taught. The database is currently available in csv, json, and sqlite formats.

The Portal Project Teaching Database’s GitHub repository can be found at: https://github.com/weecology/portal-teachingdb, where suggested changes or additions to this dataset can be requested or contributed. This database is not designed for research as it intentionally removes some of the real-world complexities. The Python code used for converting the original database to this teaching version can be found in create_portal_teach_dataset.py.

CITATION: Ernest, Morgan; Brown, James; Valone, Thomas; White, Ethan P. (2017): Portal Project Teaching Database. Figshare. https://doi.org/10.6084/m9.figshare.1314459.v6

Download Data for Spreadsheets Lesson

For the purposes of training, this data has been simplified a bit (you can still download the full dataset and work with it using exactly the same tools we will learn here). This simplified version of data is available from the Portal Project Teaching Dataset. In this lesson, you will need to download the following five files from the Portal Project Teaching Dataset:

Install Excel

Excel is commonly provided by most institutions via the Microsoft Office suite via an instututional licence. On Windows and Mac Excel can be downloaded using Microsoft Store or the App Store. On Linux systems you can use Excel in a browser (not reccomended) or use an alterntive such as LibreOffice. If you do not have acess to a Microsoft offive licence then please see the LibreOffice installation instructions.

Install LibreOffice

To interact with spreadsheets, you can use various software - for example Microsoft Excel, LibreOffice, Gnumeric, OpenOffice.org, Google Spreadsheets. Commands may differ a bit between programs, but the general ideas for thinking about spreadsheets are the same.

For this lesson, if you do not have a spreadsheet program already, you can use a free and open source tool LibreOffice as it can open Excel spreadsheets, which is the format of the data we will work with during the lesson (also all examples used refer to Excel).

Windows

Mac OS X

Linux

Data Cleaning with OpenRefine

Download Data for OpenRefine Lesson

The Portal Project Teaching Dataset is a real dataset that has been used in over 100 publications. We have simplified it for the purposes of this lesson, but you can download the full dataset (see below for details) and work with it using exactly the same tools we will learn here.

For this lesson, you will need to download the following file (remember where you downloaded the file!):

Data in some of the columns of the above file (e.g. geolocation, locality, county, country, JSON) are contrived for the purpose of the lessons and are in no way related to the original dataset.

Install OpenRefine

For this lesson you will need OpenRefine (formerly GoogleRefine) and a web browser. Download the most recent version of OpenRefine for your operating system, then follow the instructions below.

OpenRefine is a Java program that runs locally on your machine (i.e. you are not accessing a remote service on the Internet). OpenRefine for Mac come with embedded Java, on Windows please select Windows kit with embedded Java, on Linux you will need to install Java separately.

Once it is running on your machine, you access it via your browser at the address http://localhost:3333. No Internet connection is needed for this as the programme is running locally.

Windows

Mac

Linux

Text Editor

A text editor is the piece of software you use to view and write code. If you have a preferred text editor, please use it. Suggestions for text editors are, Notepad++ (Windows), TextEdit (macOS), Gedit (GNU/Linux), GNU Nano, Vim. Alternatively, there are IDE’s (integrated developer environments) that have more features specifically for coding such as VS Code; there are also IDEs specific to languages will be listed in the appropriate section(s) below.

The Bash Shell

Open a Terminal

For this lesson, first you need to be able to open a terminal:

Git Setup

Windows

We’ll be using Git Bash for both git and a shell to run it in. If you’ve already installed Git Bash then go to the next section. Otherwise, go to git for windows and click Download, then install it. Most of the options can be left on default, but be sure you check these:

Mac OS

To use Git you must install the Apple Command Line Tools, this may take a few minutes.

You can obtain these from Apple (requires your Apple ID)

Alternatively, you can install the tools from the command line:

$ xcode-select --install

Linux

Git comes pre-installed on most Linux distributions. You can test if it’s installed by running git --version. If it’s not installed, you can install it by running sudo apt-get install git or sudo yum install git, depending on your distribution.

GitHub

We’ll be using the website GitHub to host, back up, and distribute our code. You’ll need to create an account there. As your GitHub username will appear in the URLs of your projects there, it’s best to use a short, clear version of your name if you can.

Download Data for Shell Lesson

Type the following into the prompt that appears (pressing enter/return after each line):

$ cd
$ git clone https://github.com/Southampton-RSG-Training/shell-novice.git

cd will move to your home directory, and git clone will download a copy of the materials.

Alternatively, if you have SSH authentication with GitHub enabled (if you don’t know what this means don’t worry, it is covered in the Git SWC course if you want to know more!) you can use the following:

$ cd
$ git clone git@github.com:Southampton-RSG-Training/shell-novice.git

This should download all the content for the lesson to a new directory. Please let the instructors know if you run into any problems.

Version Control with Git

Introductory Data Management with R

Install R and RStudio

R is a programming language and software environment for statistical computing and graphics. The RStudio Integrated Development Environment (IDE) is a set of tools designed to help you be more productive with R.

We need to install R and RStudio: The latest links can be found on the RStudio downloads page

R

R can be found at https://cran.rstudio.com/, from here pick your OS and download the latest release, see below for direct links to your OS.

Windows

Mac OS

Linux

RStudio

Your OS should be detected and a link provided under step 2 on this page RStudio downloads page. Else select your OS from the list under All Installers.

Windows

Download and run the .exe file and follow instructions given by your OS.

Mac OS

Download the .dmg file.

Linux

Download the appropriate install file (.rpm or .deb) for your distro.