Setup for Introduction to Research Computing

Best Practices in Data Organisation Using Spreadsheets

Data for Spreadsheets Lesson

The data used in this lesson comes from a project observing a small mammal community in southern Arizona, US. This is part of a project studying the effects of rodents and ants on the plant community that has been running for almost 40 years. The rodents are sampled on a series of 24 plots, with different experimental manipulations controlling which rodents are allowed to access which plots. This is a real dataset that has been used in over 100 publications. It is published at Ecological Archives and can be found on Portal Project Database. This data is open and free to use for research purposes.

For the purposes of training, this data has been simplified a bit (you can still download the full dataset and work with it using exactly the same tools we will learn here). This simplified version of data is available from the Portal Project Teaching Dataset. In this lesson, you will need to download the following five files from the Portal Project Teaching Dataset:

Install Microsoft Excel

Microsoft Excel is commonly provided by most institutions with the Microsoft Office suite via an instututional licence. On Windows and MacOS, Microsoft Excel can be downloaded using the Microsoft Store and the App Store respectively. On Linux, you may use Microsoft Excel in a web browser. This is not reccomended and we suggest you use an alterntive such as LibreOffice instead. If you do not have access to a Microsoft Office licence, please use an alternative such as LibreOffice.

Install LibreOffice

In the lesson, Microsoft Excel is used as the spreadsheet software of choice. An alternative spreadsheet software to Microsoft Excel is LibreOffice Calc. There will be some commands and formatting options which differ between Calc and Excel, but the general workflow and ideas for thinking about data organisation in spreadsheets are the same.

Windows

MacOS

Linux

Data Cleaning with OpenRefine

Introduction to the Data for this Lesson

The data used in this lesson comes from a project observing a small mammal community in southern Arizona, US. This is part of a project studying the effects of rodents and ants on the plant community that has been running for almost 40 years. The rodents are sampled on a series of 24 plots, with different experimental manipulations controlling which rodents are allowed to access which plots. This is a real dataset that has been used in over 100 publications. It is published at Ecological Archives and can be found on Portal Project Database. This data is open and free to use for research purposes.

Download Data for OpenRefine Lesson

The Portal Project Teaching Dataset is a real dataset that has been used in over 100 publications. We have simplified it for the purposes of this lesson, but you can download the full dataset (see below for details) and work with it using exactly the same tools we will learn here.

For this lesson, you will need to download the following file (remember where you downloaded the file!):

Data in some of the columns of the above file (e.g. geolocation, locality, county, country, JSON) are contrived for the purpose of the lessons and are in no way related to the original dataset.

Install OpenRefine

For this lesson you will need OpenRefine (formerly GoogleRefine) and a web browser. Download the most recent version of OpenRefine for your operating system, then follow the instructions below.

OpenRefine is a Java program that runs locally on your machine (i.e. you are not accessing a remote service on the Internet). OpenRefine for Mac come with embedded Java, on Windows please select Windows kit with embedded Java, on Linux you will need to install Java separately.

Once it is running on your machine, you access it via your browser at the address http://localhost:3333. No Internet connection is needed for this as the programme is running locally.

Windows

Mac

Linux

Text Editor

A text editor is the piece of software you use to view and write code. If you have a preferred text editor, please use it. Suggestions for text editors are, Notepad++ (Windows), TextEdit (macOS), Gedit (GNU/Linux), GNU Nano, Vim. Alternatively, there are IDE’s (integrated developer environments) that have more features specifically for coding such as VS Code; there are also IDEs specific to languages will be listed in the appropriate section(s) below.

Building Programs with Python

Python Setup

IDEs: PyCharm, Spyder, VS Code

We use Python 3*. The “Anaconda3” package provides everything Python-related you will need for the workshop. To install Anaconda, follow the instructions below.

Some old research projects may be in Python 2 but Python 2 has been retired and new projects should be in Python 3.

Windows

Download the latest Anaconda Windows installer. Double-click the installer and follow the instructions. When asked “Add Anaconda to my PATH environment variable”, answer “yes”. It will warn you not to, but it’s required for it to be found by git bash After it’s finished, close and reopen any open terminals to reload the updated PATH and allow the installed Python to be found.

Once the Anaconda installation is finished you will be asked if you want the installer to initialize Anaconda3 by running conda init? You should select yes. Alternatively/additionally you will need to run the following command in GitBash

conda init bash

Then close and reopen GitBash.

Please test the python install open GitBash (or your favorite terminal) and run the following command to verify that the installation was successful.

cd ~
python

You can then type the following to exit:

quit()

In some cases GitBash will hang on this command and not launch the Python interpreter. 
In this case close and reopen git bash and issue the following commands:

cd ~
echo 'alias python="winpty python.exe"' >> .bashrc
source .bashrc
python

Mac OS X

Mac OS Intel

Download the latest Anaconda Mac OS X installer. Double-click the .pkg file and follow the instructions.

Mac OS M1

If you have a M1 Mac you need a specific version of Anaconda follow the link below.

M1 Compatible Anaconda

Once the Anaconda installation is finished you will be asked if you want the installer to initialize Anaconda3 by running conda init? You should select yes.

Linux

Download the latest Anaconda Linux Installer.

Install via the terminal like this (you will need to change the version number to the latest version):

First move to the folder where you downloaded the installer, this is likely to be the Downloads folder e.g.

$ cd ~/Downloads

$ bash Anaconda3-2021.11-Linux-x86_64.sh

Answer ‘yes’ to allow the installer to initialize Anaconda3 in your .bashrc.

Download Data for Python Lesson

Now we are ready to download the code that we need for this lesson. Open a terminal on your machine, and enter:

$ cd
$ git clone https://github.com/Southampton-RSG-Training/python-novice

cd will move to your home directory, and git clone will download a copy of the materials.

Introductory Data Management with R

Install R and RStudio

R is a programming language and software environment for statistical computing and graphics. The RStudio Integrated Development Environment (IDE) is a set of tools designed to help you be more productive with R.

We need to install R and RStudio: The latest links can be found on the RStudio downloads page

R

R can be found at https://cran.rstudio.com/, from here pick your OS and download the latest release, see below for direct links to your OS.

Windows

Mac OS

Linux

RStudio

Your OS should be detected and a link provided under step 2 on this page RStudio downloads page. Else select your OS from the list under All Installers.

Windows

Download and run the .exe file and follow instructions given by your OS.

Mac OS

Download the .dmg file.

Linux

Download the appropriate install file (.rpm or .deb) for your distro.