This is a short tutorial to test using Compute Canada for R computing with brms and mgcv. See also this blog post. For complete information see the extensive Compute Canada docs.

1. Log in and install modules

  1. Make sure you have an active account with Compute Canada.

  2. SSH in to a Compute Canada login node (with -Y for trusted X11 forwarding):
      ssh -Y ${username}@beluga.computecanada.ca
    

    where ${username} is your Compute Canada username (not your CCI). You could also replace beluga with another Compute Canada cluster name, if you’re not in Quebec (see here). You will be prompted to enter a password. Note: if you have entered your public ssh key on Compute Canada’s website, you won’t need a password (you can generate a key, or if you have a key on your computer already it’s probably at ~/.ssh/id_rs.pub; you can paste the contents of that file on the Compute Canada website at My Account > Manage SSH Keys, and press ‘Add Key’ and then you won’t need to use a password to login).

  3. Once you’re logged into, you will see a welcome message and a bash prompt that looks something like [username@beluga3 ~]$. You now need to load the necessary modules. Use the following command. Note: the bioconductor module is necessary for brms to work.
      module load gcc/9.3.0 r-bundle-bioconductor/3.12 r/4.1.2
    

    You may get messages about modules being replaced/reloaded, which you can ignore.

  4. Start an R session
      R
    
  5. Once R is started, install the R packages you want to use.
      install.packages(c("tidyverse", "mgcv", "brms"),
                    repos = "https://mirror.rcg.sfu.ca/mirror/CRAN/")
    

    You may get a warning message saying 'lib = " ... "' is not writeable, asking if I want to use a personal library (respond y), and then asking if you want to create the personal library directory to install packages into (respond y again). The packages will install (and take some time, like an hour). If this succeeds, you should get a message like:

      The downloaded source packages are in
      	‘/tmp/RtmpEFrlg1/downloaded_packages’
    
  6. Quit R with q(). You will return to the bash prompt on the login node.

Now it should be all set.

Note: Steps 1-4 you’ll use anytime you use Compute Canada for an R project. You should only have to run step 5 once ever (the packages will remain installed for you for all future sessions on Beluga). If you need other packages in the future, just open R and install.packages as in step 5 to install them too.

2. Test fitting models in R

When you ssh into Compute Canada, you start out in a “login node” (you’ll see the name of the particular node your on at the prompt. It will probably have a name like beluga3).

Important: A login node is just your entry point to the cluster, it is not where you run your code. Fitting models on a login node will be slow (probably not faster your laptop), and more importantly, it can make the login node unusable to other users! Don’t be rude! To run things, you must do one of the following:

  • Run things interactively. This is for data exploration and interactive debugging of a script, for instance. See Section 2.1 below.
  • Use a job script to do batch submission. This is for large computational tasks, like fitting your heavy-duty models. In general, this is the default way Compute Canada expects you’ll use the cluster. See Section 2.2 below.

2.1 Running things interactively

(skip this section if you just want to submit jobs to be run)

  1. Start in a Compute Canada login node, as above. Then, request an interactive session using the salloc command:

    salloc --x11 --time=1:0:0 --ntasks=4 --mem-per-cpu=4GB --account=${ACCOUNT} 
    

    this example requests four CPU cores for one hour, with 4GB of RAM per CPU. ${ACCOUNT} is the name of the account you want to use.

    • If you misspell or don’t specify the account, you’ll get a helpful error message telling you what accounts you are a member of. Choose one of them.

    • The --x11 flag is necessary only if you want to do interactive plotting.

    After running the salloc command, you’ll see some messages like the below. It may take a minute, but once the resources you requested are allocated you, will see a bash prompt, like [username@blg8610 ~]$

  2. Now you’re on a the “compute node”. Start up R, where you can interactively run Test 1 and Test 2, below.

  3. After running the tests below, q() to quit R, and then exit to relinquish the compute node (you’ll be returned to the login node).

Test 1: using mgcv

Let’s test fitting a GAM.

library(mgcv)
dat <- gamSim(1,n=400,dist="normal",scale=2)
b <- gam(y~s(x0)+s(x1)+s(x2)+s(x3),data=dat)
summary(b)

This should fit the model and print the summary.

and (if you used -x11 flag to salloc when you requested the allocation), test plotting the model, like so

plot(b, select=1)

This should use X11 forwarding to show a plot of the fit smooth for s(x0).

Test 2: using brms

Let’s test fitting a Bayesian regression model with brm()

library(brms)
data("kidney")
fit1 <- brm(formula = time | cens(censored) ~ age * sex + disease
            + (1 + age|patient),
            data = kidney, family = lognormal(),
            prior = c(set_prior("normal(0,5)", class = "b"),
            set_prior("cauchy(0,2)", class = "sd"),
            set_prior("lkj(2)", class = "cor")),
            warmup = 1000, iter = 2000, chains = 4, cores = 4,
            control = list(adapt_delta = 0.95))
summary(fit1)

This should compile the Stan program and fit the model, and print a summary. Works for me. Also test plotting with plot(fit1, select=1). Note that setting cores=4 tells the brms to make use of the multiple cores you requested.

When you’re done, you can quit R and exit the compute node (also, be aware that you’ll be kicked out automatically after the time you requested has been used up).

2.2 Running things in a job script

Start in a login node.

Let’s make an R script that does the same two tests above. Name the script Rtest.R, and save it in the user directory of the login node (that’ll be accessible from the compute nodes too), by running the following:

cd;
cat > Rtest.R << EOF
## testing mgcv
library(mgcv)
dat <- gamSim(1,n=400,dist="normal",scale=2)
gam1 <- gam(y~s(x0)+s(x1)+s(x2)+s(x3),data=dat)
summary(gam1)
png(file="Rtest-plots/gam1.png", width=600, height=500)
plot(gam1, select=1)
dev.off()

## testing brms
library(brms)
data("kidney")
fit1 <- brm(formula = time | cens(censored) ~ age * sex + disease
            + (1 + age|patient),
            data = kidney, family = lognormal(),
            prior = c(set_prior("normal(0,5)", class = "b"),
            set_prior("cauchy(0,2)", class = "sd"),
            set_prior("lkj(2)", class = "cor")),
            warmup = 1000, iter = 2000, chains = 4, cores = 4,
            control = list(adapt_delta = 0.95))
summary(fit1)
png(file="Rtest-plots/fit1.png", width=600, height=500)
plot(fit1, select=1)
dev.off()
EOF

Now make a job script, and save it as Rtest-job.sh in the home directory too:

cd;
cat > Rtest-job.sh << EOF
#!/bin/bash
#SBATCH --account=${ACCOUNT}
#SBATCH --time=00:15:00
#SBATCH --mem-per-cpu=4G
#SBATCH --cpus-per-task=8
#SBATCH --job-name=Rtest
#SBATCH --output=%x-%j.out
#SBATCH --mail-user=${EMAIL}
#SBATCH --mail-type=ALL
module load gcc/9.3.0 r-bundle-bioconductor/3.12 r/4.1.2
Rscript Rtest.R
EOF

This requests 15 minutes of time on a compute node with 8 CPUs, and 4GB of RAM per each. Replace ${ACCOUNT} with the account name you want to use. Replace ${EMAIL} with your email address to be alerted when the job starts/finishes/fails. The --output flag specifies the format for the output file you’ll get when your job terminates. In this case, we’re setting it to be Rtest-jobnumber.out.

Make a directory for the test plots to be put in:

mkdir Rtest-plots

Submit the job to be scheduled.

sbatch Rtest-job.sh

You’ll see something like Submitted batch job 27701226. You can use squeue or sq to see the list of queued jobs. Hopefully yours will be run soon.

When finished, the following outputs will be in the home directory

  • An out file with a name like Rtest-27701226.out, containing the stdout stuff from the R script (and if the job crashed, useful info may be in there).
  • Two plots, in the test-plots/ directory.

3. Transferring files

Beyond just running a simple test like above, you’ll likely need to transfer datasets and scripts to the cluster before running things.

To transfer files between your local machine and your home directory on Béluga

  • you can use scp (OS X/Linux) or Windows equivalents to copy files via SSH. For example, to copy a script from your laptop to your home directory on Béluga, run something like the following (on your local terminal):
    scp /path/to/myscript.R ${username}@beluga.computecanada.ca:/home/${username}/
    

    You can use it similarly to copy output files back to your local machine.

  • for large/multiple files, Compute Canada recommends the Globus system.