---
title: "Packages and provisioning"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Packages and provisioning}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

<!-- 
    This vignette builds fine locally, and can probably be built ok
    on the cluster, but it's also quite slow to build and will be subject
    to network issues/package failures so it's not totally desirable that
    it builds every commit either.
    
    In its current form, it does not use the windows cluster at all,
    and uses the example driver, just like hipercow.Rmd etc.
-->



<!-- Please edit the file in vignettes_src/ -->

Depending on where they come from and how specific you need to be on versioning, getting packages available for your cluster tasks ranges from very straightforward to a constant struggle.

We install packages into a special directory within `hipercow/` by running a special job on the cluster itself using our [`conan2`](https://mrc-ide.github.io/conan2) package.  This aims to solve most of the common situations that we have seen, and has a couple of escape hatches that we hope will help with future problems.

There are three broad approaches to installing packages that we support:

* Dependency resolution and package installation using [`pkgdepends`](https://r-lib.github.io/pkgdepends) (`method = "pkgdepends"`)
* A totally automatic method that uses `pkgdepends` but tries to just work out everything itself (`method = "auto"`)
* A bespoke script that you write yourself, and we run (`method = "script"`)
* Reproducible project environments with [`renv`](https://rstudio.github.io/renv/) (`method = "renv"`)

We expect that packages might come from any number of places:

* CRAN or Bioconductor
* A CRAN-like repository (such as an ["R universe"](https://mrc-ide.r-universe.dev) or a drat repository)
* A **public** GitHub repository, which one might ordinarily install via `remotes::install_github()`
* Local sources, available on your computer and in the hipercow directory

We don't currently support packages from private repositories, but let us know if that is a problem for you.  We have a plan to support this later in 2024.

This all ends up being a lot of flexibility, so our (current) suggestion if you are starting out is to use the `pkgdepends` method with the `pkgdepends.txt` configuration a starting point as this will be the easiest for us to replicate issues with, and should end up being the most predictable.


```r
library(hipercow)
```

# Using `pkgdepends`

This is the method that is least prescriptive on your workflow and simplest.  We recommend this as a good starting place.

There are two places that we might get lists of packages to install:

* From a file `pkgdepends.txt` in the directory that also contains the `hipercow/` directory
* Manually, within a call to `hipercow_provision()`

We'll consider these in turn here.

## A list of packages

Create a file called `pkgdepends.txt` at the top of the hipercow root.  This file has a very simple format, with each line being the name of a package, or a [`pkgdepends` "package reference"](https://r-lib.github.io/pkgdepends/reference/pkg_refs.html), plus support for comments, blank lines and specifying additional repositories to install from.

So a complex case might look like

```
# Also use the ncov-ic universe
repo::https://ncov-ic.r-universe.dev

# Specific version of malaria simulation, via a tag
mrc-ide/malariasimulation@v1.6.0

# Package from CRAN
coda
```

* Blank lines are ignored
* Lines starting with `#` are comments and are also ignored
* Entries are separated by newlines
* Lines like `coda` represent packages from CRAN or another CRAN-like repository
* Lines like `mrc-ide/malariasimulation@v1.6.0` are a pkgdepends reference and can be quite complex to control what version is installed
* Lines like `repo::https://ncov-ic.r-universe.dev` add additional CRAN-like repositories (this is the recommended way of installing some of our packages such as `odin` or `orderly2`)
* You can install packages that you have a `.tar.gz` for by adding a line like `local::./mypkg_0.1.0.tar.gz` where the path is relative to your hipercow root folder.

The [`pkgdepends` docs](https://r-lib.github.io/pkgdepends/reference/pkg_refs.html) provide lots of nice examples on package references.  Note that the `repo::<url>` line is an extension to this.



We have an new empty hipercow root, containing a simple `pkgdepends.txt` file:


```
#> .
#> ├── hipercow
#> └── pkgdepends.txt
```

The `pkgdepends.txt` file contains simply:

```
cowsay
```

We can provision this environment by running


```r
hipercow_provision()
#> ✔ Selected provisioning method 'pkgdepends'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method pkgdepends
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc77685796
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc77685796/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113025
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> 
#> ✔ Updated metadata database: 4.19 MB in 9 files.
#> 
#> ℹ Updating metadata database
#> ✔ Updating metadata database ... done
#> 
#> + cowsay     0.9.0 [bld][dl]
#> + crayon     1.5.2 [bld][dl]
#> + fortunes   1.5-4 [bld][dl]
#> + rmsfact    0.0.3 [bld][dl]
#> ℹ Getting 4 pkgs with unknown sizes
#> ✔ Got rmsfact 0.0.3 (source) (10.73 kB)
#> ✔ Got fortunes 1.5-4 (source) (192.94 kB)
#> ✔ Got crayon 1.5.2 (source) (40.57 kB)
#> ✔ Got cowsay 0.9.0 (source) (573.56 kB)
#> ℹ Building crayon 1.5.2
#> ℹ Building fortunes 1.5-4
#> ℹ Building rmsfact 0.0.3
#> ✔ Built rmsfact 0.0.3 (883ms)
#> ✔ Built fortunes 1.5-4 (1s)
#> ✔ Installed rmsfact 0.0.3  (46ms)
#> ✔ Installed fortunes 1.5-4  (29ms)
#> ✔ Built crayon 1.5.2 (1.6s)
#> ✔ Installed crayon 1.5.2  (12ms)
#> ℹ Building cowsay 0.9.0
#> ✔ Built cowsay 0.9.0 (806ms)
#> ✔ Installed cowsay 0.9.0  (17ms)
#> ✔ Summary:   4 new  in 4.4s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113025'
#> Done!
```

This launches a small task on the cluster and uses `pkgdepends` to install `cowsay` and all its dependencies.  If recent installations have been attempted, it will use cached copies, which should make things fairly snappy.

Once installed we can run a task that uses this package:


```r
id <- task_create_expr(cowsay::say("HiperMoo", "cow"))
#> ✔ Submitted task '1d520a4efa59c757917b2ad74148a48c' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc77685796' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: 1d520a4efa59c757917b2ad74148a48c
#> ℹ starting at: 2024-01-26 11:30:34.348013
#> ℹ task type: expression
#> ℹ expression: cowsay::say("HiperMoo", "cow")
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#> 
#>  ----- 
#> HiperMoo 
#>  ------ 
#>     \   ^__^ 
#>      \  (oo)\ ________ 
#>         (__)\         )\ /\ 
#>              ||------w|
#>              ||      ||
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:34.348013 (elapsed: 0.1545 secs)
```

For all methods of provisioning you can see what has been done previously by running `hipercow_provision_list()` and `hipercow_provision_check()`.  Running `hipercow_provision_list()` just lists the installations into your library:


```r
hipercow_provision_list()
#> ℹ 1 conan installation recorded
#> • 1: 20240126113025 (moments ago) [0]
```

The `hipercow_provision_check()` function also compares previously performed installations with the what `hipercow_provision()` would do with the same arguments:


```r
hipercow_provision_check()
#> ✔ Selected provisioning method 'pkgdepends'
#> ℹ 1 conan installation recorded
#> • 1: 20240126113025 (moments ago) [0] (*)
#> ℹ The entry marked with '*' matches the provided installation hash
```

You can see what is in a library by using `hipercow_provision_compare()`; this compares between two versions of the library.  Because we only have one version here the comparison will be against the empty library:


```r
hipercow_provision_compare()
#> ── Comparing conan installations ───────────────────────────────────────────────
#> • (empty installation)
#> • 20240126113025 1st; current installation (moments ago)
#> 
#> ── 4 added packages ──
#> 
#> • cowsay (0.9.0) CRAN
#> • crayon (1.5.2) CRAN
#> • fortunes (1.5.4) CRAN
#> • rmsfact (0.0.3) CRAN
```

## Manually

Sometimes, regardless of how things are installed, you need to install something manually.  For example, you just want to change the version being used of some package, or `pkgdepends` fails to resolve a nice set of dependencies from some complex interdependent set of packages and you want to install a specific version on top of everything it did.

To do this, `hipercow_provision` with `method = "pkgdepends"` and `refs` as a character vector of package references, following the same format as `pkgdepends.txt`


```r
hipercow_provision(method = "pkgdepends", refs = "cran::coda")
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method pkgdepends
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc77685796
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc77685796/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113035
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cran::coda
#> ℹ Loading metadata database
#> ✔ Loading metadata database ... done
#> 
#> + coda   0.19-4 [bld]
#> ℹ No downloads are needed, 1 pkg is cached
#> ✔ Got coda 0.19-4 (source) (74.24 kB)
#> ℹ Building coda 0.19-4
#> ✔ Built coda 0.19-4 (1.8s)
#> ✔ Installed coda 0.19-4  (13ms)
#> ✔ Summary:   1 new   1 kept  in 1.8s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113035'
#> Done!
```

Now `hipercow_provision_list()` and hipercow_provision_compare()` show we have two steps in the history of the library and we can see what has changed:


```r
hipercow_provision_list()
#> ℹ 2 conan installations recorded
#> • 1: 20240126113025 (moments ago) [-1]
#> • 2: 20240126113035 (moments ago) [0]
hipercow_provision_compare()
#> ── Comparing conan installations ───────────────────────────────────────────────
#> • 20240126113025 1st; previous installation (moments ago)
#> • 20240126113035 2nd; current installation (moments ago)
#> 
#> ── 4 unchanged packages ──
#> 
#> ℹ To show unchanged packages, print with 'show_unchanged = TRUE'
#> 
#> ── 1 added package ──
#> 
#> • coda (0.19.4) CRAN
```

# Automatically, from an environment

The `pkgdepends` approach above duplicates some information already present in the packages listed in your calls to `hipercow_environment()`, implied in the files `source()`'d at the start of your task and from the local metadata about where your packages were installed from in the first place.

We try to poke around in your script files and find all the packages that you use, then we look at your installation, and from that build a set of references that we hope will recreate the installation.



We have an new empty hipercow root, this time containing some source file `src.R`:


```
#> .
#> ├── hipercow
#> └── src.R
```

The file `src.R` contains the code that we want to run on the cluster:

```r
cowsay_fact <- function() {
  cowsay::say("catfact", "cow")
}
```

We tell hipercow about this:


```r
hipercow_environment_create(sources = "src.R")
#> ✔ Created environment 'default'
```

Now, when we run `hipercow_provision()`, `hipercow` will check through our `src.R` file and seen that we use `cowsay`


```r
hipercow_provision()
#> ✔ Selected provisioning method 'auto'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method auto
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113040
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> 
#> ✔ Updated metadata database: 4.19 MB in 9 files.
#> 
#> ℹ Updating metadata database
#> ✔ Updating metadata database ... done
#> 
#> ! Failed to update system requirement mappings, will use cached mappings.
#> + cowsay     0.9.0 [bld][dl]
#> + crayon     1.5.2 [bld][dl]
#> + fortunes   1.5-4 [bld][dl]
#> + rmsfact    0.0.3 [bld][dl]
#> ℹ Getting 4 pkgs with unknown sizes
#> ✔ Got crayon 1.5.2 (source) (40.57 kB)
#> ✔ Got rmsfact 0.0.3 (source) (10.73 kB)
#> ✔ Got fortunes 1.5-4 (source) (192.94 kB)
#> ✔ Got cowsay 0.9.0 (source) (573.56 kB)
#> ℹ Building crayon 1.5.2
#> ℹ Building fortunes 1.5-4
#> ℹ Building rmsfact 0.0.3
#> ✔ Built rmsfact 0.0.3 (880ms)
#> ✔ Built fortunes 1.5-4 (985ms)
#> ✔ Installed rmsfact 0.0.3  (44ms)
#> ✔ Installed fortunes 1.5-4  (29ms)
#> ✔ Built crayon 1.5.2 (1.5s)
#> ✔ Installed crayon 1.5.2  (34ms)
#> ℹ Building cowsay 0.9.0
#> ✔ Built cowsay 0.9.0 (803ms)
#> ✔ Installed cowsay 0.9.0  (17ms)
#> ✔ Summary:   4 new  in 4.3s
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113040'
#> Done!
```

Now our code works!


```r
id <- task_create_expr(cowsay_fact())
#> ✔ Submitted task '90d0a213194dbc448f0af38f3b5ac31e' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc1833a1ef' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: 90d0a213194dbc448f0af38f3b5ac31e
#> ℹ starting at: 2024-01-26 11:30:48.655369
#> ℹ task type: expression
#> ℹ expression: cowsay_fact()
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#> 
#>  -------------- 
#> Your cat's heart beats at a rate almost double that of yours, from 110-140 beats per minute. 
#>  --------------
#>     \
#>       \
#>         \
#>             |\___/|
#>           ==) ^Y^ (==
#>             \  ^  /
#>              )=*=(
#>             /     \
#>             |     |
#>            /| | | |\
#>            \| | |_|/\
#>       jgs  //_// ___/
#>                \_)
#>   
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:48.655369 (elapsed: 0.9717 secs)
```

If we had installed `cowsay` initially by running `remotes::install_github()` it would have noticed this and installed the package that way.  For example, the [`rfiglet` package](https://richfitz.github.io/rfiglet) is not available on CRAN and must be installed via `remotes`:

```r
remotes::install_github("richfitz/rfiglet")
```



I've adjusted our `src.R` to use `rfiglet`:

```r
cowsay_fact <- function() {
  cowsay::say("catfact", "cow")
}

figlet_date <- function() {
  print(rfiglet::figlet(as.character(Sys.Date())))
}
```

and now when we call `hipercow_provision()` you can see it installs rfiglet too, automatically finding it on GitHub!


```r
hipercow_provision()
#> ✔ Selected provisioning method 'auto'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method auto
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc1833a1ef/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113050
#> Logs from pkgdepends follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> ── repos 
#> • https://cloud.r-project.org
#> 
#> ── refs 
#> • cowsay
#> • richfitz/rfiglet@HEAD
#> ℹ Loading metadata database
#> ✔ Loading metadata database ... done
#> 
#> + rfiglet   0.2.0 [bld][cmp] (GitHub: d713c1b)
#> ℹ No downloads are needed, 1 pkg is cached
#> ✔ Got rfiglet 0.2.0 (source) (144.05 kB)
#> ℹ Packaging rfiglet 0.2.0
#> ✔ Packaged rfiglet 0.2.0 (318ms)
#> ℹ Building rfiglet 0.2.0
#> ✔ Built rfiglet 0.2.0 (979ms)
#> ✔ Installed rfiglet 0.2.0 (github::richfitz/rfiglet@d713c1b) (14ms)
#> ✔ Summary:   1 new   4 kept  in 992ms
#> 
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113050'
#> Done!
```


```r
id <- task_create_expr(figlet_date())
#> ✔ Submitted task 'eb00966e02cecd38fddb79e8b262b59d' using 'example'
task_wait(id)
#> [1] TRUE
task_log_show(id)
#> 
#> ── hipercow 0.2.33 running at '/tmp/Rtmp0FgkdQ/file1af6fc1833a1ef' ─────────────
#> ℹ library paths:
#> • /home/rfitzjoh/lib/R/library
#> • /usr/local/lib/R/site-library
#> • /usr/lib/R/site-library
#> • /usr/lib/R/library
#> ℹ id: eb00966e02cecd38fddb79e8b262b59d
#> ℹ starting at: 2024-01-26 11:30:55.72189
#> ℹ task type: expression
#> ℹ expression: figlet_date()
#> ℹ no local variables
#> ───────────────────────────────────────────────────────────────── task logs ↓ ──
#>  ____   ___ ____  _  _          ___  _      ____   __
#> |___ \ / _ \___ \| || |        / _ \/ |    |___ \ / /_
#>   __) | | | |__) | || |_ _____| | | | |_____ __) | '_ \
#>  / __/| |_| / __/|__   _|_____| |_| | |_____/ __/| (_) |
#> |_____|\___/_____|  |_|        \___/|_|    |_____|\___/
#> 
#> ───────────────────────────────────────────────────────────────── task logs ↑ ──
#> ✔ status: success
#> ℹ finishing at: 2024-01-26 11:30:55.72189 (elapsed: 0.1573 secs)
```


# Using a script

Sometimes you just need a lot more control than `pkgdepends` can provide and you want to do your own thing.  To support this, create a file `provision.R` in the root of your hipercow repository and within this write whatever you want to install packages.



Here, we have another empty hipercow root, containing a file `provision.R`:


```
#> .
#> ├── hipercow
#> └── provision.R
```

The `provision.R` file contains simply:

```r
install.packages("cowsay")
cowsay::say("Moo", "cow")
```

Here, it's very simple, and just installs the package (plus runs a simple function from it).


```r
hipercow_provision()
#> ✔ Selected provisioning method 'script'
#> /`-'\  _______  ___  ___ ____
#> \,T./ / __/ _ \/ _ \/ _ `/ _ \
#>   |   \__/\___/_//_/\_,_/_//_/
#>   |   ---- THE  LIBRARIAN ----
#> 
#> Bootstrapping from: /home/rfitzjoh/lib/R/library
#> Installing into library: hipercow/lib
#> Using method script
#> Running in path: /tmp/Rtmp0FgkdQ/file1af6fc18c44b73
#> Library paths:
#>   - /tmp/Rtmp0FgkdQ/file1af6fc18c44b73/hipercow/lib
#>   - /usr/local/lib/R/site-library
#>   - /usr/lib/R/site-library
#>   - /usr/lib/R/library
#> id: 20240126113056
#> Logs from your installation script 'provision.R' follow:
#> 
#> -------------------------------------------------------------------------------
#> 
#> 
#> > install.packages("cowsay")
#> Installing package into ‘/tmp/Rtmp0FgkdQ/file1af6fc18c44b73/hipercow/lib’
#> (as ‘lib’ is unspecified)
#> also installing the dependencies ‘crayon’, ‘fortunes’, ‘rmsfact’
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/crayon_1.5.2.tar.gz'
#> Content type 'application/x-gzip' length 40567 bytes (39 KB)
#> ==================================================
#> downloaded 39 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/fortunes_1.5-4.tar.gz'
#> Content type 'application/x-gzip' length 192938 bytes (188 KB)
#> ==================================================
#> downloaded 188 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/rmsfact_0.0.3.tar.gz'
#> Content type 'application/x-gzip' length 10728 bytes (10 KB)
#> ==================================================
#> downloaded 10 KB
#> 
#> trying URL 'https://cloud.r-project.org/src/contrib/cowsay_0.9.0.tar.gz'
#> Content type 'application/x-gzip' length 573560 bytes (560 KB)
#> ==================================================
#> downloaded 560 KB
#> 
#> * installing *source* package ‘crayon’ ...
#> ** package ‘crayon’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (crayon)
#> * installing *source* package ‘fortunes’ ...
#> ** package ‘fortunes’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (fortunes)
#> * installing *source* package ‘rmsfact’ ...
#> ** package ‘rmsfact’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (rmsfact)
#> * installing *source* package ‘cowsay’ ...
#> ** package ‘cowsay’ successfully unpacked and MD5 sums checked
#> ** using staged installation
#> ** R
#> ** inst
#> ** byte-compile and prepare package for lazy loading
#> ** help
#> *** installing help indices
#> ** building package indices
#> ** installing vignettes
#> ** testing if installed package can be loaded from temporary location
#> ** testing if installed package can be loaded from final location
#> ** testing if installed package keeps a record of temporary installation path
#> * DONE (cowsay)
#> 
#> The downloaded source packages are in
#> 	‘/tmp/RtmpDKlA7e/downloaded_packages’
#> 
#> > cowsay::say("Moo", "cow")
#> 
#>  ----- 
#> Moo 
#>  ------ 
#>     \   ^__^ 
#>      \  (oo)\ ________ 
#>         (__)\         )\ /\ 
#>              ||------w|
#>              ||      ||
#> -------------------------------------------------------------------------------
#> Writing library description to 'hipercow/lib/.conan/20240126113056'
#> Done!
```

Please be considerate and don't write scripts that will take very long to run (e.g., no longer than 10-15 minutes) or they may be killed by the cluster scheduler.  Please also let us know if you have a script that takes this long.

# Using renv

If you are using [`renv`](https://rstudio.github.io/renv/) to set up your packages, you can use this with hipercow, and this should be detected automatically.  With your renv project loaded and a lockfile created (so that `renv::status()` reports "No issues found -- the project is in a consistent state.") you should be able to run

```r
hipercow_provision()
```

and see `renv` build a new library based on your lockfile.  New tasks launched after this will use that library.

<!--
It would be nice to show renv usage here, but it's hard to get working reliably because we don't actually want renv running in the vignette process, and callr won't reliably pick up renv from a non-renv session!
-->

The workflow differs very slightly from using `renv` normally in that we take some steps to prevent `renv` updating the library automatically while your tasks run because this will result in disaster if multiple tasks trigger this in parallel.

# Some details about the process

The general hope is that relatively little configuration or specific calls are needed here, but this comes at the cost of a little magic.

If the `method` argument to `hipercow_provision()` is not given, then we use a simple heuristic to select a method:

* If a file `provision.R` is present, we use the `script` method; this takes precedence because it's the biggest, least clever hammer. You might use this as a last resort and so it should override any other method that you might be using.
* If a file `pkgdepends.txt` is present, we use the `pkgdepends` method; we think that this method is a reasonable tradeoff of predictability and usability and should cover most people's needs.
* Otherwise we'll try for an automatic installation; this is to help people get started quickly.

You can always pass the method explicitly and any arguments supported by that method (see `?hipercow_provision` for details).  For example, suppose you normally had things working with `pkgdepends` but hit trouble with some incompatible set of versions, you might write a file `fix_install.R` and run it with:

```r
hipercow_provision(method = "script", script = "fix_install.R")
```

If you interrupt the installation while it runs, it will not (currently) cancel the running task. We may change this in the future.  Please don't submit another provisioning task until yours is finished (you can use the [HPC portal](https://mrcdata.dide.ic.ac.uk/hpc/index.php) to check easily).


