orderly

Introduction

orderly is a package designed to help make analysis more reproducible. Its principal aim is to automate a series of basic steps in the process of writing analyses, making it easy to:

• track all inputs into an analysis (packages, code, and data resources)
• store multiple versions of an analysis where it is repeated
• track outputs of an analysis
• create analyses that depend on the outputs of previous analyses

With orderly we have two main hopes:

• analysts can write code that will straightforwardly run on someone else’s machine (or a remote machine)
• when an analysis that is run several times starts behaving differently it will be easy to see when the outputs started changing, and what inputs started changing at the same time

orderly requires a few conventions around organisation of a project, and after that tries to keep out of your way. However, these requirements are designed to make collaborative development with git easier by minimising conflicts and making backup easier by using an append-only storage system.

Example

To illustrate, we will start with a minimal example (you can use orderly::orderly_init to create a similar structure directly), and we will build it up to demonstrate orderly features. In the most minimal example, we want to run a script that creates a graph. It uses no external resources.

.
├── orderly_config.yml
└── src
    └── example
        ├── orderly.yml
        └── script.R

In this example, the orderly_config.yml file is completely empty, but serves to mark the root of the orderly project. We have one report, called example, and its configuration is within orderly.yml:

script: script.R

artefacts:
  - staticgraph:
      description: A graph of things
      filenames: mygraph.png
  - data:
      description: Data that went into the plot
      filenames: mydata.csv

There are two keys here:

• script the path of the script to run, script.R
• artefacts a description of the artefacts (files) that will be produced by running this script. In this case it is a graph with the filename mygraph.png

The script is plain R code:

dat <- data.frame(x = 1:10, y = runif(10))
write.csv(dat, "mydata.csv", row.names = FALSE)

png("mygraph.png")
plot(dat)
dev.off()

The R code can be as long or as short as needed and can use whatever packages it needs. orderly does not do anything with the script apart from run it so it can be formatted freely (there are no magic comments, etc). There are no restrictions on what can be done except that it must produce the artefacts listed in orderly.yml. If not, an error will be thrown describing what was missing.

id <- orderly::orderly_run("example", root = path)

## [ info       ]  Writing initial orderly archive version as 1.1.25

## [ name       ]  example

## [ id         ]  20241018-053122-7baca55c

## [ start      ]  2024-10-18 05:31:22.485569

## 
## > dat <- data.frame(x = 1:10, y = runif(10))
## 
## > write.csv(dat, "mydata.csv", row.names = FALSE)
## 
## > png("mygraph.png")
## 
## > plot(dat)

## 
## > dev.off()
## png 
##   2

## [ end        ]  2024-10-18 05:31:22.501336

## [ elapsed    ]  Ran report in 0.01576686 secs

## [ artefact   ]  mygraph.png: 9004cd59ab3226abada205a967a4dc9c
## [ ...        ]  mydata.csv: 8354220c49f0bec2665684bc288f52ca

id

## [1] "20241018-053122-7baca55c"

.
├── archive
├── draft
│   └── example
│       └── 20241018-053122-7baca55c
│           ├── mydata.csv
│           ├── mygraph.png
│           ├── orderly.yml
│           ├── orderly_run.rds
│           └── script.R
├── orderly_config.yml
└── src
    └── example
        ├── orderly.yml
        └── script.R

orderly::orderly_list_drafts(root = path)

##      name                       id
## 1 example 20241018-053122-7baca55c

orderly::orderly_commit(id, root = path)
## [ commit     ]  example/20241018-053122-7baca55c
## [ copy       ]
## [ import     ]  example:20241018-053122-7baca55c
## [ success    ]  :)
## [1] "/tmp/RtmpTaFezJ/filee0f11a16851/archive/example/20241018-053122-7baca55c"

.
├── archive
│   └── example
│       └── 20241018-053122-7baca55c
│           ├── mydata.csv
│           ├── mygraph.png
│           ├── orderly.yml
│           ├── orderly_run.rds
│           └── script.R
├── draft
│   └── example
├── orderly.sqlite
├── orderly_config.yml
└── src
    └── example
        ├── orderly.yml
        └── script.R

Creating a report

First, run orderly::orderly_new to create a directory within src. The name is important and should not contain spaces (nor should it change as this will change the key report id and you’ll lose a chain of history), then edit the file orderly.yml within that directory.

orderly::orderly_new("new", root = path)

## Created report at '/tmp/RtmpTaFezJ/filee0f11a16851/src/new'

## Edit the file 'orderly.yml' within this directory

which results in a directory structure like:

.
├── archive
│   └── example
│       └── 20241018-053122-7baca55c
│           ├── mydata.csv
│           ├── mygraph.png
│           ├── orderly.yml
│           ├── orderly_run.rds
│           └── script.R
├── draft
│   └── example
├── orderly.sqlite
├── orderly_config.yml
└── src
    ├── example
    │   ├── orderly.yml
    │   └── script.R
    └── new
        └── orderly.yml

Resources, sources and artefacts

Resources to a report are expected to be read-only files that are used by the script to produce the report. Examples of the sort of files that should be used as resources are:

• Moderately sized data files (large datasets should be accessed from a database),
• A markdown file used to create a report,
• Documentation for the report.

“Resources” cannot be modified by the report; if orderly detects that a resource has been changed an error will be thrown.

orderly will automatically detect any files named README.md in a report’s source directory and copy them to the new directory too.

resources: 
  - years.csv
  - data_dictionary.xlsx
  - report.Rmd
  - code_documentation.md

“Sources” are files containing R code that will be sourced (via the R function source()) before the main script is run. Often this file contains functions or variables used by the main script. All of the copying and sourcing will be handled by orderly itself so there is no need to explicitly source the files in the main script.

“Artefacts” are the output of the report. At least one artefact must be listed and files created during the running of the script must be included as artefacts (or deleted before the script finishes) or an error will be returned.

Examples of artefacts fields in orderly.yml:

artefacts:
  - report:
      filenames: report.html
      description: a simple report

artefacts:
  - report:
      filenames: report.html
      description: a simple report
  - data:
      description:
        - associated data sets
      filenames:
        - data_one.csv
        - data_two.csv
        - data_three.csv
        - data_four.csv

When declaring an artefact we have to specify what format the artefact is. Currently supported formats are :data, report, staticgraph, interactivegraph and interactivehtml. These tags reflect the intent of use of the file, they have no special meaning within orderly itself.

Using artefacts from other reports

It is often the case that we would like to write a report that depends on an earlier report, e.g. one report produces a large dataset and a later report produces a high level summary. orderly allows a report to directly copy an artefact file from an existing report without having to manually copy it into the report source directory. This is handled in the depends block of the report’s orderly.yml.

To use a file as a dependency it must be explicitly listed as an artefact.

An simple example might look like:

depends:
  - big-data-report:
      id: 20190425-163691-b8451bbf
      use:
        data.rds: huge-data-set.rds

This will copy the file the huge-data-set.rds from the report big-data-report with id 20190425-163691-b8451bbf and rename it data.rds. This file can then be used by the report as if it were in the source directory.

If we want a report to always use the latest version of a report big-data-report we can set the id field to latest, e.g.:

depends:
  - big-data-report:
      id: latest
      use:
        data.rds: huge-data-set.rds

This will find the most recent version of the report big-data-report and copy files from that directory.

To use multiple artefacts from a single report add the files into the use block e.g.:

depends:
  - big-data-report:
      id: latest
      use:
        data.rds: huge-data-set.rds
        pop.csv: population_data.csv

To use artefacts from multiple reports we add multiple entries to the depends field e.g.:

depends:
  - big-data-report:
      id: latest
      use:
        data.rds: huge-data-set.rds
        pop.csv: population_data.csv
  - report_two:
      id: latest
      use:
        data_b.rds: filename.rds

We can also use the same artefact from different versions of the same report. This might come up if we want to write a report that compares the output from different versions of another report. The yaml pattern for this is:

depends:
  - big-data-report:
      id: 20190425-163691-b8451bbf
      use:
        data_latest.rds: huge-data-set.rds
  - big-data-report:
      id: 20181225-172991-34c91ef1
      use:
        data_old: huge-data-set.rds

The important feature in this example is the dashes before the report name. When all the report names are different these dashes can be omitted, but they are necessary when the report depends on different versions of the same report. Since including the dashes will never cause a problem but omitting them might, we advise that they should always be included.

Parameterised reports

Sometimes it can be useful to control how a report runs by a parameter. This could be the name of a country that an analysis applies to (though we hope to develop a better interface for this soon) through to controlling the number of iterations that an analysis runs for. Parameters are declared in the orderly.yml like:

parameters:
  a:
    default: 1
  b: ~

This would declare that a report takes two parameters a (with a default of 1), and b (with no default). Running the report would then look like:

orderly::orderly_run("reportname", list(a = 10, b = 100))

These parameters are then present in the environment of the report, so the code can use values a and b.

The parameters will also be interpolated into any SQL queries before they are run, so if the orderly.yml contains:

data:
  cars:
    query: SELECT * FROM mtcars WHERE cyl > ?a

then this will be evaluated on the SQL server with a substituted in where the query says ?a (this is done with DBI::sqlInterpolate).

Using global resources

There might be files that are used in (almost) every report. Examples of these sorts of files might be document templates or organisation logos. To set up a global resource create a directory your_global_dir in <root> and the following to the orderly_config.yml:

global_resources:
  your_global_dir

Then to use any file in your_global_dir in your report add a global_resources field to that report’s orderly.yml:

global_resources:
  logo.jpg: org_logo.jpg
  latex_class.cls: org_latex_class.cls
  styles.css: org_styles.css

Currently code i.e. R source code cannot be sourced from the global resources directory. So for example utility functions common across multiple reports must be included in each report directory separately. The functionality to include global source code may be added in future versions.

Using version control

orderly is designed to work well with git (or any other version control system). The general principle is that src/ and any configuration files should be added to git, while most of the generated files (data, draft, archive, and any SQLite databases such as orderly.sqlite) should be excluded, which can be done by creating a .gitignore file.

This process can be automated by running

orderly::orderly_use_gitignore(root = path, prompt = FALSE)

## Changes to '/tmp/RtmpTaFezJ/filee0f11a16851/.gitignore'

## +  1 | # These are directories where orderly stores (potentially large)
## +  2 | # generated files - you definitely do not want these in git.
## +  3 | data
## +  4 | archive
## +  5 | draft
## +  6 | metadata
## +  7 | 
## +  8 | # Orderly will store data in a sqlite database, by default orderly.sqlite
## +  9 | *.sqlite
## + 10 | 
## + 11 | # It is suggested to use environment variables, in either
## + 12 | # orderly_envir.yml or .Renviron to store sensitive data to use with
## + 13 | # orderly - keep those out of git
## + 14 | .Renviron
## + 15 | orderly_envir.yml
## + 16 | 
## + 17 | # Other good things to exclude
## + 18 | .Rhistory
## + 19 | .Rdata
## + 20 | .Rproj.user
## + 21 | 
## + 22 | # Created by orderly.server
## + 23 | runner
## + 24 | backup

## Writing to '/tmp/RtmpTaFezJ/filee0f11a16851/.gitignore'

(the default, prompt = TRUE will request user confirmation before writing the .gitignore file).

You should arrange to back up the entire orderly directory through some other means.

If your generated files are particularly small you might leave them in git, but in our experience this will result in a git repository that is unpleasant to use.

Using SQL databases

One of the original aims of orderly was to provide a set of tools for use of SQL databases within reproducible reporting. Because the SQL database is an external global resource it is difficult to work with any concept of “versioning” from R (there is no git history, no way of easily rolling back to previous versions etc). If using a central SQL server, there is configuration that should be kept out of any analysis, particularly things like passwords. Configuration problems multiply when using both “production” and “staging” systems as we would like to be able to switch between different configurations.

database:
  source:
    driver: RPostgres::Postgres
    args:
      host: dbhost.example.org
      port: 5432
      user: myusername
      password: s3cret
      dbname: mydb

DBI::dbConnect(RPostgres::Postgres, host = "dbhost.example.org", port = 5432,
               user = "myusername", password = "s3cret", dbname = "mydb")

data:
  cars:
    query: SELECT * FROM mtcars WHERE cyl = 4
    database: source

connection:
  con: source

database:
  source:
    driver: RPostgres::Postgres
    args:
      host: localhost
      port: 5432
      user: myuser
      dbname: databasename
      password: p4ssw0rd

database:
  source:
    driver: RPostgres::Postgres
    args:
      host: $MY_DBHOST
      port: $MY_DBPORT
      user: $MY_DBUSER
      dbname: $MY_DBNAME
      password: $MY_PASSWORD

MY_DBHOST: localhost
MY_DBPORT: 5432
MY_DBUSER: myuser
MY_DBNAME: databasename
MY_PASSWORD: p4ssw0rd

vault:
  addr: https://example.com:8200

      password: VAULT:/secret/users/database_user:password

vault:
  addr: https://example.com:8200
  login: userpass
  username: alice
  password: secret
  mount: userpass

vaultr::vault_client(addr = "https://example.com:8200", login = "userpass",
                     username = "alice", password = "secret",
                     mount = "userpass")

vault:
  addr: https://example.com:8200
  login: userpass
  username: $VAULT_USER
  password: $VAULT_PASSWORD

database:
  source:
    driver: RPostgres::Postgres
    args:
      port: 5432
      user: user
      dbname: mydb
    instances:
      production:
        host: production.example.org
        password: $PASSWORD_PRODUCTION
      staging:
        host: staging.example.org
        password: $PASSWORD_STAGING
    default_instance: $DEFAULT_INSTANCE

orderly::orderly_run(name, instance = "production")

orderly::orderly_run(name, instance = "staging")

database:
  source:
    driver: RPostgres::Postgres
    args:
      port: 5432
    instances:
      production:
        host: production.example.org
      staging:
        host: staging.example.org
    default_instance: staging
  extra:
    driver: RPostgres::Postgres
    args:
      port: 5432
      host: extra.example.org

orderly::orderly_run(name, instance = "production")

database:
  source:
    driver: RPostgres::Postgres
    args:
      port: 5432
    instances:
      production:
        host: production.example.org
      staging:
        host: staging.example.org
    default_instance: staging
  extra:
    driver: RPostgres::Postgres
    args:
      port: 15432
    instances:
      production:
        host: production.example.org
      staging:
        host: staging.example.org
    default_instance: staging

orderly::orderly_run(name,
                      instance = c(source = "production", extra = "staging"))

Using environment variables and secrets

Environment variables declared in orderly.yml are made available in the report script (since orderly 1.1.11). For example, if your orderly.yml contains

environment:
  external_file: EXTERNAL_FILE_PATH
  variable_2: ENV_VAR_2

Then orderly will evaluate the variables and make them available in your script via the specified name. For example in your script you can use

external_data <- read.csv(external_file)

The expected use case for this is if you have data which you want to use in a report but do not want to be in the orderly repository (either because it is sensitive or particularly large). You can have the external files somewhere else on your machine and specify the path to them via an environment variable so they can then be accessed from the report script.

Environment variables can also be used in top-level orderly_envir.yml (since orderly 1.1.6) which can be accessed via Sys.getenv(). For example, if your orderly_envir.yml contains

DATA_URL: https://www.example.com/thedata

Then in your script you can use

url <- Sys.getenv("DATA_URL")
download.file(url)

If the values are sensitive, then this is not ideal, as you will store your values in plain text in the orderly_envir.yml. Instead, if using vault, you can use a secrets: section in orderly.yml, like:

secrets:
  password: /secret/myproject/login:value

and then an R variable password will be available to all the code in your report, containing the result of looking up /secret/myproject/login in your vault and getting the value field.

Developing a report

Because orderly works in a directory that is not the same as the source directory (e.g., draft/myreport/YYYYMMDD-HHMMSS-abcdefgh), because global resources and dependencies etc are copied in just as it is used, and because orderly takes control of things like parameters and sourcing files, it may not seem straightforward to develop a report as you would ordinarily.

In order to make this easier, orderly has a set of functions to help develop a report within the source directory. These functions are:

• orderly::orderly_develop_start which copies files into your source copy
• orderly::orderly_develop_status which reports on the status of your source
• orderly::orderly_develop_clean which cleans up files that orderly copied

This section illustrates the idea, creating a new report that will depend on an artefact from a previous report. Here is the state of our orderly tree from before:

.
├── archive
│   └── example
│       └── 20241018-053122-7baca55c
│           ├── mydata.csv
│           ├── mygraph.png
│           ├── orderly.yml
│           ├── orderly_run.rds
│           └── script.R
├── draft
│   └── example
├── orderly.sqlite
├── orderly_config.yml
└── src
    ├── example
    │   ├── orderly.yml
    │   └── script.R
    └── new
        ├── orderly.yml
        └── script.R

The new report has an orderly.yml containing

script: script.R

artefacts:
  - staticgraph:
      description: Mean of the values
      filenames: mean.txt

depends:
  example:
    id: latest
    use:
      data.csv: mydata.csv

In order to run this report, we need the file data.csv (which contains the output mydata.csv from the latest version of the example report) to exist. If we wanted to interactively develop the script.R we’d have to run orderly::orderly_run repeatedly, which will be annoying - and impractical if the report is slow to run. So we can run instead:

orderly::orderly_develop_start("new", root = path)

## [ name       ]  new

## [ depends    ]  example@20241018-053122-7baca55c:mydata.csv -> data.csv

which copies in the required artefact and we can then change the directory (using something like setwd("src/new")) and start working on the report directly.

You can see the status of the directory by running

orderly::orderly_develop_status("new", root = path)

##      filename       type present derived
## 1 orderly.yml    orderly    TRUE   FALSE
## 2    script.R     script    TRUE   FALSE
## 3    data.csv dependency    TRUE    TRUE
## 4    mean.txt   artefact   FALSE    TRUE

(the output of this object is likely to change in future versions), which shows that data.csv is present, and that it is derived - by which means that orderly knows that it should not persist in the source tree. Files marked as derived as TRUE are liable for deletion by orderly::orderly_develop_clean. The output also shows that mean.txt is not present.

If you have changed directly into the path under development (via setwd(file.path(path, "src/new"))) the you can omit the arguments and simply call

orderly::orderly_develop_status()

##      filename       type present derived
## 1 orderly.yml    orderly    TRUE   FALSE
## 2    script.R     script    TRUE   FALSE
## 3    data.csv dependency    TRUE    TRUE
## 4    mean.txt   artefact   FALSE    TRUE

You can then run your script as if it was a normal R script:

source("script.R", echo = TRUE)

## 
## > data <- read.csv("data.csv")
## 
## > writeLines(as.character(mean(data$y)), "mean.txt")

(the above code assuming that we are within src/new with the report)

After which the data.csv is present

orderly::orderly_develop_status()

##      filename       type present derived
## 1 orderly.yml    orderly    TRUE   FALSE
## 2    script.R     script    TRUE   FALSE
## 3    data.csv dependency    TRUE    TRUE
## 4    mean.txt   artefact    TRUE    TRUE

If a newer version of the upstream dependency has become available, you can update the file by running orderly::orderly_develop_start() again

id <- orderly::orderly_run("example", root = path)

## [ name       ]  example

## [ id         ]  20241018-053122-e994b997

## [ start      ]  2024-10-18 05:31:22.916884

## 
## > dat <- data.frame(x = 1:10, y = runif(10))
## 
## > write.csv(dat, "mydata.csv", row.names = FALSE)
## 
## > png("mygraph.png")
## 
## > plot(dat)

## 
## > dev.off()
## png 
##   2

## [ end        ]  2024-10-18 05:31:22.93257

## [ elapsed    ]  Ran report in 0.01568556 secs

## [ artefact   ]  mygraph.png: b0e560cb1bce2a550c3a04712ddfea73
## [ ...        ]  mydata.csv: cf2749f51ddc331368c20bd9ed659916

orderly::orderly_commit(id, root = path)

## [ commit     ]  example/20241018-053122-e994b997

## [ copy       ]

## [ import     ]  example:20241018-053122-e994b997

## [ success    ]  :)

## [1] "/tmp/RtmpTaFezJ/filee0f11a16851/archive/example/20241018-053122-e994b997"

orderly::orderly_develop_start()

## [ name       ]  new

## [ depends    ]  example@20241018-053122-e994b997:mydata.csv -> data.csv

(note that this has updated data.csv to use this new id 20241018-053122-e994b997).

Finally, you can delete the files that orderly has copied into the source tree:

orderly::orderly_develop_clean()
## [ remove     ]  data.csv
## [ ...        ]  mean.txt
orderly::orderly_develop_status()
##      filename       type present derived
## 1 orderly.yml    orderly    TRUE   FALSE
## 2    script.R     script    TRUE   FALSE
## 3    data.csv dependency   FALSE    TRUE
## 4    mean.txt   artefact   FALSE    TRUE

(this function also accepts name and root as above, required if not working in the source directory)