So long YAML and thanks for all the whitespace errors

The most obvious user-facing change is that there is (almost) no YAML, with the definition of inputs and outputs for a report now defined within an orderly file, <reportname>.R. So an orderly report that previously had an orderly.yml file that looked like

parameters:
  n_min:
    default: 10
script: script.R
source:
  - functions.R
resources:
  - metadata.csv
depends:
  raw_data:
    id: latest
    use:
      raw_data.csv: data.csv
artefacts:
  data:
    description: Processed data
    filenames: data.rds

would end up within an orderly file that looks like:

orderly2::orderly_parameters(n_min = 10)
orderly2::orderly_dependency("raw_data", "latest", 
                             files = c("raw_data.csv" = "data.csv"))
orderly2::orderly_resource("metadata.csv")
orderly2::orderly_artefact("Processed data", "data.rds")
source("functions.R")

We think this is much clearer, and comes with documentation and autocomplete support in most IDEs.

In fact, for simple reports, no special functions are required, though you’ll find that some will be useful (see vignette("orderly2"))

Some specific changes:

• you no longer need to list packages with packages:, instead just use library() as in an ordinary script. We will record the state of the session regardless so you will get a record of what was used
• you no longer need to use sources: to list scripts you want to source, instead use source() as normal
• global_resources has become orderly2::orderly_shared_resource (these aren’t really global so much as shared). Note that the directory they are in is now always shared/ at the orderly root, you may not configure it.

Database support has been moved into a plugin

In version 1, we had built-in support for accessing data from SQL databases; this has moved within the orderly.db plugin. All major features are supported.

No more commit

orderly2 no longer requires a separate orderly_commit() call after orderly_run(); we no longer make a distinction between local draft and archive packets. Instead, we have added finer-grained control over where dependencies are resolved from (locally, or from some subset of your servers), which generalises the way that draft/archive was used in practice. See ?orderly_run for more details on how dependencies are resolved.

This has implications for deleting things; the draft directory was always an easy target for deletion, but now after deletion you will need to tell orderly2 that you have deleted things. See vignette("introduction") for details on this (section “Deleting things from the archive”).

No more testing or development mode

We have had two different, but unsatisfactory, mechanisms for developing an orderly report:

• orderly_test_start (up to version 1.1.0)
• orderly_develop_start (from 1.1.0 onwards)

These worked by doing the initial setup, and copying of dependencies etc to a location you could work (a new draft for orderly_test_start and the source directory for orderly_develop_start). From orderly2 you can just work directly within the source directory, and so long as your working directory is set to src/<report-name>, all orderly2 commands will work as expected.

As with orderly1, you will need to be careful not to commit (to git) results of running your analysis, and we encourage per-report .gitignore files to help with this.

New, language-agnostic, backend

The biggest change, but perhaps the least visible, is that orderly is now built on an open spec outpack which can be implemented for any language. We will develop a Python implementation of this, and possibly other languages.

This takes control of all the metadata. As such there is a split between “orderly_” and “outpack_” functions in this package, for more information see the last section of vignette("introduction") and also vignette("outpack").

Other, smaller changes

“Global” resources have become “shared” resources, and always live in shared/ at the orderly root (i.e., this is no longer configurable). The reason for this is that we want reports to be able to be run fairly independently of the orderly2 configuration (the only exception to this are plugins). In practice people did not really vary this.