Package 'hipercow'

Description

Cancel all tasks in a bundle. This wraps task_cancel for all the ids.

Usage

hipercow_bundle_cancel(bundle, follow = TRUE, root = NULL)

Arguments

Value

A logical vector the same length as id indicating if the task was cancelled. This will be FALSE if the job was already completed, not running, etc.

Examples

cleanup <- hipercow_example_helper(runner = FALSE)

bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
hipercow_bundle_cancel(bundle)
hipercow_bundle_status(bundle)

cleanup()

Description

Create a bundle of tasks. This is simply a collection of tasks that relate together in some way, and we provide some helper functions for working with them that save you writing lots of loops. Each bundle has a name, which will be randomly generated if you don't provide one, and a set of task ids.

Usage

hipercow_bundle_create(
  ids,
  name = NULL,
  validate = TRUE,
  overwrite = TRUE,
  root = NULL
)

Arguments

Value

A task bundle object

Examples

cleanup <- hipercow_example_helper()

# Two task that were created separately:
id1 <- task_create_expr(sqrt(1))
id2 <- task_create_expr(sqrt(2))

# Combine these tasks together in a bundle:
bundle <- hipercow_bundle_create(c(id1, id2))

# Now we can use bundle operations:
hipercow_bundle_status(bundle)
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle)

cleanup()

Description

Delete one or more hipercow task bundles. Note that this does not delete the underlying tasks, which is not yet supported.

Usage

hipercow_bundle_delete(name, root = NULL)

Arguments

Value

Nothing, called for its side effect

Examples

cleanup <- hipercow_example_helper()

bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
hipercow_bundle_list()

# Retaining the ids, delete bundle
ids <- bundle$ids
hipercow_bundle_delete(bundle$name)
hipercow_bundle_list()

# The tasks still exist:
task_status(ids)

cleanup()

Description

List existing bundles

Usage

hipercow_bundle_list(root = NULL)

Arguments

Value

A data.frame with columns name and time, ordered by time (most recent first)

Examples

cleanup <- hipercow_example_helper()

# With no bundles present
hipercow_bundle_list()

# With a bundle
bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
hipercow_bundle_list()

cleanup()

Description

Load an existing saved bundle by name. This is intended for where you have created a long-running bundle and since closed down your session. See hipercow_bundle_list for finding names of bundles.

Usage

hipercow_bundle_load(name, root = NULL)

Arguments

Value

A hipercow_bundle object

Examples

cleanup <- hipercow_example_helper()

bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
name <- bundle$name

# Delete the bundle object; the bundle exists still in hipercow's store.
rm(bundle)

# With the name we can load the bundle and fetch its status
bundle <- hipercow_bundle_load(name)
hipercow_bundle_status(bundle)

# In fact, you can use just the name if you prefer:
hipercow_bundle_status(name)

cleanup()

Description

Fetch logs from tasks in a bundle.

Usage

hipercow_bundle_log_value(bundle, outer = FALSE, follow = TRUE, root = NULL)

Arguments

Value

A list with each element being the logs for the corresponding element in the bundle.

Examples

cleanup <- hipercow_example_helper(with_logging = TRUE)
bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:2))
hipercow_bundle_wait(bundle)
hipercow_bundle_log_value(bundle)

cleanup()

Description

Fetch all bundle results

Usage

hipercow_bundle_result(bundle, follow = TRUE, root = NULL)

Arguments

Value

An unnamed list, with each element being the result for each a task in the bundle, in the same order.

Examples

cleanup <- hipercow_example_helper()
bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle)

cleanup()

Description

Retry tasks in a bundle. This has slightly different semantics to task_retry(), which errors if a retry is not possible. Here, we anticipate that much of the time you will be interested in retrying some fraction of your bundle and so don't need to wait until all tasks have finished in order to retry failed tasks.

Usage

hipercow_bundle_retry(bundle, if_status_in = NULL, driver = NULL, root = NULL)

Arguments

Value

Invisibly, a logical vector, indicating which of the tasks within the bundle were retried. This means that it's not immediately obvious how you can get the new id back from the tasks, but typically that is unimportant, as all bundle functions follow retries by default.

Examples

cleanup <- hipercow_example_helper()
bundle <- task_create_bulk_expr(rnorm(1, x), data.frame(x = 1:5))
hipercow_bundle_wait(bundle)

retried <- hipercow_bundle_retry(bundle)
retried
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle, follow = FALSE)
hipercow_bundle_result(bundle, follow = TRUE)

cleanup()

Description

Fetch status for all tasks in a bundle.

Usage

hipercow_bundle_status(bundle, reduce = FALSE, follow = TRUE, root = NULL)

Arguments

Value

A character vector the same length as the number of tasks in the bundle, or length 1 if reduce is TRUE.

Examples

cleanup <- hipercow_example_helper()
bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
# Immediately after submission, tasks may not all be complete so
# we may get a mix of statuses.  In that case the reduced status
# will be "submitted" or "running", even though some tasks may be
# "success"
hipercow_bundle_status(bundle)
hipercow_bundle_status(bundle, reduce = TRUE)

# After completion all tasks have status "success", as does the
# reduction.
hipercow_bundle_wait(bundle)
hipercow_bundle_status(bundle)
hipercow_bundle_status(bundle, reduce = TRUE)

cleanup()

Description

Wait for tasks in a bundle to complete. This is the generalisation of task_wait for a bundle.

Usage

hipercow_bundle_wait(
  bundle,
  timeout = NULL,
  poll = 1,
  fail_early = TRUE,
  progress = NULL,
  follow = TRUE,
  root = NULL
)

Arguments

Value

A scalar logical value; TRUE if all tasks complete successfully and FALSE otherwise

Examples

cleanup <- hipercow_example_helper()

bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5))
hipercow_bundle_wait(bundle)
hipercow_bundle_status(bundle)

cleanup()

Description

Describe information about the cluster. This is (naturally) very dependent on the cluster but some details of the value are reliable; see Value for details.

Usage

hipercow_cluster_info(driver = NULL, root = NULL)

Arguments

Value

A list describing the cluster. The details depend on the driver, and are subject to change. We expect to see elements:

• resources: Describes the computational resources on the cluster, which is used by hipercow_resources_validate. Currently this is a simple list with elements max_ram (max RAM available, in GB), max_cores (max number of cores you can request), queues (character vector of available queues), nodes (character vector of available nodes), default_queue (the default queue). These details are subject to change but the contents should always be informative and fairly self explanatory.

• redis_url: The URL of the redis server to communicate with from outside of the cluster (i.e., from your computer), in a form suitable for use with redux::hiredis

• r_versions: A vector of R versions, as numeric_vector objects

Examples

cleanup <- hipercow_example_helper()
hipercow_cluster_info()
cleanup()

Description

Report on your hipercow configuration. We will always want you to post this along side any problems; it has lots of useful information in it that will help us see how your set up is configured.

Usage

hipercow_configuration(show = TRUE, root = NULL)

Arguments

Value

A list with a machine readable form of this information, invisibly.

Examples

cleanup <- hipercow_example_helper()
hipercow_configuration()

# If you have saved additional environments, they will be listed here:
file.create("functions.R")
hipercow_environment_create(
  name = "other",
  packages = "knitr",
  sources = "functions.R")
hipercow_configuration()

cleanup()

Description

Configure your hipercow root. hipercow_configure creates the configuration and hipercow_configuration looks it up

Usage

hipercow_configure(driver, ..., root = NULL)

Arguments

Windows

Options supported by the windows driver:

• shares: Information about shares (additional to the one mounted as your working directory) that should be made available to the cluster job. The use case here is where you need access to some files that are present on a shared drive and you will access these by absolute path (say ⁠M:/gis/shapefiles/⁠) from your tasks. You can provide a share as a windows_path object, or a list of such objects. You will not typically need to use this option.

• r_version: Control the R version used on the cluster. Typically hipercow will choose a version close to the one you are using to submit jobs, of the set available on the cluster. You can use this option to choose a specific version (e.g., pass "4.3.0" to select exactly that version).

See vignette("details") for more information about these options.

See Also

hipercow_unconfigure, which removes a driver

Examples

hipercow_configure("windows", r_version = "4.3.0")

Description

Create a new hipercow driver; this is intended to be used from other packages, and rarely called directly. If you are trying to run tasks on a cluster you do not need to call this!

Usage

hipercow_driver(
  configure,
  submit,
  status,
  info,
  log,
  result,
  cancel,
  provision_run,
  provision_list,
  provision_compare,
  keypair,
  check_hello,
  cluster_info,
  default_envvars = NULL
)

Arguments

Description

Create, update, list, view and delete environments.

Usage

hipercow_environment_create(
  name = "default",
  packages = NULL,
  sources = NULL,
  globals = NULL,
  overwrite = TRUE,
  check = TRUE,
  root = NULL
)

hipercow_environment_list(root = NULL)

hipercow_environment_delete(name = "default", root = NULL)

hipercow_environment_show(name = "default", root = NULL)

hipercow_environment_exists(name = "default", root = NULL)

Arguments

Value

Nothing, all are called for their side effects.

Examples

cleanup <- hipercow_example_helper()

# Suppose you have a file with some functions you want to use in
# your task:
writeLines("simulation <- function(n) cumsum(rnorm(n))", "myfuns.R")

# Update the default environment to include these functions (or in
# this example, just this one function)
hipercow_environment_create(sources = "myfuns.R")

# You can now use this function in your tasks:
id <- task_create_expr(simulation(5))
task_wait(id)
task_result(id)

cleanup()

Description

Create environment variables for use with a hipercow task.

Usage

hipercow_envvars(..., secret = FALSE)

Arguments

Value

A list with class hipercow_envvars which should not be modified.

Examples

# Declare environment variables as key-value pairs:
hipercow_envvars("MY_ENVVAR1" = "value1", "MY_ENVVAR2" = "value2")

# If an environment variable already exists in your environment
# and you want to duplicate this into a task, you can use this
# shorthand:
Sys.setenv(HIPERCOW_EXAMPLE_ENVVAR = "moo") # suppose this exists already
hipercow_envvars("HIPERCOW_EXAMPLE_ENVVAR")
hipercow_envvars("HIPERCOW_EXAMPLE_ENVVAR", ANOTHER_ENVVAR = "value")

# Secret envvars are still printed (at the moment at least) but
# once passed into a task they will be encrypted at rest.
hipercow_envvars("MY_SECRET" = "password", secret = TRUE)

# Secret and public environment variables should be created
# separately and concatenated together:
env_public <- hipercow_envvars("HIPERCOW_EXAMPLE_ENVVAR")
env_secret <- hipercow_envvars("MY_PASSWORD" = "secret", secret = TRUE)
c(env_public, env_secret)

# Cleanup
Sys.unsetenv("HIPERCOW_EXAMPLE_ENVVAR")

Description

Hello world in hipercow. This function sends a tiny test task through the whole system to confirm that everything is configured correctly.

Usage

hipercow_hello(progress = NULL, timeout = NULL, driver = NULL)

Arguments

Value

The string "Moo", direct from your cluster.

Examples

cleanup <- hipercow_example_helper()
hipercow_hello()

cleanup()

Description

Create a hipercow root. This marks the directory where your task information will be saved, along with a local copy of your R packages (a "library" for the cluster). Immediately after running this the first time, you probably want to run hipercow_configure() in order to control how we set up your projects network paths and R version.

Usage

hipercow_init(root = ".", driver = NULL, ...)

Arguments

Value

Invisibly, the root object

Examples

# Create an empty root
path <- withr::local_tempfile()
hipercow_init(path)

Description

Set parallel options. Having requested more than one core using hipercow_resources, here hipercow can start up a local cluster on the node you are running on, using either the future or parallel package.

Usage

hipercow_parallel(
  method = NULL,
  cores_per_process = 1L,
  environment = NULL,
  use_rrq = FALSE
)

Arguments

Details

Here, hipercow automatically does some setup work for the supported methods, to initialise a local cluster of processes that can be used with future_map or clusterApply, depending on your method.

By default, hipercow initialises a cluster with the same number of processes as the number of cores you requested using hipercow_resources. Each process here would be use a single core.

You can also call hipercow_parallel with cores_per_process, to make hipercow launch as many processes as it can with each process having the number of cores you request, with the total cores being at most what you requested with hipercow_resources.

For example, you could request 32 cores with hipercow_resources, and then call hipercow_parallel with cores_per_process = 4, and hipercow will create a local cluster with 8 processes, each of which reporting 4 cores if that process calls hipercow_parallel_get_cores.

If you did the same with cores_per_process = 5, hipercow would create 6 local processes, each reporting 5 cores, and two cores would be effectively unallocated.

Here are some brief examples; see vignette("parallel") for more details. In each example, we are looking up the process id (to show that different processes are being launched), and asking each process how many cores it should be using.

For using the future package:

resources <- hipercow_resources(cores = 4)
id <- task_create_expr(
  furrr::future_map(1:4,
    ~c(Sys.getpid(), hipercow_parallel_get_cores()),
  parallel = hipercow_parallel("future"),
  resources = resources)

where furrr must be provisioned using hipercow_provision. Here is an equivalent example with parallel:

resources <- hipercow_resources(cores = 4)
id <- task_create_expr(
  parallel::clusterApply(NULL, 1:4, function(x)
    c(Sys.getpid(), hipercow_parallel_get_cores()),
  parallel = hipercow_parallel("parallel"),
  resources = resources)

Value

A list containing your parallel configuration.

Description

Lookup number of cores allocated to the task

Usage

hipercow_parallel_get_cores()

Value

The number of cores a cluster has allocated to your task. This will be less than or equal to the number of cores on the cluster node running your task.

Description

Sets the environment variables MC_CORES, OMP_NUM_THREADS, OMP_THREAD_LIMIT, R_DATATABLE_NUM_THREADS and HIPERCOW_CORES to the given number of cores. This is used to help various thread-capable packages use the correct number of cores. You can also call it yourself if you know specifically how many cores you want to be available to code that looks up these environment variables.

Usage

hipercow_parallel_set_cores(cores, envir = NULL)

Arguments

Description

Provision a library. This runs a small task on the cluster to set up your packages. If you have changed your R version you will need to rerun this. See vignette("packages") for much more on this process.

Usage

hipercow_provision(
  method = NULL,
  ...,
  driver = NULL,
  environment = "default",
  check_running_tasks = TRUE,
  root = NULL
)

Arguments

Details

Our hope is that that most of the time you will not need to pass any options through ..., and that most of the time hipercow will do the right thing. Please let us know if that is not the case and you're having to routinely add arguments here.

Value

Nothing

Manually adding packages to an installation

One case where we do expect that you will pass options through to hipercow_provision is where you are manually adding packages to an existing library. The usage here will typically look like:

hipercow_provision("pkgdepends", refs = c("pkg1", "pkg2"))

where pkg1 and pkg2 are names of packages or pkgdepends references (e.g., username/repo for a GitHub package; see vignette("packages") for details).

Supported methods and options

There are four possible methods: pkgdepends, auto, script and renv.

The canonical source of documentation for all of these approaches is conan2::conan_configure.

pkgdepends

The simplest method to understand, and probably most similar to the approach in didehpc. This method installs packages from a list in pkgdepends.txt in your hipercow root, or via a vector of provided package references. Uses pkgdepends for the actual dependency resolution and installation.

Supported options (passed via ...)

• refs: A character vector of package references to override pkgdepends.txt

• policy: the policy argument to pkgdepends::new_pkg_installation_proposal (accepts lazy and upgrade)

auto

Uses pkgdepends internally but tries to do everything automatically based on your declared environments (see hipercow_environment_create and vignette("hipercow")) and the installation information recorded in the locally installed versions of the required packages.

This is experimental and we'd love to know how it works for you.

No options are supported, the idea is it's automatic :)

script

Runs a script (by default provision.R) on the cluster to install things however you want. Very flexible but you're on your own mostly. The intended use case of this option is where pkgdepends fails to resolve your dependencies properly and you need to install things manually. The remotes package will be pre-installed for you to use within your script.

Your script will run on a special build queue, which will run even when the cluster is very busy. However, this is restricted in other ways, allowing a maximum of 30 minutes and disallowing parallel running.

Supports one option:

• script: The path for the script to run, defaulting to provision.R

renv

Uses renv to recreate your renv environment. You must be using renv locally for this to work, and at present your renv project root must be the same as your hipercow root.

No options are currently supported, but we may pass some renv options in the future; if you need more flexibility here please let us know.

Examples

cleanup <- hipercow_example_helper()
writeLines(c("knitr", "data.table"), "pkgdepends.txt")
hipercow_provision()
hipercow_provision_list()

cleanup()

Description

Compare installations performed into your libraries by conan.

Usage

hipercow_provision_compare(curr = 0, prev = -1, driver = NULL, root = NULL)

Arguments

Value

An object of class conan_compare, which can be printed nicely.

Examples

cleanup <- hipercow_example_helper()
hipercow_provision("pkgdepends", refs = "knitr")
hipercow_provision("pkgdepends", refs = "data.table")
hipercow_provision_compare()

cleanup()

Description

List previous successful installations of this hipercow root.

Usage

hipercow_provision_list(driver = NULL, root = NULL)

hipercow_provision_check(
  method = NULL,
  ...,
  driver = NULL,
  environment = "default",
  root = NULL
)

Arguments

Value

A data.frame with columns:

• name: the name of the installation. This might be useful with conan_compare

• time: the time the installation was started

• hash: the installation hash

• method: the method used for the installation

• args: the arguments to the installation (as a list column)

• current: if using hipercow_provision_check, does this installation match the arguments provided?

This object also has class conan_list so that it prints nicely, but you can drop this with as.data.frame.

Examples

cleanup <- hipercow_example_helper()
writeLines("data.table", "pkgdepends.txt")

# Before any installation has happened:
hipercow_provision_list()
hipercow_provision_check()

# After installation:
hipercow_provision()
hipercow_provision_list()
hipercow_provision_check()

# After a different installation:
hipercow_provision("pkgdepends", refs = "knitr")
hipercow_provision_list()
hipercow_provision_check()

cleanup()

Description

Purge (delete) hipercow tasks. This is a destructive operation that cannot be undone and can have unintended consequences! However, if you are running short of space and don't want to just delete everything and start again (which is our general recommendation), this function provides a mechanism for cleaning up tasks that you no longer need.

Usage

hipercow_purge(
  task_ids = NULL,
  finished_before = NULL,
  in_bundle = NULL,
  with_status = NULL,
  dry_run = FALSE,
  root = NULL
)

Arguments

Details

Most of the arguments describe filters over your tasks. We delete the intersection of these filters (not the union), and you must provide at least one filter. So to delete all tasks that were created more than a week ago you could write:

hipercow_purge(created_before = as.difftime(1, units = "weeks"))

but to restrict this to only tasks that have also failed you could write

hipercow_purge(created_before = "1 week", with_status = "failed")

Value

A character vector of deleted identifiers, invisibly.

Consequences of deletion

A non-exhaustive list:

• If you delete a task that is part of a task_retry chain, then all tasks (both upstream and downstream in that chain) are deleted

• Once we support task dependencies (mrc-4797), deleting tasks will mark any not-yet-run dependent task as impossible, or perhaps delete it too, or prevent you from deleting the task; we've not decided yet

• You may have a bundle that references a task that you delete, in which case the bundle will not behave as expected. As a result we delete all bundles that reference a deleted task

• Deleted bundles or deleted tasks that you hold identifiers to before deletion will not behave as expected, with tasks reported missing. Restarting your session is probably the safest thing to do after purging.

• We can't prevent race conditions, so if you are purging tasks at the same time you are also retrying tasks that you will purge, you'll create tasks that we might not want to allow, and these tasks will fail in peculiar ways.

Examples

cleanup <- hipercow_example_helper()

# Here are some tasks that have finished running:
bundle <- task_create_bulk_expr(sqrt(x), data.frame(x = 1:5),
                                bundle_name = "mybundle")
hipercow_bundle_wait(bundle)

# Purge all tasks contained in any bundle starting with "my":
hipercow_purge(in_bundle = "my*")

cleanup()

Description

Specify what resources a task requires to run. This creates a validated list of resources that can be passed in as the resources argument to task_create_expr or other task creation functions.

Usage

hipercow_resources(
  cores = 1L,
  exclusive = FALSE,
  max_runtime = NULL,
  hold_until = NULL,
  memory_per_node = NULL,
  memory_per_process = NULL,
  requested_nodes = NULL,
  priority = NULL,
  queue = NULL
)

Arguments

Value

If the function succeeds, it returns a hipercow_resources list of parameters which is syntactically valid, although not yet validated against a particular driver to see if the resources can be satisfied. If the function fails, it will return information about why the arguments could not be validated. Do not modify the return value.

Windows cluster (wpia-hn)

• Cores at present must be between 1 and 32

• Memory per node (or per task) can be 512Gb at most.

• The available queues are AllNodes and Training

• The node names are between wpia-001 and wpia-070, excluding 41, 42, 49 and 50.

Linux cluster (hermod)

Coming Soon.

Examples

# The default set of resources
hipercow_resources()

# A more complex case:
hipercow_resources(
  cores = 32,
  exclusive = TRUE,
  priority = "low")

# (remember that in order to change resources you would pass the
# return value here into the "resources" argument of
# task_create_expr() or similar)

Description

Query a driver to find information about the cluster, and then validate a hipercow_resources list against that driver to see if the resources requested could be satisfied.

Usage

hipercow_resources_validate(resources, driver = NULL, root = NULL)

Arguments

Value

TRUE if the resources are compatible with this driver.

Examples

cleanup <- hipercow_example_helper()
hipercow_resources_validate(hipercow_resources(cores = 1))

# This example does not allow more than one core
tryCatch(
  hipercow_resources_validate(hipercow_resources(cores = 32)),
  error = identity)

cleanup()

Description

Create an rrq controller for your queue, and set it as the default controller. Use this to interact with workers created with hipercow_rrq_workers_submit(). Proper docs forthcoming, all interfaces are subject to some change.

Usage

hipercow_rrq_controller(
  ...,
  set_as_default = TRUE,
  driver = NULL,
  queue_id = NULL,
  root = NULL
)

Arguments

Value

An rrq::rrq_controller object.

Description

Submit workers to the cluster, use this in conjunction with hipercow_rrq_controller. A worker may sit on a single core or a whole node depending on how you set up resources. We use the rrq environment if it exists (hipercow_environment_create) otherwise we'll use the default environment.

Usage

hipercow_rrq_workers_submit(
  n,
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  timeout = NULL,
  progress = NULL,
  root = NULL
)

Arguments

Value

A data.frame with information about the launch, with columns:

• queue_id: the rrq queue id (same for all workers)

• worker_id: the rrq worker identifier

• task_id: the hipercow task identifier

• bundle_name: the hipercow bundle name (same for all workers)

Description

Remove a driver configured by hipercow_configure. This will not affect tasks already submitted with this driver, but will prevent any future tasks being submitted with it.

Usage

hipercow_unconfigure(driver, root = NULL)

Arguments

Value

Nothing, called for its side effects only.

See Also

hipercow_configuration, which shows currently enabled drivers.

Description

Cancel one or more tasks

Usage

task_cancel(id, follow = TRUE, root = NULL)

Arguments

Value

A logical vector the same length as id indicating if the task was cancelled. This will be FALSE if the task was already completed, not running, etc.

Examples

cleanup <- hipercow_example_helper()

ids <- c(task_create_expr(Sys.sleep(2)), task_create_expr(runif(1)))

# The first task may or not be cancelled (depends on if it was
# started already) but the second one will almost certainly be
# cancelled:
task_cancel(ids)

cleanup()

Description

Create a bulk set of tasks based on applying a function over a vector or data.frame. This is the bulk equivalent of task_create_call, in the same way that task_create_bulk_expr is a bulk version of task_create_expr.

Usage

task_create_bulk_call(
  fn,
  data,
  args = NULL,
  environment = "default",
  bundle_name = NULL,
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Value

A hipercow_bundle object, which groups together tasks, and for which you can use a set of grouped functions to get status (hipercow_bundle_status), results (hipercow_bundle_result) etc.

Examples

cleanup <- hipercow_example_helper()

# The simplest way to use this function is like lapply:
x <- runif(5)
bundle <- task_create_bulk_call(sqrt, x)
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle) # lapply(x, sqrt)

# You can pass additional arguments in via 'args':
x <- runif(5)
bundle <- task_create_bulk_call(log, x, list(base = 3))
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle) # lapply(x, log, base = 3)

# Passing in a data.frame acts like Map (though with all arguments named)
x <- data.frame(a = runif(5), b = rpois(5, 10))
bundle <- task_create_bulk_call(function(a, b) sum(rnorm(b)) / a, x)
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle) # Map(f, x$a, x$b)

cleanup()

Description

Create a bulk set of tasks. This is an experimental interface and does not have an analogue within didehpc. Variables in data take precedence over variables in the environment in which expr was created. There is no "pronoun" support yet (see rlang docs). Use ⁠!!⁠ to pull a variable from the environment if you need to, but be careful not to inject something really large (e.g., any vector really) or you'll end up with a revolting expression and poor backtraces. We will likely change some of these semantics later, be careful.

Usage

task_create_bulk_expr(
  expr,
  data,
  environment = "default",
  bundle_name = NULL,
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Value

A hipercow_bundle object, which groups together tasks, and for which you can use a set of grouped functions to get status (hipercow_bundle_status), results (hipercow_bundle_result) etc.

See Also

hipercow_bundle_wait, hipercow_bundle_result for working with bundles of tasks

Examples

cleanup <- hipercow_example_helper()

# Suppose we have a data.frame:
d <- data.frame(a = 1:5, b = runif(5))

# We can create a "bundle" by applying an expression involving "a"
# and "b":
bundle <- task_create_bulk_expr(sqrt(a * b), d)

# Once you have your bundle, interact with it using the bundle
# analogues of the usual task functions:
hipercow_bundle_wait(bundle)
hipercow_bundle_result(bundle)

cleanup()

Description

Create a task based on a function call. This is fairly similar to callr::r, and forms the basis of lapply()-like task submission. Sending a call may have slightly different semantics than you expect if you send a closure (a function that binds data), and we may change behaviour here until we find a happy set of compromises. See Details for more on this. The expression task_create_call(f, list(a, b, c)) is similar to task_create_expr(f(a, b, c)), use whichever you prefer.

Usage

task_create_call(
  fn,
  args,
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Details

Things are pretty unambiguous when you pass in a function from a package, especially when you refer to that package with its namespace (e.g. pkg::fn).

If you pass in the name without a namespace from a package that you have loaded with library() locally but you have not loaded with library within your hipercow environment, we may not do the right thing and you may see your task fail, or find a different function with the same name. We may change the semantics here in a future version to attach your package immediately before running the task.

If you pass in an anonymous function (e.g., function(x) x + 1) we may or may not do the right thing with respect to environment capture. We never capture the global environment so if your function is a closure that tries to bind a symbol from the global environment it will not work. Like with callr::r, anonymous functions will be easiest to think about where they are fully self contained (i.e., all inputs to the functions come through args). If you have bound a local environment, we may do slightly better, but semantics here are undefined and subject to change.

R does some fancy things with function calls that we don't try to replicate. In particular you may have noticed that this works:

c <- "x"
c(c, c) # a vector of two "x"'s

You can end up in this situation locally with:

f <- function(x) x + 1
local({
  f <- 1
  f(f) # 2
})

this is because when R looks for the symbol for the call it skips over non-function objects. We don't reconstruct environment chains in exactly the same way as you would have locally so this is not possible.

Value

A task id, a string of hex characters. Use this to interact with the task.

Examples

cleanup <- hipercow_example_helper()

# Similar to the example in task_create_call
id <- task_create_call(stats::runif, list(5))
task_info(id)
task_wait(id)
task_result(id)

# Unlike task_create_explicit, variables are automatically included:
id <- task_create_call(function(x, y) x + y, list(2, 5))
task_info(id)
task_wait(id)
task_result(id)

cleanup()

Description

Create an explicit task. Explicit tasks are the simplest sort of task in hipercow and do nothing magic. They accept an R expression (from quote or friends) and possibly a set of variables to export from the global environment. This can then be run on a cluster by loading your variables and running your expression. If your expression depends on packages being attached then you should pass a vector of package names too. This function may disappear, and is used by us to think about the package, it's not designed to really be used.

Usage

task_create_explicit(
  expr,
  export = NULL,
  envir = parent.frame(),
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Value

A task id, a string of hex characters. Use this to interact with the task.

Examples

cleanup <- hipercow_example_helper()

# About the most simple task that can be created:
id <- task_create_explicit(quote(sqrt(2)))
task_wait(id)
task_result(id)

# Variables are not automatically included with the expression:
a <- 5
id <- task_create_explicit(quote(sqrt(a)))
task_info(id)
task_wait(id)
task_result(id)

# Include variables by passing them via 'export':
id <- task_create_explicit(quote(sqrt(a)), export = "a")
task_info(id)
task_wait(id)
task_result(id)

cleanup()

Description

Create a task based on an expression. This is similar to task_create_explicit except more magic, and is closer to the interface that we expect people will use.

Usage

task_create_expr(
  expr,
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Details

The expression passed as expr will typically be a function call (e.g., f(x)). We will analyse the expression and find all variables that you reference (in the case of f(x) this is x) and combine this with the function name to run on the cluster. If x cannot be found in your calling environment we will error; this behaviour is subject to change so let us know if you have other thoughts.

Alternatively you may provide a multiline statement by using {} to surround multiple lines, such as:

task_create_expr({
  x <- runif(1)
  f(x)
}, ...)

in this case, we apply a simple heuristic to work out that x is locally assigned and should not be saved with the expression.

If you reference values that require a lot of memory, hipercow will error and refuse to save the task. This is to prevent you accidentally including values that you will make available through an environment, and to prevent making the hipercow directory excessively large. Docs on controlling this process are still to be written.

Value

A task id, a string of hex characters. Use this to interact with the task.

Examples

cleanup <- hipercow_example_helper()

# Similar to task_create_explicit, but we don't include the 'quote'
id <- task_create_expr(runif(5))
task_wait(id)
task_result(id)

# Unlike task_create_explicit, variables are automatically included:
n <- 3
id <- task_create_expr(runif(n))
task_info(id)
task_wait(id)
task_result(id)

cleanup()

Description

Create a task from a script. This will arrange to run the file script via hipercow. The script must exist within your hipercow root, but you may change to the directory of the script as it executes (otherwise we will evaluate from your current directory relative to the hipercow root, as usual).

Usage

task_create_script(
  script,
  chdir = FALSE,
  echo = TRUE,
  environment = "default",
  driver = NULL,
  resources = NULL,
  envvars = NULL,
  parallel = NULL,
  root = NULL
)

Arguments

Value

A task id, a string of hex characters. Use this to interact with the task.

Examples

cleanup <- hipercow_example_helper()

# Create a small script; this would usually be several lines of
# course.  The script will need to do something as a side effect
# to be worth calling, so here we write a file.
writeLines("saveRDS(mtcars, 'data.rds')", "script.R")

# Now create a task from this script
id <- task_create_script("script.R")
task_info(id)
task_wait(id)
task_result(id)
dir()

cleanup()

Description

Run a task that has been created by a ⁠task_create_*⁠ function, e.g., task_create_explicit(), task_create_expr(). Generally users should not run this function directly.

Usage

task_eval(id, envir = .GlobalEnv, verbose = FALSE, root = NULL)

Arguments

Value

Logical indicating success (TRUE) or failure (FALSE)

Examples

cleanup <- hipercow_example_helper(runner = FALSE)
id <- task_create_expr(runif(1), driver = FALSE)
# Status is only 'created', not 'submitted', as we did not submit
# task.  This task can never run.
task_status(id)

# Explicitly evaluate the task:
task_eval(id, verbose = TRUE)
task_result(id)

cleanup()

Description

Fetch information about a task. This is much more detailed than the information in task_status. If a task is running we also fetch the true status via its driver, which can be slower.

Usage

task_info(id, follow = TRUE, root = NULL)

Arguments

Value

An object of class hipercow_task_info, which will print nicely. This is just a list with elements:

• id: the task identifier

• status: the retrieved status

• driver: the driver used to run the task (or NA)

• data: the task data (depends on the type of task)

• times: a vector of times

• retry_chain: the retry chain (or NULL)

You can see and access these elements more easily by running unclass() on the result of task_info().

Examples

cleanup <- hipercow_example_helper()
id <- task_create_expr(runif(1))
task_wait(id)

# Task information at completion includes times:
task_info(id)

# If you need to work with these times, use the "times" element:
task_info(id)$times

# If a task is retried, this information appears as a retry chain:
id2 <- task_retry(id)
task_info(id2, follow = FALSE)
task_info(id2)

cleanup()

Description

Get the task log, if the task has produced one. Tasks run by the windows driver will generally produce a log. A log might be quite long, and you might want to print it to screen in its entirety (task_log_show), or return it as character vector (task_log_value).

Usage

task_log_show(id, outer = FALSE, follow = TRUE, root = NULL)

task_log_value(id, outer = FALSE, follow = TRUE, root = NULL)

task_log_watch(
  id,
  poll = 1,
  skip = 0,
  timeout = NULL,
  progress = NULL,
  follow = TRUE,
  root = NULL
)

Arguments

Details

The function task_log_watch has similar semantics to task_wait but does not error on timeout, and always displays a log.

Value

Depending on the function:

• task_log_show returns the log value contents invisibly, but primarily displays the log contents on the console as a side effect

• task_log_value returns a character of log contents

• task_log_watch returns the status converted to logical (as for task_wait)

Examples

cleanup <- hipercow_example_helper(with_logging = TRUE)

# Tasks that don't produce any output (print, cat, warning, etc)
# will only contain logging information from hipercow itself
id <- task_create_expr(runif(1))
task_wait(id)
task_log_show(id)

# If your task creates output then it will appear within the
# horizontal rules:
id <- task_create_expr({
  message("Starting analysis")
  x <- mean(runif(100))
  message("all done!")
  x
})
task_wait(id)
task_log_show(id)

# Use "task_log_value" to get the log value as a character vector
task_log_value(id)

# Depending on the driver you are using, there may be useful
# information in the "outer" log; the logs produced by the
# submission system before hipercow takes over:
task_log_show(id, outer = TRUE)

cleanup()

Description

Get the task result. This might be an error if the task has failed.

Usage

task_result(id, follow = TRUE, root = NULL)

Arguments

Value

The value of the queued expression

Examples

cleanup <- hipercow_example_helper()

# Typical usage
id <- task_create_expr(runif(1))
task_wait(id)
task_result(id)

# Tasks that error return error values as results
id <- task_create_expr(readRDS("nosuchfile.rds"))
task_wait(id)
task_result(id)

cleanup()

Description

Retry one or more tasks. This creates a new task that copies the work of the old one. Most of the time this is transparent. We'll document this in the "advanced" vignette once it's written.

Usage

task_retry(id, driver = NULL, resources = NULL, root = NULL)

Arguments

Details

This ends up being a little more complicated than ideal in order to keep things relatively fast, while keeping our usual guarantees about race conditions etc. Basically; retrying is the only way a task can move out of a terminal state but it still does not modify the existing task. Instead, we keep a separate register of whether a task has been retried or not. Each time we retry we write into this register. When you query about the status etc of a task you can then add a follow argument to control whether or not we check the register. We assume that you never call this in parallel; if you do then retries may be lost. You can run task_retry(NULL) to refresh the cached copy of the retry map if you need to.

Value

New identifiers for the retried tasks

Examples

cleanup <- hipercow_example_helper()

# For demonstration, we just generate random numbers as then it's
# more obvious that things have been rerun:
id1 <- task_create_expr(runif(1))
task_wait(id1)
task_result(id1)

# Now retry the task and get the retried result:
id2 <- task_retry(id1)
task_wait(id2)
task_result(id2)

# After a retry, both the original and derived tasks know about
# each other:
task_info(id1)
task_info(id2)

# By default every task will "follow" and access the most recent
# task in the chain:
task_result(id1) == task_result(id2)

# You can prevent this by passing follow = FALSE to get the value
# of this particular attempt:
task_result(id1, follow = FALSE)

# Tasks can be retried as many times as needed, creating a
# chain. It does not matter which task you retry as we always
# follow all the way to the end of the chain before retrying:
id3 <- task_retry(id1)
task_info(id1, follow = FALSE)
task_info(id3)

cleanup()

Description

Get the status of a task. See Details for the lifecycle.

Usage

task_status(id, follow = TRUE, root = NULL)

Arguments

Details

A task passes through a lifecycle:

• created

• submitted

• running

• success, failure, cancelled

These occur in increasing order and the result of this function is the furthest through this list.

Later, we will introduce other types to cope with tasks that are blocked on dependencies (or have become impossible due to failed dependencies).

Value

A string with the task status. Tasks that do not exist will have a status of NA.

Examples

cleanup <- hipercow_example_helper()

ids <- c(task_create_expr(runif(1)), task_create_expr(runif(1)))
# Depending on how fast these tasks get picked up they will be one
# of 'submitted', 'running' or 'success':
task_status(ids)

# Wait until both tasks are complete
task_wait(ids[[1]])
task_wait(ids[[2]])
# And both are success now
task_status(ids)

cleanup()

Description

Submit a task to a queue. This is a lower-level function that you will not often need to call. Typically a task will be submitted automatically to your driver on creation (e.g., with task_create_expr()), unless you specified driver = FALSE or you had not yet configured a driver.

Usage

task_submit(id, ..., resources = NULL, driver = NULL, root = NULL)

Arguments

Description

Wait for a single task to complete (or to start). This function is very similar to task_log_watch, except that it errors if the task does not complete (so that it can be used easily to ensure a task has completed) and does not return any logs.

Usage

task_wait(
  id,
  for_start = FALSE,
  timeout = NULL,
  poll = 1,
  progress = NULL,
  follow = TRUE,
  root = NULL
)

Arguments

Details

The progress spinners here come from the cli package and will respond to cli's options. In particular cli.progress_clear and cli.progress_show_after.

Value

Logical value, TRUE if the task completed successfully, FALSE otherwise.

Examples

cleanup <- hipercow_example_helper()

id <- task_create_expr(sqrt(2))
task_wait(id)

cleanup()

Description

Register DIDE windows credentials.

Usage

windows_authenticate()

Details

In order to be able to communicate with the Windows DIDE HPC system, we need to be able to communicate with the HPC portal (https::mrcdata.dide.ic.ac.uk/hpc), and for this we need your DIDE password and username. This is typically, but not always, the same as your Imperial credentials. We store this information securely using the keyring package, so when unlocking your credentials you will be prompted for your computer password, which will be your DIDE password if you use a windows machine connected to the DIDE domain, but will likely differ from either your DIDE or Imperial password if you are outside the DIDE domain, or if you don't use Windows.

Value

Nothing, this function is called for its side effect of setting or updating your credentials within the keyring.

Examples

windows_authenticate()

Description

Perform some basic checks to make that your system is configured to use the windows cluster properly. Calling this when something goes wrong is never a bad idea.

Usage

windows_check(path = getwd())

Arguments

Value

Invisibly, a logical; TRUE if all checks succeed and FALSE otherwise.

Examples

windows_check()

Description

Generate a keypair for encrypting small data to send to the windows cluster. This can be used to encrypt environment variables, and possibly other workflows in future. By default, if you have ever created a keypair we do not replace it if it already exists, unless you set update = TRUE so you may call this function safely to ensure that you do have a keypair set up.

Usage

windows_generate_keypair(update = FALSE)

Arguments

Value

Nothing, called for its side effect

Examples

# Generate a new keypair, if one does not exist
windows_generate_keypair()

Description

Describe a path mapping for use when setting up jobs on the cluster.

Usage

windows_path(path_local, path_remote, drive_remote, call = NULL)

Arguments

Examples

# Suppose you have mounted your malaria share at "~/net/malaria"
# (e.g., on a Linux machine).  You can tell the cluster to mount
# this as "M:" when running tasks by first creating a path
# mapping:
share <- windows_path("~/net/malaria",
                      "//wpia-hn.hpc.dide.ic.ac.uk/Malaria",
                      "M:")

# This share object contains information about how to relate your
# local and remote paths:
share

# When configuring the cluster you might pass this:
hipercow_configure("windows", shares = share)

Description

Report the username used to log into the web portal for use with the windows cluster. This may or may not be the same as your local username. We may ask you to run this when helping debug cluster failures.

Usage

windows_username()

Value

Your username, as a string

Examples

# Return your windows username
windows_username()