Migration from didehpc

The hipercow package shares many of its ideas from didehpc but is a new implementation, sharing little code and offering no direct migration path. If you have an existing didehpc-based project, you should keep using that and use hipercow for any new work.

If you have never used didehpc before, there is no need to read this vignette.

Differences

Some differences of note between hipercow and didehpc; if you are confused by some other difference not mentioned here please let us know.

The user interface is quite different, with free functions replacing the big queue_didehpc object. Previously you created an object (often obj) with queue_didehpc and then interacted with this by running a method (say obj$task_result()); instead in hipercow you use a free function (in this example task_result). See below for a translation of methods to new function(s).

No object corresponding to a task. Previously there were object handles that had methods. So you had a task t and fetched its result as t$result(). As with the queue, this has moved to free functions (in this example task_result()).

We no longer check that your packages have been installed. We might modify this behaviour, but the previous behaviour of not allowing the queue object to be created until packages have been installed has been stopped. Instead, we provide some tools (notably, hipercow_provision_check()) for you to check that the state of the packages is what you expect them to be. There’s nothing to stop you submitting tasks into a cluster environment that cannot run them due to missing packages, but we hope that will be more flexible.

We never load packages into your current environment, you don’t even need them installed. In didehpc when you loaded the queue, it loaded all packages in your context in your current session. This no longer happens, and the packages do not need to be installed at all.

Smaller differences include:

  • task_wait does not return the result of the task, but a boolean indicating if the task was successful

Mapping of didehpc methods to hipercow functions

queue_didehpc

  • cluster_load: TODO (mrc-4892)
  • config: hipercow_configuration()
  • context: no analogue
  • dide_id: no analogue (we might add this to task_info() though)
  • dide_log: task_log_show() with argument outer = TRUE
  • enqueue: task_create_expr()
  • enqueue_: task_create_explicit()
  • enqueue_bulk: task_create_bulk_expr()
  • initialize_context: no analogue
  • install_packages: hipercow_provision() with arguments method = "pkgdepends", refs = ...
  • lapply: task_create_bulk_call()
  • login: windows_check() (we don’t actually log in this way though)
  • mapply: task_create_bulk_call()
  • provision_context: hipercow_provision()
  • reconcile: task_info() (but only one id at a time, and only reconciles as a side effect)
  • rrq_controller:
  • stop_workers: TODO (mrc-4869)
  • submit: task_submit()
  • submit_workers: TODO (mrc-4869)
  • task_bundle_get: hipercow_bundle_load()
  • task_bundle_info: no analogue, as bundles no longer need to share anything common.
  • task_bundle_list: hipercow_bundle_list()
  • task_delete: TODO (mrc-4842)
  • task_get: no analogue (see below)
  • task_list: not implemented. Practically this was quite slow for real-world usage and not terribly useful, so we might just skip doing it. Let us know if this bothers you.
  • task_result: task_result() (but only for a single task)
  • task_status: task_status()
  • task_times: task_info()
  • unsubmit: task_cancel()

task

Running task_get, or creating a task with $enqueue() returned a task object. The methods on this correspond again to free functions in hipercow:

  • context_id: no analogue
  • expr: task_info()
  • log: task_log_show() and task_log_value(), also task_log_watch()
  • result: task_result()
  • status: task_status()
  • times: task_info()
  • wait: task_wait()

bundle

Running $task_bundle_get() or creating a bulk submission with $enqueue_bulk() or $lapply() created a task bundle object. The methods on this correspond to free functions in hipercow:

  • times: not yet implemented (will be hipercow_bundle_info)
  • results: hipercow_bundle_result()
  • wait: hipercow_bundle_wait()
  • status: hipercow_bundle_status()
  • expr: not yet implemented, may go into hipercow_bundle_info
  • function_name: not implemented

Configuration

Previously you might have used didehpc_config to change lots of bits of didehpc’s behaviour. We’ve removed most of this for now! Below is a list of arguments to didehpc_config() and

  • credentials: the first time you use hipercow you should use windows_authenticate to arrange your credentials and securely save them in your keychain.
  • home: this used to be information about where to mount your home drive, which we always did. We no longer mount it by default unless it is your working directory.
  • temp: this used to be information about the temp drive, we do not mount this by default either.
  • cluster: this was the cluster to use. That’s not currently configurable as we only support one (wpia-hn, the “new cluster”). This may reappear as a windows-specific option to hipercow_configure() later depending on how the cluster setup goes.
  • shares: this is still supported, as the shares option to hipercow_configure()
  • template: this is renamed to queue in hipercow_resources(), which is what template essentially meant. You then pass your hipercow_resources list to functions when creating tasks.
  • cores: this is the cores option in hipercow_resources().
  • nodes: this wasn’t really useful in didehpc, since it reserved nodes but could only use one of them. For now, it is not implemented in hipercow.
  • wholenode: this is more flexible now. Setting cores to Inf will request a whole node, however many cores it has. Setting exclusive to TRUE ensures that no other task runs on the node at the same time as yours, even if you request a smaller number of cores than the node has.
  • parallel: call hipercow_parallel() with the argument "future" or "parallel” and pass the result to the parallel argument when creating your task, to set up a local cluster within a node. Parallel processing can additionally be done by requesting multiple cores with hipercow_resources(), which sets useful environment variables for some thread-capable packages that use them.
  • workdir: no longer specified; the path to your working directory is determined automatically.
  • use_workers: workers are not yet supported, and the interface will change
  • use_rrq: rrq is not yet supported, and the interface will change
  • worker_timeout: workers are not yet supported, and the interface will change
  • worker_resource: workers are not yet supported, and the interface will change
  • conan_bootstrap: no longer configurable, this should just work for you now.
  • r_version: this is still supported, as the r_version option to hipercow_configure()
  • use_java: This is no longer needed; A recent Java LTS is always available for rJava and packages that use it.
  • java_home: This is no longer needed; the JAVA_HOME environment variable is automatically set to a recent Java LTS release.

Missing features

Features that we are planning to implement, but that are not yet available. If you use these you should not yet migrate to hipercow. Let us know in the Cluster channel if these are blocking you moving and it’s helpful for us to prioritise.

  • Use of workers, or with rrq in general
  • Dependencies between tasks (e.g., submit a task after another has completed); we’ll implement this but nobody was actually using it to our knowledge.

New features

  • Support for provisioning environments with renv
  • Support for an escape hatch method of provisioning with scripts; use whatever you want
  • Most users should very rarely need to enter a password
  • Most configuration now automatic
  • Simple method to retry failed (or otherwise unsatisfactory) tasks
  • Resources offer specifying a maximum runtime, memory requirements, indicating when tasks should run, lowering priority, and better parallel support.
  • Easily stream logs out from a running task