The hipercow
package
shares many of its ideas from didehpc
but
is a new implementation, sharing little code and offering no direct
migration path. If you have an existing didehpc
-based
project, you should keep using that and use hipercow
for
any new work.
If you have never used didehpc
before, there is no need
to read this vignette.
Some differences of note between hipercow
and
didehpc
; if you are confused by some other difference not
mentioned here please let us know.
The user interface is quite different, with free functions
replacing the big queue_didehpc
object. Previously
you created an object (often obj
) with queue_didehpc
and then interacted with this by running a method (say
obj$task_result()
); instead in hipercow
you
use a free function (in this example task_result
). See
below for a translation of methods to new function(s).
No object corresponding to a task. Previously there
were object handles that had methods. So you had a task t
and fetched its result as t$result()
. As with the queue,
this has moved to free functions (in this example
task_result()
).
We no longer check that your packages have been
installed. We might modify this behaviour, but the previous
behaviour of not allowing the queue object to be created until packages
have been installed has been stopped. Instead, we provide some tools
(notably, hipercow_provision_check()
) for you to check that
the state of the packages is what you expect them to be. There’s nothing
to stop you submitting tasks into a cluster environment that cannot run
them due to missing packages, but we hope that will be more
flexible.
We never load packages into your current environment, you don’t even need them installed. In didehpc when you loaded the queue, it loaded all packages in your context in your current session. This no longer happens, and the packages do not need to be installed at all.
Smaller differences include:
task_wait
does not return the result of the task, but a
boolean indicating if the task was successfuldidehpc
methods to hipercow
functionsqueue_didehpc
cluster_load
: TODO (mrc-4892
)config
: hipercow_configuration()
context
: no analoguedide_id
: no analogue (we might add this to
task_info()
though)dide_log
: task_log_show()
with argument
outer = TRUE
enqueue
: task_create_expr()
enqueue_
: task_create_explicit()
enqueue_bulk
: task_create_bulk_expr()
initialize_context
: no analogueinstall_packages
: hipercow_provision()
with arguments method = "pkgdepends", refs = ...
lapply
: task_create_bulk_call()
login
: windows_check()
(we don’t actually
log in this way though)mapply
: task_create_bulk_call()
provision_context
:
hipercow_provision()
reconcile
: task_info()
(but only one id at
a time, and only reconciles as a side effect)rrq_controller
:stop_workers
: TODO (mrc-4869
)submit
: task_submit()
submit_workers
: TODO (mrc-4869
)task_bundle_get
:
hipercow_bundle_load()
task_bundle_info
: no analogue, as bundles no longer
need to share anything common.task_bundle_list
:
hipercow_bundle_list()
task_delete
: TODO (mrc-4842
)task_get
: no analogue (see below)task_list
: not implemented. Practically this was quite
slow for real-world usage and not terribly useful, so we might just skip
doing it. Let us know if this bothers you.task_result
: task_result()
(but only for a
single task)task_status
: task_status()
task_times
: task_info()
unsubmit
: task_cancel()
task
Running task_get
, or creating a task with
$enqueue()
returned a task object. The methods on this
correspond again to free functions in hipercow
:
context_id
: no analogueexpr
: task_info()
log
: task_log_show()
and
task_log_value()
, also task_log_watch()
result
: task_result()
status
: task_status()
times
: task_info()
wait
: task_wait()
bundle
Running $task_bundle_get()
or creating a bulk submission
with $enqueue_bulk()
or $lapply()
created a
task bundle object. The methods on this correspond to free functions in
hipercow
:
times
: not yet implemented (will be
hipercow_bundle_info
)results
: hipercow_bundle_result()
wait
: hipercow_bundle_wait()
status
: hipercow_bundle_status()
expr
: not yet implemented, may go into
hipercow_bundle_info
function_name
: not implementedPreviously you might have used didehpc_config
to change
lots of bits of didehpc’s behaviour. We’ve removed most of this for now!
Below is a list of arguments to didehpc_config()
and
credentials
: the first time you use
hipercow
you should use windows_authenticate
to arrange your credentials and securely save them in your
keychain.home
: this used to be information about where to mount
your home drive, which we always did. We no longer mount it by default
unless it is your working directory.temp
: this used to be information about the temp drive,
we do not mount this by default either.cluster
: this was the cluster to use. That’s not
currently configurable as we only support one (wpia-hn
, the
“new cluster”). This may reappear as a windows-specific option to
hipercow_configure()
later depending on how the cluster
setup goes.shares
: this is still supported, as the
shares
option to hipercow_configure()
template
: this is renamed to queue
in
hipercow_resources()
, which is what template
essentially meant. You then pass your hipercow_resources list to
functions when creating tasks.cores
: this is the cores
option in
hipercow_resources()
.nodes
: this wasn’t really useful in
didehpc
, since it reserved nodes but could only use one of
them. For now, it is not implemented in hipercow.wholenode
: this is more flexible now. Setting
cores
to Inf
will request a whole node,
however many cores it has. Setting exclusive
to
TRUE
ensures that no other task runs on the node at the
same time as yours, even if you request a smaller number of cores than
the node has.parallel
: call hipercow_parallel()
with
the argument "future"
or "parallel
” and pass
the result to the parallel
argument when creating your
task, to set up a local cluster within a node. Parallel processing can
additionally be done by requesting multiple cores with
hipercow_resources()
, which sets useful environment
variables for some thread-capable packages that use them.workdir
: no longer specified; the path to your working
directory is determined automatically.use_workers
: workers are not yet supported, and the
interface will changeuse_rrq
: rrq
is not yet supported, and the
interface will changeworker_timeout
: workers are not yet supported, and the
interface will changeworker_resource
: workers are not yet supported, and the
interface will changeconan_bootstrap
: no longer configurable, this should
just work for you now.r_version
: this is still supported, as the
r_version
option to hipercow_configure()
use_java
: This is no longer needed; A recent Java LTS
is always available for rJava
and packages that use
it.java_home
: This is no longer needed; the JAVA_HOME
environment variable is automatically set to a recent Java LTS
release.Features that we are planning to implement, but that are not yet
available. If you use these you should not yet migrate to
hipercow
. Let us know in the Cluster channel if these are
blocking you moving and it’s helpful for us to prioritise.
rrq
in
generalrenv