master
branchmaster
branchKetrew is:
This is the 2.0.0
version of Ketrew, serialized content is backwards
compatible with the 1.x.x
series but the protocol is not; the EDSL API
requires minor changes (all caught by the OCaml compiler).
See also the documentation for the
master branch.
Ketrew requires at least OCaml 4.02.2 and should be able to build & work on any Unix platform.
If you have opam
up and running:
opam install ketrew
Then you need at runtime ssh
in the $PATH
.
This gets you the ketrew
executable and the ketrew_pure
and ketrew
libraries.
See the development documentation to find out how to build Ketrew (and its dependencies) from the sources.
Ketrew is very flexible and hence may seem difficult to understand, let's get a very minimalistic workflow running.
The first time you use Ketrew, you need to configure it, simplest by calling
ketrew init
, and please choose an authentication token:
rm -fr ~/.ketrew/ # if you had a previous configuration there
ketrew init --with-token my-not-so-secret-token
by default this will configure Ketrew in $HOME/.ketrew/
with a client/server
mode not using TLS on port 8756
(see ketrew init --help
you can even ask
it to generate self-signed TLS certificates).
See also the documentation
on the configuration file learn how to tweak it.
You can check that the client or the server are configured:
ketrew print-configuration
ketrew print-configuration --configuration-profile server
ketrew print-configuration -P daemon
You may then start a server:
KETREW_PROFILE=daemon ketrew start-server
and then open the GUI:
ketrew gui
which is just trying to load http://127.0.0.1:8756/gui?token=my-not-so-secret-token ☺
You can always stop the server or check its status:
ketrew stop -P daemon
ketrew status -P daemon
The ketrew submit
sub-command can create tiny workflows:
ketrew submit --wet-run --tag 1st-workflow --tag command-line --daemonize /tmp/KT,"du -sh $HOME"
The job will appear on the WebUI and you can inspect/restart/kill it.
If you don't like Web UI's you can use the text-based UI:
ketrew interact
The previous section uses ketrew submit
to launch an extremely simple
workflow, to go further we need the EDSL.
The EDSL is an OCaml library where all the functions are used to build a
workflow data-structure. Then, one function: Ketrew.Client.submit
is used to
submit workflows to the engine.
A workflow is a Graph of “targets”.
There are 3 kinds of links between targets:
Any OCaml program can use the EDSL (script, compiled, or even inside the toplevel), see the documentation of the EDSL API.
This example is a “single-target” workflow that runs an arbitrary shell command on an LSF-based cluster:
#use "topfind" #thread #require "ketrew" let run_command_with_lsf cmd = let module KEDSL = Ketrew.EDSL in let host = (* `Host.parse` takes an URI and creates a “Host” datastructue: a place to run stuff. *) KEDSL.Host.parse "ssh://user42@MyLSFCluster/home/user42/ketrew-playground/?shell=bash" (* This one is an SSH host, named `MyLSFCluster`. The directory `/home/user42/ketrew-playground/` will be used by Ketrew to monitor the jobs. *) in let program = (* A “program” is a datastructure representing “extended shell scripts”. `Program.sh` creates one out a shell command. *) KEDSL.Program.sh cmd in let lsf_build_process = (* “build process” is a method for making things: `lsf` creates a datastructure that represents a job running a `program` with the LSF scheduling engine, on the host `host`. *) KEDSL.lsf ~queue:"normal-people" ~wall_limit:"1:30" ~processors:(`Min_max (1,1)) ~host program in (* The function `KEDSL.target` creates a node in the workflow graph. This one is very simple, it has a name and a build-process, and since it doesn't have dependencies or fallbacks, it is a “single-node” workflow: *) KEDSL.target "run_command_with_lsf" ~make:lsf_build_process let () = let workflow = (* Create the workflow with the first argument of the command line: *) run_command_with_lsf Sys.argv.(1) in (* Then, `Client.submit` is the only function that “does” something, it submits the workflow to the engine: *) Ketrew.Client.submit workflow (* If Ketrew is in Standalone mode, this means writing the workflow in the database (nothing runs yet, you need to run Ketrew's engine yourself). If Ketrew is in Client-Server mode, this means sending the workflow to the server over HTTPS. The server will start running the workflow right away. *)
If you actually have access to an LSF cluster and want to try this workflow,
put it in a file my_second_workflow.ml
, and simply:
ocaml my_second_workflow.ml 'du -sh $HOME'
To learn more about the EDSL, you can also explore examples of more and more complicated workflows (work-in-progress).
From here:
src/test/Workflow_Examples.ml
for
examples and the documentation of the EDSL API.Ketrew.Lsf
or in the tests:
src/test/dummy_plugin.ml
.ketrew
as a client).Ketrew.Engine
.It's Apache 2.0.