Ketrew: Developer Documentation

Contents

Menu

Development Documentation

Build

Getting Just The Dependencies

Ketrew depends on

  • nonstd: nano-library providing a portable extract of Core_kernel
  • pvem: error monad
  • docout: logging with smart-print
  • sosa: String module/functor
  • pvem_lwt_unix: Lwt_unix wrapped in a Pvem.t (error monad) with more precise error types.
  • uri: parse RFC-3986-compliant URIs (uri itself depends on camlp4).
  • cmdliner: command line parsing
  • ppx_deriving_yojson, ppx_deriving, yojson: JSON parsing/printing and other code generation
  • cohttp.lwt, ssl, and conduit: HTTP server and client
  • findlib + dynlink: dynamic loading of plugins
  • trakeva with the sqlite backend
  • js_of_ocaml, tyxml (with reactiveData)

and uses the omake as a build system and the tool ocamlify.

At runtime, Ketrew may use an ssh client (tested only with OpenSSH; but SSH calls are quite configurable).

Build

Then you may setup and build the libraries and ketrew the command line application:

 omake

to build also all the tests, use:

omake build-all

Install

Ketrew is installed with:

 omake install BINDIR=/path/to/bin

(it will use ocamlfind to install the library and copy the executable to BINDIR).

There is also an opam file in the repository:

 opam pin add ketrew .

will pick it (Opam ≥ 1.2.0).

Build The Documentation

The documentation depends on oredoc:

omake doc

and check-out _doc/<branch>/index.html (unless branch is master, then _doc/index.html).

Tests

Automated Tests

Run the tests like this:

    ./ketrew-test [-no-color] <Test-names>

where a Test-names is one or more of

  • ALL all of the following.
  • basic-test
  • nohup-test

The cli Test

The workflow examples in the documentation are actually interactive tests (cf. ./ketrew-workflow-examples-test).

The integration Test

The integration test uses Ketrew to build Vagrant virtual machines and uses them to test further features: we test the PBS, LSF, and YARN long-running backends by creating small “virtual clusters”. You can use the commands prepare, go, and clean-up separately, or try ./ketrew-test integration (which does them all).

Dynamically Loaded Plugins

The build-system creates a plugin and a workflow which uses it:

Generating a Test Environment

In order to not impact a potential “global” installation of Ketrew, one can use:

omake test-env
Using package lwt.react as findlin-plugin
Creating cert-key pair: _test_env/test-cert.pem, _test_env/test-key.pem
Creating _test_env/configuration.ml
Creating _test_env/test-authorized-tokens
Creating _test_env/env.env

The command creates the directory _test_env/ with a preconfigured test-environment (a self-signed SSL certificate/key pair, client/server/standalone configuration file, an “authorization-tokens” configuration, … which all work together harmoniously).

Sourcing _test_env/env.env will give a few aliases to run the tests. Aliases which start with ks mean “with a standalone-mode configuration file;” those which start with kd are in client-server mode ('d' for “distributed”).

  • kscli: the standalone ketrew application.
  • kdserver: the server ketrew application.
  • kdroserver: the server ketrew application but running in “read-only” mode.
  • kddaemon: the same as kdserver but quiting the current terminal (daemon option).
  • kdclient: the client ketrew application (talking to a kdserver or kddaemon instance).
  • kstest: the cli test with a “standalone-mode” configuration file.
  • kdtest: the same cli test but as a client (again, talking to a kdserver or kddaemon instance).
  • ksintegration, and kdintegration: the integration test in standalone and client modes.
  • ksplugin_user, and kdplugin_user: the mini-workflow using the plugin.

Coverage

To generate coverage reports you need to instrument the code by recompiling from scratch using the environment variable WITH_BISECT equal to true:

omake clean
WITH_BISECT=true omake

Running the instrumented versions of the code will generate bisect*.out files when run.

Then, omake bisect-report will take these files and generate an html file in _report_dir/index.html. omake bisect-clean removes the reports and _report_dir.

To remove the instrumentation just use omake clean; omake without WITH_BISECT set to true.

Note that the Web-UI does not work when the code has been instrumented with Bisect (You'll see errors in the JS-Console: “caml_mutex_new not implemented”).

How to Release

Once we are somewhat happy about the state of the master branch (tests, documentation, issues that went into the “next release” milestone), this is the release workflow:

  • Release dependencies for which we are using unreleased features (e.g. trakeva, sosa, etc.).
  • Set version string in OMakeroot
  • Update the introductory paragraph of the README.md file for the particular version.
  • Write a human-friendly change-log (go through git history and write important changes).
  • Create the release/tag ketrew.x.y.z (put the change-log there, see releases documentation).
  • Fork the mothership opam-repository.
  • Add a new package by modifying the repository's one (see ./opam), fix the URL and the MD5 sum (from release), test the package (in a new opam-switch, or remove ocamlfind), create pull-request.
  • Add the tag as an “interesting checkout” in smondet/build-docs-workflow, then build and push the documentation (which is for now part of smondet/smondet.github.com).
  • Once the opam PR is merged, brag about it, write a blog post, start hacking on the next version.