Ketrew: Developer Documentation

Contents

Menu

Development Documentation

Build

Getting Just The Dependencies

Ketrew depends on

  • nonstd: nano-library providing a portable extract of Core_kernel
  • pvem: error monad
  • docout: logging with smart-print
  • sosa: String module/functor
  • pvem_lwt_unix: Lwt_unix wrapped in a Pvem.t (error monad) with more precise error types.
  • uri: parse RFC-3986-compliant URIs (uri itself depends on camlp4).
  • cmdliner: command line parsing
  • ppx_deriving_yojson, ppx_deriving, yojson: JSON parsing/printing and other code generation
  • cohttp.lwt, ssl, and conduit: HTTP server and client
  • findlib + dynlink: dynamic loading of plugins
  • trakeva, specifically the trakeva_of_uri library (with either the sqlite or postgresql backends).
  • js_of_ocaml, tyxml (with reactiveData)

and uses the omake as a build system and the tool ocamlify.

At runtime, Ketrew may use an ssh client (tested only with OpenSSH; but SSH calls are quite configurable).

Build

Then you may setup and build the libraries and ketrew the command line application:

 omake

to build also all the tests, use:

omake build-all

Install

Ketrew is installed with:

 omake install BINDIR=/path/to/bin

(it will use ocamlfind to install the library and copy the executable to BINDIR).

There is also an opam file in the repository:

 opam pin add ketrew .

will pick it (Opam ≥ 1.2.0).

Build The Documentation

The documentation depends on oredoc:

omake doc

and check-out _doc/<branch>/index.html (unless branch is master, then _doc/index.html).

Merlin

Simply,

omake .merlin

Tests

Automated Tests

Run the tests like this:

    ./ketrew-test [-no-color] <Test-names>

where a Test-names is one or more of

  • ALL all of the following.
  • basic-test
  • nohup-test

The cli Test

The workflow examples in the documentation are actually interactive tests (cf. ./ketrew-workflow-examples-test).

The integration Test

The integration test uses Ketrew to build Vagrant virtual machines and uses them to test further features: we test the PBS, LSF, and YARN long-running backends by creating small “virtual clusters”. You can use the commands prepare, go, and clean-up separately, or try ./ketrew-test integration (which does them all).

Dynamically Loaded Plugins

The build-system creates a plugin and a workflow which uses it:

Generating a Test Environment

In order to not impact a potential “global” installation of Ketrew, one can use:

omake test-env
Using package lwt.react as findlin-plugin
Creating cert-key pair: _test_env/test-cert.pem, _test_env/test-key.pem
Creating _test_env/configuration.ml
Creating _test_env/test-authorized-tokens
Creating _test_env/env.env

The command creates the directory _test_env/ with a preconfigured test-environment (a self-signed SSL certificate/key pair, client/server/standalone configuration file, an “authorization-tokens” configuration, … which all work together harmoniously).

Sourcing _test_env/env.env will give a few aliases to run the tests. Aliases which start with ks mean “with a standalone-mode configuration file;” those which start with kd are in client-server mode ('d' for “distributed”).

  • kscli: the standalone ketrew application.
  • kdserver: the server ketrew application.
  • kdroserver: the server ketrew application but running in “read-only” mode.
  • kddaemon: the same as kdserver but quiting the current terminal (daemon option).
  • kdclient: the client ketrew application (talking to a kdserver or kddaemon instance).
  • kstest: the cli test with a “standalone-mode” configuration file.
  • kdtest: the same cli test but as a client (again, talking to a kdserver or kddaemon instance).
  • ksintegration, and kdintegration: the integration test in standalone and client modes.
  • ksplugin_user, and kdplugin_user: the mini-workflow using the plugin.

Coverage

To generate coverage reports you need to instrument the code by recompiling from scratch using the environment variable WITH_BISECT equal to true:

omake clean
WITH_BISECT=true omake

Running the instrumented versions of the code will generate bisect*.out files when run.

Then, omake bisect-report will take these files and generate an html file in _report_dir/index.html. omake bisect-clean removes the reports and _report_dir.

To remove the instrumentation just use omake clean; omake without WITH_BISECT set to true.

Note that the Web-UI does not work when the code has been instrumented with Bisect (You'll see errors in the JS-Console: “caml_mutex_new not implemented”).

Slow Down The Server

In order to test the WebUI in more adverse conditions, the server can be asked to answer HTTP queries artificially slow:

KETREW_DEBUG_SLOW_SERVER=1.,0.5 kdserver start

The 2 floating-point numbers x,y mean that, at each HTTP request, the server will pause for Random.float x +. y seconds.

How to Release

Once we are somewhat happy about the state of the master branch (tests, documentation, issues that went into the “next release” milestone), this is the release workflow:

  • Release dependencies for which we are using unreleased features (e.g. trakeva, sosa, etc.).
  • Set version string in OMakeroot
  • Update the introductory paragraph of the README.md file for the particular version.
  • Write a human-friendly change-log (go through git history and write important changes).
  • Create the release/tag ketrew.x.y.z (put the change-log there, see releases documentation).
  • Fork the mothership opam-repository.
  • Add a new package by modifying the repository's one (see ./opam), fix the URL and the MD5 sum (from release), test the package (in a new opam-switch, or remove ocamlfind), create pull-request.
  • Add the tag as an “interesting checkout” in smondet/build-docs-workflow, then build and push the documentation (which is for now part of smondet/smondet.github.com).
  • Once the opam PR is merged, brag about it, write a blog post, start hacking on the next version.