Bundling and Distributing Complex ES6 Libraries in an ES5 World

We previously announced pileup.js, a genome viewer designed to be embedded in web applications. Since then, we have restructured the code base and created a cleaner distribution. These both make pileup.js easier to integrate into applications. We faced several challenges along the way that are likely to be encountered by anyone building and releasing modern web libraries, so this post will detail the workflow we’ve come up with to solve them.

Bundled File Distribution

Adding pileup.js to your application is as simple as including our bundled distribution file, which can be downloaded from NPM, Github, or Browserify’s CDN. That file defines a global function require, which you can use to create new pileup instances:

  

// Request the pileup object
var pileup = require("pileup");
// and create an instance
var p = pileup.create(yourDiv, { ... });

This setup is great because you need only include a single JavaScript file; in particular, you don’t have to worry about managing/including its dependencies. To make this possible, our code goes through two transformations: jstransform and browserify:

(Our current workflow for developing and distributing the pileup.js code (1,2))

JSTransform: transpiling ES6 into plain JavaScript

The pileup.js code is written in EcmaScript 6 (ES6) and JSX, and annotated with Flow types. These syntaxes are great and tend to take a lot of burden off the developers, but they also come with a necessary code transformation step, commonly referred to as transpiling. What this means in practice is that people can’t use the original code directly, since we still live in an EcmaScript 5-dominated world. For example, if you were to check out our code from GitHub, and try to use it “as is”, this is what you will get:

  

$ echo "require('./src/main/pileup.js');" | node
./pileup.js/src/main/pileup.js:21
import type {Track, VisualizedTrack} from './types';
^^^^^^
SyntaxError: Unexpected reserved word
  ...

This SyntaxError is expected, as import type syntax is part of Flow and not ES5. To make our code ES5-compatible, we have to run it through JSTransform first:

  

$ jstransform --react --harmony --strip-types --non-strict-es6module src/ dist/

This transpiles all the code from the src folder into another one. Once the transpiling is completed, you can now start using the transformed code under dist/:

  

$ echo "require('./dist/main/pileup');" | node
# No errors

As such, it does not make sense to distribute the untransformed code via NPM, nor does it make sense to keep track of the transformed code via GitHub; therefore, we ignore the dist folder via our .gitignore and list only the files under dist for release in our package.json:

  

...
"files": [
  "dist",
  "style"
],
...

Browserify: bundling everything into a single file

Even if we distribute the plain ES5 code, we still need to make things easy for people who would like to use pileup.js in their web applications. And by default, it is not possible to use a library structured as a node module as they don’t understand require statements and can’t handle a module’s multi-file structure. Browserify solves this by converting a node module into a form that can be used for web applications. It can bundle a node library and all of its dependencies into a single file by wrapping it up in an anonymous closure and optionally exporting a function (require) to access its contents. When run on a project folder, browserify takes hints from your package description (package.json) and starts the process by reading the module specified in the browser property:

...
"browser": "dist/main/pileup.js",
...

It then traverses all the code that is used by this file and its dependencies. These files get bundled in a closure and structured in a way to make dependencies work seamlessly within that single bundled file. This means that if you open the bundled pileup.js and examine its contents, you will also see the code for our dependencies, such as react, d3 and underscore.

After transpiling our code and before publishing it on NPM, we run browserify as a task and bundle the code up:

$ cd pileup.js/
# npm run browserify
$ browserify -r .:pileup -v --debug -o dist/pileup.js

This

• calls the command line browserify utility on the current folder (.);
• allows access to the main browser module via pileup alias, so that you can require('pileup');
• creates a map from the bundled file into individual source files for easier debugging;
• saves the bundled file as dist/pileup.js.

And since this new browserified file is under dist/, it also is distributed via NPM so that people don’t have to bother with all these steps and can use this file directly from the repository:

$ npm install pileup --save
$ ls node_modules/pileup/dist/*.js
node_modules/pileup/dist/pileup.js
node_modules/pileup/dist/pileup.min.js

UglifyJS: optional transformation to compress the code for production

Minification dramatically reduces the size of our bundled file. Here is the proof:

$ du -sh dist/*.js
4.2M	dist/pileup.js
556K	dist/pileup.min.js

You can see that the bundled file, pileup.js, is gigantic by web standards while the minified version, pileup.min.js, is much more reasonable. To accomplish this, we need to transform our code once more, this time through UglifyJS:

$ uglifyjs --compress --mangle -o dist/pileup.min.js dist/pileup.js

which turns the code into an ugly yet functionally-equivalent form by renaming variables and reducing whitespace in the file. This minified file is better for distribution, but it is terrible for development purposes since it is almost impossible to debug or read. Therefore, our tests and playground examples all make use of the bundled version instead of the minified one; but we encourage the use of the minified version for production purposes.

Decoupling transformations

Although the current structure of pileup.js is now relatively simple, we had to try different configurations with different setups before settling on this particular solution. Before this, we were running browserify and transforming the code at the same time using jstransform extensions. Although the final output was the same, we hit multiple issues during tests/flow-checks and reducing the size of other web applications that make use of pileup.js.

Specifically, the former was due to the difference between the way watchify and flow were parsing the code, which are both required for our test procedures. When transforming the code simultaneously with jstransform and browserify, we were asking it to alias the main module as pileup to be able to require it. However, when watchify was trying to bundle the main code together with the tests, the relative path the tests files were using to require the main module was causing trouble, therefore breaking many of our tests. If we started using the pileup alias instead of the relative paths to require the module, this time flow was complaining about missing modules. We tried to work around this problem, but it was a hack.

The latter was mainly because we were only distributing the bundled file. This meant that any other web application that was depending on our library had to deal with our massive distribution file. To better understand this problem, imagine that you are working on an application that depends on d3. Since d3 is also a dependency for pileup.js, it makes sense to not bundle two d3s into a single file when transforming the code through browserify. However, when pileup.js is already bundled together with its own dependencies, d3 has to be bundled into the application once again, causing redundancy in the production code and considerably inflating it.

Due to these problems, we decided to uncouple the jstransform and browserify steps, and start distributing the intermediate ES5 code to allow developers interact with our CommonJS-compatible module when desired.

Wrap up

Overall, after a few iterations of trial-and-error (and some level of frustration in between), we are quite happy with the end result of our refactoring to improve the way we distribute pileup.js. Thanks to these recent changes and the new structure of our module, our library now plays nicely with online editors, such as JSFiddle:

In case you missed it in the previous blog post, you can take a look at the public demo of pileup.js. If you are looking for some example code to get inspiration for your web application, our examples folder folder is a good place to start If you bump into any issues using pileup.js in your application, please let us know.