Commoner 
Commoner makes it easy to write scripts that flexibly and efficiently transpile any dialect of JavaScript into a directory structure of Node-compatible CommonJS module files.
This task is made possible by
- a declarative syntax for defining how module source code should be found and processed,
- the use of promises to manage an asynchronous build pipeline, and
- never rebuilding modules that have already been built.
The output files can be required seamlessly by Node, or served by any static file server, or bundled together using a tool such as Browserify, WrapUp, or Stitch for delivery to a web browser.
If you pass the --relativize option, Commoner also takes care to rewrite
all require calls to use relative module
identifiers,
so that the output files can be installed into any subdirectory of a
larger project, and external tools do not have to give special treatment
to top-level modules (or even know which modules are top-level and which
are nested).
Commoner was derived from an earlier, more opinionated tool called Brigade that provided additional support for packaging modules together into multiple non-overlapping bundles. Commoner grew out of the realization that many tools already exist for bundling CommonJS modules, but that fewer tools focus on getting to that point.
Installation
From NPM:
npm install commoner
From GitHub:
cd path/to/node_modules
git clone git://github.com/reactjs/commoner.git
cd commoner
npm install .
Usage
Here's the output of bin/commonize --help:
Usage: commonize [options] <source directory> <output directory> [<module ID> [<module ID> ...]]
Options:
-h, --help output usage information
-V, --version output the version number
-c, --config [file] JSON configuration file (no file means STDIN)
-w, --watch Continually rebuild
-x, --extension <js | coffee | ...> File extension to assume when resolving module identifiers
--relativize Rewrite all module identifiers to be relative
--follow-requires Scan modules for required dependencies
--cache-dir <directory> Alternate directory to use for disk cache
--no-cache-dir Disable the disk cache
--source-charset <utf8 | win1252 | ...> Charset of source (default: utf8)
--output-charset <utf8 | win1252 | ...> Charset of output (default: utf8)
In a single sentence: the commonize command finds modules with the given
module identifiers in the source directory and places a processed copy of
each module into the output directory, along with processed copies of all
required modules.
If you do not provide any module identifiers, commonize will process all
files that it can find under the source directory that have the preferred
file extension (.js by default). If your source files have a file
extension other than .js, use the -x or --extension option to
specify it. For example, --extension coffee to find .coffee files.
Output
Commoner prints various status messages to STDERR, so that you can see
what it's doing, or figure out why it's not doing what you thought it
would do.
The only information it prints to STDOUT is a JSON array of module
identifiers, which includes the identifiers passed on the command line and
all their dependencies. This array contains no duplicates.
Internally, each module that Commoner generates has a hash computed from the module's identifier, source code, and processing steps. Since this hash can be computed before processing takes place, Commoner is able to avoid processing a module if it has ever previously processed the same module in the same way.
If you dig into the
code,
you'll find that Commoner maintains a cache directory (by default,
~/.commoner/module-cache/) containing files with names like
9ffc5c853aac07bc106da1dc1b2486903ca688bf.js. When Commoner is about to
process a module, it checks its hash against the file names in this
directory. If no match is found, processing procedes and the resulting
file is written to the cache directory with a new hash. If the appropriate
hash file is already present in the cache directory, however, Commoner
merely creates a hard link between the hash file and a file with a more
meaningful name in the output directory.
When you pass the --watch flag to bin/commonize, Commoner avoids
exiting after the first build and instead watches for changes to
previously read files, printing a new JSON array of module identifiers to
STDOUT each time rebuilding finishes. Thanks to the caching of processed
modules, the time taken to rebuild is roughly proportional to the number
of modified files.
Customization
The bin/commonize script is actually quite simple, and you can write
similar scripts yourself. Let's have a look:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
var context = this;
return context.getProvidedP().then(function(idToPath) {
// If a module declares its own identifier using @providesModule
// then that identifier will be a key in the idToPath object.
if (idToPath.hasOwnProperty(id))
return context.readFileP(idToPath[id]);
});
}, function(id) {
// Otherwise assume the identifier maps directly to a filesystem path.
// The readModuleP method simply appends the preferred file extension
// (usually .js) to the given module identifier and opens that file.
return this.readModuleP(id);
});
The scriptable interface of the commoner module abstracts away many of
the annoyances of writing a command-line script. In particular, you don't
have to do any parsing of command-line arguments, and you don't have to
worry about installing any dependencies other than commoner in your
$NODE_PATH.
What you are responsible for, at a minimum, is telling Commoner how to
find the source of a module given a module identifier, and you do this by
passing callback functions to require("commoner").resolve. The script
above uses two strategies that will be tried in sequence: first, it calls
the helper function this.getProvidedP to retrieve an object mapping
identifiers to file paths (more about this below); and, if that doesn't
work, it falls back to interpreting the identifier as a path relative to
the source directory.
Now, you might not care about this.getProvidedP. It's really just a
proof of concept that Commoner can support modules that declare their own
identifiers using the // @providesModule <identifier> syntax, and I
included it by default because it doesn't make a difference unless you
decide to use @providesModule. If you don't like it, you could write an
even simpler script:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
return this.readModuleP(id);
});
The point is, it's entirely up to you to define how module identifiers are interpreted. In fact, the source you return doesn't even have to be valid JavaScript. It could be CoffeeScript, or LESS, or whatever language you prefer to write by hand. Commoner doesn't care what your source code looks like, because Commoner allows you to define arbitrary build steps to turn that source code into plain old CommonJS.
Let's consider the example of using LESS to write dynamic CSS
modules. First, let's apply what we already know to give special meaning
to .less files:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readModuleP(id);
});
function isLess(id) {
return /\.less$/i.test(id);
}
All this really accomplishes is to avoid appending the .js file
extension to identifiers that already have the .less extension.
Now we need to make sure the contents of .less files somehow get
transformed into plain old CommonJS, and for that we need
require("commoner").process:
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readModuleP(id);
}).process(function(id, source) {
if (isLess(id))
return compileLessToJs(source);
return source;
});
How should compileLessToJs be implemented? At a high level, I propose
that we generate a CommonJS module that will append a new <style> tag to
the <head> the first time the module is required. This suggests to me
that we need to take the CSS generated by LESS and somehow embed it as a
string in a CommonJS module with a small amount of boilerplate JS.
Here's a first attempt:
function compileLessToJs(less) {
var css = require("less").render(less);
return 'require("css").add(' + JSON.stringify(css) + ");";
}
Implementing a css module with an appropriate add method is an
exercise that I will leave to the reader (hint: you may find this
StackOverflow answer useful).
This almost works, but there's one problem: require("less").render does
not actually return a string! For better or worse, it passes the compiled
CSS to a callback function, which would make our task extremely painful
if Commoner were not deeply committed to supporting asynchronous
processing.
Commoner uses promises for asynchronous control flow, so we need to return
a promise if we can't return a string immediately. The easiest way to make
a promise is to call this.makePromise in the following style:
#!/usr/bin/env node
require("commoner").resolve(function(id) {
if (isLess(id))
return this.readFileP(id);
}, function(id) {
return this.readModuleP(id);
}).process(function(id, source) {
if (isLess(id)) {
return this.makePromise(function(nodeStyleCallback) {
compileLessToJs(source, nodeStyleCallback);
});
}
return source;
});
function compileLessToJs(less, callback) {
require("less").render(less, function(err, css) {
callback(err, 'require("css").add(' + JSON.stringify(css) + ");")
});
}
And we're done! This example was admittedly pretty involved, but if you
followed it to the end you now have all the knowledge you need to write
source files like sidebar.less and require them from other modules by
invoking require("sidebar.less"). (And, by the way, embedding dynamic
CSS modules in your JavaScript turns out to be an excellent idea.)
Generating multiple files from one source module
Commoner is not limited to generating just one output file from each
source module. For example, if you want to follow best practices for
producing source maps, you probably want to create a .map.json file
corresponding to every .js file that you compile.
Recall that normally your .process callback returns a string (or a
promise for a string) whose contents will be written as a .js file in
the output directory. To write more than one file, just return an object
whose keys are the file extensions of the files you want to write, and
whose values are either strings or promises for strings representing the
desired contents of those files.
Here's an example of generating two different files for every source
module, one called <id>.map.json and the other called <id>.js:
require("commoner").resolve(function(id) {
return this.readModuleP(id);
}).process(function(id, source) {
var result = compile(source);
return {
".map.json": JSON.stringify(result.sourceMap),
".js": [
result.code,
"//# sourceMappingURL=" + id + ".map.json"
].join("\n")
};
});
Note that
return {
".js": source
};
would be equivalent to
return source;
so you only have to return an object when you want to generate multiple
files. However, the .js key is mandatory when returning an object.
For your convenience, if you have a sequence of multiple processing functions, the values of the object returned from each step will be resolved before the object is passed along to the next processing function, so you can be sure all the values are strings (instead of promises) at the beginning of the next processing function.
Configuration
Of course, not all customization requires modifying code. Most of the time, in fact, configuration has more to do with providing different dynamic values to the same code.
For that kind of configuration, you don't need to modify your Commoner
script at all, because Commoner scripts accept a flag called --config
that can either specify a JSON file or (if --config is given without a
file name) read a string of JSON from STDIN.
Examples:
bin/commonize source/ output/ main --config release.json
bin/commonize source/ output/ main --config debug.json
echo '{"debug":false}' | bin/commonize source/ output/ main --config
echo '{"debug":true}' | bin/commonize source/ output/ main --config /dev/stdin
This configuration object is exposed to the .resolve and .process
callbacks as this.config. So, for example, if you wanted to implement
minification as a processing step, you might do it like this:
require("commoner").resolve(function(id) {
return this.readModule(id);
}).process(function(id, source) {
if (this.config.debug)
return source;
return minify(source);
});
Perhaps the coolest thing about the configuration object is that Commoner generates a recursive hash of all its properties and their values which is then incorporated into every module hash. This means that two modules with the same identifier and identical source code and processing steps will have distinct hashes if built using different configuration objects.
Custom Options
You can define custom options for your script by using the option function.
require("commoner").resolve(function(id) {
return this.readModule(id);
}).option(
'--custom-option',
'This is a custom option.'
).process(function(id, source) {
if (this.options.customOption) {
source = doCustomThing(source);
}
return source;
});
For more information of the options object available inside the process function see Commander.