through2

A tiny wrapper around Node streams.Transform (Streams2) to avoid explicit subclassing noise

Inspired by Dominic Tarr's through in that it's so much easier to make a stream out of a function than it is to set up the prototype chain properly: through(function (chunk) { ... }).

Note: A Streams3 version of through2 is available in npm with the tag "1.0" rather than "latest" so an npm install through2 will get you the current Streams2 version (version number is 0.x.x). To use a Streams3 version use npm install through2@1 to fetch the latest version 1.x.x. More information about Streams2 vs Streams3 and recommendations here.

fs.createReadStream('ex.txt')
  .pipe(through2(function (chunk, enc, callback) {

    for (var i = 0; i < chunk.length; i++)
      if (chunk[i] == 97)
        chunk[i] = 122 // swap 'a' for 'z'

    this.push(chunk)

    callback()

   }))
  .pipe(fs.createWriteStream('out.txt'))

Or object streams:

var all = []

fs.createReadStream('data.csv')
  .pipe(csv2())
  .pipe(through2.obj(function (chunk, enc, callback) {

    var data = {
        name    : chunk[0]
      , address : chunk[3]
      , phone   : chunk[10]
    }

    this.push(data)

    callback()

  }))
  .on('data', function (data) {
    all.push(data)
  })
  .on('end', function () {
    doSomethingSpecial(all)
  })

Note that through2.obj(fn) is a convenience wrapper around through2({ objectMode: true }, fn).

API

through2([ options, ] [ transformFunction ] [, flushFunction ])

Consult the stream.Transform documentation for the exact rules of the transformFunction (i.e. this._transform) and the optional flushFunction (i.e. this._flush).

options

The options argument is optional and is passed straight through to stream.Transform. So you can use objectMode:true if you are processing non-binary streams (or just use through2.obj()).

The options argument is first, unlike standard convention, because if I'm passing in an anonymous function then I'd prefer for the options argument to not get lost at the end of the call:

fs.createReadStream('/tmp/important.dat')
  .pipe(through2({ objectMode: true, allowHalfOpen: false }, function (chunk, enc, cb) {

    this.push(new Buffer('wut?'))
    cb()

  })
  .pipe(fs.createWriteStream('/tmp/wut.txt'))

transformFunction

The transformFunction must have the following signature: function (chunk, encoding, callback) {}. A minimal implementation should call the callback function to indicate that the transformation is done, even if that transformation means discarding the chunk.

To queue a new chunk, call this.push(chunk)—this can be called as many times as required before the callback() if you have multiple pieces to send on.

If you do not provide a transformFunction then you will get a simple simple pass-through stream.

flushFunction

The optional flushFunction is provided as the last argument (2nd or 3rd, depending on whether you've supplied options) is called just prior to the stream ending. Can be used to finish up any processing that may be in progress.

through2.ctor([ options, ] transformFunction[, flushFunction ])

Instead of returning a stream.Transform instance, through2.ctor() returns a constructor for a custom Transform. This is useful when you want to use the same transform logic in multiple instances.

var FToC = through2.ctor({objectMode: true}, function (record, encoding, callback) {
  if (record.temp != null && record.unit = "F") {
    record.temp = ( ( record.temp - 32 ) * 5 ) / 9
    record.unit = "C"
  }
  this.push(record)
  callback()
})

// Create instances of FToC like so:
var converter = new FToC()
// Or:
var converter = FToC()
// Or specify/override options when you instantiate, if you prefer:
var converter = FToC({objectMode: true})

See Also

• through2-map - Array.prototype.map analog for streams.
• through2-filter - Array.prototype.filter analog for streams.
• through2-reduce - Array.prototype.reduce analog for streams.
• through2-spy - Wrapper for simple stream.PassThrough spies.

License

through2 is Copyright (c) 2013 Rod Vagg @rvagg and licenced under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.