Scramjet Framework Modules

A Scramjet Framework module is a simple module that exposes a single function that performs some operations with the stream. The API is designed so that these modules can be easily used on any stream you like without needing to use scramjet.

Using Framework modules

Using a scramjet module is fairly simple, first install it as a dependency:

$ npm i -s scramjet-module

Next simply use it in your code:

DataStream.from("scramjet-module/hello-read") // Loads and sources the stream from the `hello-read`
  // scramjet module of `scramjet-module` package
  .use("scramjet-module/hello-transform"); // This loads and uses `hello-transform` from the same package.

Scramjet modules are resolved automatically in relation to the calling file, just like node.js require method.

Incidentally scramjet is not needed as a peer to use a scramjet module. The API is thought in a way so it will be fully interoperable with any stream you like, so you can simply use them with another stream and they always return a standard Node.js readable stream. This means you can use it with any stream - maybe gulp?

import { helloRead } from "scramjet-module";

helloRead(gulp.src(somePath)).pipe(gulp.target(targetPath));

Another option

Writing a sample read module

Here's how a minimal read module may look like:

import {StringStream, createReadModule} from "scramjet";

export default const ndjsonStream = createReadModule(
    (file) => StringStream
        .from(fs.createReadStream(file))
        .split(/\r?\n/)
        .parse(x => JSON.parse(x))
);

Or in commonjs:

const { StringStream, createReadModule } = require("scramjet");

module.exports = createReadModule((file) =>
  StringStream.from(fs.createReadStream(file))
    .split(/\r?\n/)
    .parse((x) => JSON.parse(x))
);

Here's how it would be used:

import readFile from "scramjet-read-ndjson";

readFile("./data.ndjson").each((x) => console.log(x.id));

Or even easier with DataStream.from:

DataStream.from("scramjet-read-ndjson").each((x) => console.log(x.id));

Usage with CLI

Scramjet provides a simple CLI mechanism that allows chaining number of modules one after another without writing any code.

$ npm i -g scramjet-cli
$ scramjet -i 'scramjet-hello/read' 'scramjet-hello/transform1' [...] -o './file'

You can see more information at Scramjet CLI.

A sample transform module

Here's how a minimal transform module may look like:

import {StringStream, createTransformModule} from "scramjet";

export default const ndjsonStream = createTransformModule(
    (stream, hello = "Hello, {}!") => stream
        .stringify(name => hello.replace('{}', name.toString()))
);

Or in commonjs:

const { StringStream, createTransformModule } = require("scramjet");

module.exports = createTransformModule((stream, hello = "Hello, {}!") =>
  stream.stringify((name) => hello.replace("{}", name.toString()))
);

Publishing scramjet modules

In general scramjet modules can be published as standard npmjs packages.

If your package contains multiple modules, you may want to set meaningful names so that when your users require the package they'll know what they are referring to:

import commonlogTransform from "scramjet-common-log/transform";
import svgRead from "scramjet-csv/read";