About Scramjet Framework

Version 4

What does it do?

Scramjet is a fast, simple, functional reactive stream programming framework written on top of node.js object streams. The code is written by chaining functions that transform the streamed data, including well known map, filter and reduce and fully compatible with ES7 async/await. Thanks to it some built in optimizations scramjet is much faster and much much simpler than similar frameworks when using asynchronous operations.

The main advantage of scramjet is running asynchronous operations on your data streams. First of all it allows you to perform the transformations both synchronously and asynchronously by using the same API - so now you can "map" your stream from whatever source and call any number of API's consecutively. And if you're after some heavy maths there's an option of running your stream as multi-threaded!

Example

How about a full API to API migration, reading a long list of items from one API and checking them one after another, pushing them to another API? With simultaneous request control? And outputting the log of the conversion? Easy!

const fetch = require("node-fetch");
const get = async (url, options = {}) => (await fetch(url, options)).json;
const { StringStream } = require("scramjet");

StringStream.from(
  // fetch your API to a scramjet stream
  () => get("https://api.example.org/v1/shows/list")
)
  .setOptions({ maxParallel: 4 }) // set your options
  .lines() // split the stream by line
  .parse((line) => {
    // parse strings to data
    const [id, title, url] = line.split(",");
    return { id, title, url };
  })
  .map(async (myShow) =>
    get({
      // use asynchronous mapping (for example send requests)
      uri: `http://api.local/set/${myShow.id}`,
      body: JSON.stringify(myShow),
    })
  )
  .stringify((resp) => `+ Updated "${resp}"`)
  .catch((err) => `! Error occured ${err.uri}`) // handle errors
  .append("\n")
  .pipe(process.stdout); // use any stream

Here you can find a most basic guide on how to execute the above example starting from just having access to some command line: Scramjet from Scratch

Execution and deployment

You can now run stream processing programs with our Scramjet Transform Hub. It will allow you to deploy and execute programs on local and remote environments of your choice and it's as easy as:

npm i -g @scramjet/sth @scramjet/cli
scramjet-transform-hub &
si run <path-to-your-program-dir>

See more info:

• On our website Scramjet.org,
• The documentation repository Scramjetorg/scramjet-cloud-docs
• The source code and development guide Scramjetorg/transform-hub

Scramjet Framework core

Don't like dependencies? Scramjet packs just a couple of those, but if you are really really annoyed by second depth of deps, please try Scramjet-core.

Only the most vital methods there, but the library is dependency free.

Scramjet Framework pre-release

Scramjet is a simple reactive stream programming framework. The code is written by chaining functions that transform the streamed data, including well known map, filter and reduce (in JS and Python) or each (in C++).

The main advantage of Scramjet is running asynchronous operations on your data streams concurrently. It allows you to perform the transformations both synchronously and asynchronously by using the same API - so now you can "map" your stream from whatever source and call any number of API's consecutively.

This is a pre-release of the next major version (v5) of JavaScript Scramjet Framework.

Originally written on top of node.js object streams, Scramjet is now being ported into cpp. This is what is happening in this repository. There is also JavaScript/TypeScript and Python version available.

We are open to your feedback! We encourage you to report issues with any ideas, suggestions and features you would like to see in this version. You can also upvote (+1) existing ones to show us the direction we should take in developing Scramjet Framework.

License and contributions

As of version 2.0 Scramjet is MIT Licensed.

Help wanted

The project need's your help! There's lots of work to do - transforming and muxing, joining and splitting, browserifying, modularizing, documenting and issuing those issues. If you want to help and be part of the Scramjet team, please reach out to us on discord or email us.

Donation

Do you like this project? It helped you to reduce time spent on delivering your solution? Support us with at Github Sponsors