Quick Start - Framework from scratch

This guide shows how to start using scramjet from scratch - meaning on a bare os. With this guide you'll be able to create and execute your own scramjet pipeline on any server or computer

Step 1: Install prerequisites

Scramjet is based on node.js, so we'll need to install that before scramjet - the easiest way to do that is using nvm so first install that:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
source ~/.bashrc # you can restart your terminal as well.

nvm is a cool little tool that allows you to automatically install node.js on a machine - all you need is ssh access, not even root access is needed.

There's a windows version of nvm. It has some small differences, but I hope this guide does work with it too. If not - please report an issue.

Next we need to install the current version of node, or any you like - scramjet is compatible with all supported node versions (currently 8, 10, 12).

nvm install node # instead of node you can set this to any version you like, 8.16.0 for instance.

And now you'll have node up and running in order to confirm that let's try:

node -v
# v12.8.0

Step 2: create a node project

There's a lot of IDE's out there - I'd recommend VSCode and GitHub's Atom. There's some who'd prefer Webstorm or Sublime and those who can actually exit vim without restarting the whole server - in the end you will need a good text editor and who am I to judge? Choose your weapon of choice and let's now start cracking:

First let's create a directory and enter it (I'll use command line, but if you're using the IDE's browser of Nautilus that's fine too):

mkdir ~/src/my-workflow/
cd ~/src/my-workflow/

That done we need to create a node package. This is not a necessary step, but it'll allow us to share our work with other later.

npm init # now we'll be asked a couple questions - about name etc. It's ok to just keep pressing enter.

After that you'll see a new file called package.json that will hold out package data. Now let's install scramjet:

npm install --save scramjet # this installs and adds scramjet to package.json so others know it's needed to run

Now after that's done we can create a simple scramjet workflow in a javascript file - for instance index.js - for example lets download a list of files fetched from an external server:

#!/usr/bin/env node
// the first line above allows us to execute this file

const { StringStream } = require("scramjet");
const { get: get_ } = require("https");
const { createWriteStream } = require("https");

// here's a simple wrapper to fetch stream from any server - you can use this or axios, request or node-fetch.
const get = (url, options = {}) =>
  new Promise((resolve) => get(url, options, resolve));

StringStream.from(async () => get("https://www.some.server.net/some.file.txt"))
  .lines()
  .parse((x) => x.split("\t"))
  .map(async ([url, data, name]) => [await get(`${url}?${data}`), name])
  .each(([name]) => console.log(`downloading ${name}...`))
  .each(
    ([data, name]) =>
      new Promise((resolve, reject) =>
        createWriteStream(name).on("finish", resolve).on("error", reject)
      )
  )
  // the cool thing here is that you're writing a couple files at the same time
  // see the docs (www.scramjet.org) for `maxParallel`.
  .each(([name]) => console.log(`downloaded ${name}.`))
  .run()
  .catch((e) => {
    console.error(e.stack);
    throw e;
  });

Now let's allow to execute the file (you don't need to do that on Windows) by running chmod +x index.js and now we can run this:

./index.js