Sequence Lifecycle

There are four major steps in the Sequence lifecycle in the Scramjet Cloud Platform

Sequence Source code - your code in development form as text files
Sequence Package - the built program compiled with it's dependencies in one file
Sequence, The - the program sent to the Platform and ready for execution
Sequence Instance - an executed program while it's running

This document describes the process and their internal stages.

Source code

In this phase the program is being written by the developer and may contain additional files or items that you need for development. The minimal source code would consist of:

A source file (js for instance):
module.exports = async function () { // do stuff here. };
A basic configuration file - package.json - this always needs to be defined, no matter the language.
{ "name": "@example/my-sequence", "version": "0.1.1", "main": "./index.js" }

And that's it.

You may consider using build scripts to create a structure as above and remove the unneeded development files. Task runners like Yarn, Gulp or similar are quite good choices.

Sequence Package

A Sequence Package is a tar.gz file that contains:

The above source code or a compiled version of your program
Additional files needed by the program to run (dependencies)
The definition in package.json.

These items put together in a tarball will allow all needed files to be sent to the platform. Once you have a directory with contents as mentioned above you can just create a tarball, but if you don't remember the commandline and/or would prefer to have Scramjet to run some basic checks, we've prepared a command to get you to the next stage in lifecycle:

si seq pack <directory> [-o <output.tar.gz>]

Important: this won't install your dependencies, so make sure you do pip install -r requirements.txt or npm i wherever it might be needed.

The Sequence

A package becomes a sequence via the process of <drumroll> uploading it to the platform. It's that easy:

si seq send <file.tar.gz>|-

If you ommit the filename and instead add the - as shown above, the last sequence packed with si will be sent.

All the sequences sent to Scramjet Cloud Platform can be listed, removed and most importantly invoked to become an Instance

Sequence Instance

When the sequence is started it becomes an instance - you can think of this like a program window, you can start as much as you want. You can start as many instances as you like within the limits of your platform Space or your Self-Hosted Hub.

In order to start your instance you can use the API, the web Console or this simple CLI command:

si seq start <id>|- [...arguments]

You can pass different argument every time you start a new Sequence Instance and use those in your implementation. This means within a sequence like this:

module.export => async function* (initialNumber = 0, length = 1000) {
  for (let i = initialNumber; i < initialNumber + length; i++)
    yield i;
}

The output of the sequence can be controlled by the arguments, so using this command:

si seq start <id> 3000 10
si inst output -

You will see the output like this:

What happens when the sequence ends

When the sequence ends it's process gets removed along with any parts the sequence left behind.

The output, logs and stdio will be accessible for a short time period for inspection.

Javascript Guide

Packaging a sequence is literally just compressing it into a tar.gz file. If your code doesn't use any dependencies you can simply run tar czf /path/to/package.tar.gz . and you're set. We suggest that you use the si command for packing, it'll do some extra checks to make sure that your package contains the needed files:

si seq pack

Currently supported runners, node.js and python, will run your program based on the main entry in package.json - this is clarified below.

What Sequence Package should contains to operate properly?

index.js / index.py - the main file with user's program logic,
package.json - manifest file that describes this particular Sequence,
dependencies* - preinstalled all dependencies that are needed to execute the Sequence,
any additional files your code needs (data files, text files, libs and binaries** etc.)

Dependencies such as node_modules in case of JavaScript or __pypackages__ in Python package.

Be aware that native binary may not work well with different architectures, we'll try to provide a solution for this in future releases.

The code

In order for a sequence to work correctly your code needs to expose a function that Scramjet Transform Hub or Scramjet Cloud Platform will execute. The function can do anything you want, you can also import any libraries or even perform computations as long as the function is exposed.

The signature of the function is as follows:

module.exports = async function (input, ...args) {
    return new Promise();
};

The exact type definition can be read here

When the program is started by issuing the si seq start <id> arg1 arg2 command, the function will get invoked with arguments from command line (you can do that via API as well). Mind you, over CLI all arguments will be strings.

Package.json file

All Scramjet Sequences need a package.json - it's a small file typically used by node.js, but since it's a standard format we currently require you to add this json to the tar.gz packed file.

Example: JavaScript

    {
        "name": "package-name",
        "version": "0.0.1",
        "main": "index",
        "dependencies": {
            "package": "0.0.1"
        },
        "engines": {
            "node": "^16.0.0"
        },
        "config": {
            "arg": "value"
        }
    }

Dependencies

Scramjet Sequences are self-contained programs - meaning that in order to run it we need to receive all the code including the dependencies in the packaged tar.gz file. Typically this will mean installing your dependencies in the directory and packing them in the tar.gz file at the same time.

In node.js you need to add your dependencies in package.json in the dependencies tree and then to download and install them run the following command:

npm install

Code examples

JavaScript – it looks trivial!

Data Consumer
module.exports = async function (input) { for await (const id of input) { yield await saveEntryToDB(id); } };
Data Producer
module.exports = async function* (_stream) { while (true) { yield* await getPageFromAPI(); await wait(1000); } };
Data Transformer
module.exports = async function* (input) { for await (const id of input) { yield await getDBEntryForId(id); } };
Data Operator
module.exports = async function (_stream) { while (true) { await callSimpleAPI(); await wait(1000); } };

TypeScript Development Guide

si seq pack

Currently supported runners, node.js and python, will run your program based on the main entry in package.json - this is clarified below.

What Sequence Package should contains to operate properly?

index.js - the main file with user's program logic,
package.json - manifest file that describes this particular Sequence,
dependencies* - preinstalled all dependencies that are needed to execute the Sequence,
any additional files your code needs (data files, text files, libs and binaries** etc.)

Dependencies such as node_modules in case of JavaScript or __pypackages__ in Python package.

Be aware that native binary may not work well with different architectures, we'll try to provide a solution for this in future releases.

The code

The signature of the function is as follows:

module.exports = async function (input, ...args) {
    return new Promise();
};

Package.json file

All Scramjet Sequences need a package.json - it's a small file typically used by node.js, but since it's a standard format we currently require you to add this json to the tar.gz packed file.

Example: TypeScript

    {
        "name": "package-name",
        "version": "0.0.1",
        "main": "index",
        "dependencies": {
            "package": "0.0.1"
        },
        "engines": {
            "node": "^16.0.0"
        },
        "config": {
            "arg": "value"
        }
    }

Dependencies

In node.js you need to add your dependencies in package.json in the dependencies tree and then to download and install them run the following command:

npm install

Code examples

Python Guide

si seq pack

Currently supported runners, node.js and python, will run your program based on the main entry in package.json - this is clarified below.

What Sequence Package should contains to operate properly?

index.py - the main file with user's program logic,
package.json - manifest file that describes this particular Sequence,
dependencies* - preinstalled all dependencies that are needed to execute the Sequence,
any additional files your code needs (data files, text files, libs and binaries** etc.)

Dependencies such as node_modules in case of JavaScript or __pypackages__ in Python package.

Be aware that native binary may not work well with different architectures, we'll try to provide a solution for this in future releases.

The code

The signature of the function is as follows:

def run(context, input, *args):
    return Stream

Package.json file

All Scramjet Sequences need a package.json - it's a small file typically used by node.js, but since it's a standard format we currently require you to add this json to the tar.gz packed file.

Example: Python

    {
        "name": "package-name",
        "version": "0.0.1",
        "main": "./index.py",
        "dependencies": {
            "package": "0.0.1"
        },
        "assets": [
            "data.json"
        ],
        "engines": {
            "python3": "3.9.0"
        }
    }

Dependencies

In python you typically create a file called requirements.txt and list the dependencies there. Then you need to install them so that they'll be visible to your program:

pip3 install -r requirements.txt

For convienience the program is started with a __pypackages__ dir on PYTHON_PATH, so you can install your dependencies in there for clearer contents of the package. Then the command line will be:

pip3 install -t __pypackages__/ -r requirements.txt

Code examples

Data Consumer
async def run(context, input): async for id in input: await save_entry_to_db(id)
Data Producer
async def run(context, input): while True: async for result in await get_page_from_api(): yield result await asyncio.sleep(1)
Data Transformer
async def run(context, input): async for id in input: yield await get_db_entry_for_id(id)
Data Operator
async def run(context, input): while True: await call_simple_api() await asyncio.sleep(1)

Sequence Lifecycle

Source code

Sequence Package

The Sequence

Sequence Instance

What happens when the sequence ends

Javascript Guide

The code

Package.json file

Dependencies

Code examples

JavaScript – it looks trivial!

TypeScript Development Guide

The code

Package.json file

Dependencies

Code examples

Python Guide

The code

Package.json file

Dependencies

Code examples

Was it helpful?

Didn't find information needed?

Sequence Lifecycle

Source code

Sequence Package

The Sequence

Sequence Instance

What happens when the sequence ends

Javascript Guide

The code

Package.json file

Dependencies

Code examples

JavaScript – it looks trivial!​

TypeScript Development Guide

The code

Package.json file

Dependencies

Code examples

Python Guide

The code

Package.json file

Dependencies

Code examples

Was it helpful?

Didn't find information needed?

JavaScript – it looks trivial!