Transform Hub

Prerequisites

Scramjet's Transform Hub installation requires the following 3 step:

1. Setup a Linux machine (local UNIX/Linux OS, Vitrual Machine)
2. Install Docker on Linux machine
3. Install Node.js on the machine. Currently it is recommended to install Node.js version 18.x LTS.

---

Installation

On a new terminal execute the below command lines for Scramjet's Transform Hub (STH) installation:

• Install Scramjet Transform Hub and STH CLI:

  npm i -g @scramjet/sth @scramjet/cli

• Run Scramjet's Transform Hub:

  scramjet-transform-hub # or sth

More detailed information on how to install Scramjet's Transform Hub can be found at Scramjet's Transform Hub GitHub repository.

---

The Sequence

Scramjet's Sequences are usually the user's source code with all its required dependencies packed and ready to be executed on Scramjet's Transform Hub. With this set of algorithmic steps, a Sequence enhancement through Scramjet's Framework provides the ability to produce, consume and transform data. Scramjet's Sequences allow the user's demand to iterate over data asynchronously. This section illustrates a series of steps that will ensure the execution of your Typescript/JavaScript Sequence successfully on the Scramjet Transform Hub.

For demonstrational purposes, Scramjet's audio2text-input TypeScript sample found on GitHub provides a sneak preview of the prerequisite files a Sequence should contain to be able to be run on STH and, demonstrates the integration of AssemblyAI code for speech recognition in a Scramjet Sequence which can be run by a user through the below command. A user must include their AssemblyAI token once executing the start Sequence command. As a reminder escaping the quotation marks is a must for this command. For more information on si command-lines check CLI reference section.
Prerequisite files of a Sequence written in JavaScript:

• index.js - The main file with the user's logic
• package.json - manifest file that describes this particular Sequence

The audio2text-input was written in TypeScript. Therefore, transpiling is required before being packed and sent to STH as a Sequence. However, it is possible to write code of a Sequence in plain JavaScript or Python. The Templates section illustrates the content of each prerequisite file.

---

Run a Sequence

Five steps to deploy a Sequence on STH:

1. Pack a Sequence into a *.tar.gz package
   Each Sequence must be packaged as a *.tar.gz before sending it to the Transform Hub. A Sequence is a TAR file or an archive, as it is only a collection of multiple files put together inside a single file. And a GZ file which is a compressed file zipped using the gzip algorithm.

   si sequence pack <path/to/folder>

2. Start Scramjet's Transform Hub
   Prior to sending a Sequence to Scramjet's Transform Hub, the STH must run by executing the below command lines on terminal #1. Before running Scramjet's Transform Hub, check whether the environment configuration setting is assigned to development by si config print; If Environment Configuration is for production, use the first of the command lines below.
   

   si c set env development // to switch between the Cloud Platform and the local Transform Hub environment
   info

   Three types of adapters for STH to run with. Determine the adapter while starting STH through --runtime-adapter <adapter> whether docker as the default one if pre-installed , process or kubernetes.

   DEVELOPMENT=true scramjet-transform-hub // or sth

3. Send the Sequence package
   While STH is still running on terminal #1, send the Sequence audio2text.tar.gz or any other Sequence by executing the below command line on terminal #2. A return output with a globally unique identifier number (GUID) assigned to this Sequence shall be used for starting the Sequence as an Instance once or multiple times.

   info

   For debugging the request, use the below config command line.

   si config set log --debug true

   $ si sequence send <path/to/filename.tar.gz>

   Request ok: http://127.0.0.1:8000/api/v1/sequence status: 202 Accepted
   SequenceClient {
     _id: 'xl1200x0-883l-883n-883c-1200v0o1200n',
     host: HostClient { apiBase: 'http://127.0.0.1:8000/api/v1' },
     sequenceURL: 'sequence/xl1200x0-883l-883n-883c-1200v0o1200n'
   }

4. Run the Sequence
   Adding the Sequence-id to the below command line on terminal #2 starts a Sequence while STH is still running on terminal #1. A return output with a globally unique identifier number (GUID) identifies the Instance-id. The hyphen alias (-) assumes the last Sequence-id.

   $ si sequence start <Sequence-id> --args [\"<AssemblyAI-token>\"] # or $ si seq start -

   Request ok: http://127.0.0.1:8000/api/v1/sequence/xl1200x0-883l-883n-883c-1200v0o1200n/start status: 200 OK
   InstanceClient {
     host: HostClient { apiBase: 'http://127.0.0.1:8000/api/v1' },
     _id: '117fxlrs-fxdl-fxbr-vrsc-hd1130vrscdx',
     instanceURL: 'instance/117fxlrs-fxdl-fxbr-vrsc-hd1130vrscdx'
   }

5. Send input data to the Sequence
   An Instance can receive data as input after the start. For the sample audio2text-input, use terminal #2 to send the audio file to the Instance while STH runs on terminal #1.

   si instance input <Instance-id> /path/to/file -e -t application/octet-stream

6. Get the Instance output
   The output command line returns a value from an Instance if assigned. In the case of the audio2text-input sample, once the Instance has completed its task, a return output text is retrieved through the following command line.
   

   si instance output <Instance-id> | tee filename.txt
   warning

   Instance Output is a streaming endpoint where data are consumed when they are read.

---