Application Protocol Interface (API) Reference

Introduction
Space operations
Hub operations
Sequence operations
Instance operations
Topics operation on data

Introduction

Scramjet Cloud Platform API supports the following methods:

[ GET ] - all operations requesting data from the Scramjet Cloud Platform (SCP),
[ POST ] - creating new Hub, Sequence, Sequence Instances or for messaging and streaming data to a Sequence,
[ DELETE ] - removeing operations eg. Hub or Sequence.

API Base URL examples

# Self Hosted Hub

http://127.0.0.1:11000/api/v1/

# Scramjet Cloud Platform Space

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1

# Scramjet Cloud Platform Hub

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/sth/:id/api/v1/

To communicate with the Cloud Platform API, please remember to provide authorization Bearer Token in Headers.

Example cURL

curl -s -k -X GET https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/list -H 'accept: */*' -H 'authorization: Bearer {Token}' -H 'content-type: application/json'

No token provided or wrong token response

{
  "message": "Forbidden"
}

Space operations

API Base SCP Space URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1

[ GET ] {API Base}/version - show space version

Parameters

No parameters

Responses

Status: 200 - Success

{
  "service": "@scramjet/manager",
  "apiVersion": "v1",
  "version": "0.22.0",
  "build": "8b39d26"
}

[ GET ] {API Base}/load - show space load info

Parameters

No parameters

Responses

Status: 200 - Success

{
  "avgLoad": 0.15,
  "currentLoad": 10.34503968983302,
  "memFree": 27063754752,
  "memUsed": 16267390976,
  "fsSize": [
    {
      "fs": "/dev/mapper/ubuntu--vg-ubuntu--lv",
      "type": "ext4",
      "size": 51716325376,
      "used": 11687919616,
      "available": 37864574976,
      "use": 23.59,
      "mount": "/etc/hosts"
    }
  ]
}

[ GET ] {API Base}/list - list space Hubs

Parameters

No parameters

Responses

Status: 200 - Success

[
  {
    "id": "sth-6cd78f999b-ntdtw",
    "info": {
      "created": "2022-04-07T11:42:59.185Z",
      "lastConnected": "2022-04-07T11:42:59.187Z"
    },
    "healthy": true,
    "isConnectionActive": true
  }
]

[ GET ] {API Base}/sth/:id/info - show the Hub info

Parameters

No parameters

Responses

Status: 200 - Success

{
  "id": "sth-0",
  "info": {
    "created": "2022-07-08T08:54:53.394Z",
    "lastConnected": "2022-07-18T15:10:37.567Z",
    "lastDisconnected": "2022-07-18T15:10:37.323Z"
  },
  "healthy": true,
  "isConnectionActive": true
}

Hub operations

API Base SCP Hub URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/sth/:id/api/v1/

[ GET ] {API Base}/version - show the Hub version

Parameters

No parameters

Responses

Status: 200 - Success

{
  "service": "@scramjet/host",
  "apiVersion": "v1",
  "version": "0.24.2",
  "build": "source"
}

[ GET ] {API Base}/config - show the Hub config

Parameters

No parameters

Responses

Status: 200 - Success

{
  "logLevel": "trace",
  "logColors": true,
  "cpmUrl": "mm-api.org-5fa96f7c-058h-9o59-8645-y7d4b0af2717.svc.cluster.local:11000",
  "cpmId": "org-5fa96f7c-058h-9o59-8645-y7d4b0af2717-manager",
  "docker": {
    "prerunner": {
      "image": "scramjetorg/pre-runner:0.24.2",
      "maxMem": 128
    },
    "runner": {
      "image": "",
      "maxMem": 512,
      "exposePortsRange": [30000, 32767],
      "hostIp": "0.0.0.0"
    },
    "runnerImages": {
      "python3": "scramjetorg/runner-py:0.24.2",
      "node": "scramjetorg/runner:0.24.2"
    }
  },
  "identifyExisting": false,
  "host": {
    "hostname": "0.0.0.0",
    "port": 8000,
    "apiBase": "/api/v1",
    "instancesServerPort": 8001,
    "infoFilePath": "/tmp/sth-id.json",
    "id": "sth-0"
  },
  "instanceRequirements": {
    "freeMem": 256,
    "cpuLoad": 10,
    "freeSpace": 128
  },
  "safeOperationLimit": 512,
  "instanceAdapterExitDelay": 9000,
  "runtimeAdapter": "kubernetes",
  "startupConfig": "",
  "exitWithLastInstance": false,
  "heartBeatInterval": 10000,
  "kubernetes": {
    "namespace": "org-5fa96f7c-058h-9o59-8645-y7d4b0af2717",
    "sthPodHost": "sth-0.msth.org-5fa96f7c-058h-9o59-8645-y7d4b0af2717.svc.cluster.local",
    "runnerImages": {
      "python3": "scramjetorg/runner-py:0.24.2",
      "node": "scramjetorg/runner:0.24.2"
    },
    "timeout": "0",
    "runnerResourcesRequestsCpu": "0.01",
    "runnerResourcesRequestsMemory": "8M",
    "runnerResourcesLimitsCpu": "0.565",
    "runnerResourcesLimitsMemory": "1024MB"
  }
}

[ GET ] {API Base}/load-check - monitor CPU, memory and disk usage metrics on the Hub machine

Parameters

No parameters

Responses

Status: 200 - Success

{
  "avgLoad": 0.04,
  "currentLoad": 2.34338747099768,
  "memFree": 26667257856,
  "memUsed": 16138383360,
  "fsSize": [
    {
      "fs": "/dev/mapper/ubuntu--vg-ubuntu--lv",
      "type": "ext4",
      "size": 51716325376,
      "used": 11699834880,
      "available": 37852659712,
      "use": 23.61,
      "mount": "/etc/hosts"
    }
  ]
}

[ GET ] {API Base}/log - show logs form Hub

Parameters

No parameters

Responses

Content-type: application/octet-stream

Status: 200 - Success

2021-11-19T16:04:47.094Z log (object:Hub) Hub main called.
2021-11-19T16:04:47.100Z info (object:SocketServer) Server on: /tmp/scramjet-socket-server-path
2021-11-19T16:04:47.104Z info (object:Hub) API listening on: 127.0.0.1:8000
2021-11-19T16:05:08.228Z info (object:Hub) New sequence incoming...
2021-11-19T16:05:08.229Z log (object:LifecycleDockerAdapterSequence) Docker sequence adapter init.
2021-11-19T16:05:08.229Z log (object:DockerodeDockerHelper) Checking image scramjetorg/pre-runner:0.12.2
2021-11-19T16:05:12.234Z info (object:LifecycleDockerAdapterSequence) Docker sequence adapter done.
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Volume created. Id:  c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Starting PreRunner { image: 'scramjetorg/pre-runner:0.12.2', maxMem: 128 }
2021-11-19T16:05:13.536Z log (object:DockerodeDockerHelper) Checking image scramjetorg/runner:0.12.2
2021-11-19T16:05:16.670Z info (object:SequenceStore) New sequence added: c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:16.672Z info (object:Hub) Sequence identified: {
  container: {
    image: 'scramjetorg/runner:0.12.2',
    maxMem: 512,
    exposePortsRange: [ 30000, 32767 ],
    hostIp: '0.0.0.0'
  },
  name: '@scramjet/multi-outputs',
  version: '0.12.2',
  engines: {},
  config: {},
  sequencePath: 'index.js',
  packageVolumeId: 'c50fe4d3-89cc-4685-a82a-16cbc744733d'
}
2021-11-19T16:05:16.691Z debug (object:Hub) Request date: 2021-11-19T16:05:08.239Z, method: POST, url: {API Base}/sequence, status: 202

Status 404 Not Found

Sequence operations

API Base SCP Space URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1

[ GET ] {API Base}/sequences - show all Sequences on Space

Parameters

No parameters

Responses

Status: 200 - Success

["a54675bd-fed8-4b67-9c3e-47c3302083a7", "00539df4-d0e2-4190-8ea9-52a2a22c3122"]

API Base SCP Hub URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/sth/:id/api/v1/

[ GET ] {API Base}/sequences - show all Sequences saved on the Hub

Parameters

No parameters

Responses

Status: 200 - Success

[
  {
    "id": "a54675bd-fed8-4b67-9c3e-47c3302083a7",
    "config": {
      "type": "kubernetes",
      "entrypointPath": "index.js",
      "version": "0.13.1",
      "name": "@scramjet/simple-counter-js",
      "id": "a54675bd-fed8-4b67-9c3e-47c3302083a7",
      "sequenceDir": "/root/.scramjet_k8s_sequences/a54675bd-fed8-4b67-9c3e-47c3302083a7",
      "engines": {
        "node": ">=14"
      }
    },
    "instances": []
  },
  {
    "id": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
    "config": {
      "type": "kubernetes",
      "entrypointPath": "./index",
      "version": "0.13.1",
      "name": "@scramjet/hello",
      "id": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
      "sequenceDir": "/root/.scramjet_k8s_sequences/00539df4-d0e2-4190-8ea9-52a2a22c3122",
      "engines": {}
    },
    "instances": ["bde7e8a1-0aa6-41ae-b922-f29eaf7dbe26"]
  }
]

[ POST ] {API Base}/sequence - add new Sequence

Parameters

Name	Description	Type	Required
file	compressed package in tar.gz format	binary	yes
appConfig	additional package.json config file	json	no

Responses

Status: 202 - accepted

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0" // sequence id
}

[ POST ] {API Base}/sequence/:id/start - start chosen Sequence

Parameters

Name	Description	Type	Required
appConfig	additional package.json config file	json	no
args	additional arguments that instance should starts with	json	no

Responses

Status: 200 - Success

{
  "result": "success",
  "opStatus": "OK",
  "id": "c9803353-f44e-4e75-8b2e-11a8b07924c7"
}

[ DELETE ] {API Base}/sequence/:id - delete a Sequence by id

Parameters

No parameters

Responses

Status: 200 - Success

{
  "opStatus": "OK",
  "id": "a54675bd-fed8-4b67-9c3e-47c3302083a7"
}

Status 409 - conflict - the instance is still running

{
  "error": "Can't remove sequence in use."
}

Instance operations

API Base SCP Space URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1

[ GET ] {API Base}/instances - show list of Instances on Space

Parameters

No parameters

Responses

Status: 200 - Success

[
  [
    ["66e0e68f-7ad8-4c90-9233-23748c5f2445"],
    ["bde7e8a1-0aa6-41ae-b922-f29eaf7dbe26"]
  ]
]

API Base SCP Hub URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/sth/:id/api/v1/

[ GET ] {API Base}/instances - show list of Instances running on the Hub

Parameters

No parameters

Responses

Status: 200 - Success

[
  {
    "id": "bde7e8a1-0aa6-41ae-b922-f29eaf7dbe26",
    "appConfig": {},
    "sequence": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
    "created": "2022-07-19T21:07:11.043Z",
    "started": "2022-07-19T21:07:13.668Z",
    "status": "running"
  },
  {
    "id": "217ff6c1-2054-46e4-8f69-d5151cd313a7",
    "appConfig": {},
    "sequence": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
    "created": "2022-07-19T22:08:46.577Z",
    "started": "2022-07-19T22:08:49.121Z",
    "status": "running"
  },
  {
    "id": "fe314445-ccd7-41f5-bff4-37163cd21940",
    "appConfig": {},
    "sequence": "a54675bd-fed8-4b67-9c3e-47c3302083a7",
    "created": "2022-07-19T22:09:36.852Z",
    "started": "2022-07-19T22:09:39.444Z",
    "status": "running"
  }
]

[ GET ] {API Base}/instance/:id - show data of chosen Instance

Parameters

No parameters

Responses

Status 200 Accepted

{
  "id": "bde7e8a1-0aa6-41ae-b922-f29eaf7dbe26",
  "appConfig": {},
  "sequence": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
  "created": "2022-07-19T21:07:11.043Z",
  "started": "2022-07-19T21:07:13.668Z",
  "status": "running"
}

Status 404 Not Found - when the Instance is not found, for example: the Instance was already stopped.

[ POST ] {API Base}/instance/:id/_stop - end instance gracefully and prolong operations or not for task completion

Parameters

Name	Description	Type	Required
timeout	The number of milliseconds before the Instance will be killed. Default: 7000ms.	number	no
canCallKeepalive	If set to true, the instance will prolong the running. Default: false.	boolean	no

Responses

Status: 200 - Success

{
  "code": 0,
  "type": "string",
  "message": "string"
}

[ POST ] {API Base}/instance/:id/_kill - end instance gracefully waiting for unfinished tasks

Parameters

No parameters

Responses

Status: 202 - accepted

{
  "opStatus": "Accepted",
  "id": "bde7e8a1-0aa6-41ae-b922-f29eaf7dbe26",
  "appConfig": {},
  "sequence": "00539df4-d0e2-4190-8ea9-52a2a22c3122",
  "created": "2022-07-19T21:07:11.043Z",
  "started": "2022-07-19T21:07:13.668Z",
  "status": "killing"
}

[ GET ] {API Base}/instance/:id/health - check status about Instance health

Parameters

No parameters

Responses

Status: 200 - Success

{
  "healthy": true
}

Events

Cloud Platform allows interaction (communication) with an instance by sending events with or without any information defined in the object. Furthermore, it lets providing and receiving streams with all kinds of data files.

Event contains <eventName>, <handler> with optional <message> of any type: string, num, json obj, array, etc..

[ POST ] {API Base}/instance/:id/_event - send event to the Instance

Parameters

Name	Type	Description	Required
`eventName`	`string`	Name of an event	true
`message`	`string`	JSON formatted event payload	false

Responses

Content-type: application/octet-stream

[ GET ] {API Base}/instance/:id/event - get the data stream with the events from the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] {API Base}/instance/:id/once - get the last event sent by the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

Input/Output (I/O)

The data can be sent to the input stream of the Instance where it can be consumed. Everything the Instance writes to its output stream also can be consumed through the endpoint.

[ POST ] {API Base}/instance/:id/input - send data to the Instance

Parameters

No parameters

Responses

Status: 200 - Success

Status: 406 - Not Acceptable - with an error message "Input provided in other way." This status code is returned when the Instance expects the input to be provided from the Topic API.

[ GET ] {API Base}/instance/:id/output - get the output stream from the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] {API Base}/instance/:id/stdin - process.stdin

Parameters

No parameters

Responses

Status: 200 - Success

[ GET ] {API Base}/instance/:id/stdout - process.stdout

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] {API Base}/instance/:id/stderr - process.stderr

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] {API Base}/instance/:id/log - stream all instance logs

Parameters

No parameters

Responses

Content-type: application/octet-stream

Status: 200 - Success

2021-11-19T16:12:22.948Z log (Sequence) 42
2021-11-19T16:12:23.949Z log (Sequence) 41
2021-11-19T16:12:24.950Z log (Sequence) 40
2021-11-19T16:12:25.951Z log (Sequence) 39
2021-11-19T16:12:26.952Z log (Sequence) 38
2021-11-19T16:12:27.952Z log (Sequence) 37
2021-11-19T16:12:28.953Z log (Sequence) 36
2021-11-19T16:12:29.953Z log (Sequence) 35

Status 404 Not Found

Topics operation on data

API Base SCP Hub URL

https://api.beta.scramjet.cloud/api/v1/space/:id/api/v1/sth/:id/api/v1/

Type of Headers

"x-end-stream" - close topic stream [optional, boolean]. If x-end-stream header value is true, the topic stream is closed after processing this request. The default value is false.
"content-type" - specify stream content type [optional, boolean]. The content-type header specifies data type of this topic. The recognized values are: text/x-ndjson, application/x-ndjson, application/x-ndjson, text/plain, application/octet-stream. The default value is application/x-ndjson.

[ POST ] {API Base}/topic/:name - sends data to the Topic, if it does not exist, the Topic is created

Parameters

No parameters

Request Headers

Content-type: application/x-ndjson

Responses

Status: 200 - Success - when data to topic is sent with the header indicating the end of data Status: 202 - accepted - when data to topic is sent without the header indicating the end of data (default)

[ GET ] {API Base}/topic/:name - get data from the Topic

Parameters

No parameters

Responses

Topic data stream.

Status: 200 - Success

{
  "source": "Twitter",
  "id": "850006245121695778",
  "content": "Natural wetlands make up ~30% of global total CH4 emissions",
  "user": {
    "id": 1234994945,
    "name": "Climate Change Conference",
    "screen_name": "Climate Change"
  }
}

[ GET ] {API Base}/topis - list Hub Topics

Parameters

No parameters

Responses

Status: 200 - Success

[
  {
    "contentType": "application/x-ndjson",
    "stream": {
      "_readableState": {
        "objectMode": false,
        "highWaterMark": 16384,
        "buffer": {
          "head": {
            "data": {
              "type": "Buffer",
              "data": [
                123,
                10,
                32,
                32,
                32
                //..
              ]
            },
            "next": {
              "data": {
                "type": "Buffer",
                "data": [10]
              },
              "next": null
            }
          },
          "tail": {
            "data": {
              "type": "Buffer",
              "data": [10]
            },
            "next": null
          },
          "length": 2
        },
        "length": 51,
        "pipes": [],
        "flowing": null,
        "ended": false,
        "endEmitted": false,
        "reading": true,
        "constructed": true,
        "sync": false,
        "needReadable": true,
        "emittedReadable": false,
        "readableListening": false,
        "resumeScheduled": false,
        "errorEmitted": false,
        "emitClose": true,
        "autoDestroy": true,
        "destroyed": false,
        "errored": null,
        "closed": false,
        "closeEmitted": false,
        "defaultEncoding": "utf8",
        "awaitDrainWriters": null,
        "multiAwaitDrain": false,
        "readingMore": false,
        "dataEmitted": false,
        "decoder": null,
        "encoding": null
      },
      "_events": {},
      "_eventsCount": 1,
      "_writableState": {
        "objectMode": false,
        "highWaterMark": 16384,
        "finalCalled": false,
        "needDrain": false,
        "ending": false,
        "ended": false,
        "finished": false,
        "destroyed": false,
        "decodeStrings": true,
        "defaultEncoding": "utf8",
        "length": 0,
        "writing": false,
        "corked": 0,
        "sync": false,
        "bufferProcessing": false,
        "writecb": null,
        "writelen": 0,
        "afterWriteTickInfo": null,
        "buffered": [],
        "bufferedIndex": 0,
        "allBuffers": true,
        "allNoop": true,
        "pendingcb": 0,
        "constructed": true,
        "prefinished": false,
        "errorEmitted": false,
        "emitClose": true,
        "autoDestroy": true,
        "errored": null,
        "closed": false,
        "closeEmitted": false
      },
      "allowHalfOpen": true
    },
    "localProvider": "api",
    "topic": "currency"
  }
]