mediawiki

Sequence that keeps printing mediawiki event stream.

Description

  • Name: @scramjet/mediawiki
  • Version: 0.23.0
  • Language: typescript
  • Author: iaforek
  • Tags: easy,MediaWiki,Wikipedia,Big Data,data operator

Running

❗ Remember to setup transform-hub locally or use the platform's environment for the sequence deployment.

Open the terminal and run the following commands:

# go to 'mediawiki' directory cd typescript/mediawiki # install dependencies npm install # transpile TS->JS to dist/ npm run build # deploy the Sequence from the dist/ directory, which contains transpiled code, package.json and node_modules si seq deploy dist # See output si inst output - # Check console.log messages si inst stdout - # Check console.error messages si inst stderr - # Send event, e.g. `drain`: `si inst emit <instance-id> drain "{}"` si inst event emit - <event> <payload>

💡NOTE: Command deploy performs three actions at once: pack, send and start the Sequence. It is the same as if you would run those three commands separately:

si seq pack dist/ -o mediawiki.tar.gz # compress 'dist/' directory into file named 'mediawiki.tar.gz' si seq send mediawiki.tar.gz # send packed Sequence to STH, this will output Sequence ID si seq start - --args [\"<search>\"] # start the Sequence, this will output Instance ID. Search is optional and can be used to filter out results, e.g. "data.server_name === 'en.wikipedia.org'"

Example Event

ID

[ { "topic":"eqiad.mediawiki.recentchange", "partition":0, "timestamp":1631710626001 }, { "topic":"codfw.mediawiki.recentchange", "partition":0, "offset":-1 } ]

DATA

{ "$schema":"/mediawiki/recentchange/1.0.0", "meta":{ "uri":"https://en.wikipedia.org/wiki/Bernie_Ecclestone", "request_id":"41ba28c2-c8ce-4331-9e5d-96ca10e116ef", "id":"48ae7d49-e28d-4ad5-9922-d2972e492ab7", "dt":"2021-09-15T12:57:06Z", "domain":"en.wikipedia.org", "stream":"mediawiki.recentchange", "topic":"eqiad.mediawiki.recentchange", "partition":0, "offset":3299628122 }, "id":1423007653, "type":"edit", "namespace":0, "title":"Bernie Ecclestone", "comment":"/* Racial statement */Fixed typo", "timestamp":1631710626, "user":"2A02:C7F:2CA7:F700:DC87:1AF6:9C2B:4497", "bot":false, "minor":false, "length":{ "old":59388, "new":59393 }, "revision":{ "old":1043848398, "new":1044480964 }, "server_url":"https://en.wikipedia.org", "server_name":"en.wikipedia.org", "server_script_path":"/w", "wiki":"enwiki", "parsedcomment":"<span dir=\"auto\"><span class=\"autocomment\"><a href=\"/wiki/Bernie_Ecclestone#Racial_statement\" title=\"Bernie Ecclestone\">→‎Racial statement</a>: </span>Fixed typo</span>" }