EWAI-GEN

EWAI-GEN is a PTD message generator utility (simulator) to help create dummy message data sets to test EWAI with. Messages can be generated in either JSON or TEXT format.

See also:

1. Message formats,
2. EWAI Architecture

EWAI-GEN also supports a "dry run" mode, which can generate message files for replay later without actually sending the to EW-Messaging (see --dryrun option).

USAGE:

Run from root of repo, just as an example, which pulls settings from the config1.env file:

node dist/src/ewaigen/index.js --config ./src/ewaigen/config/config1.env

EWAI-GEN uses a config file to specify how to generate messages. An example is given here:

# specify the EWNS of the EWAI Instance (marketplace) the messages are going to. Note that
# an EWAI instance coresponds 1:1 to a marketplace
EWAI_INSTANCE_NAME="market1.apps.adivate.iam.ewc"

# specify the EWNS of the data asset in that marketplace messages are going to:
EWAI_ASSET="hydro1.market1.adivate.ewc"

# specify the fully qualified (relative or absolute path) to the message template file to be used
# note that this path must be relative to where you are running ewaigen/index.js!
# The template to use to generate messages. JSON and TEXT messages can be created.
# For Json, make sure the template is a VALID json file with .json extension recommended.
# For Text, use .txt or some other extension as appropriate. if Text template, it's just
# loaded and the replacement fields (if any, see below) are substituted and message created.
# For Json, the JSON template is parsed and loaded so it must be a valid SINGLE Json object file,
# which again can have replacement fields (see below)
MSG_TEMPLATE_FILE="./src/ewaigen/templates/template1.json"

# specify the ETH DID for the DER which is sending the messages. Note that this DID must
# have been provisioned in EW-Switchboard prior with the proper roles required to
# a) publish to the marketplace instance, and b) publish to the asset also:
DER_DID="did:ethr:0x1234..."

# specify the ETH PRIVATE KEY for the DER DID (this is used to sign messages):
DER_PRIVATE_KEY="1111..."

# A CSV file with header row (containing field names) which contains the values
# you want to substitute in the message parameters specified in the template being
# used. You can use this to replay a repeatable series of data into messages.
# optional
INPUT_CSV_FILE=""

# specify a limit on the number of messages produced:
LIMIT=10000

# specify a file to save the messages to (if you want them saved for later replay)
# for example, you can use dry run to create a message file of 100 messages that
# you save to a file and then later replay live
# optional
SAVE_TO_FILE=""

# specify the file you are replaying messages from (if any):
# if provided, the messages are read from this input file in either Text or Json
# format based on file extension type. Note that the messages CAN still have
# replacement parameters in them (and they will still be replaced). Also note
# that if a replay file is specified, it takes precedence over the template
# parameter (i.e. template parameter is ignored if specified also because
# it is not needed - a replay file has all the messages already in it in the
# proper format, presumably from some prior run reading a template and
# output to a save file)
# optional
REPLAY_FROM_FILE=""

# specify the delay between messages. Values < 1 are considered ms. Values > 1 are considered secs:
# optional, defaults to 1
DELAY=5

# specify whether to suppress login/info output:
QUIET=false

# by default, a Json template file is checked to make sure it's actually a valid Json document before using
# you can use this flag to suppress that test (not recommended, except for dev/debug usage)
DISABLE_JSON_TEMPLATE_VALIDATION=false

# specify whether to create the messages but not actually send them (hence a dry run). This flag
# is often used in combination with SAVE_TO_FILE to create a message file for later replay:
DRYRUN=false

# specify the EWC chain to use:
EWAI_EWC_RPC_URL="https://volta-rpc.energyweb.org"
EWAI_EWC_CHAIN_ID=73799

# specify the location of the EW-Messaging Server to use to send the messages:
EWAI_MESSAGING_SERVER="http://localhost:3000/graphql"
EWAI_MESSAGING_BROKER_ROLE="messagebroker.roles.messaging.apps.energyweb.iam.ewc"

# specify the I-AM caching server to use:
#EWAI_EWC_IAM_CACHE_URL="https://volta-iam-cacheserver.energyweb.org/"
EWAI_EWC_IAM_CACHE_URL="https://volta-identitycache.energyweb.org/"

A message template file is used to create the messages, which supports random number generation and substitution of other fields. An example to generate text messages is:

did="<<did()>>"|asset="<<asset()>>"|time="<<timestamp()>>";lat="<<chance.floating({ min: -90, max: 90, fixed: 5 })>>";long="<<chance.floating({ min: -180, max: 180, fixed: 5 })>>";minKw="<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>";maxKw="<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>";avgKw="<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>";sensorId="<<chance.integer({ min: 0, max: 2000 })>>"

An example to generate Json messages is:

{
    "ts": "<<timestamp().toString()>>",
    "ewai": {
        "did": "<<did().toString()>>",
        "asset": "<<asset().toString()>>"
    },
    "data": {
        "loc": {
            "lat": "<<chance.floating({ min: -90, max: 90, fixed: 5 })>>",
            "long": "<<chance.floating({ min: -180, max: 180, fixed: 5 })>>"
        },
        "minKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
        "maxKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
        "avgKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
        "sensorId": "<<chance.integer({ min: 0, max: 2000 })>>"
    }
}

An example to generate Json messages and read PTD data from an input CSV file:

{
    "ts": "<<csv('ts').toString()>>",
    "ewai": {
        "did": "<<did().toString()>>",
        "asset": "<<asset().toString()>>"
    },
    "data": {
        "loc": {
            "lat": "<<csv('lat')>>",
            "long": "<<csv('long')>>"
        },
        "minKw": "<<csv('minKw')>>",
        "maxKw": "<<csv('maxKw')>>",
        "avgKw": "<<csv('avgKw')>>",
        "sensorId": "<<csv('sensorId')>>"
    }
}

SAMPLE FILES

The templates used can contain fields (replacement tokens) which are generated and substituted at run-time. A sample Json template using all randomly created values (template1.json):

{
"ts": "<<timestamp().toString()>>",
"data": {
"loc": {
"lat": "<<chance.floating({ min: -90, max: 90, fixed: 5 })>>",
"long": "<<chance.floating({ min: -180, max: 180, fixed: 5 })>>"
},
"minKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
"maxKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
"avgKw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>",
"sensorId": "<<chance.integer({ min: 0, max: 2000 })>>"
}
}

This allows for dynamic and random data messages to be generated. Replacement tokens have the format "\<\<token spec>>". If you want the output to be sent back inside double quotes, add toString() to the end. This allows the tokens to be placed inside real .json files (as you see above) which validate in VS code and other editors to help with manual editing/creation. If you don't use toString() at the end, the value returned will be whatever JavaScript evals the code inside the token to be (number, date, float, boolean, etc.), and it's your job to make sure it works properly in the template (regarding final Json syntax etc).

Various replacement functions are supported:

• timestamp() => returns current date/time in ISO8601 format
• timestamp().toString() => returns current date/time in ISO8601 format surrounded by double quotes
• csv('fieldname') => returns the CSV value in the column of the input csv file specified
• csv('fieldname').toString() => returns the CSV value in the column of the input csv file specified surrounded by double quotes
• chance.xxxxxx() => uses the chance JS library (see: https://chancejs.com/) to create random values, whatever you put in will be eval'd at run-time and put in that replacement field.
• chance.xxxxxx().toString() => the chance call returned inside double quotes
• customfunc(whatever) => You can also add your own custom funcs to the code (see src/ewaigen/customfuncs.ts)
• customfunc(whatever).toString() => calls your func and returns value inside double quotes

Some example templates are provided in src/ewaigen/templates directory. Note that a single template can mix and read from any/all of these at the same time:

• randomly generated values (timestamp, chance),
• csv input files values (csv...), and
• custom func values

A sample Json template that pulls it's field values from a CSV file which must also have been specified by the input csv file parameter (template1_csv.json):

{
"ts": "<<csv('ts').toString()>>",
"data": {
"loc": {
"lat": "<<csv('lat')>>",
"long": "<<csv('long')>>"
},
"minKw": "<<csv('minKw')>>",
"maxKw": "<<csv('maxKw')>>",
"avgKw": "<<csv('avgKw')>>",
"sensorId": "<<csv('sensorId')>>"
}
}

A sample TEXT template that pulls it's field values from a CSV file which must also have been specified by the input csv file parameter (template1_csv.txt):

time="<<csv('ts')>>";lat="<<csv('lat')>>";long="<<csv('long')>>";minKw="<<csv('minKw')>>";maxKw="<<csv('maxKw')>>";avgKw="<<csv('avgKw')>>";sensorId="<<csv('sensorId')>>"

A sample input CSV data file (which would match template1_csv.json and also template1_csv.txt templates)

ts,lat,long,avgKw,maxKw,minKw,sensorId
"2020-11-26T17:26:10.075Z",21.266,-95.012,65.28,181.56,694.86,751
"2020-11-26T17:26:11.075Z",-57.367,163.005,658.17,388.21,529.85,1791
"2020-11-26T17:26:12.075Z",-19.055,78.62,95.48,432.54,866.18,103
"2020-11-26T17:26:13.075Z",88.533,178.224,549.33,658.91,833.68,1891
"2020-11-26T17:26:14.075Z",-61.445,179.625,259.16,133.64,534.52,743
"2020-11-26T17:26:15.075Z",96.17,40.31,74.06,53.29,63.99,37
"2020-11-26T17:26:16.075Z",71.97,67.09,47.48,69.69,84.43,80
"2020-11-26T17:26:17.075Z",72.88,18.5,51.55,44.55,90.32,16
"2020-11-26T17:26:18.075Z",44.08,65.49,46.66,11.06,82.04,77
"2020-11-26T17:26:19.075Z",88.45,53.52,51.11,11.3,24.83,71

Another Json template example (template2.json):

{
    "ts": "<<timestamp().toString()>>",
    "lat": "<<chance.floating({ min: -90, max: 90, fixed: 5 })>>",
    "long": "<<chance.floating({ min: -180, max: 180, fixed: 5 })>>",
    "sensorId": "<<chance.integer({ min: 0, max: 2000 })>>",
    "temp": "<<chance.floating({ min: 0, max: 100, fixed: 2 })>>",
    "kw": "<<chance.floating({ min: 0, max: 1000, fixed: 2 })>>"
}

See more example files in the ewaigen folder in the repo.

SETUP IF USING NATS & JETSTREAM

<<<This is now deprecated, as EW-Messaging is to be used instead of NATS directly. Note that EW-Messaging actually uses NATS for its transport layer.>>>

# start the Jetstream server (instructions for my localhost DEV Box only!!)

cd ~/Projects/nats/nats-server
./nats-server -js -DV

# SETUP NATS STREAMS AND CONSUMERS:

#

# by convention all NATS stream and consumer names are ALL UPPERCASE

#

# EWAI_INSTANCE = A Unique name for this EWAI instance in the form of a EWNS (e.g. market1.apps.bigco.iam.ewc)

# (note: for dev, you could use something like localhost.ewai.iam.ewc which becomes SDAN LOCALHOST_EWAI_EWC)

# EDAN = An ENERGY DATA ASSET NAME is uniquely specified by an EWNS name (e.g. windfarm1.windco.ewc)

# SDAN = NATS Stream name for the EDAN, will be created by uppercasing EDAN and replacing “.” with “\_“ (e.g. WINDFARM1_WINDCO_EWC)

# SDANs must be unique across all the entire subsystem (since EWNS are unique, this won't be a problem)

# devices will send events into {EWAI_INSTANCE}.{SDAN} subject (e.g. MARKET1_APPS_BIGCO_IAM_EWC.WINDFARM1_WINDCO_EWC).

# EWAI Listener (Message Consumer) will subscribe to {EWAI*INSTANCE}.* (e.g. MARKET1*APPS_BIGCO_IAM_EWC.*)

nats str add {EWAI*INSTANCE} --subjects "{EWAI_INSTANCE}.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=-1 --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="1h"
e.g. nats str add EWAI --subjects "MARKET1*APPS_BIGCO_IAM_EWC.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=-1 --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="1h"

# consumers, define a pull based consumer (MONITOR name is hardcoded in the source right now, although there's nothing special to the name being used, could be anything you want such as LISTENER, FEED, etc...)

nats con add {EWAI_INSTANCE} MONITOR --filter '' --ack explicit --pull --deliver all --sample 100 --max-deliver=-1 --replay instant
e.g. nats con add MARKET1_APPS_BIGCO_IAM_EWC MONITOR --filter '' --ack explicit --pull --deliver all --sample 100 --max-deliver=-1 --replay instant

###

So, to just setup NATS/JETSTREAM FOR DEV RUN THESE 2 COMMANDS ONLY:

###

nats str add MARKET1_APPS_BIGCO_IAM_EWC --subjects "MARKET1_APPS_BIGCO_IAM_EWC.\*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=-1 --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="1h"
nats con add MARKET1_APPS_BIGCO_IAM_EWC MONITOR --filter '' --ack explicit --pull --deliver all --sample 100 --max-deliver=-1 --replay instant

# stuff below is just notes on CLI things (not used by EWAI code anywhere)

# get the messages in the MONITOR consumer:

nats con next {EWAI_INSTANCE} MONITOR
e.g. nats con next MARKET1_APPS_BIGCO_IAM_EWC MONITOR
for dev use: nats con next MARKET1_APPS_BIGCO_IAM_EWC MONITOR

nats str purge {EWAI_INSTANCE}
e.g. nats str purge MARKET1_APPS_BIGCO_IAM_EWC

# don't run these next lines, but you could send messages from the NATS cli also (not just ewaigen)

// while true
// do
// nats pub {EWAI_INSTANCE}.{SDAN} "{\"id\":\"did:ethr:0x1095C1c4CbF6F5fc9225cda21c822b240DB91eE4\",\"ts\":\"\$(date -u +"%Y-%m-%dT%H:%M:%SZ")\",\"data\":{\"minKwh\":100,\"maxKwh\":125,\"avgKwh\":110}}"
// e.g. nats pub MARKET1_APPS_BIGCO_IAM_EWC.WINDFARM1_WINDCO_EWC "{\"ts\":\"\$(date -u +"%Y-%m-%dT%H:%M:%SZ")\",\"data\":{\"minKwh\":100,\"maxKwh\":125,\"avgKwh\":110}}"
// sleep 5
// done