...
This page documents the planning and analysis of StackStorm for this story.
Findings
These are just my feelings after doing the prototyping work in this sprint.
Surprises
The polling sensor code is run on the polling interval, whether or not there are triggers configured.
Many different kinds of state in a sensor: sensor configuration, trigger configuration, persistent state, object state.
Pros
StackStorm was very easy to get running using Docker. It does more-or-less what you expect it to do, in more-or-less the way you expect it to.
The sensors and actions are run in separate processes. This protects the system from misbehaving code in the packs. It shouldn't be possible to induce a complete failure of the StackStorm system by putting broken code into it.
While Sensors need to be written in Python to inject events into the system, Actions can be written in a variety of systems, including simply being HTTP endpoints or bash scripts. This adds a lot of flexibility to the integration platform.
StackStorm offers a comprehensive UI. While not the most polished thing in the world, it's very very good.
Cons
First, I found that developing a sensor was not straightforward. Each time I changed the sensor code, I had to reinstall the pack. There may be a more efficient way to do this but it wasn't made clear in the documentation I saw.
Second, there are a lot of ways to fail at making a sensor. It wasn't obvious how to debug it. I wound up tailing the logs. Several times I got into a situation or state where the web service backing the UI was returning a 500—such as when a sensor changed its API significantly. These were resolved by using the command line tool st2.
Third, the developer documentation—while extensive—still failed to include Python API documentation. I wound up reading the source code to existing sensors to discover what the right way to use the APIs was.
StackStorm terminology
Amygdala is a little shy on terminology. It essentially has one major interface, which is Loader and whose role is basically to sit on a certain channel of an AMQP connection and do something with the events that come in.
...
More information about this can be found in the StackStorm "About" documentation.
Alternatives
Amygdala Migration
The purpose of this section is to draft out what Amygdala processes exist, and how to convert them to StackStorm processes.
...
Option 3 seems like it would be the most irritating and least helpful, except for the fact that it would be totally sufficient for this problem and requires no particular state.
Queue Runner/Loaders
These come in two flavors: ALMA and VLBA. They share a similar architecture, which is very interesting and cool (I wrote the first one) but mainly exists to enable throttling of executions at a certain number, which is an intrinsic functionality of Workspaces (capability queue limits). So it should not be necessary to bring much of the complexity along for the migration.
ALMA
Despite the names, AlmaReingestionQueueLoader actually accounts for three things:
- Reingestion of raw ASDMs from the ALMA instrument
- Reingestion of calibrations from ALMA
- Marking ASDMs as not calibrateable
What is needed to implement this is:
- Source of ALMA ASDMs to load
- Source of ALMA calibrations to load
- Action of marking ASDM as not calibrateable
VLBA
The VLBA system is simpler, it only handles the appearance of VLBA data to ingest. There is no automatic calibration of VLBA data. What is needed to implement this is therefore only a source of VLBA data to load.
StackStorm Development Notes
There is not a super adequate amount of developer API documentation inside the Python code for StackStorm. A certain amount of reverse engineering was required to create the new file detector. Here are some notes about what I discovered while doing this implementation.
Sensors
You're going to get your sensor instantiated exactly once.
A lot of weird stuff about the API is somewhat explained by the fact that your sensor can define many trigger types. Each trigger type has a parameter type and a payload type. These are expressed using JSON Schema. So the Python code winds up accepting and producing lots of dictionaries representing JSON.
Triggers get manufactured by StackStorm when the user configures a rule to depend on a trigger type coming out of your sensor, but they aren't instances of a real class, just a dictionary. When this happens, StackStorm will call add_trigger on your sensor instance with a trigger dictionary. The most interesting slot of this to you will be the trigger's parameter, which will have whatever parameters you defined in the parameter schema in the YAML file. So for the directory watcher, this is the directory to check. It could be other things too, like the glob string to use.
It's important to hold onto this configuration somewhere because StackStorm will then issue update and delete calls with the same dictionary, and you need the dictionary to call sensor_service.dispatch with. It's these calls that create events in the StackStorm system which the action in a rule will catch.
State
In terms of saving state, using your sensor instance isn't super safe, because the service could go down and come back. Helpfully, StackStorm provides a key-value store that is keyed to your instance. Unhelpfully, it's a key-value store, so you'll have to invent keys for whatever properties you want to hold on to for longer than the life of a single instance.
Development Cycle
The easiest thing to do, following the advice of StackStorm Docker, is this:
- Clone the docker repo
- Run
docker compose up -d . - Develop your pack in the
packs.devdirectory inside the cloned Docker repo - Run
docker-compose exec st2client bashand then change to the directorypacks.dev - Run
st2 pack install file://$PWD/<your-pack> - Navigate to the UI at http://localhost and create a rule using your actions and sensors
Whenever you modify the pack sourcecode, you'll need to rerun st2 pack install file://$PWD/<your pack> . In another console, follow the logs with: docker compose logs -f
This is fairly onerous. I am not entirely sure what can be done about it at this time.
StackStorm Checklist
- Sensor: Filesystem monitor trigger
- Action: Workspaces Capability create-and-run
- Action: Workspaces Workflow create-and-run
- Sensor: ALMA ASDMs to load
- Sensor: ALMA calibrations to load
- Sensor: VLBA data to load
- Action: Mark ASDM as not calibrateable