Create a modeler plugin
Now that we’ve created a NwsStation
component type, we need to create
a modeler plugin to create locations in the database. We’re dealing with
a custom HTTP API, so we’ll want to base our modeler plugin on the
PythonPlugin
class. This gives us full control of both the collection
and processing of the modeling data.
The modeler plugin will pass each state the user has specified in the
zNwsStates
property to the National Weather Service's Stations API
endpoint to retrieve the list of weather observation stations, and some
basic information about each.
Use the following steps to create our modeler plugin.
-
Make the directory that will contain our modeler plugin.
mkdir -p $ZP_DIR/modeler/plugins/NWS
-
Create
__init__.py
ordunder-init
files.touch $ZP_DIR/modeler/__init__.py touch $ZP_DIR/modeler/plugins/__init__.py touch $ZP_DIR/modeler/plugins/NWS/__init__.py
These empty
__init__.py
files are mandatory if we ever expect Python to import modules from these directories. -
Create
$ZP_DIR/modeler/plugins/NWS/Stations.py
with the following contents."""Models locations using the National Weather Service API.""" # stdlib Imports import json import urllib # Twisted Imports from twisted.internet.defer import inlineCallbacks, returnValue, DeferredList from twisted.web.client import getPage # Zenoss Imports from Products.DataCollector.plugins.CollectorPlugin import PythonPlugin class Stations(PythonPlugin): """NWS Stations modeler plugin.""" relname = 'nwsStations' modname = 'ZenPacks.training.NWS.NwsStation' requiredProperties = ( 'zNwsStates', ) deviceProperties = PythonPlugin.deviceProperties + requiredProperties @inlineCallbacks def collect(self, device, log): """Asynchronously collect data from device. Return a deferred.""" log.info("%s: collecting data", device.id) NwsStates = getattr(device, 'zNwsStates', None) if not NwsStates: log.error( "%s: %s not set.", device.id, 'zNwsStates') returnValue(None) requests = [] responses = [] for NwsState in NwsStates: if NwsState: try: response = yield getPage( 'https://api.weather.gov/stations?state={query}' .format(query=urllib.quote(NwsState))) response = json.loads(response) responses.append(response) except Exception, e: log.error( "%s: %s", device.id, e) returnValue(None) requests.extend([ getPage( 'https://api.weather.gov/stations/{query}' .format(query=urllib.quote(result['properties']['stationIdentifier'])) ) for result in response.get('features') ]) results = yield DeferredList(requests, consumeErrors=True) returnValue((responses, results)) def process(self, device, results, log): """Process results. Return iterable of datamaps or None.""" rm = self.relMap() (generalResults, detailedRawResults) = results detailedResults = {} for result in detailedRawResults: result = json.loads(result[1]) id = self.prepId(result['properties']['stationIdentifier']) detailedResults[id] = result['properties'] for result in generalResults: for stationResult in result.get('features'): id = self.prepId(stationResult['properties']['stationIdentifier']) zoneLink = detailedResults.get(id, {}).get('forecast', '') countyLink = detailedResults.get(id, {}).get('county', '') rm.append(self.objectMap({ 'id': id, 'station_id': id, 'title': stationResult['properties']['name'], 'longitude': stationResult['geometry']['coordinates'][0], 'latitude': stationResult['geometry']['coordinates'][1], 'timezone': stationResult['properties']['timeZone'], 'county': countyLink.split('/')[-1], 'nws_zone': zoneLink.split('/')[-1], })) return rm
While it looks like there’s quite a bit of code in this modeler plugin, a lot of that is the kind of error handling you’d want to do in a real modeler plugin. Let’s walk through some of the highlights.
-
Imports
We import the standard
json
module because the Weather Underground API returns JSON-encoded responses that we’ll want to convert to Python data structures.We import
inlineCallbacks
,returnValue
, andDeferredList
because thePythonPlugin.collect
method should return aDeferred
so that it can be executed asynchronously byzenmodeler
. You don’t need to useinlineCallbacks
, but I find it to be a nice way to make Twisted’s asynchronous callback-based code look more procedural and be easier to understand. I recommend Dave Peticolas’ excellent Twisted Introduction for learning more about Twisted.inlineCallbacks
is covered in part 17. ImportingDeferredList
is necessary because we're going to return a list, rather than a single value for our deferred.We also import Twisted’s
getPage
function. This is an extremely easy to use function for asynchronously fetching a URL.We import
PythonPlugin
because it will be the base class for our modeler plugin class. It’s the best choice for modeling data from HTTP APIs. -
Stations
classRemember that your modeler plugin’s class name must match the filename or Zenoss won’t be able to load it. So because we named the file
Stations.py
we must name the classStations
. -
relname
andmodname
propertiesThese should be defined in this way for modeler plugins that fill a single relationship like we’re doing in this case. It states that this modeler plugin creates objects in the device’s
nwsStations
relationship, and that it creates objects of theZenPacks.training.NWS.NwsStation
type within this relationship.Where does
relname
come from? It comes from theNwsDevice 1:MC NwsStation
relationship we defined inzenpack.yaml
. Because it’s a to-many relationship to theNwsStation
type,zenpacklib
will name the relationship by lowercasing the first letter and adding ans
to the end to make it plural.Where does
modname
come from? It will be <name-of-zenpack>.<name-of-class>. So because we defined theNwsStation
class inzenpack.yaml
, and the ZenPack’s name isZenPacks.training.NWS
, themodname
will beZenPacks.training.NWS.NwsStation
. -
deviceProperties
propertiesThe class’
deviceProperties
property provides a way to get additional device properties available to your modeler plugin’scollect
andprocess
methods. The default properties that will be available for aPythonPlugin
are:id
,manageIp
,snmpLastCollection
,snmpStatus
, andzCollectorClientTimeout
. Our modeler plugin will also need to know what values the user has set forzNwsStates
. So we add those to the defaults. -
collect
MethodThe
collect
method is somethingPythonPlugin
has, but other base modeler plugin types likeSnmpPlugin
don’t. This is because you must write the code to collect the data to be processed, and that’s exactly what you should do in thecollect
method.While the
collect
method can return either normal results or aDeferred
, it is highly recommend to return aDeferred
to keepzenmodeler
from blocking while yourcollect
method executes. In this example we’ve decorated the method with@inlineCallbacks
and have returned our data at the end withreturnValue(rm)
. This causes it to return aDeferred
. By decorating the method with@inlineCallbacks
we’re able to make an asynchronous request to the NWS API withresponse = yield getPage(...)
.The first thing we do in the
collect
method is log an informational message to let the user know what we’re doing. This log will appear inzenmodeler.log
, or on the console if we runzenmodeler
in the foreground, or in the web interface when the user manually remodels the device.Next we make sure that the user as configured at least one state in
zNwsStates
. This is mandatory because this controls what stations we will be querying for. Then we iterate through the states listed inzNwsStates
, and query for the weather observation stations in each of those states. We use list comprehension across those results to create a list of queries for the detailed data for each of those stations, and then create aDeferredList
with those queries, so Twisted can perform them asynchronously. We then return both the original results, and theDeferredList
as a tuple. -
process
methodThe
process
method is where you take the data in theresults
argument and process it into DataMaps to return.We create
rm
which is a common convention we use in modeler plugins and stands forRelationshipMap
. Because we set therelname
andmodname
class properties this will create aRelationshipMap
with it’srelname
andmodname
set to the same.Now we iterate through each result in the
DeferredList
we created before. We're just creating a dictionary by parsing the JSON in each result so we can use this to supplement the information from the list of weather stations.Next we will append an
ObjectMap
torm
with some key properties set.-
id
is mandatory and should be set to a value unique to all components on the device. If you look back the exampleStations
response you’ll see that thestationIdentifier
property is useful for this purpose. Note thatprepId
should always be used forid
. It will make any string safe to use as a Zenoss ID. -
title
gets set to the name of the weather station. It’s usually a good idea to explicitly set it as we’re doing here. It should be a human-friendly label for the component. The location’sname
is a good candidate for this. It will look something likeDallas/Fort Worth International Airport
. -
timezone
is another property we defined just for informational purposes. -
station_id
is another property we defined. It’s purely informational and will simply be shown to the user when they’re viewing the location in the web interface. It stores the same value asid
, but makes it more convenient to see in the grid. -
longitude
is another property we defined just for informational purposes. -
latitude
is another property we defined just for informational purposes. -
county
is another property we defined just for informational purposes. Note that we had to get this from the detailed results list by parsing a string to remove everything up to the last/
. -
country_code
is another property we defined just for informational purposes. Note that we had to get this from the detailed results list by parsing a string to remove everything up to the last/
.
-
-
-
Restart Zenoss.
After adding a new modeler plugin you must restart Zenoss. During development like this, it would be enough to just restart Zope and zenhub with the following commands.
serviced service restart zope serviced service restart zenhub
That’s it. The modeler plugin has been created. Now we just need to do some Zenoss configuration to allow us to use it.