Create a modeler plugin

Now that we’ve created a NwsStation component type, we need to create a modeler plugin to create locations in the database. We’re dealing with a custom HTTP API, so we’ll want to base our modeler plugin on the PythonPlugin class. This gives us full control of both the collection and processing of the modeling data.

The modeler plugin will pass each state the user has specified in the zNwsStates property to the National Weather Service's Stations API endpoint to retrieve the list of weather observation stations, and some basic information about each.

Use the following steps to create our modeler plugin.

Make the directory that will contain our modeler plugin.
```
mkdir -p $ZP_DIR/modeler/plugins/NWS
```
Create __init__.py or dunder-init files.
```
touch $ZP_DIR/modeler/__init__.py
touch $ZP_DIR/modeler/plugins/__init__.py
touch $ZP_DIR/modeler/plugins/NWS/__init__.py
```
These empty __init__.py files are mandatory if we ever expect Python to import modules from these directories.
Create $ZP_DIR/modeler/plugins/NWS/Stations.py with the following contents.
```
"""Models locations using the National Weather Service API."""

# stdlib Imports
import json
import urllib

# Twisted Imports
from twisted.internet.defer import inlineCallbacks, returnValue, DeferredList
from twisted.web.client import getPage

# Zenoss Imports
from Products.DataCollector.plugins.CollectorPlugin import PythonPlugin


class Stations(PythonPlugin):

    """NWS Stations modeler plugin."""

    relname = 'nwsStations'
    modname = 'ZenPacks.training.NWS.NwsStation'

    requiredProperties = (
        'zNwsStates',
        )

    deviceProperties = PythonPlugin.deviceProperties + requiredProperties

    @inlineCallbacks
    def collect(self, device, log):
        """Asynchronously collect data from device. Return a deferred."""
        log.info("%s: collecting data", device.id)

        NwsStates = getattr(device, 'zNwsStates', None)
        if not NwsStates:
            log.error(
                "%s: %s not set.",
                device.id,
                'zNwsStates')

            returnValue(None)
        requests = []
        responses = []

        for NwsState in NwsStates:
            if NwsState:
                try:
                    response = yield getPage(
                        'https://api.weather.gov/stations?state={query}'
                        .format(query=urllib.quote(NwsState)))
                    response = json.loads(response)
                    responses.append(response)
                except Exception, e:
                    log.error(
                        "%s: %s", device.id, e)
                    returnValue(None)

                requests.extend([
                    getPage(
                            'https://api.weather.gov/stations/{query}'
                            .format(query=urllib.quote(result['properties']['stationIdentifier']))
                    )
                    for result in response.get('features')
                ])
        results = yield DeferredList(requests, consumeErrors=True)
        returnValue((responses, results))

    def process(self, device, results, log):
        """Process results. Return iterable of datamaps or None."""
        rm = self.relMap()

        (generalResults, detailedRawResults) = results

        detailedResults = {}
        for result in detailedRawResults:
            result = json.loads(result[1])
            id =  self.prepId(result['properties']['stationIdentifier'])
            detailedResults[id] = result['properties']
        for result in generalResults:
            for stationResult in result.get('features'):
                id = self.prepId(stationResult['properties']['stationIdentifier'])
                zoneLink = detailedResults.get(id, {}).get('forecast', '')
                countyLink = detailedResults.get(id, {}).get('county', '')

                rm.append(self.objectMap({
                    'id': id,
                    'station_id': id,
                    'title': stationResult['properties']['name'],
                    'longitude': stationResult['geometry']['coordinates'][0],
                    'latitude': stationResult['geometry']['coordinates'][1],
                    'timezone': stationResult['properties']['timeZone'],
                    'county': countyLink.split('/')[-1],
                    'nws_zone': zoneLink.split('/')[-1],
                    }))

        return rm
```
While it looks like there’s quite a bit of code in this modeler plugin, a lot of that is the kind of error handling you’d want to do in a real modeler plugin. Let’s walk through some of the highlights.
1. Imports
  
  We import the standard json module because the Weather Underground API returns JSON-encoded responses that we’ll want to convert to Python data structures.
  
  We import inlineCallbacks, returnValue, and DeferredList because the PythonPlugin.collect method should return a Deferred so that it can be executed asynchronously by zenmodeler. You don’t need to use inlineCallbacks, but I find it to be a nice way to make Twisted’s asynchronous callback-based code look more procedural and be easier to understand. I recommend Dave Peticolas’ excellent Twisted Introduction for learning more about Twisted. inlineCallbacks is covered in part 17. Importing DeferredList is necessary because we're going to return a list, rather than a single value for our deferred.
  
  We also import Twisted’s getPage function. This is an extremely easy to use function for asynchronously fetching a URL.
  
  We import PythonPlugin because it will be the base class for our modeler plugin class. It’s the best choice for modeling data from HTTP APIs.
2. Stations class
  
  Remember that your modeler plugin’s class name must match the filename or Zenoss won’t be able to load it. So because we named the file Stations.py we must name the class Stations.
3. relname and modname properties
  
  These should be defined in this way for modeler plugins that fill a single relationship like we’re doing in this case. It states that this modeler plugin creates objects in the device’s nwsStations relationship, and that it creates objects of the ZenPacks.training.NWS.NwsStation type within this relationship.
  
  Where does relname come from? It comes from the NwsDevice 1:MC NwsStation relationship we defined in zenpack.yaml. Because it’s a to-many relationship to the NwsStation type, zenpacklib will name the relationship by lowercasing the first letter and adding an s to the end to make it plural.
  
  Where does modname come from? It will be <name-of-zenpack>.<name-of-class>. So because we defined the NwsStation class in zenpack.yaml, and the ZenPack’s name is ZenPacks.training.NWS, the modname will be ZenPacks.training.NWS.NwsStation.
4. deviceProperties properties
  
  The class’ deviceProperties property provides a way to get additional device properties available to your modeler plugin’s collect and process methods. The default properties that will be available for a PythonPlugin are: id, manageIp, snmpLastCollection, snmpStatus, and zCollectorClientTimeout. Our modeler plugin will also need to know what values the user has set for zNwsStates. So we add those to the defaults.
5. collect Method
  
  The collect method is something PythonPlugin has, but other base modeler plugin types like SnmpPlugin don’t. This is because you must write the code to collect the data to be processed, and that’s exactly what you should do in the collect method.
  
  While the collect method can return either normal results or a Deferred, it is highly recommend to return a Deferred to keep zenmodeler from blocking while your collect method executes. In this example we’ve decorated the method with @inlineCallbacks and have returned our data at the end with returnValue(rm). This causes it to return a Deferred. By decorating the method with @inlineCallbacks we’re able to make an asynchronous request to the NWS API with response = yield getPage(...).
  
  The first thing we do in the collect method is log an informational message to let the user know what we’re doing. This log will appear in zenmodeler.log, or on the console if we run zenmodeler in the foreground, or in the web interface when the user manually remodels the device.
  
  Next we make sure that the user as configured at least one state in zNwsStates. This is mandatory because this controls what stations we will be querying for. Then we iterate through the states listed in zNwsStates, and query for the weather observation stations in each of those states. We use list comprehension across those results to create a list of queries for the detailed data for each of those stations, and then create a DeferredList with those queries, so Twisted can perform them asynchronously. We then return both the original results, and the DeferredList as a tuple.
6. process method
  
  The process method is where you take the data in the results argument and process it into DataMaps to return.
  
  We create rm which is a common convention we use in modeler plugins and stands for RelationshipMap. Because we set the relname and modname class properties this will create a RelationshipMap with it’s relname and modname set to the same.
  
  Now we iterate through each result in the DeferredList we created before. We're just creating a dictionary by parsing the JSON in each result so we can use this to supplement the information from the list of weather stations.
  
  Next we will append an ObjectMap to rm with some key properties set.
  - id is mandatory and should be set to a value unique to all components on the device. If you look back the example Stations response you’ll see that the stationIdentifier property is useful for this purpose. Note that prepId should always be used for id. It will make any string safe to use as a Zenoss ID.
  - title gets set to the name of the weather station. It’s usually a good idea to explicitly set it as we’re doing here. It should be a human-friendly label for the component. The location’s name is a good candidate for this. It will look something like Dallas/Fort Worth International Airport.
  - timezoneis another property we defined just for informational purposes.
  - station_id is another property we defined. It’s purely informational and will simply be shown to the user when they’re viewing the location in the web interface. It stores the same value as id, but makes it more convenient to see in the grid.
  - longitude is another property we defined just for informational purposes.
  - latitude is another property we defined just for informational purposes.
  - county is another property we defined just for informational purposes. Note that we had to get this from the detailed results list by parsing a string to remove everything up to the last /.
  - country_code is another property we defined just for informational purposes. Note that we had to get this from the detailed results list by parsing a string to remove everything up to the last /.
Restart Zenoss.

After adding a new modeler plugin you must restart Zenoss. During development like this, it would be enough to just restart Zope and zenhub with the following commands.
```
serviced service restart zope
serviced service restart zenhub
```

That’s it. The modeler plugin has been created. Now we just need to do some Zenoss configuration to allow us to use it.