Add a local_statsd configuration to PaaS manifest template

Running `statsd_exporter` alongside the app process allows us to get
StatsD metrics pushed by workers to Prometheus.

This requires adding a route to the worker instances and binding the
RE prometheus discovery service. So this approach won't work for API
and admin since they already have `gunicorn` bound to the `$PORT`.

Since we're not ready to switch all apps to Prometheus metrics at once
and we don't currently have a way to push statsd metrics to multiple
destination we're using a configuration setting in the manifest template
to switch individual workers in specific environments.

`local_statsd` contains a list of environments where the app should
use local `statsd_exporter` for pushing statsd metrics instead of
HostedGraphite.

manifest.yml.j2

@@ -9,8 +9,8 @@
 
   'notify-delivery-celery-beat': {'memory': '128M'},
   'notify-delivery-worker-database': {},
-  'notify-delivery-worker-research': {},
+  'notify-delivery-worker-research': {'local_statsd': ['preview', 'staging']},
-  'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '3G'},
+  'notify-delivery-worker-sender': {'disk_quota': '2G', 'memory': '3G', 'local_statsd': ['preview', 'staging']},
   'notify-delivery-worker-periodic': {},
   'notify-delivery-worker-priority': {},
   'notify-delivery-worker': {},
@@ -37,6 +37,10 @@ applications:
       {%- for route in app['routes'][environment] %}
       - route: {{ route }}
       {%- endfor -%}
+    {%- elif environment in app.get('local_statsd', []) -%}
+    health-check-type: none
+    routes:
+      - route: {{ CF_APP }}-{{ environment }}.cloudapps.digital
     {%- else -%}
     health-check-type: none
     no-route: true
@@ -45,6 +49,7 @@ applications:
     services:
       - notify-db
       - logit-ssl-syslog-drain
+      {% if environment in app.get('local_statsd', []) %}- notify-prometheus{% endif %}
 
     env:
       NOTIFY_APP_NAME: {{ app.get('NOTIFY_APP_NAME', CF_APP.replace('notify-', '')) }}
@@ -70,7 +75,13 @@ applications:
       AWS_ACCESS_KEY_ID: '{{ AWS_ACCESS_KEY_ID }}'
       AWS_SECRET_ACCESS_KEY: '{{ AWS_SECRET_ACCESS_KEY }}'
 
+      {% if environment in app.get('local_statsd', []) %}
+      STATSD_HOST: "localhost"
+      STATSD_PREFIX: ""
+      {% else %}
+      STATSD_HOST: "statsd.hostedgraphite.com"
       STATSD_PREFIX: '{{ STATSD_PREFIX }}'
+      {% endif %}
 
       ZENDESK_API_KEY: '{{ ZENDESK_API_KEY }}'