Add original file data to job downloads

When downloading a report of a which messages from a job have been
delivered and which have failed we currently only include the Notify
data. This makes it hard to reconcile or do analysis on these reports,
because often the thing that people want to reconcile on is in the data
they’ve uploaded (eg a reference number).

Here’s an example of a user talking about this problem:

> It would also be helpful if the format of the delivery and failure
> reports could include the fields from the recipient's file. While I
> can, of course,  cross-reference one report with the other it would be
> easier if I did not have to. We send emails to individuals within
> organisations and it is not always easy to establish the organisation
> from a recipient's email address. This is particularly important when
> emails fail to be delivered as we need to contact the organisation to
> establish a new contact.

– ticket 677

We’ve also seen it when doing research with a local council.

This commit takes the original file, the data from the API, and munges
them together.

app/utils.py

@@ -17,6 +17,7 @@ import pytz
 import yaml
 from flask import abort, current_app, redirect, request, session, url_for
 from flask_login import current_user
+from notifications_utils.recipients import RecipientCSV
 from notifications_utils.template import (
     EmailPreviewTemplate,
     LetterImageTemplate,
@@ -121,33 +122,43 @@ def get_errors_for_csv(recipients, template_type):
 
 def generate_notifications_csv(**kwargs):
     from app import notification_api_client
+    from app.main.s3_client import s3download
     if 'page' not in kwargs:
         kwargs['page'] = 1
 
-    if kwargs['job_id']:
-        fieldnames = ['Row number', 'Recipient', 'Template', 'Type', 'Job', 'Status', 'Time']
+    if kwargs.get('job_id'):
+        original_file_contents = s3download(kwargs['service_id'], kwargs['job_id'])
+        original_upload = RecipientCSV(
+            original_file_contents,
+            template_type=kwargs['template_type'],
+        )
+        original_column_headers = original_upload.column_headers
+        fieldnames = ['Row number'] + original_column_headers + ['Template', 'Type', 'Job', 'Status', 'Time']
     else:
         fieldnames = ['Recipient', 'Template', 'Type', 'Job', 'Status', 'Time']
 
     yield ','.join(fieldnames) + '\n'
 
     while kwargs['page']:
-        # if job_id then response looks different
         notifications_resp = notification_api_client.get_notifications_for_service(**kwargs)
         notifications = notifications_resp['notifications']
-        if kwargs['job_id']:
+
+        if kwargs.get('job_id'):
             for notification in notifications:
-                    values = [
-                        notification['row_number'],
-                        notification['recipient'],
-                        notification['template_name'],
-                        notification['template_type'],
-                        notification['job_name'],
-                        notification['status'],
-                        notification['created_at']
-                    ]
-                    line = ','.join(str(i) for i in values) + '\n'
-                    yield line
+                original_row = original_upload[notification['row_number'] - 1]
+                values = [
+                    notification['row_number'],
+                ] + [
+                    original_row.get(header)
+                    for header in original_column_headers
+                ] + [
+                    notification['template_name'],
+                    notification['template_type'],
+                    notification['job_name'],
+                    notification['status'],
+                    notification['created_at']
+                ]
+                yield ','.join(map(str, values)) + '\n'
         else:
             # Change here
             for notification in notifications: