Added the letter_rate table to the list of tables which does not get
deleted after each test run and changed the tests to use the real letter
rates.
Also removed the letter rate DAO since this was only being used in
tests, so was no longer needed.
We now support letters of up to 5 sheets long, so we need to store the
rates for 4 and 5 sheet letters (both crown and non-crown) in the
`letter_rates` table.
We have a few old jobs which don’t have a `processing_started` date.
This means that they always sort to the top of the jobs list in admin,
no matter how old they are. We think this is due to an old bug where
jobs would not be updated if a deploy was in progress.
This commit backfills the `processing_started` data for these jobs,
which will be roughly accurate. Complete accuracy is not the goal;
having these jobs not sort to the top of the list is.
This will affect 5 jobs across 3 services on production:
```sql
select service_id, job_status, created_at, updated_at, processing_started, processing_finished, notification_count, notifications_sent, notifications_delivered, notifications_failed from jobs where processing_started is null and job_status = 'in progress';
```
```
service_id | job_status | created_at | updated_at | processing_started | processing_finished | notification_count | notifications_sent | notifications_delivered | notifications_failed
--------------------------------------+-------------+----------------------------+----------------------------+--------------------+---------------------+--------------------+--------------------+-------------------------+----------------------
d47e5a1b-a04b-4398-8935-c8a266ce1d44 | in progress | 2017-09-29 13:49:41.512356 | 2017-10-01 02:01:05.281162 | | | 10615 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-09-29 09:25:39.802623 | 2017-09-29 16:01:02.154291 | | | 10240 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-09-29 09:31:52.455919 | 2017-09-29 16:01:01.990054 | | | 9930 | 0 | 0 | 0
128b91b6-2996-4107-bb65-51b7c24a728d | in progress | 2017-08-22 08:15:39.125999 | 2017-08-22 16:01:07.758805 | | | 6967 | 0 | 0 | 0
95316ff0-e555-462d-a6e7-95d26fbfd091 | in progress | 2016-05-27 14:44:18.114564 | 2016-06-13 00:18:14.542795 | | | 2742 | 2238 | 525 | 1713
(5 rows)
```
Both service api tasks work fine if the object is unexpectedly deleted
halfway through - they both check to see if the api details are still
in the DB before trying to send the request.
it now uses the ft_notification_status table - nice and quick. if you
ask for the current tax year it'll top up the data with numbers from
the notification table.
the data is in exactly the same shape as the old endpoint - no changes
to the admin app are necessary
By adding the service id, the query performance has improved greatly. It went from 6200ms to 0.04ms.
This should stop the 500s when a template is deleted.
Test notifications are only stored in the notification table, not the
notification_history table. We were deciding which table to query for
the platform admin stats based on the start date passed to the DAO
function. This means that if the start date given was more than 7 days
ago and the end date was within the last 7 days, test notifications were
not being returned. We now use the end date to choose which table to
query which means that test notifications always get returned when they
should.
Created a platform-stats blueprint and moved the new platform stats
endpoint to the new blueprint (it was previously in the service
blueprint). Since the original platform stats route and the new platform
stats route are now in different blueprints, their view functions can
have the same name without any issues.
Moved the `fetch_new_aggregate_stats_by_date_range_for_all_services`
DAO function from the services DAO to the Notifications DAO since this
function queries the `notification` and `notification_history` tables.
Also added a test to check that the data returned from the function
takes BST into account.
Added in a new endpoint and DAO function to provide the data for the new
platform admin statistics page. The DAO method gets different data from
the Notifications / NotificationHistory table and also groups it differently.
The old endpoint has not been deleted yet to allow the numbers on the
old and new pages to be compared.
Our application servers and celery workers write logs both to a
file that is shipped to CloudWatch and to stdout, which is picked
up by CloudFoundry and sent to Logit Logstash.
This works with gunicorn and single-worker celery deployments, however
celery multi daemonizes worker processes, which detaches them from
stdout, so there's no log output in `cf logs` or Logit.
To fix this, we start a separate tail process to duplicate logs written
to a file to stdout, which should be picked up by CloudFoundry.
