If there are no files to delete we won't get an excpetion.
Wrap the delete file in a try/except to avoid stopping the entire task.
Fix the missing slash for the file name.
when a test letter is created on dev or preview, we upload a file to
the dvla ftp response bucket, to test that our integration with s3
works. s3 triggers an sns notification, which we pick up, and then we
download the file and mark the letters it mentions as delivered.
However, if two tests run at the same time, they'll create the same
file on s3. One will just overwrite the next, and the first letter will
never move into delivered - this was causing functional tests to
intermittently fail.
This commit makes the test letter task check if the file exists - if it
does, it moves back one second and tries again. It tries this thirty
times before giving up.
All of our uploads now have the metadata about the job set on them in
S3. So this commit moves to using that metadata, if it’s there, instead
of the data in the body of the post request.
The aim of this is to stop the admin app having to post this data, which
means that it won’t have to keep this data in the session for the
while doing the file upload flow.
- Also convert the files info to upper() for comparison rather than lower
because original file names are in upper case. The unit tests contain examples of the returned lists.
this means that if the task is accidentally ran twice (eg we autoscale
notify-celery-worker-beat to 2), it won't send letters twice.
Additionally, update some function names and config variables to make
it clear that they are referring to letter jobs, rather than all letter
content
- brings boto S3 into new AWS folder
- CSV processing utils method
Rejigs the jobs rest endpoint - removes some now unused endpoints,
Calls to the task with the job, job processing in task, delegating SMS calls to the sms task
