In response to: [^1][^2][^3].
Originally there were three tests:
- One was testing the "get_user_template_folders" method directly.
- The other two were testing "get_template_folders", which calls
the "_user_" method if a user is passed - this is what the tests
were primarily checking, by passing or not passing a User object.
The two tests of "get_template_folders" also implicitly checked
that it filtered folders based on a "parent_folder_id". I wanted
to emulate the tests but it makes more sense to simplify them, as
the methods are now only used by TemplateList internally.
To simplify, we just keep just two tests to check that the overall
set of folders is filtered based on the presence of a User. We do
lose the implicit check of filtering by "parent_folder_id", but:
- We are still checking this implicitly in the sense that the loop
to generate the overall set of folders terminates. If the method
didn't do any filtering, the loop would be infinite.
- We need to acknowledge that these tests were incomplete to begin
with. There is also lots of coverage in higher level tests [^4],
although it's harder to reason exactly what is covered.
[^1]: https://github.com/alphagov/notifications-admin/pull/4258#discussion_r890144076
[^2]: https://github.com/alphagov/notifications-admin/pull/4258#discussion_r890151220
[^3]: https://github.com/alphagov/notifications-admin/pull/4258#discussion_r890147506
[^4]: 1787f9f42f/tests/app/main/views/test_template_folders.py
This gives us the performance gains identified in [^1] for the test
service described in the spike:
- user_template_folders - from 10s to a little above 3s on its own
- get_templates_and_folders - from 10s to below 6s on its own
In combination, these two uses of caching reduce the test page load
time from 10s to a little above 3s. This is slightly higher than in
the spike PR due to all the extra work we're doing to generate the
"move to" list of folders, as described in a previous commit.
The render time is unchanged for services with few folders. We start
to see the benefit of this change at around 200 templates + folders,
with no evidence that any service will experience worse render times,
despite the extra work we're doing from previous commits.
[^1]: https://github.com/alphagov/notifications-admin/pull/4251
This is a straight move with a few minor tweaks:
- Some references to self.X in the code from Service now need to
become self.service.X.
- Some reference to self.service.Y in TemplateList can now become
self.Y for the methods that were migrated.
The remaining _template_ methods in Service are still called from
multiple places and there's no performance gain to motivate moving
them out of the Service class, which is now a more manageable size.
This is part of the overall migration of the "_template_folders"
methods to the TemplateList class. Moving the existing tests now
will make the actual migration easier to follow.
To emulate the second and third tests, we need to grab a specific
folder from the TemplateList and then look at its folders - these
are set based on "get_template_folders" as before.
TemplateListServices are used when we want to show the service
as an additional layer of hierarchy when a user copies a template,
potentially across services [^1].
Normally a TemplateFolder is given "folders" and "templates" by
TemplateList [^2]; TemplateListService was doing it the other
way round and getting its own instead.
Using a subclass of TemplateList means we can make the approach
consistent, which will support the caching approach later on, as
well as simplifying how we work with templates and folders.
[^1]: 2e637f801f/app/main/views/templates.py (L356)
[^2]: bef0382cca/app/models/template_list.py (L31-L36)
This is slightly less efficient than getting the folder dicts from
"get_user_template_folders" directly, since:
- TemplateList returns both templates and folders.
- TemplateList encapsulates the dicts in model classes.
We'll compensate for this later on:
- We'll introduce a new caching approach to make the call fast.
- We'll expose a property to avoid the "if" in the comprehension.
Part of moving "get_template_folders" et al. into TemplateList so we
can cache it more effectively. This is slightly less efficient as
iterating a TemplateList will instantiate an object for each item
in the folder; but the difference is minimal.
Note that:
- The default template_type for TemplateList is "all".
- We need to pass realistic template "JSON" in the test now.
This is a very low impact bug since a user can always create such
templates after their service is live and not be subject to checks
we do before that point. Still, we may as well fix it.
The main benefit of this change is actually to contribute towards
moving methods like "get_templates" out of the Service class so
we can simplify and cache their results more effectively.
Note: I wanted to simplify the Service class further as the two
"has_" properties are only used once in the app code. Unfortunately
they are tightly coupled in one of the tests as well [^1].
[^1]: bef0382cca/tests/app/main/views/service_settings/test_service_settings.py (L1961-L1962)
`.xlsm` files are like `.xlxs` files but with macros enabled. They store
data in the same XML-based format as `.xlsx` files.
Pyexcel will try to use the xlrd package to parse `.xlsm` files. This
used to work because xlrd used to support reading `.xlsx` files. xlrd
has dropped support for `.xlsx` files in version 2 because of security
concerns. This means that when pyexcel asks xlrd to parse a `.xlsm` file
it causes an error.
This commit adds some branching to force `.xlsm` files to be opened
with pyexcel-xlsx instead, which does support `.xlsx` files.
At the moment if a user is pending we don’t show the ‘change’ link.
This is unhelpful because:
- there’s no way to remove this user
- there’s no way to change their phone number, if the reason that
they are still pending is because they’ve been unable to receive
the two factor code at the number they first provided
Direct string comparison in multiple places is prone to typos. It
also means that a consumer of the class needs to know that whether
a user is pending or active is held in the `state` property, which
is an implementation detail.
To keep the conditionals in the Jinja template more readable, this
commit moves the logic into a method on the model, where it can
be split over multiple statements and lines.
Implementing `__lt__` makes objects sortable.
Rather than reimplementing `__lt__` each time we want to make an object
sortable by name, we can inherit from the extant `SortByNameMixin`.
for a job to be finished there are two requirements:
* the status to be "finished"
* the percentage complete to be 100%
The job status is set to finished when the process_job task finishes
(even though not all process_row may have finished). The
percentage_complete is calculated by comparing the number of
notifications in the database with the number of rows in the
spreadsheet.
This was inadvertently changed from an "and" to an "or" clause two years
ago. This meant that people could download a report when the status was
finished but not all notifications were present in the database. Lets
change it back.
7d52ac97f1 (diff-44b012cad205379c481bed244ddb2294bae5ee85dcd01f4aee932a2bd85b67b2L86-R100)
When we combine simplified polygons from different areas we try to use
the polygons that are defined in metres as a speed optimisation.
This doesn’t work when those polygons are in several different
projections because the distance in metres is relative to the edge of
the projection. Just naively choosing one projection means that some
of the polygons will shift position on the earth by 100s of kilometers
once they are converted back to degrees.
To avoid this we need to:
- check for situations where we have polygons in multiple different
projections
- transforms these polygons back into degrees and let `Polygons` pick a
common projection for all of them before doing any
merging/smoothing/etc.
These are no longer used anywhere. Note that pyexcel-xlsx is still
a necessary dependency: it acts as an implicit plugin to pyexcel
that, surprisingly, allows pyexcel to...read excel files [1].
[1]: https://github.com/pyexcel/pyexcel-xlsx#as-a-pyexcel-plugin
Looking up the coordinate reference system for a given polygon’s bounds
is surprisingly expensive.
We can make things run faster by precomputing it at the time of
generating the simplified polygons, then storing it in the SQLite
database alongside the polygon data.
Transforming full resolution polygons to linear coordinates is somewhat
expensive. We can make things run faster by:
- looking at `simple_polygons` which have fewer points and are therefore
faster to transform.
- reusing polygons that have already been transformed where possible
Brings in https://github.com/alphagov/notifications-utils/pull/889/files
At the moment, we are not doing any transformation of features before
applying geometric algorithms to them. This is, in effect, assuming that
the earth is flat.
This new version of utils implements the transformation of our polygons
to a Cartesian plane. In other words, it converts them from being
defined in spherical degrees to metres.
For the admin app this means we need to convert places where the code
expects things to be measured in degrees to work in metres instead.
This makes it clearer that this model collection isn’t the organisations
for a user or a service or some other entity, like most model
collections are.
It will also lets us make a separate Organisations model, without the
name conflicting.
The model layer shouldn’t need to be concerned with sorting. For
services this means we can make a `SerialisedModelCollection` rather
than writing a manual loop.
