This changeset increases the worker process memory slightly in staging to help with running out of memory issues we are starting to see with our testing.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset bumps the memory of the API app processes to 1.5G as they are now starting to run out of memory due to recent app changes with gunicorn and gevent.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset makes a few adjustments to our sandbox environment config to make the sandbox deployments of the API much easier. It does this with the following:
* Adds a few environment variable values to the sandbox.yml file to cover the ones that were not there previously
* Adds a new script that can be run in place of memorizing any commands
* Adds documentation on how to configure and run the sandbox environment
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
The Celery workers are not using as much memory as we had been anticipating, and we are also hitting our memory quota. This adjusts them down a bit to free up a bit more.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset bumps our production app memory for the API to 3 GB available in anticipation of the shift of the job cache being managed by the application itself instead of a worker process.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset increases the number of Celery workers available to our app in the production environment. It also bumps the amount of memory available to them to 3G each.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset adds a bit more memory to the staging worker instance, and it bumps the number of worker instances to 2 in staging and demo.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset adjusts our production manifest settings to strike a balance with memory usage and app/worker instances. We are currently running into a memory quota but would still like to increase our memory for improved app performance and stability.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset updates the memory available to the API app and workers to be 4 GB to improve app stability and performance.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset adjusts our worker process memory to 1G from 512MB in production. We recently saw memory-related crashes for the Celery worker processes, meaning they did not have enough available to them. This was compounded by an ongoing platform issue with cloud.gov due to Cloud Foundry VMs operating with a Linux kernel that has a memory allocation issue.
Fixes for the Linux kernel and Cloud Foundry VMs are in flight and cloud.gov is tracking this closely, but we can help ourselves in the mean time and given the increased usage with pilot partners, it makes sense to adjust this anyway.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset adjusts the REDIS_ENABLED environment variable to match how the admin app is set up to make sure the API properly connects to the Redis service.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset switches AWS service touchpoints to use their FIPS-enabled counterparts. Note that S3 has some specific configuration associated with it.
This changeset also updates our allow ACLs to cover the FIPS-enabled endpoints. We should investigate removing the non-FIPS endpoints as a part of this.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
This changeset adds another endpoint to the staging egress proxy to enable access to AWS CloudWatch logs.
NOTE: We will have to probably do this for production as well.
Signed-off-by: Carlo Costino <carlo.costino@gsa.gov>
