Title: infrastructure – Make Openverse – WordPress.org --- # Tag Archives: infrastructure [ ](https://profiles.wordpress.org/aetherunbound/) [Madison Swain-Bowden](https://profiles.wordpress.org/aetherunbound/) 8:35 pm _on_ September 2, 2022 Tags: [airflow ( 3 )](https://make.wordpress.org/openverse/tag/airflow/), [data-refresh ( 2 )](https://make.wordpress.org/openverse/tag/data-refresh/), ecs, infrastructure, [openverse ( 186 )](https://make.wordpress.org/openverse/tag/openverse/) # 󠀁[Applying ECS to the ingestion server/data refresh](https://make.wordpress.org/openverse/2022/09/02/applying-ecs-to-the-ingestion-server-data-refresh/)󠁿 This was a passing thought I had that I wanted to note somewhere. Currently the [ingestion server](https://github.com/WordPress/openverse-api/tree/v2.5.11/ingestion_server) is a small [Falcon](https://falcon.readthedocs.io/en/stable/) app that runs most aspects of the data refresh, but then also (in staging/prod) interacts with a fleet of “indexer worker” EC2 instances when performing the Postgres -> Elasticsearch indexing. We have plans for moving the data refresh steps from the ingestion server into Airflow. Most of these steps are operations on the various databases, so they’re not very processor-intensive on the server end. However, the indexing steps _are_ intensive, which is why they’re spread across 6 machines in production (and even then it can take a number of hours to complete). We could replicate this process in Airflow by setting up Celery-based workers so that the tasks run on a separate instance from the webserver/scheduler. Ultimately I’d like to go this route (or use something like the [ECS Executor](https://github.com/aelzeiny/airflow-aws-executors) rather than Celery), but that’s a non-trivial effort to complete. One other way we could accomplish this would be to use ECS tasks! We could have a container defined specifically for the indexing step, which expects to receive the range on which to index and all necessary connection information. We could then kick off `_n_` of those jobs using the [`EcsRunTaskOperator`](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/ecs.html#run-a-task-definition), and wait for completion using the [`EcsTaskStateSensor`](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/ecs.html#aws-ecs-task-state-sensor) to determine when they complete. This could be done in our current setup without any new Airflow infrastructure. It’d also allow us to remove the indexer workers, which currently sit idle (albeit in the `stopped` state) in EC2 until they are used. [#airflow](https://make.wordpress.org/openverse/tag/airflow/), [#data-refresh](https://make.wordpress.org/openverse/tag/data-refresh/), [#ecs](https://make.wordpress.org/openverse/tag/ecs/), [#infrastructure](https://make.wordpress.org/openverse/tag/infrastructure/), [#openverse](https://make.wordpress.org/openverse/tag/openverse/) * [Login to Reply](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fopenverse%2F2022%2F09%2F02%2Fapplying-ecs-to-the-ingestion-server-data-refresh%2F%23respond&locale=en_US) [ ](https://profiles.wordpress.org/aetherunbound/) [Madison Swain-Bowden](https://profiles.wordpress.org/aetherunbound/) 8:12 pm _on_ June 17, 2022 Tags: api, infrastructure, [openverse ( 186 )](https://make.wordpress.org/openverse/tag/openverse/) # 󠀁[Mitigating out of terms API usage](https://make.wordpress.org/openverse/2022/06/17/mitigating-out-of-terms-api-usage/)󠁿 Yesterday at 20:20 UTC, we released [version 2.5.5 of our API](https://github.com/WordPress/openverse-api/releases/tag/v2.5.5)! Along with a few dependency upgrades and DevEx improvements/fixes, this release also brings [an important change regarding anonymous API requests](https://github.com/WordPress/openverse-api/pull/727). After v2.5.5, any media searches that are made without an APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. key cannot request more than 20 results per page. This change was made in order to mitigate behavior we were seeing on the API which was adversely affecting performance for other users, our capacity to update the data that backs OpenverseOpenverse **Openverse **is a search engine for openly-licensed media, including images and audio. Find Openverse on [GitHub](https://github.com/WordPress?q=openverse) and at [https://openverse.org](https://openverse.org)., and our ability to deployDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. new changes to the API. Our API [Terms of Service](https://wordpress.github.io/openverse-api/terms_of_service.html#licensed-uses-and-restrictions) state: > – A user must adhere to all rate limits, registration requirements, and comply > with all requirements in the Openverse API documentation; > – A user must not scrape the content in the Openverse Catalog; > – A user must not use multiple machines to circumvent rate limits or otherwise > take measures to bypass our technical or security measures; > – A user must not operate in a way that negatively affects other users of the > API or impedes the WordPress Foundation’s ability to provide its services; ## Background Beginning around May 18th, we saw a significant increase in traffic. ![](https://make.wordpress.org/openverse/files/2022/06/image-1.png) Total requests made to api.openverse.engineering over the last 30 days While the digital demographics (browser, user agent, OS, device type, etc.) were quite varied, one feature stuck out – these requests were all being made with the` page_size=500` parameter. ![](https://make.wordpress.org/openverse/files/2022/06/image-4.png) Total requests made to [api.openverse.engineering](http://api.openverse.engineering) over the last 30 days **using the `page_size=500` parameter** Over the course of the last 30 days, these requests constituted almost 80% of our total traffic! While our application is designed to handle this many requests, it is _not _designed to handle each request querying for 500 results per page (the default page size is 20). As such, this had created significant strain on our Elasticsearch cluster and eventually caused disruptions in the API’s ability to serve results. The image below combines a few of our monitoring tools to show a general correlation between the `page_size=500` requests and our Elasticsearch resource utilization. ![](https://make.wordpress.org/openverse/files/2022/06/image-5.png) Request count compared to Elasticsearch resource utilization Even before this release, our application was set up to throttle individual, anonymous users to [1 request/second](https://api.openverse.engineering/v1/#section/Introduction). These `page_size=500` requests were coming from a myriad of different hosts; the initiator was able to circumvent the individual throttles by employing a large number of machines (also known as a [botnet](https://en.wikipedia.org/wiki/Botnet)). These machines were also predominantly tied to a single data center and a single [ASN](https://www.thousandeyes.com/learning/glossary/as-autonomous-system), which led us to believe this was orchestrated by a single user. This behavior was clearly in violation of our Terms of Service, since it was: 1. Not using a registered API key for high-volume use 2. Scraping data from Openverse 3. Using multiple machines to circumvent the application throttles 4. Consuming significant enough resources that it impacted other users of Openverse ## Mitigation As mentioned above, we deployedDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. a change which would now return a [401 Unauthorized](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401) for any anonymous requests to the API that included a `page_size` greater than the default of 20. Almost immediately after deployment, we saw this mitigation take effect when observing request behavior: ![Screenshot of a Cloudflare analytics page. The graph in the center shows total requests with page_size=500, separated by status code over 6 hours. A consistent number of requests (split between 301 and 200) can be seen starting at 9:00 PST. At 13:00 PST, the number of 401 requests begins to overtake the number of 200 requests. After 13:15, the number of 200 requests drops to zero and all requests returned are 401s.](https://make.wordpress.org/openverse/files/2022/06/image-6.png) Total number of `page_size=500` requests made over the course of 6 hours, separated by return status code In the above graph, you can see where we deployed v2.5.5 (~13:00 PST) – the number of `200 OK` responses decreased, and the number of `401 Unauthorized` responses increased significantly! Eventually all of the `page_size=500` requests were being rejected as unauthorized. With this change, we were able to successfully mitigate the botnet and return our resource consumption to typical levels. This can be seen easily with a few Elasticsearch metrics: ![](https://make.wordpress.org/openverse/files/2022/06/image-8-633x1024.png) Elasticsearch metrics over the last 12 hours While the intention behind Openverse is to make openly licensed media easy to access, we don’t currently have the capacity to enable users to access the entire dataset at once. We do plan on exploring options for this in the future. We’re pleased that this mitigation was successful, and we will continue to be vigilant in ensuring uninterrupted access to Openverse for our users! [#openverse](https://make.wordpress.org/openverse/tag/openverse/), [#infrastructure](https://make.wordpress.org/openverse/tag/infrastructure/), [#api](https://make.wordpress.org/openverse/tag/api/) * [Login to Reply](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fopenverse%2F2022%2F06%2F17%2Fmitigating-out-of-terms-api-usage%2F%23respond&locale=en_US)