openverse

Merged PRs

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.

• #4603: Limit access for content moderators
• #4635: Remove client application headers from API responses
• #4640: Publish changelog for api-2024.07.20.10.58.11
• #4649: Use a separate logger for dead links check that Sentry will ignore
• #4650: Publish changelog for api-2024.07.23.07.16.35
• #4656: Update media list admin view to add ES-powered search

Catalog

• #4652: Remove env from Airflow alerts, improve ES alert username & icon
• #4661: Fix update query and increase simultaneous tasks of the `add_license_url` DAG

Documentation

• #4639: Publish changelog for frontend-2024.07.20.10.58.11
• #4640: Publish changelog for api-2024.07.20.10.58.11
• #4650: Publish changelog for api-2024.07.23.07.16.35
• #4659: Nuxt 3 robots.txt fixes

Frontend

• #4639: Publish changelog for frontend-2024.07.20.10.58.11
• #4653: Fix HTMLHTML HTML is an acronym for Hyper Text Markup Language. It is a markup language that is used in the development of web pages and websites. creator crediting incorrect image tags
• #4659: Nuxt 3 robots.txt fixes
• #4660: Remove the single-result endpoints from the robots.txt

Infra

• #4652: Remove env from Airflow alerts, improve ES alert username & icon

Management

• #4625: Migrate automations to PDM
• #4626: Migrate `utilities/generate_test_locales` to PDM
• #4632: Fix version qualifiers in `pyproject.toml` files
• #4657: Remove the "2 Reviews Needed" column automation from the PR board

Closed issues

API

• #3640: Finalise access control for Django admin content moderator users
• #4350: Reenable Science Museum provider in Django admin
• #4569: Remove client application headers from API responses

Catalog

• #4477: Airflow should support dark mode

Frontend

• #2150: Further accessbility improvements to search result titles
• #4602: Openverse.org embeds are not functioning

Infra

• #4613: Remove environment name from Airflow Alerts message username

openverse-infrastructure

Merged PRs

Catalog

• #984: Bump ingestion server and catalog

Infra

• #986: Use correct variable name for Airflow deployment recovery
• #991: Fix GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ branch protections

Ingestion Server

• #984: Bump ingestion server and catalog

Management

• #988: 🔄 synced file(s) with WordPress/openverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.
• #989: Only require a single review
• #990: 🔄 synced file(s) with WordPress/openverse

Closed issues

API

• #971: Uncaught `ValueError` in API with malformed URI escape sequences in query string

Infra

• #985: Airflow deployment unable to rollback copied config if restart fails

#openverse, #week-in-openverse

openverse

Merged PRs

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.

• #4567: Expand `ov` documentation and replace environment setup docs with `ov` instructions
• #4582: Update pook to v2
• #4598: Decrease link validation timeout
• #4605: Remove `proxy` service and make API NGINXNGINX NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers. https://www.nginx.com/. setup similar to the frontend
• #4612: Publish changelog for api-2024.07.15.03.59.12

Catalog

• #4567: Expand `ov` documentation and replace environment setup docs with `ov` instructions
• #4582: Update pook to v2
• #4601: Create a dataclass for indexer worker TaskStatus
• #4610: Add `catalog_cleaner` DAG
• #4615: Update ingestion server removal IP with EC2 approach
• #4629: Update dependency apacheApache Apache is the most widely used web server software. Developed and maintained by Apache Software Foundation. Apache is an Open Source software available for free.-airflow to v2.9.3 [SECURITY]
• #4634: Publish changelog for catalog-2024.07.19.17.07.23

Documentation

• #4567: Expand `ov` documentation and replace environment setup docs with `ov` instructions
• #4585: Make running Playwright to update snapshots easier
• #4605: Remove `proxy` service and make API NGINX setup similar to the frontend
• #4610: Add `catalog_cleaner` DAG
• #4611: Publish changelog for frontend-2024.07.15.03.59.13
• #4612: Publish changelog for api-2024.07.15.03.59.12
• #4615: Update ingestion server removal IP with EC2 approach
• #4633: Publish changelog for ingestion_server-2024.07.19.17.07.25
• #4634: Publish changelog for catalog-2024.07.19.17.07.23

Frontend

• #4543: Make the sources tables responsive using CSSCSS CSS is an acronym for cascading style sheets. This is what controls the design or look and feel of a site.
• #4567: Expand `ov` documentation and replace environment setup docs with `ov` instructions
• #4585: Make running Playwright to update snapshots easier
• #4611: Publish changelog for frontend-2024.07.15.03.59.13
• #4627: Remove unnecessary locales downloads from CI
• #4628: Update pnpm and use corepack

Infra

• #4598: Decrease link validation timeout

Ingestion Server

• #4615: Update ingestion server removal IP with EC2 approach
• #4631: Bump sentry-sdk from 2.7.1 to 2.8.0 in /ingestion_server
• #4633: Publish changelog for ingestion_server-2024.07.19.17.07.25

Management

• #4567: Expand `ov` documentation and replace environment setup docs with `ov` instructions
• #4585: Make running Playwright to update snapshots easier
• #4609: Bump jinja2 from 3.1.3 to 3.1.4 in /utilities/generate_test_locales
• #4622: Migrate `utilities/project_planning` to PDM
• #4623: Migrate `utilities/dead_link` to PDM
• #4624: Migrate `utilities/provider_tallies` to PDM
• #4627: Remove unnecessary locales downloads from CI
• #4628: Update pnpm and use corepack

Closed issues

API

• #4332: Limit the number of concurrent dead link requests
• #4507: Decrease dead link verification request timeout

Catalog

• #3415: Use the `batched_update` DAG with stored CSVs to update Catalog URLs
• #4534: Create a dataclass for indexer worker TaskStatus

Frontend

• #470: Sources table does not show the full URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org in mobile size
• #4621: Stop unnecessary downloads of translations for storybook build

Infra

• #4507: Decrease dead link verification request timeout

Management

• #3807: Replace `pnpm/action-setup` with Corepack
• #4535: Make running Playwright to update snapshots easier

openverse-infrastructure

Merged PRs

API

• #962: Fix API deployment timeout; add nginx container healthchecks
• #978: Decomission alternative prod subdomains; blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. malicious traffic

Frontend

• #978: Decomission alternative prod subdomains; block malicious traffic

Infra

• #962: Fix API deployment timeout; add nginx container healthchecks
• #978: Decomission alternative prod subdomains; block malicious traffic
• #980: Enable ASG-less immutable-ec2-service and convert indexer worker pools to use it
• #983: Add malicious IP

Closed issues

API

• #959: Consider dropping production alias domain names

Frontend

• #959: Consider dropping production alias domain names

Infra

• #959: Consider dropping production alias domain names

#openverse, #week-in-openverse

openverse

Merged PRs

Catalog

• #4552: Split `batched_update` DAG into automated and manual DAGs

Documentation

• #4552: Split `batched_update` DAG into automated and manual DAGs

Frontend

• #4584: Fix TypeError by checking if duration is not Finite before setting cu…
• #4591: Remove all usage of jest

Management

• #4560: Make `ov` base image updates overall more convenient and take immediate effect
• #4591: Remove all usage of jest
• #4606: Bump certifi from 2024.2.2 to 2024.7.4 in /automations/python
• #4607: Bump tqdm from 4.66.2 to 4.66.3 in /utilities/generate_test_locales
• #4608: Bump certifi from 2024.2.2 to 2024.7.4 in /utilities/generate_test_locales

Closed issues

Catalog

• #4457: Separate the batched update DAG into manually and automatically triggered DAGs

Frontend

• #4156: TypeError: Failed to set the 'currentTime' property on 'HTMLMediaElement': The provided double value is non-…

Management

• #4545: Install pipx packages outside the dev-env volume

openverse-infrastructure

Merged PRs

Infra

• #960: Fix Nuxt 3 frontend alarm
• #966: Add all initial datasources to Grafana
• #972: BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. Internet Archive from single and search views
• #976: Remove usage of `terraform-aws-modules/vpc/aws`

Management

• #977: 🔄 synced file(s) with WordPress/openverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.

Closed issues

Infra

• #953: Set up Grafana Cloud for initial access and use by Openverse maintainers
• #969: Remove `terraform-aws-modules/vpc/aws` usage

#openverse, #week-in-openverse

openverse

Merged PRs

Analytics

• #4550: Fix Plausible setup after domain was already set
• #4568: Specify pull policy for `openverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.-` images

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.

• #4499: Include xml in frontend attribution options
• #4530: Drop FK constraint on media_obj in MediaDecisionThrough, update backfillmoderationdecision command
• #4536: Do not use unstable pook reference for API
• #4540: Add linting for Dockerfiles
• #4544: Create sensitive and deleted media models for decisions
• #4547: Publish changelog for api-2024.06.24.18.01.42
• #4551: Shorten PDM hash to first 8 characters
• #4554: Remove backfillmoderationdecision management command after production run
• #4568: Specify pull policy for `openverse-` images

Catalog

• #4475: Add DAG to decode and deduplicate image tags with escaped literal unicode sequences
• #4495: Fix placing test S3 data into MinIO
• #4497: Add CI/CD and PDM to new indexer worker
• #4526: Fix separators in catalog and dev-env images and dev-env volume
• #4540: Add linting for Dockerfiles
• #4555: Ensure plpython3u exists in live db when using it
• #4557: Remove `trim_and_deduplicate_tags` DAG after successful run
• #4568: Specify pull policy for `openverse-` images

Documentation

• #4417: Implementation Plan: Augment the catalog database with suitable Rekognition tags
• #4475: Add DAG to decode and deduplicate image tags with escaped literal unicode sequences
• #4546: combine frontend testing documentation pages
• #4547: Publish changelog for api-2024.06.24.18.01.42
• #4548: Publish changelog for frontend-2024.06.24.18.01.44
• #4557: Remove `trim_and_deduplicate_tags` DAG after successful run
• #4562: Publish changelog for frontend-2024.06.26.17.18.17

Frontend

• #4291: Display generated tags separately
• #4497: Add CI/CD and PDM to new indexer worker
• #4499: Include xml in frontend attribution options
• #4509: Replace "Over…" language with more precise "Top…"
• #4516: Add caching to frontend NginxNGINX NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers. https://www.nginx.com/. configuration
• #4523: Fix possible TypeError when accessing properties of `route.value`
• #4540: Add linting for Dockerfiles
• #4548: Publish changelog for frontend-2024.06.24.18.01.44
• #4549: Re-add tags page text
• #4559: Fix flaky VCollectionHeader snapshot tests
• #4562: Publish changelog for frontend-2024.06.26.17.18.17

Infra

• #4516: Add caching to frontend Nginx configuration

Ingestion Server

• #4471: Remove single quotes in values of Ingestion Server's TSV files
• #4529: Upload Ingestion Server's TSV files to AWS S3 (skip tags)

Management

• #4497: Add CI/CD and PDM to new indexer worker
• #4526: Fix separators in catalog and dev-env images and dev-env volume
• #4539: Add dev tools jq and HTTPie to `ov`
• #4540: Add linting for Dockerfiles
• #4546: combine frontend testing documentation pages
• #4568: Specify pull policy for `openverse-` images

Closed issues

API

• #4430: Attribution: XML/RDF/Turtle please.
• #4454: Determine if all tags in the catalog database have an associated provider
• #4512: The `AbstractMediaDecisionThrough` class and its inheriting classes shouldn't use actual foreign keys to media tables
• #4513: Creating `MediaDecision` has no effect on deindexed actions

Catalog

• #663: Upgrade catalog to Python 3.11
• #1464: Create a DAG to log and report code review response times
• #4199: Remove and de-duplicate tags with leading/trailing whitespace
• #4454: Determine if all tags in the catalog database have an associated provider
• #4494: Test S3 inaturalist files are not found in MinIO

Documentation

• #4040: Implementation Plan: Augment the catalog database with suitable Rekognition tags
• #4514: Combine frontend testing documentation pages

Frontend

• #461: Add a message to inform the user about more filters when one media type is chosen
• #2130: Update sensitive browsing designs to allow re-blurring of search results
• #2213: Frontend local dev error `Cannot convert undefined or null to object`
• #4192: Displaying machine-generated content
• #4379: Write a page describing the machine-generated tags for the frontend
• #4430: Attribution: XML/RDF/Turtle please.
• #4470: Add caching of static assets to frontend Nginx
• #4522: TypeError: Cannot read properties of undefined (reading 'name') in `useMatchRoute()`
• #4558: vcollectionheader storybook visual regression test broken

Infra

• #4470: Add caching of static assets to frontend Nginx

Ingestion Server

• #3912: Upload Ingestion Server's TSV files to AWS S3

openverse-infrastructure

Merged PRs

API

• #951: Fix and improve api-management-command script

Frontend

• #940: Cache frontend assets at edge for 3 days

Infra

• #924: Add `StatusCheckFailed` alarms for EC2 services
• #940: Cache frontend assets at edge for 3 days
• #942: Add Grafana PDC
• #944: Touch up indexer worker pools to match IP requirements
• #947: Bypass WAF for Cloudflare Access services
• #948: Ignore changes to `actions_enabled` on externally controlled alarm
• #951: Fix and improve api-management-command script
• #956: BlockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. malicious ASNs and UA string pattern 2024-06-27/28 incident

Management

• #955: 🔄 synced file(s) with WordPress/openverse

Closed issues

API

• #950: Disable migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. running during management command executions

Frontend

• #927: Change frontend edge caching rules

Infra

• #254: Audit logging costs and find savings
• #792: Add EC2 instance state change monitor
• #927: Change frontend edge caching rules
• #941: Wire up Grafana PDC
• #943: Add Cloudflare WAF skip rule for Airflow
• #950: Disable migration running during management command executions

#openverse, #week-in-openverse

openverse

Merged PRs

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.

• #4415: Add `backfillmoderationdecision` management command
• #4444: More precisely handle waveform generation failures
• #4467: Publish changelog for api-2024.06.07.17.19.06

Catalog

• #4068: Add verbose logging option to `ProviderDataIngester`
• #4429: Add DAG to trim and deduplicate tags
• #4447: Capture thumbnails during europeana ingestion
• #4460: Update the 'updated_on' column during popularity refresh
• #4481: Moved by tag from the fuzzy match group to exact match

Documentation

• #4441: Add favicon to Storybook
• #4466: Publish changelog for frontend-2024.06.07.17.19.06
• #4467: Publish changelog for api-2024.06.07.17.19.06
• #4485: Order quickstart links, add missing catalog link
• #4487: Publish changelog for ingestion_server-2024.06.13.17.07.56

Frontend

• #4441: Add favicon to Storybook
• #4442: Tags page copy
• #4466: Publish changelog for frontend-2024.06.07.17.19.06

Ingestion Server

• #4487: Publish changelog for ingestion_server-2024.06.13.17.07.56

Management

• #4441: Add favicon to Storybook
• #4472: Fix ov corepack and pdm existence issues

Closed issues

API

• #3641: Create `ModerationDecision` backfill management command
• #4218: Audio waveform should return 424 instead of 500 when waveform cannot be generated
• #4474: The API `result_count` is no more than 240 for unauthenticated requests

Catalog

• #1420: Add verbose logging option to `ProviderDataIngester`
• #4403: Capture thumbnails during Europeana ingestion
• #4453: Remove deny-listed tags in the catalog with the `batched_update` DAG
• #4464: Move "by" tag contains filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. to tag exact match filter

Documentation

• #4479: Link to the catalog quickstart guide from the central quickstart page

Infra

• #2037: Move OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. API and catalog to `openverse.org` subdomains
• #4489: Environment variables set when running `ov` not passed to the container

Management

• #4468: `ov` will hang silently if `corepack` is used and there is an update to PNPM
• #4469: `ov` does not capture error output if `pdm` not installed on host

openverse-infrastructure

Merged PRs

Infra

• #920: Remove openverse.engineering Cf Access rules and update documentation
• #928: Move Nuxt 3 to prod, create new listener rule for split testing

Management

• #926: 🔄 synced file(s) with WordPress/openverse

Closed issues

Infra

• #609: Use pre-commit and lint setup identical to the monorepo
• #785: Remove any remaining Cloudflare resources from `openverse.engineering` zone

Management

• #438: Enable merge queues and require PRs to be up-to-date before merging
• #609: Use pre-commit and lint setup identical to the monorepo

#openverse, #week-in-openverse

openverse

Merged PRs

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways.

• #4402: Rename ContentProvider to ContentSource
• #4419: Update docker.io/redis Docker tag to v7.2.5
• #4434: Publish changelog for api-2024.06.03.15.35.02
• #4440: Handle tags without provider in media admin view

Catalog

• #4366: Add catalog media properties documentation

Documentation

• #4366: Add catalog media properties documentation
• #4432: Update docs to recommend blobless cloning strategy
• #4435: Add a link to the committer announcements in the committer docs
• #4436: Update assets in the documentation
• #4448: Updated Playwright Codegen broken link
• #4449: Jest docs broken link fixed

Frontend

• #4420: Update pnpm to v9.1.4
• #4423: Update Node.js to v20.14.0
• #4424: Update dependency @playwright/test to v1.44.1
• #4425: Update dependency eslint-pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party-tsdoc to ^0.3.0
• #4426: Update dependency prettier-plugin-tailwindcss to v0.6.1
• #4428: Ensure required DB extension is installed before attempting to setup plausible
• #4431: Add Nuxt 3 folders to gitignore
• #4433: Publish changelog for frontend-2024.06.03.15.35.03
• #4437: Delete `frontend/src/stories/` directory
• #4445: Update pnpm to v9.2.0

Ingestion Server

• #4418: Update dependency elasticsearch to v8.13.2
• #4443: Revert "Save cleaned data of Ingestion Server to AWS S3 (#4163)"

Management

• #4392: Add load testing script for frontend
• #4416: Move NGINXNGINX NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers. https://www.nginx.com/.-based services out of the API profile
• #4421: Update workflows
• #4438: Overhaul the complete labelling system
• #4450: Fix incorrect brackets in PR automation
• #4451: Update pr_automations.yml with missing character
• #4462: Bump tornado from 6.4 to 6.4.1 in /utilities/project_planning

Closed issues

API

• #3943: Implement logging for moderation events
• #3944: Implement and surface value-based deferred metrics
• #3946: Implement and surface list-based deferred metrics
• #4289: CI + CD builds `nginx` image during API up
• #4346: Rename the `ContentProvider` model to `ContentSource`
• #4439: `/api/api/admin/media_report.py, line 387, in change_view` can fail if the tag does not have a provider

Catalog

• #2187: Create the media properties description file
• #4255: iNaturalist is no longer able to access S3

Documentation

• #4329: Dramatically improve cloning speed for contributors
• #4395: Add a favicon to our Docs site

Frontend

• #3972: Update references to audio works to use "audio track(s)"
• #4391: Create a script for load-testing the frontend

Management

• #1968: Implementation Plan: Computer vision metadata for content reports
• #3823: Seek alternatives to `banyan/auto-label`
• #4203: Stack label is not applied to contributor PRs
• #4391: Create a script for load-testing the frontend
• #4400: Local Plausible setup can fail

openverse-infrastructure

Merged PRs

Infra

• #916: Redirect all .engineering API requests
• #918: Add nuxt-preview cache rule
• #921: Update .engineering to .org redirect to exclude GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ media inserter requests
• #922: Bypass cache and WAF for non-production frontends with load testing UA string

Management

• #923: Add Princewill Onyenanu (madewithkode) as a committer

Closed issues

API

• #781: Open PR in Gutenberg to point integration to `api.openverse.org`
• #782: Open PR to point Jetpack integration to api.openverse.org
• #783: Remove headerHeader The header of your site is typically the first thing people will experience. The masthead or header art located across the top of your page is part of the look and feel of your website. It can influence a visitor’s opinion about your content and you/ your organization’s brand. It may also look different on different screen sizes. check from Cloudflare redirect rule

Infra

• #779: Redirect production API requests to `api.openverse.org` when a special testing header is present
• #784: Replace API openverse.engineering Cloudflare domain records with noops
• #787: Downgrade openverse.engineering Cloudflare plan to the free tier
• #917: Add cache rules for `nuxt-preview.openverse.org` to not cache it in Cloudflare

Management

• #740: PR labeller should apply stack labels for infrastructure repo

#openverse, #week-in-openverse