Zack Krida 8:46 pm on May 20, 2024
Tags: openverse-priorities ( 18 ), priorities ( 17 )

Openverse Monthly Priorities Meeting 2024-06-05

OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. contributors will host a community meeting to discuss priorities for June at 1500 UTC on June 5th, 2024.

A sync video chat link will be provided in the #openverse channel of the Making WordPress Chat. We hope to see you there!

You can read the ongoing notes document for these meetings here.

This meeting in particular will also serve as a mid-year check in for the project and our goals.

#openverse-priorities, #priorities

Zack Krida 7:27 pm on June 5, 2024
Summary

OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. contributors discussed priorities for June in this meeting. View the realtime notes here and a summary below. For context on the different statuses, see our documentation.

Completed Projects
- Clearly document all media properties (moved to Success) – The final PR for the catalog was merged today!
- Move Openverse API and catalog to openverse.org subdomains (moved to Shipped) – After a small hiccup with the coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. editor integration, these changes are live. The final PR to clean up resources in our infrastructure repo is under review.
In Progress Projects

Incorporate Rekognition data into the catalog #431

APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. work is complete. Frontend work is nearly complete, with copy for the generated tags page under review. The IP for the incorporation of the tags into the catalog is open and under discussion: #4417. Discussion there has raised important points concerning the data normalization project (#430) and the removal of the ingestion server (#3925).

Because this IP has raised so many new ideas and considerations, we discussed the importance of making sure to remain focused and only implement changes necessary to the completion of the project. Other ideas should be carefully documented but not rushed into.

Sensitive content report moderation workflow #383

This project is progressing nicely with a lot of recent work on new Django UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing. views. @zackkrida raised an idea to ask @sarayourfriend about: The remaining implementation plan involves AWS Rekognition and relates to some of the topics discussed in the “Augment the catalog database with suitable Rekognition tags”. Could there be benefit to extracting this piece of the project into a standalone, follow-up issue, and removing it as a requirement of the “Sensitive content report moderation workflow”? No one felt strongly that this is important but we thought that it could make sense to wait on it.

Data normalization #430

We acknowledged some delays and setbacks in this project, along with a lot of learning resulting from those concerns. The remaining work for this project entails creating three batched_update DAG runs with the goal of:
- Updating the catalog rows from generated TSVs, excluding tags #3415
- Decoding and de-duplicating existing tags
- Removing deny-listed tags from the catalog
@krysal has also created a PR to revert the saving of cleanup TSV files: #4443

Finally, we proposed refining the scope of this project to remove all cleanup steps from the Data refresh process but to preserve the filtering of tags based on their accuracy thresholds. The accuracy threshold filtering will now be preserved and iterated on as part of the Removal of the ingestion server #3925 project. This also spawned discussion of some related side work:
- A document/visualization describing our current and desired ETL steps all throughout the project
- A number of issues around possible changes to the batched update, including splitting manual & automated runs into separate DAGs, increased reporting for manually run batched updates, a document defining how batched updates should be run, and possibly a document which serves as an “execution log” for batched updates (we might be able to use airflow directly for this, especially if we split the runs out). This will all be captured in a batched update milestone, which isn’t necessarily part of a project but just as a mechanism to group the work together.
Removal of the ingestion server #3925

Development continues here, with the now added dimension of explicitly including tag filtering as part of the new (sans-ingestion-server) data refresh process.

Nuxt 3 migration #411

Nuxt 3 was deployedDeploy Launching code from a local development environment to the production web server, so that it's available to visitors. to a testing branch, updated, and load tested in comparison with our staging site. Testing indicated that by default Nuxt 3’s memory usage appears to be better than that of Nuxt 2. These early tests are promising! Next steps include:
1. Add Sentry to make the Nuxt 3 branch have exactly the same functionality as the current production app.
2. Build Nuxt 3 app with all available locales
3. Increase the CPU of nuxt-preview and run the load testing with all locales.
4. Finally, Route some portion of the production traffic to the Nuxt 3 app to test production loads.
Upcoming Projects

We didn’t have time to discuss upcoming work in depth, but did mention focusing on “search relevancy” projects after the current batch of projects is resolved. We also reclassified the project Expand Content Providers Metadata as a “search experience” project rather than a relevancy project.

Welcome!