Zack Krida 2:25 pm on November 16, 2022

Thinking towards 2023

The OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. community has been discussing plans for upcoming projects and our vision for the coming year in a few different places. I’d like to suggest we focus all remaining discussion here.

I am personally curious around the following guiding questions:

1. What could we approve on from the approach taken in our 2022 roadmap?
2. What guiding, big-picture goals should inform our work in 2023?
3. What projects might we take on, related to those goals?

I’d like to propose we take about 3 weeks to discuss these topics in depth, supplemented with synchronous sessions. We can move our community “priorities” meeting one week earlier to accommodate this. I’ll post about that shortly.

Zack Krida 2:25 pm on November 16, 2022

1. What could we approve on from the approach taken in our 2022 roadmap?
- Zack Krida 2:27 pm on November 16, 2022
  
  It will be best to be strategic about lowering the number of things we accomplish and increasing the utility of the things we do accomplish. We should commit to about 50% fewer projects than we planned for 2022. We were very ambitious and still learning how to work together as a team when we envisioned some of these specific projects within the big-picture goals.
  
  Additionally, what we listed as ‘potential projects’ for 2023 were not actually projects in a useful sense. Instead, they were more like concepts or aspects of the project which would need attention. I think we’d benefit greatly from taking the approach we’ve gravitated towards: making projects as small and focused as possible.
  - Madison 9:19 pm on November 22, 2022
    Looking back on 2022, I think the framing of the 4 overarching goals was useful. I agree that more formalized projects would be appropriate, and with the reduction in expected projects.
    
    I don’t think that “Add new media types” should be its own overarching goal for next year. I would like to put 3D models on the 2023 plan, but even then would it just be “3D Models”? We had spoken previously about adding them just to the catalog and/or APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. since the frontend would be a nontrivial effort, but there were also discussions about waiting until we can do this work across the stack at the same time so as to coordinate the data fields.
    
    Many of the projects listed in the 2022 plan were also abandoned (in the sense that I don’t think we plan on doing this work anymore at all, beyond de-prioritizing), namely:
    
    Explore existing Rekognition data from Creative Commons [Q2-Q4] Not started
    
    Restore the Common Crawl pipeline to ingest images from CC providers [Q3] Not started
    
    I think the other infrastructure goals could be carried forward, but we’d want to be more specific about each of the projects. Perhaps:
    
    Migrate API to ECS and automate deployments
    
    Migrate ingestion server to ECS and automate deployments (perhaps we’d want to prioritize moving this functionality to Airflow instead and eradicating the ingestion server?)
    
    Implement Grafana/Prometheus and tie in metrics from each service in our stack
    - sarayourfriend 2:05 am on December 5, 2022
      
      > Explore existing Rekognition data from Creative Commons [Q2-Q4] Not started
      
      I sure hope we do intend to come back to this dataset and make a definitive decision about whether this kind of thing can be useful for OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.. In particular, I think we should consider this as one of many potential avenues for improving result relevancy that would need to be prioritised along with other approaches to result relevancy improvements. That is, it is one option that we need to understand it better to be able to correctly prioritise further work on it compared to other relevant work.
      
      Questions that might be useful to ask of this data set:
      
      1. How reliable is it?
      2. How free of scandal is it, considering the issues with AI image classification in the past?
      3. What risk tolerance do we have vis a vis 1 and 2.
      4. Can we get a sense of how effective/useful the data would be for our existing relevancy approach (full text search of title, tags, and description with the title weighted more heavily) compared to other potential ways of improving relevancy. For example, could using this dataset be more or less useful than more dynamic score calculation using derived result usefulness/accuracy based on analytics data?
      
      If we have a strong gut feeling that this dataset is disqualified or extremely unlikely to not be useful, then that feels like an important thing to document explicitly so that we can correctly prioritise it. Otherwise, I fear it will remain an obscure dataset that we do not well understand the benefits or risks of and will continue to be pushed down without clear explanation or reason.
      
      The fact is that the grant was several thousands of dollars of computation that was run. It would be a shame for that data to go to waste or for its usefulness to be further delayed merely because we do not well understand the implications of using it or how to use it.
    - sarayourfriend 2:18 am on December 5, 2022
      
      Regarding the infrastructure points you brought up:
      
      Migrate ingestion server to ECS and automate deployments (perhaps we’d want to prioritize moving this functionality to Airflow instead and eradicating the ingestion server?)
      
      The latter is my preference, though it’s roughly a distinction without meaning given that I think the current thinking on our team is to utilise ECS as the basis for our ingestion server worker processes. I could be wrong about that though.
      
      However, given the ingestion server kind of “just works” at the moment and its deployment is neither often nor exceptionally painful (despite being quite manual) my sense is that the time put towards changing the ingestion process so dramatically would be better put towards exploring, understanding, and designing improvements to our data pipelines that would result in faster and more maintainable overall data ingestion from extraction to loading (rather than just changing how loading works). Personally I would be keen to see ingestion changed to an incremental approach (I’ll again mention UPDATE_INDEX here as something we could/probably should prioritise), and for our data warehousing strategy to be more expansive.
      
      In that respect, I hope the data normalisation project and the changes planned to our data model bring up particular pain points that need to be addressed. I’m sure the folks working closely on the catalogue are already able to identify many of these.
      Migrate APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. to ECS and automate deployments
      
      Certainly this should be prioritised towards the start of the year. It has a tremendous quality of life impact and also automatically increases visibility into certain aspects of our services that could introduce easy cost saving measures. Specifically we would see RAM and CPU usage for our API boxes and could save on smaller allocations. Plus the project is already relatively well understood with few outstanding questions and is already under way.
      Implement Grafana/Prometheus and tie in metrics from each service in our stack
      
      I’d like to see this happen as well. Realistically we should stage this per-service as ‘each service in our stack’ is a pernicious concept that has bitten infrastructure projects before. Each service, or more likely each basis of deployment (ECS, EC2, etc) will require different approaches. Given the dramatic changes in our infrastructure since I wrote the original RFC for this at the beginning of 2022, I want to re-do the planning for the metrics infrastructure entirely. The precise metrics that we gather per-service should also be investigated per-service. There are only a handful of metrics relevant to both Django and Nuxt (overall response time is about all I can think of off the top of my head, but I bet there one or two more). Airflow and ingestion seem likely to be even more diverse.
      - Madison 9:18 pm on December 13, 2022
        
        Thank you for the call-out around the ingestion server! You’re right, investigations into the UPDATE_INDEX approach and changes to our refresh approach more broadly are likely to be more beneficial areas of focus than moving the service wholesale to a different platform.
- Zack Krida 3:57 pm on November 16, 2022
  
  Something I also think we missed in 2022 was making sure the devex and interal improvements we’ve made were captured in our planning.
  
  This work is often done out of necessity, but is also an easy thing to gravitate towards when goals are not clear, so being intentional about planning it will help us make sure we’re not doing too much or not enough of this kind of work.
  - Madison 9:05 pm on November 22, 2022
    
    Definitely agree here, are there any larger projects we’d want to capture, along with general devex improvement and library maintenance? Perhaps the monorepo move?
    - Olga Bulat 2:03 pm on December 5, 2022
      
      I’m not sure if this is a “larger project”, but we definitely need to plan to migrate to Nuxt 3 by the end of next year at the latest. Without it, we cannot use many libraries that support VueVue Vue (pronounced /vjuː/, like view) is a progressive framework for building user interfaces. https://vuejs.org/. 3 or do not support Nuxt 2.
      - sarayourfriend 4:10 am on December 6, 2022
        
        Whether this is a larger project will likely depend on whether Nuxt bridge actually works. If it doesn’t then we’ll need to accept either a re-write to Nuxt 3 or a re-write using a different VueVue Vue (pronounced /vjuː/, like view) is a progressive framework for building user interfaces. https://vuejs.org/. SSR framework like vite-ssr. My gut tells me neither would be trivial…
  - Dhruv Bhanushali 4:28 pm on November 23, 2022
    
    Yes, I think when planning for the larger user-facing goals, we don’t give a lot of thought on how the development process changes and evolves and can introduce metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. work to streamline that. While some small improvements here and there cannot be foreseen, bigger changes like the monorepo migrationMigration Moving the code, database and media files for a website site from one server to another. Most typically done when changing hosting companies. need to be accounted for in any timelines we draw up.
- Francisco 9:14 am on November 24, 2022
  
  It will be best to be strategic about lowering the number of things we accomplish and increasing the utility of the things we do accomplish
  
  I agree very much with this point. And from a design perspective, some interaction improvements are simple to tackle and have a big impact on the browsing experience. It would be great to set up a revisit session to identify those pain points and prioritize them according to subsequent goals.
  
  On the other hand, and since the project’s coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. is displaying relevant content for visitors, I would keep the “Improve result quality and relevancy” as the backbone of all the following tasks.
Zack Krida 2:26 pm on November 16, 2022

2. What guiding, big-picture goals should inform our work in 2023?
- Zack Krida 2:29 pm on November 16, 2022
  
  Two goals I’d like to suggest for 2023:
  
  Quantify our work. We intend to ship analytics tracking for the frontend and the api in the next few months. We would like to use this data to inform future projects but also much more clearly track the growth of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. in terms of daily users, page views, and so on, which we do not currently have accurate reporting on.
  
  Spread awareness of Openverse. We want to expand knowledge of Openverse as a first-class media search engine to users outside of the WordPress and Creative Commons communities. We will do this through increased text content on Openverse.org. Examples might include a dedicated ‘about’ landing page which explains the project, or a dedicated landing page on community and contributing which acts as a visually-appealing hub for our content spread across GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ and Make.
  - beccawidom 6:27 pm on November 22, 2022
    
    I’m really excited about the quantifying our goals piece! It’s both incredibly exciting and a little daunting to think about an almost blank slate.
  - sarayourfriend 2:24 am on December 5, 2022
    
    We want to expand knowledge of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. as a first-class media search engine
    
    For this to be possible we first need to make Openverse a “first-class media search engine”. Given the known issues with result relevancy, I do not think Openverse is there yet, despite having a tremendous potential for getting there. You could not easily replace Google/DuckDuckGo/etc image search with Openverse’s image search, for example.
    
    Openverse has unique use cases though, creative and educational ones. It would be good to more clearly highlight those.
    
    Regarding the first point, work quantification: this is great. It is in line with what I pushed for so strongly earlier this year in our previous planning. We should be wary of continuing to make improvements of which we cannot quantify the effectiveness.
    
    Is it possible Openverse could undergo observed user-testing sometime next year?
    - Francisco 10:02 am on December 5, 2022
      
      Is it possible OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. could undergo observed user-testing sometime next year?
      
      We have discussed running tests before, and +1 to set it for next year plan. But the expected outcome is crucial.
      
      User research is undoubtedly beneficial, but only with clear goals of what we want to discover. Otherwise, the efforts of scheduling session(s), running them, collecting data, analyzing data, and suggesting the next steps are pointless.
      
      In my experience running research sessions, I have seen many user tests failing because of the lack of goal and scope. Nonetheless, I do agree with incorporate it in 2023’s plan and then planning the details of what we want to cover.
      - Olga Bulat 2:16 pm on December 5, 2022
        
        Can we add “Plan user testing” as a goal for the next year?
        Some questions I would like to get answers to are:
        
        how do the users use the media they find on the site? Do they often right-click and save or go to the provider site? How do they add attribution?
        
        how do users use the ways of refining search to improve the results they get?
        
        I have a hunch that adding pages for the providers (where, for example, the user can see the latest media from that provider, and can search within the provider’s media) and adding links to the tags (so that clicking on them opens a page with only media that is marked with those tags) will improve the search experience greatly, even before we can improve search relevancy in the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. layer. But it would be really nice to have some metrics before and after to really understand if this is true.
      - sarayourfriend 11:20 pm on December 5, 2022
        
        +1 to what @olgabulat said. @fcoveram do you think this is something we could discuss when we’re mostly together in Istanbul? I’d like to understand better what kinds of up-front planning are necessary for user-testing as I’ve never been involved in that kind of thing before, just on the back-end of seeing the results.
      - Francisco 11:16 am on December 6, 2022
        
        Those questions are a great start @olgabulat I have some assumptions about current features as well. And yes, metrics within a funnel is definitely a great input to understand how visitors interact with the content they find.
        
        Sure @sarayourfriend Happy to set a session to talk about this more deeply. Before any discussion, take a look at this diagram that summarizing accurately the range of research methods. Each one requires a different plan, tools to use, and participants. The image comes from this post if you would like to dive in.
  - cogdog 3:12 pm on December 9, 2022
    
    Spread awareness of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.. We want to expand knowledge of Openverse as a first-class media search engine to users outside of the WordPress and Creative Commons communities.
    
    As someone who can’t help much on the technical side, this is something I can do. I am less concerned as being “first-class” as affirming the value of knowing with full confidence the rights and means to credit shared media, and an appreciation for the collections included. I find very few people in my world even know this exists.
    - Zack Krida 3:12 pm on December 13, 2022
      
      Thank you, @cogdog! I’m curious if there are any resources or things that would help people share and spread the word about OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org., or things you’d like to be able to share that do not exist yet.
      - cogdog 5:13 pm on December 13, 2022
        
        Not too sure, there’s just not much awareness out there, and certainly you are doing the right thing to work with Nate at Creative Commons.
        
        Maybe some short how to videos, or some showcase of real world search methods and their results that demonstrate effective strategies.
        
        Not sure of value, but what if there was a way (beyond what I know to copy the URLURL A specific web address of a website or web page on the Internet, such as a website’s URL www.wordpress.org) to share a specific search result, like
        
        What’s in the OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.? See all public domain wide orientation images of dogs from the Cleveland Museum https://wordpress.org/openverse/search/image?q=dog&license=pdm,cc0&source=clevelandmuseum
        
        Or even from a specific result, if there was a means to show the route by which it was found, from say this image, a cut/paste or link sharable:
        
        I found “Bullfights: They Loose Dogs on the Bull” https://wordpress.org/openverse/image/bcf8e5ca-1cac-4332-81b3-c7dffb96f99b in the Openverse by searching for public domain wide orientation images of dogs from the Cleveland Museum
        
        Shrug if this would be used?
- Zack Krida 5:33 pm on November 17, 2022
  
  Outside of measuring and advocating for OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org., the main goal I see for us is to make Openverse better. The biggest win I see there is to improve result quality and relevancy. A goal around doing so would make a lot of sense to me.
- Madison 9:24 pm on November 22, 2022
  
  A bigger picture goal I’d like to see is something along the lines of “reducing data inertia”. Changes to our data are either incredibly difficult or (in some cases we’ve seen) currently infeasible. Having a better picture of how we could make data changes would fold into this.
  
  I could also see efforts to reduce the latency of the data refresh fall under this goal as well.
  - sarayourfriend 2:25 am on December 5, 2022
    
    Agreed! I wrote a bit about how we could prioritise this as an infrastructure project here: https://make.wordpress.org/openverse/2022/11/16/thinking-towards-2023/#comment-177
- Madison 9:31 pm on November 22, 2022
  
  One other thought, content safety (both for maintainers/administrators and for our userbase) should also be captured as a high-level big-picture goal.
  - sarayourfriend 2:27 am on December 5, 2022
    
    Agreed. Not just content safety, but we must also establish a process for regularly reviewing media reports. Because this will require significant collaboration with people who do not currently actively work on OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org., it would be good for these conversations to at least start sooner than later.
- Krystle Salazar 9:28 pm on November 23, 2022
  I agree with you both on setting our big-picture goals around:
  - Quantify our work
  - Spread awareness of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.
  - Improve the search experience making browsing on Openverse safer and with a higher quality of results
  - Continuos improvement of internal processes (Ingestion, etc.)
- Francisco 9:15 am on November 24, 2022
  
  In line with the previous questions, two points come up to me:
  
  1. Improving search results: Visitors will use the content display and become recurrent users only if OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. brings up content that matches their mental model of what they expect to find. Poor results would imply never return to the site.
  2. Expanding Openverse use: By integrating Openverse with other tools, we are introducing using content in a trustworthy manner.
  
  Openverse in GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ as editing is a common context for needing media content and a handy tool. By tackling the integration, we are expanding its uses and connecting with the community and potential contributors.
- Olga Bulat 2:26 pm on December 5, 2022
  
  Some areas I think we should work on next year:
  1. Improving search results.
  2. Improving user search experience on the front end: make it easier to search by provider/ tag / creator.
  3. Spread awareness of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org.. This would include work on SEO, GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ integration, and browser extension. Outreach to providers and projects like adding provider pages and promoting them on social media would also fall under this goal.
  4. Improving the state of content safety.
  5. Quanitfying our work.
  - Francisco 11:25 am on December 6, 2022
    
    3. Spread awareness of OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org..
    
    I love this point. And I think we should at least participate in two events talking about Openverse significance and how to contribute.
    
    The upcoming milestone that occurs to me is submitting a proposal to the Mozfest 2023, but if we add this point to the 2023 goals, we can plan the details later.
Zack Krida 2:26 pm on November 16, 2022

3. What projects might we take on, related to those goals?
- beccawidom 6:29 pm on November 22, 2022
  
  I do think that some analytics infrastructure would help a lot with improving result quality and relevancy. I would also nominate some data quality metrics (e.g. using dbt tests or great expectations) if we can, though I think those might raise some infrastructure questions too…
  - Madison 9:25 pm on November 22, 2022
    
    +1 to this! Data quality alerts would be incredible for us.
- Madison 9:29 pm on November 22, 2022
  A few projects that come to mind on the “reduce data inertia” front (mentioned here):
  - Reduce data refresh latency by updating indices rather than recreating them
  - Improve or decouple popularity calculation step so data refreshes are not contingent on it
  - Explore alternate data storage mechanisms (e.g. parquet)
  - stacimc 10:27 pm on November 22, 2022
    
    Big +1 to all of this.
    - beccawidom 12:16 pm on December 4, 2022
      
      Big +1 from me too!
  - sarayourfriend 2:30 am on December 5, 2022
    
    +1 from me as well. In particular, I don’t fully understand where the friction is with the first especially given we already have the UPDATE_INDEX action. I’ve been bringing up the potential for using it for several months now (since at least mid-August).
    - stacimc 12:16 am on December 7, 2022
      
      To be clear, I don’t think there is friction to this. There is a lot of support for everything listed here, and I at least am even particularly excited about a potential quick win with UPDATE_INDEX. There just has not been time to prioritize it, while a lot of work has been happening in the Catalog. I would love to prioritize this in 2023.
- Francisco 9:15 am on November 24, 2022
  
  For the search results improvement, I rely on folks’ suggestions as the infrastructure work seems the most demanding one. And I agree with Madison in addressing the safety of the content displayed. For this aspect, I have no solid thoughts on the next steps.
  
  Regarding expanding OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. use, I would push further the GutenbergGutenberg The Gutenberg project is the new Editor Interface for WordPress. The editor improves the process and experience of creating new content, making writing rich content much simpler. It uses ‘blocks’ to add richness rather than shortcodes, custom HTML etc. https://wordpress.org/gutenberg/ integration as editing is a common context for needing media content and a handy tool. By tackling the integration, we are also connecting with the community and reaching potential contributors.
  
  Lastly, the bookmark feature developed in the browser extension sounds appealing to retake as it sets the baseline for future openverse integrations. It will allow users to save and reuse content handily across different touchpoints.
  - Zack Krida 10:42 pm on December 6, 2022
    
    Very interesting idea to implement the bookmarking feature. I haven’t thought about it in some time. It is an interesting ‘first step’ towards user generated content and users interacting with OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. in ways besides searching.
- Zack Krida 7:22 pm on November 29, 2022
  
  Regarding content safety, I thought of a project related to scanning and labeling results as mature.
  
  Fundamentally, this would work similarly to dead link handling.
  
  Here’s the basic idea:
  
  1. We add an `?enable_mature_scanning=true` flag to APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. requests, available to a certain class of authenticated users. this would be to enable our frontend access specifically but could perhaps be expanded later on.
  2. When this flag is enabled, each individual result thumbnail is scanned against a 3rd party image analysis service, like the Microsoft computer Vision API, which could classify them as explicit, violent, etc. we then store the id and metadata about the explicitness in redis, which we check first for each request.
  3. On the frontend, we read this mature information to:
  – Blur single results, and prompt the user to “enable mature content” if they would like to see the result. This helps prevent the sharing of explicit images
  – Exclude images from frontend results if the `mature` filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. is set. Perhaps also show some text somewhere indicating (X number of mature results were omitted / are hidden) and so on.
  
  Depending on the quality and detail these types of APIs provide, we could also automatically report all of these kinds of images through our API reporting system, so they are officially marked as mature regardless of their redis cache status. And for cases where a piece of content is so egregious to the point of being illegal or something we wouldn’t even show under the ‘mature’ flag, we can remove it completely.
  - Madison 7:34 pm on November 29, 2022
    
    That’s a really interesting idea 😮
    - Zack Krida 9:45 pm on November 29, 2022
      
      My main concern would be that of performance, of course, but we’d have to test it.
      - Madison 10:06 pm on November 29, 2022
        
        I think we’d also need to be careful and considerate about which APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways./model we use for this purpose.
      - Zack Krida 10:20 pm on November 29, 2022
        
        Yes! We need to hold any programmatic content moderation we might use to the same standards and policies we would (and should) define for human moderators.
  - sarayourfriend 2:42 am on December 5, 2022
    
    I think it’s a nice solution, but making 20+ requests to an external APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. that will process and classify an image sounds prohibitive from a performance perspective. Storing the result in Redis also seems like it would further exacerbate issues we have with dead link filtering causing blank result pages (like with SMK, IIRC) and deep pagination. I believe the benefit for using Redis with dead links is that the condition could be temporary, so we periodically re-check results and can easily change the period of time they’re cached. We also only ever check results that might actually be returned for a query, so we don’t waste time checking things that aren’t ever surfaced in results. Whether an image is “mature” doesn’t change though, unless cultural norms change, but usually those are pretty slow. Storing this data in Elasticsearch would be ideal.
    
    This impresses upon me the importance of solving asynchronous and massive data processing that we have yet to tackle but will probably become more necessary if we want to have accurate metadata like file types, sizes, dimensions, waveform, etc.
    
    The significant benefit of only checking results that would actually be surfaced, however, is lost. That is a shame! Perhaps there is a hybrid approach then, which I think we could apply to dead links as well. Store the initial result in Redis and then have an ETL process that pulls the data from Redis, back into the catalogue. One problem with this approach are that Redis records are not time stamped nor is the pagination cursor stable for wildcard queries. In that case, we might want to store a boolean flag in Redis for whether something is mature that can be used for the query mask, but store the actual result for the item in Postgres (given we can trivially timestamp there) and ETL out of Postgres into the catalogue. Dead links could have a similar approach, though with some kind of “expiration” so that they get periodically re-checked, but still get entirely excluded from the ES results rather than returned filtered out through the query mask and probably back-filling.
    - Zack Krida 11:00 pm on December 6, 2022
      
      I think it’s a nice solution, but making 20+ requests to an external APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. that will process and classify an image sounds prohibitive from a performance perspective.
      
      Many of these APIs have batch endpoints to which you can send a list of images in one httpHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. request. Not saying this would alleviate all performance concerns but it could certainly help.
      
      Storing this data in Elasticsearch would be ideal.
      
      This makes sense. I was definitely piggy-backing on the dead link logic and suggested redis as a result of that.
      
      Store the initial result in Redis and then have an ETL process that pulls the data from Redis, back into the catalogue.
      
      This sounds like a refined version of what I was proposing. Do you think by just posting things to the API reporting endpoint, which would allow for manual verification of the images (should we want to do that) we could avoid developing a new ETL pipeline? The frontend result pages are small and I’d imagine that often there’s only 1-2 mature results on a given page. I would imagine the API could handle the traffic in realtime with direct post requests to report each image as they are analyzed.
    - Zack Krida 4:26 pm on December 7, 2022
      
      An additional thought here: It’s reasonable to only apply this mature analysis to images which come from ‘social media’ providers, at present Wikimedia Commons and Flickr. Flickr tends to dominate our returned results at present but as we experiment with authority data and other result boosting measures that may change as well, further reducing the number of images which need this analysis applied.
  - sarayourfriend 2:44 am on December 5, 2022
    
    I think we’d also need to be careful and considerate about which APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways./model we use for this purpose.
    
    I can’t reply to your comment @aetherunbound because of thread depth, so I started a new thread. What do you mean by this? Do you mean which classification model we use? Just wanted to clarify if you meant which Django model (in which case I didn’t understand what you’re getting at).
    - Zack Krida 10:43 pm on December 6, 2022
      
      I believe @aetherunbound is referring to potential biases and limitations in AI image analysis.
      - Madison 9:26 pm on December 13, 2022
        
        That was my thinking!
  - sarayourfriend 2:53 am on December 6, 2022
    
    I was looking through older issues in our repositories and found this issue as well: https://github.com/WordPress/openverse-api/issues/7
    
    The same solution occurred to me late yesterday afternoon as my mind was wandering in and out of work and I forgot to mention it here. This seems like a pretty easy initial pass at getting clearer mature/NSFW data, but I’d ask that we use a word list that is maintained by someone we trust. Perhaps this kind of thing already exists for a WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ site or maybe Automattic is willing to donate resources for this.
    - Zack Krida 10:50 pm on December 6, 2022
      
      This is a great idea for an initial pass; it will be very crucial to find a respected and solid list. The example list in the issue has innumerable problems and oddities.
      
      In general however, I think it would be okay if we accidentally mark some works as ‘mature’ even if they aren’t. This doesn’t mean they won’t be discoverable, just that they’ll be behind a flag. If an image is so inappropriate that it’s not meant for inclusion in OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. at all, it could then be additionally reported by a user or detected by some other mechanism we implement.
      
      A related idea: We could improve the information provided by the ‘mature’ filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output.. It would be ideal if we could show a count of the # of mature results being omitted from a search. At a minimum we need to reinstate the mature toggle as it’s not currently available on the frontend at all.
- stacimc 12:43 am on December 7, 2022
  Speaking from the Catalog side, we’re carrying these projects from 2022 into next year:
  - Data normalization
  - DAG stability
  I suspect those projects will dominate the first quarter of the year.
  
  Beyond, I’m personally most interested in the data inertia ideas described by @aetherunbound in https://make.wordpress.org/openverse/2022/11/16/thinking-towards-2023/#comment-154. I think at minimum updating rather than recreating indices, and perhaps exploring a much larger reimagination of the data pipeline.
  
  I’m also very interested in data quality alerts as mentioned by @beccawidom. For the quantifying our work goal, I would love to get some dashboards for the health of our provider DAGs. We have very poor insight into, for example, when a DAG is taking an anomalously long time to run, or when the number of records ingested is historically unusual. Some small additions here could dramatically improve error detection.
  - Zack Krida 4:27 pm on December 7, 2022
    
    The data inertia work sounds quite compelling, especially if we manage a quick win with updating instead of replacing indicies while we explore a larger reimplementation.

Welcome!