Welcome!

Openverse is a search engine for openly-licensed media.

The OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. team builds the Openverse Catalog, APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways., and front-end application, as well as integrations between Openverse and WordPress. Follow this site for updates and discussions on the project.

You can also come chat with us in #openverse on the Make WP Chat. We have a weekly developer chat at 15:00 UTC on Mondays.

If you’re a new contributor, welcome! Have a look at our good first issues or our guide for new contributors.

Time: Wednesday, September 28th at 20:00 UTC
Attendees: @beccawidom, @krysal, @sarayourfriend, @aetherunbound

We held a retrospective for discussing & processing the recent iNaturalist provider ingestion script contribution from @beccawidom. We spent several minutes writing sticky notes of our thoughts, then spent a few more minutes grouping them and voting on them. We then had a discussion on the various groups and the points/action items which arose from each group.

We did not get to all groups and also wanted to allow for asynchronous discussion. Below are the notes from that meeting, but I will also create discussion comments for each major group so we can aggregate responses in that manner.

Stickies

Notes

Big Picture Questions

What are the big questions?
- Are we going to stay in Postgres?
  - If we’re switching the infrastructure to be flat-files, should we spend time optimizing it? -> Yes
  - Though we want to remove Postgres, we should plan for it to stick around for a few years
- Is non-APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. consumption/bulk-import a priority going forward? -> Yes

Communication transparency

Moving a lot of our internal communication to the Make WP blog would be nice, but we need better tools for facilitating that.
DMs vs public is a hard distinction for new contributors
- It can be difficult to determine what is a Python question vs an OpenverseOpenverse Openverse is a search engine for openly-licensed media, including images and audio. Find Openverse on GitHub and at https://openverse.org. question
- Public discussions could generally be better since more people could contribute
- Rebecca feels confident now to be able to throw some code into GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ with potential questions and using that as a vector for communication & discussion
- What would help improve this process or make it easier for people to get more comfortable with the various communication channels?
  - Community meeting & prioritization meetings help make maintainers more personable and approachable
  - Quarterly “contributor dayContributor Day Contributor Days are standalone days, frequently held before or after WordCamps but they can also happen at any time. They are events where people get together to work on various areas of https://make.wordpress.org/ There are many teams that people can participate in, each with a different focus. https://2017.us.wordcamp.org/contributor-day/ https://make.wordpress.org/support/handbook/getting-started/getting-started-at-a-contributor-day/.”? Potentially across timezones, like the PyCascades sprints!
Public “project thread” seemed desirable for Rebecca, maybe project threads in general could maybe go in the Make blog so the public could see what were working on
- GitHub issues precludes nested comments because everything is flat, discussions could be used but we only have 1 level of nesting
- What tool could we use to do project threads publicly?
  - Could we use P2P2 P2 or O2 is the term people use to refer to the Make WordPress blog. It can be found at https://make.wordpress.org/.? Could Make become P2? Could we have a public P2?
- We do have GitHub issues or milestones that reflect the project thread, but still don’t have the content/conversationality of the project thread

Documentation

Lots of different things were new for Rebecca at the same time, not just Openverse-specific
- Perhaps we could have a “contributor onboarding process”?

#community-contributors, #inaturalist, #retrospective

Madison 1:00 am on October 5, 2022

Communication Transparency
Madison 1:01 am on October 5, 2022

Documentation
- Madison 11:35 pm on October 6, 2022
  
  @beccawidom gave us some extra thoughts via another channel, I’ve codified them in this issue:
  
  https://github.com/WordPress/openverse/issues/313
Madison 1:01 am on October 5, 2022

Review & merge process
- Madison 7:35 pm on October 6, 2022
  
  I hope the review process throughout this wasn’t too frustrating for you Rebecca. The great thing about working on something out-of-band of the explicit maintainer priority list is that, as a project, we’re able to accomplish more than we set out to do! The downside to that is that it can feel like we can de-emphasize reviewing/engaging in this work because it’s not on our priority list. I definitely don’t want that to be the case, so I wonder how that could be improved in the future. How did you feel about the timeline @beccawidom? Did you find you were frequently waiting on the maintainers as you were working on pieces? Do you feel like you would have been comfortable pinging for help in the #openverse SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. channel if you needed it?
  - beccawidom 9:14 pm on October 6, 2022
    
    I do feel comfortable pinging for help, yes. My ideal volunteer project is important, but not urgent. It seems like adding iNaturalist data fit in that space. Does that seem right to you @madison?
    
    I have a couple of things that are in process right now, and I’m not as clear about how they fit in terms of importance/urgency, but they’re all interesting so I’m happy to keep plugging away. 🙂
    - Madison 9:24 pm on October 6, 2022
      
      I think I would feel totally comfortable trusting you with more urgent work, as you’ve been very communicative about your availability in the past! But your current level of in-progress work also feels solid to me 😄 Don’t let that stop you from taking on other issues though! I personally always love a “quick-win PR” 🙂
Madison 1:01 am on October 5, 2022

Complicated data flows
- Madison 7:37 pm on October 6, 2022
  
  One thing that’s clear to us is that the ingestion & data refresh process is quite opaque, both to contributors and maintainers. The DAG we have for the data refresh helps with that clarity, but I think the general efforts we have around reducing refresh complexity & bringing those steps into Airflow where possible will help with this endeavor.
Madison 1:02 am on October 5, 2022

Performance/feasibility
Madison 1:02 am on October 5, 2022

New Provider Type By Contributor
- Madison 6:58 pm on October 6, 2022
  
  I think my biggest comment here is just kudos to @beccawidom for opening a totally new door for us regarding bulk insertions. The clear next step here IMO is to move away from python operations entirely where possible if we’re already in SQL-land, which is where I think we’re headed with this PR!
  - beccawidom 9:33 pm on October 6, 2022
    
    Yeah, I’ve mentioned dbt, and I think it’s really worth looking especially as they add python capabilities. If not now, then maybe if/when we add more bulk data providers, outside of the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. scraping approach.
    
    I wonder if it would work for me to propose a stripped down (narrower, hopefully more performant and more reliably populated) version of the catalog `image` table, that we might be able to refresh with data from past provider loads already stored on s3.