Title: Next steps for Walters Art Museum data Author: Madison Swain-Bowden Published: September 30, 2022 --- [ ](https://profiles.wordpress.org/aetherunbound/) [Madison Swain-Bowden](https://profiles.wordpress.org/aetherunbound/) 11:10 pm _on_ September 30, 2022 Tags: [data-normalization ( 4 )](https://make.wordpress.org/openverse/tag/data-normalization/), [provider ( 2 )](https://make.wordpress.org/openverse/tag/provider/)    # Next steps for Walters Art Museum data Today I attempted to refactor the Walters Art Museum provider APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. script (see [this GitHub issue](https://github.com/WordPress/openverse-catalog/issues/597)). While working on this refactor, I noticed that I could neither use the [testing sandbox provided by the API](http://api.thewalters.org/help/index) nor [create a user account to receive an API key](https://github.com/WaltersArtMuseum/walters-api/issues/75). We have tried [reaching out a number of times](https://github.com/WordPress/openverse-catalog/issues/386) over the past year to ask for the CC Search API key to no avail. As it stands, we have no way of confirming that the API could be accessible once this DAG is turned on. We only have 16,948 records in the catalog/API (confirmed in both places). The last update to the API codebase was made on [August 7th, 2015](https://github.com/WaltersArtMuseum/walters-api/commits/master), and the last update to any of our data was December 1st, 2020. The media that our data references still exists AFAICT. Given all this context, I propose that we: 1. Create a one-off script to populate `height`, `width`, `filesize`, and `filetype`( see the [filesize/filtype](https://github.com/WordPress/openverse-catalog/issues/536) and [height/width](https://github.com/WordPress/openverse-catalog/issues/647) backfill GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged by the repository owner. [https://github.com/](https://github.com/) issues). This can likely be done without an API key using the direct image URLs we have in our database. 2. Move the Walters provider script into the [Retired DAGs directory](https://github.com/WordPress/openverse-catalog/tree/main/openverse_catalog/dags/retired) and decommission the DAG. It does not seem likely that API will become accessible to us again in the near future. The backfills described above would at least allow us to have the minimum data we’d like to have now as part of our ongoing [data normalization effort](https://github.com/WordPress/openverse-catalog/milestone/4) and allow us to continue to serve the data we have in the API. What do y’all think? [#data-normalization](https://make.wordpress.org/openverse/tag/data-normalization/), [#provider](https://make.wordpress.org/openverse/tag/provider/) ### Share this: * [ Threads ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=threads) * [ Mastodon ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=mastodon) * [ Bluesky ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=bluesky) * [ Facebook ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=facebook) * [ X ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=x) * [ LinkedIn ](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?share=linkedin) * [Login to Reply](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fopenverse%2F2022%2F09%2F30%2Fnext-steps-for-walters-art-museum-data%2F%23respond&locale=en_US) 1. ![](https://secure.gravatar.com/avatar/af35a1eee8924bfa7bca41d4845f317a01749f7d475d459cb56354984d011849? s=32&d=identicon&r=g) 2. [Zack Krida](https://profiles.wordpress.org/zackkrida/) 12:56 am _on_ October 1, 2022 3. I think this sounds like a solid plan. One signal I’d be curious to identify as to the ‘health’ or sustainability of the project is if there are a significant number of objects in their APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. that we haven’t ingested yet. 4. It definitely looks like this project has stalled though and I’m happy to go ahead with your proposal. 5. * ![](https://secure.gravatar.com/avatar/c4575c45d6e3cc466883f6adc0e1006d4407f9aa3108bc8bb8e743423aeab3c1? s=32&d=identicon&r=g) * [Madison](https://profiles.wordpress.org/aetherunbound/) 1:27 am _on_ October 1, 2022 * Great point! Interestingly enough, it looks like there was a “bulk data import” done in the repo [in February 2015](https://github.com/WaltersArtMuseum/walters-api/commit/243792177592b5bec80d9c2d4856e72084dedfed). It looks like there are about 15,322 objects in the `objects/data/` directory: * ``` $ find objects/data/ -type f | wc -l 15322 ``` * This looks like _less_ than what we have in the catalog, so it seems like we’re likely close to up-to-date if not fully up-to-date. 6. ![](https://secure.gravatar.com/avatar/0fff3c26b28921573731dc81a4c968c65118153b8beface810032b6fb6908be4? s=32&d=identicon&r=g) 7. [beccawidom](https://profiles.wordpress.org/beccawidom/) 3:49 pm _on_ October 4, 2022 8. It sounds like a good plan to me! Cycling through to get the file info might give some useful info about staleness more generally. 9. ![](https://secure.gravatar.com/avatar/2695b831202728c6a5e6bbe3f68cb72d2b659e46fd48a987665cb06042c8b99a? s=32&d=identicon&r=g) 10. [Krystle Salazar](https://profiles.wordpress.org/krysal/) 5:04 pm _on_ October 4, 2022 11. It’s sad that Museum APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. is no longer maintained (as far as we know), but aside from that, the plan sounds good to me if the images are still being served. 12. ![](https://secure.gravatar.com/avatar/92f412e95ba034e7471156b35055553bf1d294c499345c388e266caeca811bfa? s=32&d=identicon&r=g) 13. [stacimc](https://profiles.wordpress.org/stacimc/) 5:05 pm _on_ October 4, 2022 14. It’s a shame the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. is no longer supported, but thank you for this write up — I’m glad to have found this out so we can address it. Your plan sounds like the best way forward. ### Leave a Reply 󠀁[Cancel reply](https://make.wordpress.org/openverse/2022/09/30/next-steps-for-walters-art-museum-data/?output_format=md#respond)󠁿 You must be [logged in](https://login.wordpress.org/?redirect_to=https%3A%2F%2Fmake.wordpress.org%2Fopenverse%2F2022%2F09%2F30%2Fnext-steps-for-walters-art-museum-data%2F&locale=en_US) to post a comment. # Post navigation [← Openverse Monthly Priorities Meeting 2022-10-05](https://make.wordpress.org/openverse/2022/09/30/openverse-monthly-priorities-meeting-2022-10-05/) [iNaturalist Retrospective Recap →](https://make.wordpress.org/openverse/2022/10/05/inaturalist-retrospective-recap/)