Tag Archives: provider

Madison Swain-Bowden 9:02 pm on October 10, 2022
Tags: 3d-models, , provider   

Next steps for Thingiverse data

As we close out the provider DAG refactor milestone, one of the providers we intended to refactor was Thingiverse. Thingiverse currently doesn’t have a legacy provider DAG configuration, but we do have data in the API under the image media type for it.

What complicates Thingiverse is that it is fundamentally a source of CC licensed 3D models, and we already have it slated as a 3D model provider once that project goes underway, yet we currently have existing data in the images table for it.

That to say, I’m struggling to determine what our next steps should be for this provider. It seems like it would not be ideal to continue ingesting this provider under the “image” media type, given that we’ll want to distinguish it as a 3D model provider later down the line. However, I also have some hesitancy around deleting the 32,659 records we already have in the image table as folks may be referencing those results. On the other hand it is only 32k records though and the results themselves are probably not as useful as images because they’re all 3D renders (example); perhaps it’s reasonable to remove them in favor of including them in the 3D models index down the line.

I’m open to other thoughts, but I’d like to propose that we:

  1. Retire the existing provider script (3D models will look similar but will likely grab a lot of other fields).
  2. Delete the thingiverse data from the image table in the catalog.
  3. Remove the “Thingiverse” provider from the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. search results via the Django Admin UIUI UI is an acronym for User Interface - the layout of the page the user interacts with. Think ‘how are they doing that’ and less about what they are doing..

Do folks have concern with these steps? Any alternative proposals?

#data-normalization #3d-models #provider

Madison Swain-Bowden 11:10 pm on September 30, 2022
Tags: , provider   

Next steps for Walters Art Museum data

Today I attempted to refactor the Walters Art Museum provider APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. script (see this GitHub issue). While working on this refactor, I noticed that I could neither use the testing sandbox provided by the API nor create a user account to receive an API key. We have tried reaching out a number of times over the past year to ask for the CC Search API key to no avail.

As it stands, we have no way of confirming that the API could be accessible once this DAG is turned on. We only have 16,948 records in the catalog/API (confirmed in both places). The last update to the API codebase was made on August 7th, 2015, and the last update to any of our data was December 1st, 2020. The media that our data references still exists AFAICT.

Given all this context, I propose that we:

  1. Create a one-off script to populate height, width, filesize, and filetype (see the filesize/filtype and height/width backfill GitHubGitHub GitHub is a website that offers online implementation of git repositories that can easily be shared, copied and modified by other developers. Public repositories are free to host, private repositories require a paid subscription. GitHub introduced the concept of the ‘pull request’ where code changes done in branches by contributors can be reviewed and discussed before being merged be the repository owner. https://github.com/ issues). This can likely be done without an API key using the direct image URLs we have in our database.
  2. Move the Walters provider script into the Retired DAGs directory and decommission the DAG.

It does not seem likely that API will become accessible to us again in the near future. The backfills described above would at least allow us to have the minimum data we’d like to have now as part of our ongoing data normalization effort and allow us to continue to serve the data we have in the API.

What do y’all think?

#data-normalization, #provider