Adding subtitles to the WordPress.tv videos

Captions are a text version of the speech and non-speech audio information needed to understand the content. They are synchronized with the audio and are usually shown in a media player when users turn them on.

Web accessibilityAccessibility Accessibility (commonly shortened to a11y) refers to the design of products, devices, services, or environments for people with disabilities. The concept of accessible design ensures both “direct access” (i.e. unassisted) and “indirect access” meaning compatibility with a person’s assistive technology (for example, computer screen readers). (https://en.wikipedia.org/wiki/Accessibility) is essential for people with disabilities and useful for all, so adding captions to the WordPress.tv videos would improve the usability of this platform.

We have used Otter.ai (English) & Sonix.ai (multilingual) to generate caption text files, then edit and upload them manually. These options have a cost, paid by some 5ftf company, so we used them only in a few videos.

In this post, I am going to explain how to test Whisper, a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification. Whisper is open-source.

Install

You can get all the information about the installation here.

You need to have Python 3 and Pip installed. If you have a Mac M1, take a look at this link before installing this tool.

python3 -m pip install git+https://github.com/openai/whisper.git

Install ffmpeg:

brew install ffmpeg

Using Whisper

Getting the captions

I am going to use 3 videos in 3 languages to test the tool:

Be advised that the files you will get from wordpress.tv are the video without audio (1) and the audio files (2). You have to use the audio file (or a video with audio), because the video without audio breaks the extraction (and, of course, doesn’t work without audio). The next screenshot was taken from the Google Chrome inspector for A chat with Matt Mullenweg: WordCamp US 2022 Q&A.

I have downloaded these videos to my laptop:

wget https://videos.files.wordpress.com/BJyPZJNx/video-8cdb521345_mp4_hd.mp4 -O matt.mp4
wget https://videos.files.wordpress.com/MpsA8VfU/video-d420f9c984_mp4_hd.mp4 -O RocioIsotta.mp4
wget https://videos.files.wordpress.com/AVCv2TIJ/video-3a8375257e_mp4_hd.mp4 -O NuriaMiriam.mp4

Now I am going to run Whisper with these 3 videos:

whisper matt.mp4 --language English
whisper RocioIsotta.mp4 --language Spanish
whisper NuriaMiriam.mp4 --language Galician

Each execution generates 3 files:

Getting the translations

Adding --task translate, Whisper will translate the speech into English:

whisper RocioIsotta.mp4 --language Spanish --task translate

This command only generates the translation, not the text files in the original language, so I had to run the command without the  --task translate parameter. I have to research if it is possible to do both actions at the same time.

Time execution

To have some reference values of the time it takes to run these processes, I used the time command. My laptop is a MacBook Pro M1 2020 with 16 GB RAM. Really, the time command is then actually performed by the ZSH shell.

time whisper matt.mp4 --language English
10409.95s user 3461.08s system 217% cpu 1:46:22.95 total

time whisper RocioIsotta.mp4 --language Spanish
4222.15s user 1253.41s system 164% cpu 55:31.56 total

time whisper NuriaMiriam.mp4 --language Galician
7320.41s user 2412.34s system 204% cpu 1:19:10.39 total

time whisper RocioIsotta.mp4 --language Spanish --task translate
3813.41s user 1197.22s system 221% cpu 37:42.37 total

You can see the process takes some time, so if we are going to use these files in a WordCampWordCamp WordCamps are casual, locally-organized conferences covering everything related to WordPress. They're one of the places where the WordPress community comes together to teach one another what they’ve learned throughout the year and share the joy. Learn more., we need to process them before, maybe using a script running the night before, extracting the subtitles from all videos inside a folder.

Files

These are the files if you want to review the result:

Be advised that the name of the files inside RocioIsotta-en.zip and RocioIsotta.zip are the same, but with different content: one with subtitles in English and the other with subtitles in Spanish.

Usages and conclusion

The captions are not perfect, but they have good quality, so they can be a good starting point to work in the WordCamps (TV table in the Translation Day) or by the community who uploads the videos to WordPress.tv. They can edit the caption files and get the subtitles in the original language and in English, so we can make videos more accessible to the community with this open-source tool.

Update

@akirk mentioned this interesting post about how to run Whisper with GitHub Issues/Actions. Could be interesting to adapt this workflow to wordpress.tv.

+make.wordpress.org/accessibility/

+make.wordpress.org/tv/

#captions, #subtitles

Subtitles for the 4.7 Release Video

Dear polyglots,

The WordPress 4.7 release video finalized and presented tonight during the State of the WordState of the Word This is the annual report given by Matt Mullenweg, founder of WordPress at WordCamp US. It looks at what we’ve done, what we’re doing, and the future of WordPress. https://wordpress.tv/tag/state-of-the-word/., so let’s get this going and translate it to as many languages as we can.

Here are the instructions to subtitle the video:

1. Open the release video project on Amara

2. To add your language click “Add a new language”

3. Select your language. This will load the subtitle editor.

4. On the left side of the subtitle editor you’ll see the English subtitles. On the right side you can translate into your own language. Follow the instructions to add the subtitles and sync them.

5. Add the translations to your language, sync them and don’t forget to

6. Save and Publish the subtitles.

If your language’s language code is not supported by Amara, you can add the subtitle file here and we will try and add them manually to WordPress.tv.

The deadline for translating the video is Tuesday, December 6th. I realise this is very short notice, but we will be grateful for any help that we can get.

Thank you to everyone in advance. Your work is so important for so many people around the globa

Love,

P.

#subtitles, #video, #wordpress-4-7

#release

Subtitles for the 4.6 Release Video

Dear polyglots,

The WordPress 4.6 release video is being finalised as we speak but the voice over is already available for translation, so let’s get this going and translate it to as many languages as we can. The 4.5 release video was subtitled into 31 languages but I know we can do better than that 🙂

Here are the instructions to subtitle the video:

1. Open the release video project on Amara

2. To add your language click “Add a new language”

3. Select your language. This will load the subtitle editor.

4. On the left side of the subtitle editor you’ll see the English subtitles. On the right side you can translate into your own language. Follow the instructions to add the subtitles and sync them.

5. Add the translations to your language, sync them and don’t forget to

6. Save and Publish the subtitles.

If your language’s language code is not supported by Amara, you can add the subtitle file here and we will try and add them manually to WordPress.tv.

Our jazz legend is male.

The deadline for translating the video is Monday, Aug 15th, midnight CET.

Thank you to everyone who’ll help us get the release video to more people around the world.

Love,

P.

#release, #subtitles

Subtitles for the 4.5 Release Video

Apologies to the delay to getting the release video to you! The revised deadline for submitting subtitles is Monday 11th April.

1. The release video is here.

2. To add your language click “Add a new language” (left hand side of the screen)

3. Select your language. This will load the subtitle editor.

4. On the left you’ll see the English subtitles. On the right you can translate into your own language.

5. Translate! You can find help on using Amara here.

6. Save.

I will download all of the subtitles in the morning (UTC) on 11th April, so if you would like for your language to be included please add them before then.

If you would like to translate the .ttml file directly you can grab a copy of the subtitle file in English. If you do that, please leave a message here with a link to the file.

A few notes:

  • the jazz legend is male
  • if you have a language or language variation that isn’t supported by Amara you can drop the subtitle file on this thread and I’lll grab it here.

#subtitles

Hello Polyglots Just a heads up that I’ll…

Hello Polyglots!

Just a heads-up that I’ll be making the next release video available for subtitles on 4th April, and the deadline for delivery will be April 8th.

Process will be as usual – I’ll upload everything to Amara and you can use that to produce the subtitles.

Thanks!

#subtitles

Hello everyone Time for subtitling This time the…

Hello everyone!

Time for subtitling 🙂 This time the video isn’t quite there – I have a few more things to do; but it’s possible to get it subtitled anyway as the timestamps won’t be changing on the voice over track. Here’s what needs to be done:
1. Jazzer to be added
2. Tagline to be added – I’ve added the tagline in the English subtitles so you can just translate that.
3. I also need to add the opening and closing tags.

You can find the video here. I’ve added the English subtitles so please just add the ones in your language. I need them done by noon GMT on Monday 7th December.

#subtitles

Hi polyglotters WordPress 4.4 is fast approaching which…

Hi polyglotters!

WordPress 4.4 is fast approaching which means it’ll soon be subtitling time! I just wanted to give you advance warning and some dates.

  • the video will be available for subtitling by 3rd December
  • I will need all subtitles by 7th December

I will let you know the gender of the jazzer this time so you can translate accordingly 🙂

#subtitles

Subtitles for the 4.3 Release Video

As promised, it’s time to subtitle the WordPress 4.3 release video. We’re going to do it a bit differently this time. When you’re done, please do provide feedback on the process as it’ll help us to refine it again next time.

1. You’ll find the audio for the release video here

2. To add your language click “Add a new language” (left hand side of the screen)

3. Select your language. This will load the subtitle editor.

4. On the left you’ll see the English subtitles. On the right you can translate into your own language.

5. Translate! You can find help on using Amara here. (note: there’s a 5 second intro in the music so it takes a few seconds to get going.)

6. Save.

I will download all of the subtitles in the morning (UTC) on 15th August, so if you would like for your language to be included please add them before then.

If you would like to translate the .ttml file directly you can grab a copy of the subtitle file in English. If you do that, please leave a message here with a link to the file.

I’ll hang out in the polyglots SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. channel for the next few days so feel free to pingPing The act of sending a very small amount of data to an end point. Ping is used in computer science to illicit a response from a target server to test it’s connection. Ping is also a term used by Slack users to @ someone or send them a direct message (DM). Users might say something along the lines of “Ping me when the meeting starts.” me with any questions.

#subtitles

Subtitles for WordPress 4.3

Hi polyglotters!

The WordPress 4.2 video was the first one that we created subtitles for. It was great to be able to provide subtitles, although I think we had varying degrees of success with the overall workflow. We’d like to provide subtitles again for WordPress 4.3.

I’d like to refine the workflow and make it a better and smoother experience. You can find the workflow that we used for 4.2 here.

In brief, what we did was either:
1. Translate through a web interface, or
2. Translate the XML file directly

The files were then emailed to me.

I would be okay with using the same approach again this time but not everyone found the experience to be smooth. So if you have feedback and a suggestion for a better workflow that we can implement, please let me know on this thread.

Some dates for your diary:

  • the final cut of the video will be made available by 12th August for translation
  • I will need to receive subtitles by 15th August.

#subtitles