Platform-level Closed Captioning

[carb0n]

TL;DR: We should have the same functionality of live captions (observed in Windows, MacOS, YouTube, etc.) integrated into the Steam platform itself.

---

Why?

For context, I am a profoundly Deaf person who relies on subtitles/captions 100% in my everyday life and I use ASL (American Sign Language) to communicate with my peers.

Consider this scenario: As a millennial myself, some of my favorite games during the PS1 era included the Crash Bandicoot series, which never had subtitles/captions at the time (which is predictable, given the era and the limitations).

However, I was let down when I realized that when they released the remastered Crash Bandicoot: N-Sane Trilogy including all three classic games packed into one, for all platforms - with no captions at all. Nothing.

I believe I had the same problem with Spyro: Reignited Trilogy but they patched it in later on, which was great—considering I also had no knowledge of what the hell was happening in the original Spyro games for PS1 (and I realized from playing the trilogy how much information I missed out).

Criticism convinced the team behind the Spyro remaster to patch in subtitles, but nothing happened for the Crash trilogy which is strange considering Naughty Dog is pretty accessibility-forward, but then again, Vicarious Visions is behind the remaster.

---

My solution recently was to just simply activate the Windows’ built-in Live Caption feature which revealed so much dialogue and information unbeknownst to myself in the past and I could have sworn Aku Aku said something about… facebook… while mumbling to himself? Either that, or the captions got something wrong.

However, as an avid Steam Deck user, I play the majority of the Crash remastered games on the couch and that’s when I realized it—I can’t use Windows’ Live Caption in that situation and this is where it would be beneficial for Steam to have their own platform-wide Live Captioning feature. By allowing it to exist on the platform, users can toggle it from the overlay or in the Steam settings and be able to understand anything, including several titles (old and new) without being left out!

Someone might argue that if this happened, developers may ease back on their accessibility initiatives as a result, but I think this is highly unlikely, and we are always going to have some games that fall through the accessibility cracks and we need a catch-all solution at the Steam level—not to mention the fact we can play several retro games on the Deck / Big Picture mode.

This would take nothing away from players (who do not care for it) and give back everything to players, like myself, who really need it.

Thanks for reading, and here’s to hoping Steam acknowledges & supports this!

[Ben Lubar]

You can see whether a game has captions in any given language on the sidebar of the store page. Apart from very old games, I don't think I've recently seen a game that has spoken dialogue but no captions. (At least, none that I've been interested in playing.) In my experience, automated captioning is much less accurate than captions made by hand.

If a system-wide automated captioning solution exists for Linux (I haven't searched for one) you could run the Steam Deck in desktop big picture mode and use that.

[Ettanin]

who pays for the captioning licenses?
would the publishers even consent to the integration of such systems into their games?
What would it cost for Steam to obtain a live captioning service to be used by thousands or millions of potential Steam users?

To me it sounds like an accessibility feature that needs to be integrated by the developer themselves rather than by Steam.

[cSg|mc-Hotsauce]

โพสต์ดั้งเดิมโดย carb0n:

Platform-level Closed Captioning

TL;DR: We should have the same functionality of live captions (observed in Windows, MacOS, YouTube, etc.) integrated into the Steam platform itself.

---

Why?

For context, I am a profoundly Deaf person who relies on subtitles/captions 100% in my everyday life and I use ASL (American Sign Language) to communicate with my peers.

Consider this scenario: As a millennial myself, some of my favorite games during the PS1 era included the Crash Bandicoot series, which never had subtitles/captions at the time (which is predictable, given the era and the limitations).

However, I was let down when I realized that when they released the remastered Crash Bandicoot: N-Sane Trilogy including all three classic games packed into one, for all platforms - with no captions at all. Nothing.

I believe I had the same problem with Spyro: Reignited Trilogy but they patched it in later on, which was great—considering I also had no knowledge of what the hell was happening in the original Spyro games for PS1 (and I realized from playing the trilogy how much information I missed out).

Criticism convinced the team behind the Spyro remaster to patch in subtitles, but nothing happened for the Crash trilogy which is strange considering Naughty Dog is pretty accessibility-forward, but then again, Vicarious Visions is behind the remaster.

---

My solution recently was to just simply activate the Windows’ built-in Live Caption feature which revealed so much dialogue and information unbeknownst to myself in the past and I could have sworn Aku Aku said something about… facebook… while mumbling to himself? Either that, or the captions got something wrong.

However, as an avid Steam Deck user, I play the majority of the Crash remastered games on the couch and that’s when I realized it—I can’t use Windows’ Live Caption in that situation and this is where it would be beneficial for Steam to have their own platform-wide Live Captioning feature. By allowing it to exist on the platform, users can toggle it from the overlay or in the Steam settings and be able to understand anything, including several titles (old and new) without being left out!

Someone might argue that if this happened, developers may ease back on their accessibility initiatives as a result, but I think this is highly unlikely, and we are always going to have some games that fall through the accessibility cracks and we need a catch-all solution at the Steam level—not to mention the fact we can play several retro games on the Deck / Big Picture mode.

This would take nothing away from players (who do not care for it) and give back everything to players, like myself, who really need it.

Thanks for reading, and here’s to hoping Steam acknowledges & supports this!

Live captioning even on YT is hilariously bad, especially with music videos. How could Valve do better than Google but with 1/100 of MS and Google's $.

[carb0n]

โพสต์ดั้งเดิมโดย Ben Lubar:

You can see whether a game has captions in any given language on the sidebar of the store page. Apart from very old games, I don't think I've recently seen a game that has spoken dialogue but no captions. In my experience, automated captioning is much less accurate than captions made by hand.

You're right and I know Live Captions (windows) is not accurate, but it's better than nothing!

I learned some lore and info from the (floating head) cutscenes in the Crash games thanks to the mostly inaccurate live captions, which still relayed enough information to enhance my experience, than no subtitles at all - which is kind of my point :)

[Ben Lubar]

โพสต์ดั้งเดิมโดย Ettanin:

who pays for the captioning licenses?
would the publishers even consent to the integration of such systems into their games?

To me it sounds like an accessibility feature that needs to be integrated by the developer themselves rather than by Steam.

The kind of live closed captions they're talking about are done by a speech recognition algorithm, just running on the audio from your speakers rather than from your microphone. There's not some team of people somewhere typing the captions on a special keyboard like you might have for a TV program.

[Ettanin]

โพสต์ดั้งเดิมโดย Ben Lubar:

โพสต์ดั้งเดิมโดย Ettanin:

who pays for the captioning licenses?
would the publishers even consent to the integration of such systems into their games?

To me it sounds like an accessibility feature that needs to be integrated by the developer themselves rather than by Steam.

The kind of live closed captions they're talking about are done by a speech recognition algorithm, just running on the audio from your speakers rather than from your microphone. There's not some team of people somewhere typing the captions on a special keyboard like you might have for a TV program.

Of course it's by AI, which itself would be subject to licensing costs.

[Brian9824]

I mean it would be nice, but its unlikely as its a massive expenditure for something already available to the majority of people.

Not sure if there are any existing linux based solutions though, as partnering with an already established solution is far more cost effective then building one from scratch

[Ben Lubar]

โพสต์ดั้งเดิมโดย carb0n:

โพสต์ดั้งเดิมโดย Ben Lubar:

You can see whether a game has captions in any given language on the sidebar of the store page. Apart from very old games, I don't think I've recently seen a game that has spoken dialogue but no captions. In my experience, automated captioning is much less accurate than captions made by hand.

You're right and I know Live Captions (windows) is not accurate, but it's better than nothing!

I learned some lore and info from the (floating head) cutscenes in the Crash games thanks to the mostly inaccurate live captions, which still relayed enough information to enhance my experience, than no subtitles at all - which is kind of my point :)

Hmm, https://flathub.org/apps/net.sapples.LiveCaptions might be installable in desktop mode without any tinkering

[Ben Lubar]

โพสต์ดั้งเดิมโดย Ettanin:

โพสต์ดั้งเดิมโดย Ben Lubar:

The kind of live closed captions they're talking about are done by a speech recognition algorithm, just running on the audio from your speakers rather than from your microphone. There's not some team of people somewhere typing the captions on a special keyboard like you might have for a TV program.

So it's AI then.

Not always, but nowadays neural networks do produce better results than older, more manually designed speech recognition algorithms.

It's not the same kind of "AI" that's stealing artwork and telling people to put glue on pizza, though.

[Brian9824]

โพสต์ดั้งเดิมโดย Ben Lubar:

โพสต์ดั้งเดิมโดย carb0n:

You're right and I know Live Captions (windows) is not accurate, but it's better than nothing!

I learned some lore and info from the (floating head) cutscenes in the Crash games thanks to the mostly inaccurate live captions, which still relayed enough information to enhance my experience, than no subtitles at all - which is kind of my point :)

Hmm, https://flathub.org/apps/net.sapples.LiveCaptions might be installable in desktop mode without any tinkering

Thats exactly what i was wondering if any solutions already existed on linux

[carb0n]

โพสต์ดั้งเดิมโดย Ben Lubar:

If a system-wide automated captioning solution exists for Linux (I haven't searched for one) you could run the Steam Deck in desktop big picture mode and use that.

โพสต์ดั้งเดิมโดย Ben Lubar:

Hmm, https://flathub.org/apps/net.sapples.LiveCaptions might be installable in desktop mode without any tinkering

Now this is a thought - I will have to check this out, I already knew Live Captions were not really an option for most Linux distros but I didn't think to check GitHub for projects, thanks for the idea! I will explore this.