[SOLVED] Some letters (i) are not seen correctly on different Languages

[dragut]

I just start Dominus Galaxia and I am at New Game screen.

I believe this issue I have is due to Turkish keyboard I use as language setting, although I play games in English language (Most of my games are in English as preferred setting, but my computer language is in Turkish).

Previously I had similar issue as a beta-tester in a different game, it was due to default JAVA locale of local machine. For Turkish language it was using its locale and some fonts in the game didn't recognize the glyphs(letters) and thus did not render.
So basically setting explicitly to english locale as default solved the issue. Please check.

Here I share a screenshot of what I see:
https://www.imageupload.net/image/B4Egd

[hurleybird]

You Turks and your i's :P

I've been able to reproduce, and it looks like an issue with Text Mesh Pro, where it does render i's correctly, *except* for lowercase i's when SC flag (all uppercase, but non-capital letters are smaller) is set. I'll forward the bug along. Guessing it shouldn't take too long to get a resolution.

[dragut]

Haha :D But I prefer using the games in English, for this reason your game is one of the rare instances I encountered. But this is only bug I encountered, amazing job you did.

I also use Text Mesh Pro, maybe it can be because of font selection. Some fonts do not cover many letters, therefore that creates the problem. I suggest adding some replacement letters to Text Mesh Pro (there is a feature of TextMeshPro in doing that, I should re-watch the tutorial, he tells it in it) which general fonts (like Arial, TimesNewRoman, etc) can patch the font you choose to use. If needed, I can give more info after studying TextMeshPro's that function again. Also I believe to a computer where local language is English, it chooses the letters correctly but where the locale language is Turkish, it tries to choose Turkish versions by mistake and that creates problem.

[hurleybird]

Another developer (or at least hobbyist), awesome! In that case, I can get a bit more technical :)

I actually can't recall what the letters stand for, but the SC tag is the one that displays all characters as uppercase versions of themselves, but the characters that were previously lowercase are smaller.

I think the issue is that when you use the SC tag, .ToUpper() is called without much thought as to culture. So the i becomes İ, which doesn't have a corresponding glyph in the text asset. Nor is that glyph included in any of the default assets.

So, there are a few solutions I can think of. Most direct would be adding the 'İ' glyph (but using the same visual as the 'I' glyph). In the past, editing fonts has seemed like a giant headache so I don't think I'll do that. That, and it's inefficient to duplicate the same glyph.

The easiest fix would just be to force one culture. Eg.

Thread.CurrentThread.CurrentCulture = new CultureInfo("en-US");

But I'm super hesitant to do that. That changes the culture for the thread, not the process, and such a schism could eventually lead to some truly nasty issues.

A really hacky, but effective, solution would be to just go around and replacing all the i's with I's for any TextMesh where the SC tag is used. Of course, it's easy to miss some or to reintroduce issues in the future. (Edit: Brain fart, if I did this the I's would appear large).

Another would be just to abandon the SC tag :P

Assuming that Stephen (TMPro guy) agrees that this is a problem, he might just use invariant culture for the .ToUpper() (or whatever is causing the issue) call... which would be better in 99.9% of cases, except maybe for if you have a Turkish language game that actually wants to use the İ glyph with the SC tag (do you think that's a real consideration?). I think the best thing would just to have a setting for TMPro that picks the default culture (eg. current, invariant, en-US, etc.).

[dragut]

Sure! That's a pleasure for me.

I made an experiment (writing the word "Illinois") SC tag turns all letters to uppercase properly in TextMeshPro default LiberationSans. But when I wrote "ı" letter (in Turkish), it made it square. So I dare to think, if you wrote ı in the menu text that should happen, but of course as you don't use Turkish keyboard that should be impossible.

(1) I don't know the issues you may face so please forgive my ignorance on this topic, but I do believe that forcing one culture for text would be the best. Only, if a Spanish, French, Turkish user chooses another language in the game (I didn't check if the game has localization) then the culture will be changed.

(2) Otherwise, when you add new text, stories, etc to the game, you always have to remember upper I/İ subject in mind and have to make replacements just for the sake of Turkish language, I will be sad for you for having such trouble for us. Later, if you add stories, new texts, this will be a headache, so please think a different solution than this.

(3) Another approach would be, TextMeshPro-Font Asset Creator (Window>TextMeshPro>Font Asset Creator) could be the remedy. You surely processed your font in his window in the very beginning, you can make an experiment here again by adding Custom Characters (ıİ). When I was looking for a quick reminder tutorial for myself in order to talk you about this properly, I encountered this nice video, I wanted to share with you, the Custom font set he created looks really cool. As we can change Font in PNG, what if changing İ and ı to > I and i there (after you included them in Custom Characters)? After doing that, maybe we can experiment your menu again if your current fonts will try to look like Turkish fonts again so they can always look like English.

My suggestions at this point goes with (1) and (3) which (1) looks easier. I say again, I have many Unity games on this same computer at the moment and only this game have this problem right now. For this reason I consider if other developers (like me) don't use culture info at all or if used, enforcing en-US only.

Yes I agree, if Stephen can fix that by picking local or en-US or another forced culture, that would be best. I am only a little worried if Stephen would do that, I remember he didn't answer my last email properly and omitted details I was looking for, but I hope he adds such change you suggest because that seems really necessary.

[hurleybird]

Originally posted by dragut:

Sure! That's a pleasure for me.

I made an experiment (writing the word "Illinois") SC tag turns all letters to uppercase properly in TextMeshPro default LiberationSans. But when I wrote "ı" letter (in Turkish), it made it square. So I dare to think, if you wrote ı in the menu text that should happen, but of course as you don't use Turkish keyboard that should be impossible.

Interesting. Can confirm that small i with SC renders as squares for me with the default LiberationSans asset and TMPro 2.0.1. Might be that those default assets are different/older/generated differently? But then I would expect that you would see İ and not I... Will investigate more on my end. Just in case your Unity editor isn't using tr-TR for some reason, can you try adding: Thread.CurrentThread.CurrentCulture = new CultureInfo("tr-TR"); to an Awake() method and retesting?

Originally posted by dragut:

(1) I don't know the issues you may face so please forgive my ignorance on this topic, but I do believe that forcing one culture for text would be the best. Only, if a Spanish, French, Turkish user chooses another language in the game (I didn't check if the game has localization) then the culture will be changed.

My reservation is that I can't set a culture just for UI (CurrentUICulture seems to do nothing as far as Unity, or at least TMPro is concerned). And my understanding is that when you change Thread.CurrentCulture you're only changing the culture for that thread, not for the entire process. So the doomsday scenario in my head is that you could have some data that is dependant on culture being accessed on a thread using an incompatible culture.

Originally posted by dragut:

Another approach would be, TextMeshPro-Font Asset Creator (Window>TextMeshPro>Font Asset Creator) could be the remedy. You surely processed your font in his window in the very beginning, you can make an experiment here again by adding Custom Characters (ıİ). When I was looking for a quick reminder tutorial for myself in order to talk you about this properly, I encountered this nice video, I wanted to share with you, the Custom font set he created looks really cool. As we can change Font in PNG, what if changing İ and ı to > I and i there (after you included them in Custom Characters)? After doing that, maybe we can experiment your menu again if your current fonts will try to look like Turkish fonts again so they can always look like English.

Right, that's actually a relatively good idea. Editing fonts in my experience is a giant PITA (but maybe I just haven't found the right tools). Going in and changing the generated .png should be straightforward though. Of course, would need to remember to do this any time a font asset is generated or regenerated.

Originally posted by dragut:

I have many Unity games on this same computer at the moment and only this game have this problem right now.

I wonder if any of those use TMPro with the SC tag enabled anywhere? If one comes to mind, that would be interesting. Could give those devs a ring to see what they did.

[dragut]

When I tested SC again now in default LiberationSans, it turns i to I correctly but not "İ". When I write ı (not "i"), it turns to I correctly again.
Furthermore, my TextMeshPro is pretty old, 1.0.54 (Dec 2016) which I am currently on LTS 2017.4.35. Therefore such regression is strange if 2.0.1 doesn't do this properly. (I didn't update, because of this principle: "if something is working good, don't touch it" due to many past headaches related to updates)

Yes my Unity is using en-EN fully. And I tested as you asked using:
System.Threading.Thread.CurrentThread.CurrentCulture = new CultureInfo("tr-TR");
Unfortunately nothing changed.

I mentioned a video in the previous message but I noticed I didn't post the link by mistake, so here it is: https://www.youtube.com/watch?v=nBcP7FaDPE0

If you are hesitant to enforce en-EN culture, then alternative approach could be adding an extra "I" gylphs for each Turkish İ and ı variants through TextMeshPro Font asset creation window.

Originally posted by hurleybird:

Originally posted by dragut:

I have many Unity games on this same computer at the moment and only this game have this problem right now.

I wonder if any of those use TMPro with the SC tag enabled anywhere? If one comes to mind, that would be interesting. Could give those devs a ring to see what they did.

I didn't notice if same problem happened in rest of the game except very few screens. I checked the game and browsed almost all panels quickly now and UPPERCASE titles are seen in everywhere and I is properly displayed. But in Diplomacy>Stats panel and New Game>Advanced Options and Quit to Windows Panel has I-Square problem (Uppercase, not UPPERCASE), which you can write all those with lowercase-SC or UPPERCASE like you did in rest of your UI (large majority of UI titles). As the use of SC is so little and if you don't plan to rely on SC often, I believe this is neglectable, what do you think?

Also there is a 4th approach I can suggest now, what do you think of using this kind of fonts for titles instead (SC-Native, hahah):
https://www.1001fonts.com/sf-intellivised-font.html
And I believe this font example would look really cool in the game in my very humble opinion (IMVHO).

[hurleybird]

Originally posted by dragut:

Also there is a 4th approach I can suggest now, what do you think of using this kind of fonts for titles instead (SC-Native, hahah):
https://www.1001fonts.com/sf-intellivised-font.html
And I believe this font example would look really cool in the game in my very humble opinion (IMVHO).

At first blush, I rather like those. I'll give them a try :)

[hurleybird]

Oh, this should be *sort of* fixed in the new version, where the I's will render but in the wrong font. Getting them to render in the correct font slipped my mind this time.

[dragut]

That's okay, I will be checking, thanks for the follow-up.

[dragut]

Update: I checked and everything looks fine now, which "I" (in SC) are rendered as "İ" at the moment (Normally I prefer I, not İ to be honest for keeping SC in English context, but current look is acceptable.)