Localizing DW2 for non-Latin character sets

[elliotgibbs]

Hi all

This is a heads-up explaining a new feature coming in the next version of DW2 that will provide better support for languages with non-Latin alphabets.

Unfortunately the window for releasing public beta updates has closed for this year, so this update won't be available until January (version 1.2.8.0).

But this information might help anyone who is considering how to include a custom font that supports other character sets.

See the attached document for an outline of the general process. In summary you need to perform the following steps:

1. translate all of the text in the various XML and text files (as with any Latin language)
   
2. create a Stride bundle with a font that supports your character set
   
3. switch the font bundle used in the game to your new custom font bundle

Currently step 3 (load the custom font bundle) requires modding using Harmony. The Chinese localization mod does this.

However from version 1.2.8.0 onwards there will be a new command line argument that allows loading a custom font bundle. No code modding is required for this, just add the command line argument. This is outlined in the attached document on pages 9 and 10.

In the meantime the first portion of the document may be helpful for anyone wondering how to make a Stride bundle containing the custom font.

Feel free to ask any questions.

Thanks
Elliot

https://www.matrixgames.com/forums/download/file.php?id=1222327

[miimii]

Hi
I am trying to translate this game into Japanese.

I have two problems at the moment.

The first is a problem with line breaks.

English automatically breaks lines with a space, but Japanese does not put a space between each word.

Some other games have a zero-width space per character so that line breaks are automatically inserted.

I tried this in this game, but zero-width spaces did not cause a line break.

The second problem is that the tour pop-up window does not disappear.

When clicking on the icon in the top left corner, each window extends downwards, but the tour pop-up appears on the right side and does not disappear.

In GameText.txt, the display is as follows.

;A tour is available for {0}. \nClick the button to launch the tour
;Click to launch a tour of {0}

If it is left in English, there is no problem, but if translated into Japanese, the pop-up remains displayed.

The problem with line breaks in particular cannot be solved at the moment without manually inserting spaces, which is troubling.

Please help.

[elliotgibbs]

Hi milmil

Thanks very much for your work on the Japanese localization.

Do you know the character code for the zero-width space that you are trying to use? We could add that to our list of characters that we use for line breaks.

For the tours, it's probably easiest to just leave the titles untranslated for now. We will eventually get a solution to this, but at the moment we are relying on the English text to reference the tour steps.

Thanks
Elliot

[miimii]

Hi elliotgibbs

Thank you for your answer.

The character code for zero width spaces is U+200B in Unicode.

I will do as you say about the tour.

[elliotgibbs]

In the next build I've added support for U+200B as another character to indicate a valid place for line-breaks (in addition to the existing space character). Not sure how to test this, but you can give it a try when it's out, probably at the end of this week.

[miimii]

Hi elliotgibbs

Thanks for the beta update.
I have confirmed that I can break lines with zero-width spaces.

However, zero-width spaces appear the same width as normal spaces.

Short sentences often have a width of zero, but longer sentences, such as those containing line breaks, will have the same width as a space.

Is it possible to improve this?

[elliotgibbs]

In the next build (1.2.9.0) I have improved this so that long text with word-wrap will properly render zero-width spaces instead of standard full-width spaces.

Look out for this build at the end of next week.

[miimii]

Hi elliotgibbs

Thanks for the beta update.

I have checked and there is still space width available.

In the screenshot of the link below, there are spaces in the text on the right and in the button text.

https://steamcommunity.com/sharedfiles/filedetails/?id=3427418366

Is it possible to improve this?

[elliotgibbs]

I've checked the text rendering for that specific passage (the Human intro text) using your Japanese text from the mod, and it looks to be rendering properly to me in version 1.2.9.0.

To explain how the zero-width spaces are used: when rendering text with word-wrap (i.e. long multi-line text) then the spaces are used to determine when to break onto a new line. When rendering CJK characters that use zero-width spaces, no additional full-width spaces are added.

So any spacing you are seeing is coming from the font itself. I note that the comma and full-stop characters seem to have extra space after them, but as I understand it that is normal.

[miimii]

Hi elliotgibbs

GameEvents files using zero-width spaces had not yet been uploaded.

The version you have checked is the version with manual single-byte spaces.

I have just uploaded a file with zero-width spaces, so please check it when you have time.

[elliotgibbs]

Thanks for that upload. Checking with that I can see the problem: some of your paragraphs start with a zero-width space.

To optimize text rendering performance I'm character-sniffing for CJK characters and only enable zero-width space line breaks when I detect a paragraph using CJK. In the next build I'll add the zero-width space itself as a valid indicator of CJK text.

In the meantime you can achieve the same result by removing the zero-width spaces from the start of paragraphs.

[miimii]

Hi elliotgibbs

I have confirmed that this works well once the zero-width space is removed from the beginning of the paragraph.

However, this is not a problem if the paragraph starts with kanji, but does not work if it starts with hiragana.

Japanese is a messy language, with a mixture of kanji, hiragana and katakana...

Would adding the zero-width space itself as a valid indicator for CJK text solve the problem?

[elliotgibbs]

Ok, from here: https://www.unicode.org/charts/

...it looks like Hiragana starts at decimal 12353, Katakana starts at 12448 and Hangul starts at 4352. Looks like some kana characters around 6900 also.

Does that sound right?

I'll extend the range of character codes to cover all of the above. Maybe just everything at 4352 and above.

[miimii]

Hi elliotgibbs

I think that's right.

We don't need Kana around 6900.

[alnigor]

I would just like to say a word of thanks from myself as a translator moder for adding the possability to inject non-standard fonts. And want to remind you of the three problems faced by those who want to translate your wonderful game into their own language.

General issue wich need improvement - there are terms in the GameText.txt that are used in the game in different situations/contexts, but are spelled identically and use one and the same entry in the said file, like "dismiss, general....". This makes sense in English, but not in other languages. Often you need to use completely different words (translate) for different situations.

Second issue: translating of terms 'Race, Expansion, Proximity and Government' displayed in the game-start settings-window "Start New game - Other Empires" in the top-menu-line below "Add new Empire" is immposible. As I understand that words takes from dll file, but if we translate it in that file game crashes (not loaded).

And thrid wich also so upset: First greeting-/starting-message of ALL races at the beginig of new game (right after the beginning of a new game, when the players are informed about their race, etc) can't be translated also due to that text don't located in any of the files we have access.