Pocket Plane Group

Friends and Neighbors => Weimer Republic (WeiDU.org) => WeiDU => Topic started by: jastey on November 14, 2013, 01:59:48 PM

Title: Question to LAM bgee_language
Post by: jastey on November 14, 2013, 01:59:48 PM
As far as I see, you defined a WeiDU Macro called bgee_language that takes the tra-files out of tra\bgee\%LANGUAGE% instead of tra\%LANGUAGE% ?

Will this also work in combination with USING, e.g.


  ACTION_IF GAME_IS ~tob bg2ee~ THEN BEGIN
    COMPILE ~C#Ajantis_SvA\Dialogues\ToB_only.d~
USING ~C#Ajantis_SvA\Tra\%s\C#AjanJ_pr.tra~
  END

Will the correct tra files be used in this case, as well? Is there a description to this macro somewhere? I didn't find any by using a search engine.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 14, 2013, 04:51:45 PM
It's actually not a WeiDU macro. It's part of RR. It's defined in "rr/lib/lib.tpa":
Code: [Select]
DEFINE_ACTION_MACRO bgee_language BEGIN
  ACTION_IF GAME_IS ~bgee bg2ee~ BEGIN
    LOAD_TRA ~rr/tra/bgee/english/game.tra~
    LOAD_TRA ~rr/tra/bgee/%LANGUAGE%/game.tra~
  END
END

It will unfortunately do nothing for instances of USING, AUTO_TRA and other instances of LOAD_TRA. It just affects "setup.tra" (the tra files you declare as part of the LANGUAGE statement).

There's a not completely terrible solution for USING and LOAD_TRA. You declare a %tra_path% variable in your ALWAYS block and change your USINGs and LOAD_TRAs into using the %tra_path% variable instead of hardcoding the path.
Code: [Select]
ALWAYS
  ACTION_IF GAME_IS ~bgee bg2ee~ BEGIN
    OUTER_SPRINT tra_path ~mymod/tra/bgee~
  END ELSE BEGIN
    OUTER_SPRINT tra_path ~mymod/tra~
  END
END

// elsewhere

COMPILE ~mymod/foo.d~ USING ~%tra_path%/%LANGUAGE%/foo.tra~
LOAD_TRA ~%tra_path%/%LANGUAGE%/bar.tra~
The saving grace here is that %tra_path% is not something you will have to undo if and when a proper solution materialises.

I'm afraid there's no good solution for AUTO_TRA (at least, not that I can think of), since AUTO_TRA declares the path unconditionally, before you get the chance to define anything yourself.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 15, 2013, 10:31:52 AM
Thank you! Macros scare me so much I didn't even start to look in the libraries. Thank you for the suggested code, too.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 15, 2013, 11:01:12 AM
No good solution for AUTO_TRA:

Do I see it right that my possibilities are either:

1. put all tra lines in the setup.tra (There are 53 tra files, the thought alone scares me to death.) and use the bgee_language macro

2. use your suggested code above and either specify every single tra file by USING or load the tra file before compiling the .d by LOAD_TRA,  (There are 53 tra files in my mod and the thought alone...)

Is there any possibility to automate the USING / LOAD_TRA command using something like "SPRINT dfile_name ~%SOURCE_RES%~" or something like that..?

Does anyone have a better idea for this?

Title: Re: Question to LAM bgee_language
Post by: DavidW on November 15, 2013, 01:43:37 PM
I have a better idea (that falls short of being optimal): treat (e.g.) "German (for original game)" and "German (for Enhanced Edition)" as different languages.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 15, 2013, 01:57:02 PM
While mucking around with Edwin Romance's tra files (w/ AUTO_TRA), I wrote some functions that automate the AUTO_TRA -> USING conversion, with a fairly minimal amount of extra typing. Soon as I have tested them I'll post them and see if they're to anyone's taste.
Well, that didn't work. Compiling D files one by one (like you have to do with USING) runs afoul of all the D/DLG interdependencies.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 15, 2013, 02:13:20 PM
I love you both. :)

Very eager to hear more about your functions, Wisp, but DavidW: your idea is also truly genius. (Now I only need a mass converting tool I actually get to run.)
Title: Re: Question to LAM bgee_language
Post by: cmorgan on November 16, 2013, 03:07:38 PM
Still struggling with this -

http://forum.baldursgate.com/discussion/12677/localisation-of-mods-for-bgee#latest

Declaring and then loading using LOAD_TRA does not seem to do the job.

DavidW's idea works, I assume becasue something hinkey is going on within LANGUAGE.

So,
on BGII:EE and WeiDU v235,
Code: [Select]
ALWAYS
ACTION_IF GAME_IS ~bg2ee~ BEGIN
OUTER_SPRINT tra_version ~c-aranw_cp1252~
END ELSE BEGIN
OUTER_SPRINT tra_version ~c-aranw_utf8nb~
END
END

LANGUAGE ~English_BG2EE~ ~English~ ~aranw\tra\english\c-aranw_utf8nb.tra~
LANGUAGE ~English_BG2~ ~English~ ~aranw\tra\english\c-aranw_cp1252.tra~
results in a working .tp2

but using LOAD_TRA and USING ~aranw/tra/%LANGUAGE%/%tra_version%.tra~ in multiple configurations does not seem to work .

Specifically, with

Code: [Select]
ALWAYS
ACTION_IF GAME_IS ~bg2ee~ BEGIN
OUTER_SPRINT tra_version ~c-aranw_cp1252~
END ELSE BEGIN
OUTER_SPRINT tra_version ~c-aranw_utf8nb~
END
        LOAD_TRA  ~aranw\tra\english\%tra_version%.tra~
END

 LANGUAGE ~English~ ~English~ ~aranw\tra\english\%tra_version%.tra~


results in the following:

Code: [Select]
WeiDU v 23500 Log

 C:\Program Files (x86)\BeamDog\Games\00546\setup-aranw.exe
[./chitin.key] loaded, 843510 bytes
[./chitin.key] 185 BIFFs, 59857 resources
[.\weidu.conf] loaded, 17 bytes
[lang\en_us/dialog.tlk] loaded, 13189457 bytes
[lang\en_us/dialog.tlk] 114144 string entries
Using Language [English]
[English] has 1 top-level TRA files

Using lang\en_us/dialog.tlk

Would you like to display the readme? [Y]es [N]o
ERROR: No translation provided for @1
Continuing despite error.

Install Component [UNDEFINED STRING:   @1]?
[I]nstall, or [N]ot Install or [Q]uit? ERROR: No translation provided for @157
Continuing despite error.
ERROR: No translation provided for @160
Continuing despite error.
ERROR: No translation provided for @161
Continuing despite error.
ERROR: No translation provided for @162
Continuing despite error.
ERROR: No translation provided for @164
Continuing despite error.
ERROR: No translation provided for @167
Continuing despite error.
ERROR: No translation provided for @168
Continuing despite error.
ERROR: No translation provided for @169
Continuing despite error.
ERROR: No translation provided for @170
Continuing despite error.
ERROR: No translation provided for @171
Continuing despite error.
Saving This Log:

WeiDU Timings
load TLK                         0.000
parsing .log files               0.000
unmarshal KEY                    0.016
loading files                    0.016
Parsing TP2 files                0.016
stuff not covered elsewhere      0.031
unmarshal TLK                    0.078
TOTAL                            0.156
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 16, 2013, 03:18:22 PM
Still struggling with this
LANGUAGE is evaluated before ALWAYS, so you can't put variables in there like that. If you are not using AUTO_TRA, I would suggest you use something similar to bgee_language (invoked in your ALWAYS block; overrides the relevant tra files loaded by LANGUAGE) and %tra_path% (set in your ALWAYS block; used to softcode your tra files in USING and LOAD_TRA).
I've been meaning to write a summary of ways of dealing with the charset issue, but I've been super busy and haven't yet had time.
Title: Re: Question to LAM bgee_language
Post by: cmorgan on November 16, 2013, 03:50:01 PM
So in order to do this correctly,

1. Declare LANGUAGE with a "hardcoded" default (so the .tp2 can fall back on it)
2. Override the default language in the ALWAYS block.

Code: [Select]
ALWAYS
ACTION_IF GAME_IS ~bg2ee~ BEGIN
OUTER_SPRINT tra_version ~c-aranw_utf8nb~
    LOAD_TRA ~aranw\tra\english\c-aranw_utf8nb.tra~
    LOAD_TRA ~aranw\tra\%LANGUAGE%\c-aranw_utf8nb.tra~
END ELSE BEGIN
OUTER_SPRINT tra_version ~c-aranw_cp1252~
    LOAD_TRA ~aranw\tra\english\c-aranw_cp1252.tra~
    LOAD_TRA ~aranw\tra\%LANGUAGE%\c-aranw_cp1252.tra~
END
END

LANGUAGE ~English~ ~English~ ~aranw\tra\english\c-aranw_cp1252.tra~


Edit: Tested - works. It loads the tra twice, which is fine by me. Checking the dialog.tlk shows that the lines are appended just fine.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 17, 2013, 02:52:31 AM
Do I see it right that the setup.tra (where the mod title is taken from) has to be in the path specified by LANGUAGE?

Or, that ALWAYS is processed after the BEGIN of the mod?

I tried copying the correct tras into the LANGUAGE folder before the install, having empty language folders at the beginning and "bgii" and "utf8" language subfolders. I tried using the ALWAYS block for copying the tras into the folders specified by the LANGUAGE path, but it gave me an "unspecified string" instead of the mod name.

I tried specifying the setup.tra in the ALWAYS block but this didn't work, either (the mod name string is unknown):

Code: [Select]
ALWAYS
ACTION_IF GAME_IS ~bgee bg2ee~ BEGIN

LOAD_TRA ~C#Ajantis_SvA\Tra\utf8\english\Setup-C#Ajantis_SvA.tra~
LOAD_TRA ~C#Ajantis_SvA\Tra\utf8\%LANGUAGE%\Setup-C#Ajantis_SvA.tra~

COPY ~C#Ajantis_SvA\Tra\utf8\%LANGUAGE%~ ~C#Ajantis_SvA\Tra\%LANGUAGE%~

END ELSE BEGIN

LOAD_TRA ~C#Ajantis_SvA\Tra\bgii\english\Setup-C#Ajantis_SvA.tra~
LOAD_TRA ~C#Ajantis_SvA\Tra\bgii\%LANGUAGE%\Setup-C#Ajantis_SvA.tra~

END
END

I like the idea of copying the correct tras into the LANGUAGE folder, as it saves me a lot of hassle. How would I do that so the installer knows the mod's name?
Sorry for all my questions, I am still trying to understand all this.

EDIT: The reason why I want to have both folders is the re-installability. If I have the windows-1252 encoded files in the LANGUAGE folder and the the UTF-8 ones get copied there upon install, I fear the windows-1252 files are lost after uninstallation of the mod?
This could be solved via to-and fro-copying upon install and uninstall, but I fear I don't really know what I am doing.

EDIT2: Or maybe my copy command doesn't work. I used
Code: [Select]
ALWAYS
ACTION_IF GAME_IS ~bgee bg2ee~ BEGIN
COPY ~C#Ajantis_SvA\Tra\utf8\%LANGUAGE%~ ~C#Ajantis_SvA\Tra\%LANGUAGE%~
END ELSE BEGIN
COPY ~C#Ajantis_SvA\Tra\bgii\%LANGUAGE%~ ~C#Ajantis_SvA\Tra\%LANGUAGE%~
END
END

but it doesn't copy any files into the %LANGUAGE% folder.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 17, 2013, 06:19:12 AM
Hey, neat idea! Yeah, copying works, and as far as workarounds go, it's pretty unintrusive.

Using Edwin Romance as an example:
Code: [Select]
AUTO_TRA ~edwin/tra/%s~ // Your normal AUTO_TRA declaration (not that we have a choice)

ALWAYS
  ACTION_IF GAME_IS ~bg2ee~ BEGIN
    OUTER_SPRINT tra_path ~%MOD_FOLDER%/tra/bgee~
  END ELSE BEGIN
    OUTER_SPRINT tra_path ~%MOD_FOLDER%/tra~
  END

  /*
   * Loads UTF8-encoded tra files to replace those loaded by LANGUAGE
   */
  DEFINE_ACTION_MACRO bgee_language BEGIN
    ACTION_IF GAME_IS ~bgee bg2ee~ BEGIN
      LOAD_TRA ~%MOD_FOLDER%/tra/bgee/english/game.tra~
      LOAD_TRA ~%MOD_FOLDER%/tra/bgee/%LANGUAGE%/game.tra~
    END
  END

  /*
   * Copies UTF8-encoded tra files over the normal ones, for use with AUTO_TRA
   */
  DEFINE_ACTION_FUNCTION autotra_workaround BEGIN
    COPY ~%MOD_FOLDER%/tra/bgee/%LANGUAGE%~  ~%MOD_FOLDER%/tra/%LANGUAGE%~
  END

  LAM bgee_language
  LAF autotra_workaround END
END

/*
 * In LANGUAGE, tra files have been split into different files depending on what they are used for
 * Strings that are displayed in the game need to follow the game's encoding
 * Strings that are displayed during the installation (component names, etc.) need to follow the OS's encoding
 * Additionally, since different OSs use different encodings, there is need for one file per OS
 * Accordingly:
 * game.tra: encoded in whatever charset is used by the target game, UTF8 and "normal" copies are needed
 * setup-win32.tra: encoded in whatever local charset is used by Windows for the relevant language
 * setup-osx.tra: encoded in UTF8
 * setup-unix.tra: encoded in UTF8
 * This is obviously as optional as you want to make it, and multiple setup-*.tra files can be safely omitted
 * for languages or translations that don't need special encodings (typically English, for example)
 */
LANGUAGE ~American English~
         ~english~
         ~edwin/tra/english/game.tra~ // Holds text displayed in the game
         ~edwin/tra/english/setup.tra~ // Holds text displayed during the installation

LANGUAGE ~Francaise~
         ~french~
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/french/game.tra~
         ~edwin/tra/french/setup-%WEIDU_OS%.tra~

LANGUAGE ~Espanol~
         ~spanish~
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/spanish/game.tra~
         ~edwin/tra/spanish/setup-%WEIDU_OS%.tra~

LANGUAGE ~Russian~
         ~russian~
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/russian/game.tra~
         ~edwin/tra/russian/setup-%WEIDU_OS%.tra~

LANGUAGE ~Deutsch~
         ~german~
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/german/game.tra~
         ~edwin/tra/german/setup-%WEIDU_OS%.tra~

LANGUAGE Polish
         polish
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/polish/game.tra~
         ~edwin/tra/polish/setup-%WEIDU_OS%.tra~

LANGUAGE ~Simplified Chinese~
         schinese
         ~edwin/tra/english/game.tra~
         ~edwin/tra/english/setup.tra~
         ~edwin/tra/schinese/game.tra~
         ~edwin/tra/schinese/setup-%WEIDU_OS%.tra~

// further down

//Dialogues (compiled with AUTO_TRA)
COMPILE
~edwin/dlg/eredwinp.d~
~edwin/dlg/eredwinj.d~
~edwin/dlg/ervicon.d~
~edwin/dlg/ermazzy.d~
~edwin/dlg/eranomen.d~
~edwin/dlg/erelvira.d~
~edwin/dlg/erlovetalk2.d~
~edwin/dlg/erterl02.d~
~edwin/dlg/eredwina2.d~
~edwin/dlg/eredwina.d~
~edwin/dlg/erlovetalk.d~
~edwin/dlg/erimoen.d~

// yet further down

// Compiled with USING; %tra_path% is redundant in this case, but I'm including it as a demonstration
COMPILE ~edwin/dlg/erlovetalk25.d~ USING
          ~%tra_path%/english/erlovetalk25.tra~
          ~%tra_path%/%LANGUAGE%/erlovetalk25.tra~
Title: Re: Question to LAM bgee_language
Post by: jastey on November 17, 2013, 06:33:30 AM
Thank you very much for this.

I have more questions.  :-[

In your example, what happens upon uninstall from a BGII:EE game? Will the former encoded tra files be in the LANGUAGE folder again?

Question to game.tra: Is this a standard like setup.tra? Because in my case, setup.tra also contains text that will show in the game (item descriptions, etc.). Is this externalized in game.tra for the current WeiDU? I totally missed this.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 17, 2013, 06:53:30 AM
In your example, what happens upon uninstall from a BGII:EE game? Will the former encoded tra files be in the LANGUAGE folder again?
Yes, the "normal" TRA files will be restored upon uninstallation.

Question to game.tra: Is this a standard like setup.tra? Because in my case, setup.tra also contains text that will show in the game (item descriptions, etc.). Is this externalized in game.tra for the current WeiDU? I totally missed this.
"setup.tra" is just a convention. You can use LANGUAGE to load whatever TRA files you wish, they just can't overlap, or they'll shadow one another. In this case, you split your TRA file into two parts, encode them separately and load them at the same time (in LANGUAGE). The end result is identical to if you had had a single TRA file, except that you can use different encodings. Your new "game.tra" contains the text for creature names, item descriptions etc., and "setup.tra" is left with component names and such.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 17, 2013, 11:04:27 AM
Thank you! Heaven! Peace! It is working!  8)
Title: Re: Question to LAM bgee_language
Post by: Isaya on November 17, 2013, 12:51:30 PM
The BG1 NPC beta version, which uses on the fly conversion of the tra files, uses a similar method in order to keep the files in the directory specified through LANGUAGE and maintain compatibility with AUTO_TRA. Although, instead of COPY, it uses MOVE, which is also reversed at uninstall. Another difference is that I used MOVE on a file by file basis.

The example of BG1 NPC also taught me another lesson: if your mod has several components, beware of ALWAYS, as it is run with every component. So you're going to copy the tra files over and over and your backup directory will grow unexpectedely.

In BG1 NPC, due to the fact that the first component, required, evaluates tra files using the suffix _tmp and generates tra files without the _tmp part, I used an ACTION_IF to check for the existence of the resulting file to determine whether the operation had already been done and avoid repeating it. In case of on the fly conversion, this is very important since the file in the directory could already be converted and iconv doesn't behave well when you tell it to convert from CP1252 while it's actually already UTF-8, as the secondary byte in UTF8 for a special character can be seen as a special character as well in CP1252, so it converts it and the result is garbage and probably invalid UTF-8 code.

When using premade separate sets for CP1252 and UTF-8 in a multi-component mod, I would still advise some protection be used in ALWAYS. I'm not sure it's possible to use a variable. I have the feeling they are reset when we move to another component. I believe creating an fake file after copying the tra files and checking for its presence to avoid copying again would work.

Here is an example of how it's done in BG1 NPC (this code is slightly different from the GitHub code as I added Linux and Mac handling, and I didn't submit it to cmorgan yet)
Code: [Select]
ACTION_IF FILE_EXISTS_IN_GAME ~neera.dlg~ THEN BEGIN
  /*Tell the player it is using BG:EE stuff */
  PRINT @5000
  INCLUDE ~bg1npc\lib\liam_bgee_vars.tpa~
  OUTER_SPRINT ~bgee~ ~bgee/~

  // BG:EE tra conversion
  // Character encoding for BG II for various languages: ensure the name matches the menu choices
  ACTION_DEFINE_ASSOCIATIVE_ARRAY languageencoding BEGIN
"french" => "CP1252"
"spanish" => "CP1252"
  END
  OUTER_SPRINT initialencoding ~~
  ACTION_PHP_EACH languageencoding AS lang => encoding BEGIN
    ACTION_IF ("%LANGUAGE%" STRING_EQUAL_CASE ~%lang%~ = 0) THEN BEGIN
      OUTER_SPRINT initialencoding ~%encoding%~
    END
  END
  // Isaya: check for existence of BG1NPC/TRA/%LANGUAGE%/BG1NPC.tra as a marker that the mod
  // has already installed at least one component, so conversion is already done, in order
  // to avoid repeating for each component (as a backup of all tra files is then done in each
  // subdirectory of backup)
  // This is a handy trick with BG1 NPC, since BG1NPC.tra is created when the mod is installed
  // and removed with the mod. But it cannot be used as a generic trick for other mods.
  ACTION_IF NOT (FILE_EXISTS_IN_GAME ~bg1npc/tra/%LANGUAGE%/BG1NPC.tra~) AND
            (STRING_LENGTH ~%initialencoding%~ != 0) THEN BEGIN
    ACTION_IF ("%WEIDU_OS%" STRING_EQUAL_CASE ~WIN32~ = 0) THEN BEGIN
      // Only convert for the language selected by passing the directory as parameter
      AT_NOW ~bg1npc/conv_tra.bat %LANGUAGE% %initialencoding%~
    END
    ACTION_IF ("%WEIDU_OS%" STRING_EQUAL_CASE ~OSX~ = 0) OR ("%WEIDU_OS%" STRING_EQUAL_CASE ~UNIX~ = 0) THEN BEGIN
      // Only convert for the language selected by passing the directory as parameter
      AT_NOW ~bg1npc/conv_tra.sh %LANGUAGE% %initialencoding%~
    END
    // Isaya: all tra files have to be renamed from .tra_utf8 to .tra!
    // Otherwise the tra files for dialogs are still with the Windows 1252 encoding
    MOVE ~bg1npc/tra/%LANGUAGE%/bg1npc_tmp.tra_utf8~ ~bg1npc/tra/%LANGUAGE%/bg1npc_tmp.tra~
    ... MOVE all tra files

    LOAD_TRA ~bg1npc/tra/%LANGUAGE%/setup.tra~
    PRINT ~re-loaded %LANGUAGE% TRA files~
  END

END ELSE BEGIN
  ACTION_IF FILE_EXISTS_IN_GAME ~FW0100.are~ THEN BEGIN
    /* Tell the player it is using Tutu stuff */
    PRINT @1000
    INCLUDE ~bg1npc\lib\g3_tutu_cpmvars.tpa~
    OUTER_SPRINT ~bgee~ ~~
  END ELSE BEGIN
    ACTION_IF FILE_EXISTS_IN_GAME ~AR7200.are~ THEN BEGIN
        /* Tell the player it is using BGT stuff */
        PRINT @1001
        INCLUDE ~bg1npc\lib\g3_bgt_cpmvars.tpa~
        OUTER_SPRINT ~bgee~ ~~
        /* Tell the player it is not Tutu or BGT */
    END ELSE BEGIN FAIL ~Please install on BG:EE, Tutu or BGT.~
    END
  END
END
with conv_tra.bat
Code: [Select]
:: Use a parameter to the script (%1) to specify the directory that must be converted
:: Use a parameter to the script (%2) to specify the encoding of the files to convert

:: Prevent display of DOS command lines
@echo off

:: Since the pattern matching in Windows considers that "fu.tra_utf8" also matches "*.tra", we have to
:: handle the file extension by ourselfves in order to avoid all those tra_utf8_utf8 etc files
:: Use %%~ni instead of %%~nxi (x is for extension) and specify the tra extension
for %%i in (bg1npc\tra\%1\*.tra) do bg1npc\iconv -f %2 -t UTF-8 "bg1npc\tra\%1\%%~ni.tra" > "bg1npc\tra\%1\%%~ni.tra_utf8"
and conv_tra.sh
Code: [Select]
# Use a parameter to the script ($1) to specify the directory that must be converted
# Use a parameter to the script ($2) to specify the encoding of the files to convert
for file in $(ls bg1npc/tra/$1/*.tra)
do
  iconv -f "$2" -t UTF-8  "$file" > "$file"_utf8
done
I couldn't test conv_tra.sh in a real installation as I don't have the game under my Linux virtual machine but I checked it the script standalone nonetheless.


Wisp, ever since I wrote the code to use MOVE in V231 and now in V235, I'm finding that the foo.tra_utf8 file I ask to move are actually copied under the new name but foo.tra_utf8 is not removed. Do you know if it's this a bug or if there's a reason for that?
Title: Re: Question to LAM bgee_language
Post by: jastey on November 17, 2013, 01:34:17 PM
Thank you for the heads up in regard to a mod with several components, and for sharing the code.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 17, 2013, 01:59:37 PM
Yeah, the copying should best be done only once. Variables are not unset between components, but you can install one component, quit and install another, so using a marker file is better.

Wisp, ever since I wrote the code to use MOVE in V231 and now in V235, I'm finding that the foo.tra_utf8 file I ask to move are actually copied under the new name but foo.tra_utf8 is not removed. Do you know if it's this a bug or if there's a reason for that?
MOVE only moves if the destination file does not exist. Otherwise it copies.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 17, 2013, 04:44:46 PM
I'm not entirely sure it is post-worthy, but here is an Edwin-Romance-styled autotra_workaround with a marker file:
Code: [Select]
  DEFINE_ACTION_FUNCTION autotra_workaround BEGIN
    ACTION_IF !FILE_EXISTS ~%MOD_FOLDER%/tra/%LANGUAGE%/utf8.mrk~ BEGIN
      COPY ~%MOD_FOLDER%/tra/bgee/%LANGUAGE%~  ~%MOD_FOLDER%/tra/%LANGUAGE%~
      COPY_EXISTING sw1h01.itm ~%MOD_FOLDER%/tra/%LANGUAGE%/utf8.mrk~
    END
  END

If it's desirable I'll try to find the time to summarise the current state of the art in dealing with the UTF8 situation (essentially, summarising and highlighting parts of this thread). This is likely as good as it is going to get until such time, if ever, that WeiDU is able to do charset conversions.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on November 17, 2013, 11:49:22 PM
Thank you very much for researching the problem, everyone! A tutorial will be very welcome. :)

('If ever' part is a little sad. :( I have no idea how to convert Polish/French/Italian files, since I can't check the result, not knowing the language. I can convert Russian files, sure).

Title: Re: Question to LAM bgee_language
Post by: Cahir on November 18, 2013, 12:40:47 AM
Kulyok I can check Polish part for you just tell me exactly what to do:)
Title: Re: Question to LAM bgee_language
Post by: Kulyok on November 18, 2013, 01:07:09 AM
Thank you, I'll be happy to ask for your help(once I know more about what's going on, how to do stuff and when we should expect multilanguage BG2EE releases - since I don't know about them much)!

Title: Re: Question to LAM bgee_language
Post by: Cahir on November 18, 2013, 01:36:32 AM
Ok just shoot me PM when You figure everything out ;)
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 18, 2013, 01:43:23 AM
('If ever' part is a little sad. :( I have no idea how to convert Polish/French/Italian files, since I can't check the result, not knowing the language. I can convert Russian files, sure).
Well, I don't want to get anyone's hopes up prematurely. And even if WeiDU will be able to offer something substantial to help with this, it won't be magic. At a minimum, WeiDU would have to know the charset used for the input text (and it would have to be a single charset). So it would operate under pretty much the same constraints you're under when you use iconv today.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on November 18, 2013, 07:16:45 AM
I'm looking at Edwin Romance's files right now, and it looks elegant and simple. I hope can replicate that(I don't have /tra folder, so I'll just create /BG2EE/German and such).

But what exactly should I do with the .tra files I have, how do I get BG2EE files out of them? I obviously can't read anything, but I see the visual difference between the files, so maybe I can do this. Btw, is there a way to convert all the files at once, not one-by-one?
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 18, 2013, 09:23:48 AM
But what exactly should I do with the .tra files I have, how do I get BG2EE files out of them? I obviously can't read anything, but I see the visual difference between the files, so maybe I can do this.
There are a few things that go into it.
First, you need to know which charset you are converting from. For English, German, French, Spanish and Italian this is typically CP1252 (the exception is text encoded for the pre-Vista Windows command prompt). For Polish and (I think) Czech it's CP1250 and for Russian it's CP1251. Simplified Chinese has been CP936 for the mods I have worked with. Traditional Chinese is tricky, because the exact encoding used is non-standard and can, to my knowledge, only be produced by traditionally Chinese Windows (because Microsoft botched the implementation). I think Japanese uses SHIFT JIS and Korean uses CP949.

However, there's a landmine here, because some translators have embedded text in another charset in the same file, because Windows prior to Vista used a different charset for its command prompt. In these cases, you need to (I would recommend it) split the files into one per charset, which is probably not something most people are equipped to do for languages other than their own. In other words, you may need a translator to fix the mess.

Related to the landmine, I would recommend that you place all strings that are used during the installation (component names, strings that are PRINTed, etc.) in a separate file, because only game text should be encoded in UTF-8 for BG(II)EE. Having them in separate files makes it much easier to handle.

Then, when you know which charset the text is encoded in, there are (probably) multiple programs you can use to make a copy of the file encoded in UTF-8. Beyond iconv (vide infra), I couldn't name any, however. Oh, and it's also important to note that it should be UTF-8 without any damned, completely redundant and asinine byte-order mark (BOM).

Btw, is there a way to convert all the files at once, not one-by-one?
Sure. You can, for example, use something like Isaya's batch file. iconv can be downloaded from here (http://gnuwin32.sourceforge.net/packages/libiconv.htm) (you want the zip with the binaries; the program is located in the bin/ directory, and you need the two dll files as well). Unlike Isaya's batch file, however, I would recommend that you organise your files with directories instead of weird filename extensions. That's what directories are for.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 18, 2013, 10:44:38 AM
Kulyok: I use CpConverter for the convertion, but you need Windows with .Net for it: http://sourceforge.net/projects/cp-converter/files/cp-converter/CpConverter%20V%200.1.5.0/

Wisp (and everyone else): I think Polish has different charsets for BG1 and BGII, too. I don't know which, though.

Title: Re: Question to LAM bgee_language
Post by: Kulyok on November 18, 2013, 11:43:48 AM
Thank you very much, folks. Yeah, I guess a translator's help is what we need - I dearly hope that various language communities would lend a hand.

(In Russian, I try to enforce the "do not translate installation lines, ever, leave them in simple English" rule, because we've had the problem with different encodings on different platrofm for a long time).
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 18, 2013, 11:54:34 AM
Wisp (and everyone else): I think Polish has different charsets for BG1 and BGII, too. I don't know which, though.
It's probably CP852, but I suppose someone would have to confirm it.
Title: Re: Question to LAM bgee_language
Post by: Cahir on November 18, 2013, 01:33:08 PM
Wisp, jastey, probably best person to ask about polish encoding used in vanilla BG1 is Zed Nocear. He's an expert in modding old BG1 and can most certainly confirm if this is CP852 or other encoding. But as I can see he is not active at PPG  :-[

All I could find is that polish developer CD Projekt used unusual encoding, but I couldn't find which exactly.
Title: Re: Question to LAM bgee_language
Post by: cmorgan on November 18, 2013, 08:19:28 PM

Wait, back up a sec, please, Wisp...

Related to the landmine, I would recommend that you place all strings that are used during the installation (component names, strings that are PRINTed, etc.) in a separate file, because only game text should be encoded in UTF-8 for BG(II)EE. Having them in separate files makes it much easier to handle.

Just to be clear on this, the game engine for BGII:EE uses cp1252.
The text shown/used in-game uses UTF-8(no BOM).

So... my mod, which uses attempts to use one single .tra, is getting away with declaring a cp1252 in LANGUAGE and then "swapping in" one in UTF-8(no BOM).

But if I want to be able to support multiple languages correctly, I really do need to split out into at least two .tra files:

1.  setup.tra [everything declared and referenced inside of the .tp2 or related processing files like .tpa .tpp .tph] (which would come encoded differently in different languages and platforms)
2. mymod.tra [everything called by .d and .baf to be placed into the dialog.tlk and displayed ingame] (which need to be set up into at least two copies per language; one {example:english=cp1252} for the older games, and one {ALL LANGUAGES THE SAME=UTF-8(no BOM)} for BGII:EE.

Or I need to set up a minimum of two files
1. setup.tra [as above]
2. mymod.tra [encoded in cp1252 and then convert them on the fly using a variant of Isaya's method, which is relying on outside processing].

I think I have this right, but if I don't, please correct me.

I ask this because it means untangling .tra materials, some of which, like the declared strings for soundset references in the .tp2, etc, show up in the game... so I am trying to figure out if all the .tp2 actions are pulling form the correct .tra.

Or is it that only the strings displayed in the cmd screen are the ones that need to be declared initially in LANGUAGE, and that everything actually processed pulls from the ALWAYS?

(For BG1NPC, I am throwing myself on Isaya's mercy. But for my mod, with 1.4mb .tra file, I want to set things up right from the start.)
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 19, 2013, 02:13:47 AM
Just to be clear on this, the game engine for BGII:EE uses cp1252.
The text shown/used in-game uses UTF-8(no BOM).
The original engine uses CP1252 (et al.). BGII: EE uses UTF-8. (It sounds like that is what you meant, but just to be clear.)

1.  setup.tra [everything declared and referenced inside of the .tp2 or related processing files like .tpa .tpp .tph] (which would come encoded differently in different languages and platforms)
2. mymod.tra [everything called by .d and .baf to be placed into the dialog.tlk and displayed ingame] (which need to be set up into at least two copies per language; one {example:english=cp1252} for the older games, and one {ALL LANGUAGES THE SAME=UTF-8(no BOM)} for BGII:EE.
Not everything referenced inside TP2 files should go into setup.tra. Strings for SAY, for example, need to go into mymod.tra, since they are used in the game rather than during the installation. setup.tra should contain only those strings that are printed to the command prompt.

I ask this because it means untangling .tra materials, some of which, like the declared strings for soundset references in the .tp2, etc, show up in the game... so I am trying to figure out if all the .tp2 actions are pulling form the correct .tra.
I don't know about Aran in particular, but I'd expect the vast bulk of your text is dialogue and other types of game content, with two handfuls or so of strings used during the installation. Untangling the strings is not necessarily some huge task. You can likely just create a new tra file and move your dozen (or whatever) setup strings into it. No need to disturb the rest.

Or is it that only the strings displayed in the cmd screen are the ones that need to be declared initially in LANGUAGE, and that everything actually processed pulls from the ALWAYS?
Perhaps I'm confusing things, but technically, the only strings that need to be loaded by LANGUAGE are those that are used while the user is selecting components. In terms of syntax, it's TP2 flags, components and component flags that need LANGUAGE. Actions, patches and values can use strings loaded in other ways.
Title: Re: Question to LAM bgee_language
Post by: cmorgan on November 19, 2013, 06:48:42 AM
Thanks - that clears it up for me - I thought anything declared in a .tp2/.tpa/et al could not be overridden.

So, anything printed to the cmd window at install/deinstall/fail (components, patch_print, print) goes in setup.tra. Anything (even in the .tp2) that is actually being installed to the game (SAY NAME! ~foo~ [soundref] ) etc. goes in other .tras.

Title: Re: Question to LAM bgee_language
Post by: jastey on November 19, 2013, 01:58:03 PM
Cahir: I PMed Zed Nocear at G3 about his way to convert string chars for BGQE - neat little tpa that replaces special characters with the one of the right encoding upon install. If you have other means of contacting him (Polish forum etc.), I'd be happy if you could give him a call. :)
Title: Re: Question to LAM bgee_language
Post by: Cahir on November 19, 2013, 02:27:29 PM
Cahir: I PMed Zed Nocear at G3 about his way to convert string chars for BGQE - neat little tpa that replaces special characters with the one of the right encoding upon install. If you have other means of contacting him (Polish forum etc.), I'd be happy if you could give him a call. :)

jastey: I sent PM to Zed Nocear on Children of Bhaal (http://athkatla.cob-bg.pl/index.php) forum. He visits it from time to time. If he responds to me, I'll let you know.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 19, 2013, 02:30:18 PM
Thank you!
Title: Re: Question to LAM bgee_language
Post by: jastey on November 19, 2013, 05:23:08 PM
I have the problem that OUTER_SPRINT "tra_path" isn't set for bgee, the installer takes the LANGUAGE one whatever I do (cannot LOAD_TRA from %tra_path%, cannot do USING). I'd need a second pair of eyes for this.
http://kerzenburg.baldurs-gate.eu/files/jastey/Setup-AjantisBG1.tp2
Thank you in advance.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 20, 2013, 03:10:07 AM
I have the problem that OUTER_SPRINT "tra_path" isn't set for bgee, the installer takes the LANGUAGE one whatever I do (cannot LOAD_TRA from %tra_path%, cannot do USING). I'd need a second pair of eyes for this.
http://kerzenburg.baldurs-gate.eu/files/jastey/Setup-AjantisBG1.tp2
Thank you in advance.
Your logic is a bit convoluted and excessively nested, and it'd be easier to follow if you made use of INCLUDE [1], but it is correct and the BGEE block is the only block to evaluate on BG: EE and %tra_path% evaluates to ~AjantisBG1/translations/bgee~ afterwards. How is the problem manifesting itself? Have you confirmed that it is using the wrong tra_path (e.g., by PRINTing the value), or are you inferring this from the installed state of the mod? If it is the latter, are you sure your TRA files really are encoded in UTF-8?

1.
This is equivalent, but simplified:
Code: [Select]
  ACTION_IF GAME_IS ~tutu tutu_totsc~ BEGIN
    INCLUDE ~ajantisBG1/somedir/tutu.tpa~
  END ELSE ACTION_IF GAME_IS ~bgt~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bgt.tpa~
  END ELSE ACTION_IF GAME_IS ~bg1 totsc~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bg1.tpa~
    ACTION_IF GAME_IS ~totsc~ BEGIN
      INCLUDE ~ajantisBG1/somedir/bg1_totsc.tpa~
    END ELSE BEGIN
      INCLUDE ~ajantisBG1/somedir/bg1_bg1.tpa~
    END
  END ELSE ACTION_IF GAME_IS ~bgee~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bgee.tpa~
  END

You can also remove the ELSEs entirely (except for the plain-BG1 vs. BG1-TotSC one; you can also move this conditional into bg1.tpa instead of having 3 files for BG1):
Code: [Select]
  ACTION_IF GAME_IS ~tutu tutu_totsc~ BEGIN
    INCLUDE ~ajantisBG1/somedir/tutu.tpa~
  END

  ACTION_IF GAME_IS ~bgt~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bgt.tpa~
  END

  ACTION_IF GAME_IS ~bg1 totsc~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bg1.tpa~
    ACTION_IF GAME_IS ~totsc~ BEGIN
      INCLUDE ~ajantisBG1/somedir/bg1_totsc.tpa~
    END ELSE BEGIN
      INCLUDE ~ajantisBG1/somedir/bg1_bg1.tpa~
    END
  END

  ACTION_IF GAME_IS ~bgee~ BEGIN
    INCLUDE ~ajantisBG1/somedir/bgee.tpa~
  END
Title: Re: Question to LAM bgee_language
Post by: jastey on November 20, 2013, 03:22:21 AM
Thank you for looking into this. Sorry for the state of the tp2, the mod is rather old and I don't want to change too much because it is running. I do use INCLUDE and libs for any newer mods, though!

I will try again tonight with your code. I am sure it took the wrong path because I made textual changes to the ones it should take (before the installation) like "lalala" as mod title but it never showed (that would be the LOAD_TRA) and I double - tripple checked it is UTF-8 without BOM encoded for the other tra files (for the USING) but in the game the dialogues always freezed the game, meaning it took the wrongly encoded ones. It never occurred to me I could print the variable value, but I will do that tonight.

The problem is probably something really stupid like me editing the wrong (tp2) file or something (although I was sure to do the right thing, of course, or I would have changed it), because from the code I really don't see how it wouldn't work. Thank you for your confirmation and for your code suggestion.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 20, 2013, 03:34:31 AM
EDIT: Actually I did not edit the mod title, as this is always taken from the LANGUAGE path, but I edited the "EE install detected..." line, and this edit never showed.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 20, 2013, 04:05:05 AM
EDIT: Actually I did not edit the mod title, as this is always taken from the LANGUAGE path, but I edited the "EE install detected..." line, and this edit never showed.
If you edited @14 in the TRA file in the BGEE subdirectory, the edit would not show when ALWAYS was evaluated, since the BGEE-encoded TRA files are not loaded until afterwards (at the end of the ALWAYS block).
Title: Re: Question to LAM bgee_language
Post by: jastey on November 20, 2013, 04:08:02 AM
Of course. I really could have thought that far by myself. :(
Nevertheless the freezing of the game was real.
I will retry tonight, starting be re-creation of the UTF-8 coded tras.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 20, 2013, 02:34:21 PM
I don't think this %tra_path% overwriting of the AUTO_TRA works. Did anyone confirm this?

The install takes the path specified by AUTO_TRA for USING, no matter what I put in for tra_path.

This is what I did:
My tp2 now looks like this. In the tpas, the first OUTER_SPRINT definition is the ~tra_path~. For bgee, it is  OUTER_SPRINT "tra_path" "AjantisBG1/translations/bgee"

Installation takes the wrong tra files (the one of the path defined by AUTO_TRA).

Quote
ALWAYS

ACTION_IF GAME_IS ~tutu tutu_totsc~ THEN BEGIN
    INCLUDE ~AjantisBG1/install/ajantis_tutu.tpa~

    /* Tell the player it is using Tutu */
    PRINT @1
END

ACTION_IF GAME_IS ~bgt~ THEN BEGIN
    INCLUDE ~AjantisBG1/install/ajantis_bgt.tpa~

     /* Tell the player it is using BGT stuff */
      PRINT @2
END



/* vanilla BG1, general */
ACTION_IF GAME_IS ~bg1 totsc~ THEN BEGIN
PRINT @3

   INCLUDE ~AjantisBG1/install/ajantis_bg1.tpa~

/* BG1 with TotSC installed */
   ACTION_IF GAME_IS ~totsc~ THEN BEGIN
   INCLUDE ~AjantisBG1/install/bg1.tpa~
   END ELSE BEGIN

/* BG1 without TotSC installed */
   INCLUDE ~AjantisBG1/install/totsc.tpa~
   END
END

  ACTION_IF GAME_IS ~bgee~ THEN BEGIN
   INCLUDE ~AjantisBG1/install/ajantis_bgee.tpa~
        PRINT @14
END
END


AUTO_TRA ~AjantisBG1/Translations/%s~


LANGUAGE ~English~
         ~English~
         ~AjantisBG1/Translations/English/Setup-AjantisBG1.tra~

LANGUAGE ~Francais~
         ~French~
         ~AjantisBG1/Translations/French/Setup-AjantisBG1.tra~

LANGUAGE ~Deutsch~
         ~German~
         ~AjantisBG1/Translations/German/Setup-AjantisBG1.tra~


BEGIN @0





  /* STATE.IDS patching to ToB - thanks, Cam, if you read it */
  /* adds custom IsValidForPartyDialogue state */
  APPEND ~STATE.IDS~ ~0x80101FEF CD_STATE_NOTVALID~ UNLESS ~CD_STATE_NOTVALID~


   COMPILE EVALUATE_BUFFER ~AjantisBG1/Friendship/Ajantis_friendship_d.d~
USING ~%tra_path%/%LANGUAGE%/Ajantis_friendship_d.tra~



ACTION_IF GAME_IS ~tutu tutu_totsc bgt bgee~ THEN BEGIN

COMPILE EVALUATE_BUFFER ~AjantisBG1/Friendship/Tutu_Interjections.d~
USING ~%tra_path%/%LANGUAGE%/BG_Interjections.tra~

I changed the AUTO_TRA path to translations/bgee/%LANGUAGE% (tra_path was still translations/bgee/) -> right dialogue file was taken.
I left the AUTO_TRA with translations/%LANGUAGE% -> %tra_path% was ignored and the dialogue file from the translations/%LANGUAGE% was taken.
I changed the tra_path to the "wrong" one and defined AUTO_TRA to the bgee-one: tra_path was ignored, the path specified by AUTO_TRA was used for USING.

I checked all instances in game with a modified text line (UTF-8 file was changed text line, original file was correct text line.).

This is bad. At least I know now that I am not completely crazy.
Title: Re: Question to LAM bgee_language
Post by: cmorgan on November 20, 2013, 06:55:07 PM
I can't exactly see what is going on, Jastey, but I am not seeing a problem in behavior *without* AUTO_TRA. I'll try to set up a test tomorrow if noone gets around to figuring out what is going on.

With the following in my .tp2

Code: [Select]
VERSION @30001

README ~aranw/docs/readme-aranw-%LANGUAGE%.html~ ~aranw/docs/readme-aranw.html~

ALWAYS
INCLUDE ~aranw/lib/regexp.tph~ /* extra regexp vars */

ACTION_IF GAME_IS ~bg2ee~ BEGIN
OUTER_SPRINT tra_version ~c-aranw_utf8nb~
    LOAD_TRA ~aranw\tra\english\c-aranw_utf8nb.tra~
    LOAD_TRA ~aranw\tra\%LANGUAGE%\c-aranw_utf8nb.tra~
END ELSE BEGIN
OUTER_SPRINT tra_version ~c-aranw_cp1252~
    LOAD_TRA ~aranw\tra\english\c-aranw_cp1252.tra~
    LOAD_TRA ~aranw\tra\%LANGUAGE%\c-aranw_cp1252.tra~
END
END

LANGUAGE ~English~
         ~english~
         ~aranw\tra\english\c-aranw_cp1252.tra~ // Holds text displayed in the game, including SAY and such in the TP2 et al
         ~aranw\tra\english\setup_%WEIDU_OS%.tra~ // Holds text displayed during the installation: TP2, et al

BEGIN @30010
REQUIRE_PREDICATE GAME_IS ~bg2 tob bgt bg2ee~ @30002

And the files in C:\Program Files (x86)\BeamDog\Games\00546\aranw\tra\english

c-aranw_cp1252.tra
c-aranw_utf8nb.tra
setup_win32.tra
setup_osx.tra
setup_unix.tra

I can see the original declared file loading, then being overridden:

Code: [Select]
WeiDU v 23600 Log

 C:\Program Files (x86)\BeamDog\Games\00546\setup-aranw.exe
[./chitin.key] loaded, 843510 bytes
[./chitin.key] 185 BIFFs, 59857 resources
[.\weidu.conf] loaded, 17 bytes
[lang\en_us/dialog.tlk] loaded, 13498915 bytes
[lang\en_us/dialog.tlk] 116169 string entries
Using Language [English]
[English] has 2 top-level TRA files
[aranw\tra\english\c-aranw_cp1252.tra] has 13364 translation strings
[aranw\tra\english\setup_win32.tra] has 120 translation strings

Using lang\en_us/dialog.tlk

Would you like to display the readme? [Y]es [N]o
BIFF may be in hard-drive CD-path [./lang/en_us/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [./cache/DATA/AREAS.BIF]
[./DATA/AREAS.BIF] 2545144 bytes, 318 files, 0 tilesets
BIFF may be in hard-drive CD-path [./lang/en_us/DATA/25AREAS.BIF]
BIFF may be in hard-drive CD-path [./cache/DATA/25AREAS.BIF]
[./DATA/25AREAS.BIF] 659172 bytes, 85 files, 0 tilesets
BIFF may be in hard-drive CD-path [./lang/en_us/DATA/NEERA.BIF]
BIFF may be in hard-drive CD-path [./cache/DATA/NEERA.BIF]
[./DATA/NEERA.BIF] 146175540 bytes, 1859 files, 11 tilesets

Install Component [Install Aran Whitehand for SoA and ToB]?
[I]nstall, or [N]ot Install or [Q]uit?
Installing [Install Aran Whitehand for SoA and ToB] [Beta_4]
loading 1 tra file
[aranw\tra\english\c-aranw_utf8nb.tra] has 13364 translation strings
loading 1 tra file
[aranw\tra\english\c-aranw_utf8nb.tra] has 13364 translation strings

* * * * *

WeiDU says that this game has BGII:EE content.

The version of Aran Whitehand being installed is:

Beta_4

Weidu vars on this install:
Language = english
Architecture = x86
Operating System = win32
User Directory = C:\Users\Stephen\Documents/Baldur's Gate II - Enhanced Edition
Save Game Single Player Directory = C:\Users\Stephen\Documents/Baldur's Gate II - Enhanced Edition/save
Save Game Multi Player Directory = C:\Users\Stephen\Documents/Baldur's Gate II - Enhanced Edition/mpsave
Running .exe = C:\Program Files (x86)\BeamDog\Games\00546\setup-aranw.exe

<< snip for space >>

Installing SoA dialog...
Compiling 1 dialogue file ...
[aranw/tra/english/c-aranw_utf8nb.tra] has 13364 translation strings
Processing 1 dialogues/scripts ...
Adding C-ARAN to internal list of available DLGs
Adding C-ARANJ to internal list of available DLGs

<< snip for space >>

Installing ToB dialog...
Compiling 1 dialogue file ...
[aranw/tra/english/c-aranw_utf8nb.tra] has 13364 translation strings
Processing 1 dialogues/scripts ...
Adding C-ARN25A to internal list of available DLGs

BGII:EE detected. Managing journal entries.
Processing quests and journals
[aranw/tra/english/c-aranw_utf8nb.tra] has 13364 translation strings
BIFF may be in hard-drive CD-path [./lang/en_us/DATA/BGEEUI.BIF]
BIFF may be in hard-drive CD-path [./cache/DATA/BGEEUI.BIF]
[./DATA/BGEEUI.BIF] 77256 bytes, 3 files, 0 tilesets
Processing quests and journals
[aranw/tra/english/c-aranw_utf8nb.tra] has 13364 translation strings
[./override/bgee.sql] loaded, 49610 bytes

BGII:EE content is being installed.
Compiling 1 dialogue file ...
[aranw/tra/english/c-aranw_utf8nb.tra] has 13364 translation strings


On the regular BGII - ToB install,

Code: [Select]
WeiDU v 23600 Log

 G:\BGII - SoA\setup-aranw.exe
[./CHITIN.KEY] loaded, 590467 bytes
[./CHITIN.KEY] 182 BIFFs, 41787 resources
[./Autorun.ini] loaded, 1452 bytes
[./baldur.ini] loaded, 3512 bytes
Possible HD/CD Path: [G:\BGII - SoA\]
Possible HD/CD Path: [G:\BGII - SoA\CD1\]
Possible HD/CD Path: [G:\BGII - SoA\CD2\]
Possible HD/CD Path: [G:\BGII - SoA\CD2\]
Possible HD/CD Path: [G:\BGII - SoA\CD3\]
Possible HD/CD Path: [G:\BGII - SoA\CD4\]
Possible HD/CD Path: [G:\BGII - SoA\CD5\]
[./Keymap.ini] loaded, 5615 bytes
[./Mplaynow.ini] loaded, 230 bytes
[./dialog.tlk] loaded, 10666500 bytes
[./dialog.tlk] 87406 string entries
Using Language [English]
[English] has 2 top-level TRA files
[aranw\tra\english\c-aranw_cp1252.tra] has 13364 translation strings
[aranw\tra\english\setup_win32.tra] has 120 translation strings

Would you like to display the readme? [Y]es [N]o
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0  0 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0  3 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 100 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 101 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 102 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 103 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 104 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 106 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 107 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 108 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 109 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 110 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 111 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 112 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 113 Installed
BG2FIXPACK/SETUP-BG2FIXPACK.TP2  0 114 Installed
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD5\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD4\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD3\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD2\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD2\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\CD1\/DATA/AREAS.BIF]
BIFF may be in hard-drive CD-path [G:\BGII - SoA\/DATA/AREAS.BIF]
[G:\BGII - SoA\/DATA/AREAS.BIF] 2438836 bytes, 307 files, 0 tilesets
[./override/ar6111.are] loaded, 2776 bytes

Install Component [Install Aran Whitehand for SoA and ToB]?
[I]nstall, or [N]ot Install or [Q]uit? [./override/ar6111.are] loaded, 2776 bytes

Installing [Install Aran Whitehand for SoA and ToB] [Beta_4]
[./override/ar6111.are] loaded, 2776 bytes
loading 1 tra file
[aranw\tra\english\c-aranw_cp1252.tra] has 13364 translation strings
loading 1 tra file
[aranw\tra\english\c-aranw_cp1252.tra] has 13364 translation strings

* * * * *
[./override/ar6111.are] loaded, 2776 bytes
[./override/ar6111.are] loaded, 2776 bytes

WeiDU says that this game has BGII - ToB content.
[./override/ar6111.are] loaded, 2776 bytes
[./override/ar6111.are] loaded, 2776 bytes

The version of Aran Whitehand being installed is:

Beta_4

Weidu vars on this install:
Language = english
Architecture = x86
Operating System = win32
User Directory = .
Save Game Single Player Directory = ./save
Save Game Multi Player Directory = ./mpsave
Running .exe = G:\BGII - SoA\setup-aranw.exe


* * * * *

<< snip for space>>

Installing SoA dialog...
Compiling 1 dialogue file ...
[aranw/tra/english/c-aranw_cp1252.tra] has 13364 translation strings

<< snip for space>>

Installing ToB dialog...
Compiling 1 dialogue file ...
[aranw/tra/english/c-aranw_cp1252.tra] has 13364 translation strings
etc.

Perhaps check to make sure the @ refs are all correct?
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 21, 2013, 01:12:51 AM
The install takes the path specified by AUTO_TRA for USING, no matter what I put in for tra_path.
Yeah, AUTO_TRA takes precedence over all other ways of loading TRAs. Sorry, I should have made that connection right away. I'll document this behaviour.
You can still use the COPY trick.
Title: Re: Question to LAM bgee_language
Post by: jastey on November 21, 2013, 10:19:29 AM
Ah, so the USING + tra_path can only be used without AUTO_TRA, so for cmorgan's case (one tra file for all texts) this works (good to know!).

I solved the problem for Ajantis using copy, it's just that I wasn't aware the AUTO_TRA overrides any path given for USING.

So, theoretically, I could do COMPILE ~mymod/d/dfile.d~ USING ~otherdfile.tra~ and it would take the file out of the folder specified by AUTO_TRA. ?

I know that (now). I was assuming I could specify a different path for USING - my mistake.
Title: Re: Question to LAM bgee_language
Post by: Cahir on November 24, 2013, 12:03:41 PM
Ok I received reply from Zed Nocear about encoding used by CD Projekt in Polish version of BG1. Apparently it's not cp852 coding, but something different not default for Polish DOS (cp852) encoding.

Here is a table Zed provided. Hope it helps.

number of sequence in Font BAM

          bg1       bg2       cp852
               (windows-1250) (DOS)

Ą         228       164       164
Ć         229       197       143
Ę         230       201       168
Ł         231       162       157
Ó         232       210       224
Ń         233       208       227
Ś         234       139       151
Ź         235       142       141
Ż         236       174       189
ą         237       184       165
ć         238       229*      134
ę         239       233*      169
ł         240       178       136
ń         241       240*      228*
ó         242       242       162
ś         243       155       152
ź         244       158       171
ż         245       190       190

* - is a number in BG2/cp852 coding, which corresponds with coding of different letter in BG1, so you should be careful with it.

I hope it helps.
Title: Re: Question to LAM bgee_language
Post by: Wisp on November 24, 2013, 01:12:19 PM
Apparently that's ISO-IR-179, which is apparently a supplement to ISO/IEC-8859-13, but that's about as much as I can find out about it. CD Projekt sure went off the beaten path with that one.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on December 24, 2013, 06:44:50 AM
Yay, I've got Russian and Polish files for Xan! I'm going to add two new translations - RussianEE and PolishEE, so Auto_TRA can get these.
Am I doing the right thing? Is there an easier way? (worries)
Title: Re: Question to LAM bgee_language
Post by: Kulyok on December 24, 2013, 07:32:45 AM
I, um, wonder: is it possible to make LANGUAGE backwards compatible with another optional pair of arguments, something like

LANGUAGE ~Russian (for normal BG2 and BG2EE)~ ~Russian~ ~Xan/Russian/Setup-Xan.tra~ ~RussianEE~ ~Xan/RussianEE/Setup-Xan.tra~

?

Title: Re: Question to LAM bgee_language
Post by: Wisp on December 25, 2013, 04:53:29 PM
Yay, I've got Russian and Polish files for Xan! I'm going to add two new translations - RussianEE and PolishEE, so Auto_TRA can get these.
Am I doing the right thing? Is there an easier way? (worries)
I'd use a single LANGUAGE entry and use the COPY trick (described here (http://forums.pocketplane.net/index.php/topic,28771.msg333342.html#msg333342) and here (http://forums.pocketplane.net/index.php/topic,28771.msg333334.html#msg333334)).

I, um, wonder: is it possible to make LANGUAGE backwards compatible with another optional pair of arguments
It's (probably) possible, but I'd rather not do it until I've given up on a more transparent solution.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on December 26, 2013, 12:15:49 AM
I fear that I, um, totally do not understand the code you're pointing me to, so I fear it's not very transparent for me personally(and it looks like you're overwriting default translation files with BGEE translation files, which doesn't sit well with me - whichever files I put into my mod folder are sacrosanct and never should be overwritten). I think I'm going to pause until the end of the holidays and then decide.
Title: Re: Question to LAM bgee_language
Post by: Wisp on December 26, 2013, 12:19:17 PM
I think I'm going to pause until the end of the holidays and then decide.
Maybe I'll have explained it better in the promised tutorial by then.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on January 15, 2014, 12:56:59 AM
I'm back from the holidays, and I'm going to re-read this stuff again, but I fear I'm going just to create another language (Polish-BGEE) for the time being.
Title: Re: Question to LAM bgee_language
Post by: jastey on March 29, 2014, 11:32:43 AM
I don't think the %tra_path% workaround works, tbh.

This does not work (it doesn't find the strings for the component's name), the file ~bg1re/tra/english/setup.tra~ exists (working with BG1):

Code: [Select]
BACKUP ~modfolder/install/backup~
AUTHOR ~blabla~ //for bugreports

//MODDER
VERSION ~no. x~

ALWAYS

 ACTION_IF GAME_IS ~bgee~ BEGIN
    OUTER_SPRINT tra_path ~bg1re/tra/utf8~
  END ELSE BEGIN
    OUTER_SPRINT tra_path ~bg1re/tra~
END

LANGUAGE ~English~
         ~english~
         ~%tra_path%/english/setup.tra~ ~%tra_path%/english/setup_game.tra~

BEGIN @5


If you spot the evident, please let me know. I was hoping to spare myself the copying of tra files.
Title: Re: Question to LAM bgee_language
Post by: Wisp on March 30, 2014, 04:49:38 AM
ALWAYS is executed after LANGUAGE (it is the first thing it does after BEGIN <component>), so you can't use %tra_path% in LANGUAGE like that, no.
Title: Re: Question to LAM bgee_language
Post by: jastey on March 30, 2014, 06:56:04 AM
Ah, thank you.
What I did now is the opposite: I am copying files over for both formatted tra-files. (My initially intention was to prevent one set of files to be temporarily overwritten while the mod is installed, as I tend to edit my texts while e.g. testing.). So, now AUTO_TRA points to a folder tra/autotra/%s - this is the one the current set of tra-files will be copied to depending on the game; and in addition I have the classic set in tra/classic/%LANGUAGE% and the utf-8 formatted tras in tra/utf8/%LANGUAGE%.

The only drawback is that there needs to be the setup.tra with the mod's name in the tra/autotra/%LANGUAGE% folder, or the mod won't have a name.

This looks like this:

Code: [Select]
ALWAYS
[lots of OUTER_SPRINT definition and patching of area scripts etc.pp ]

/* Handling of correctlky formatted tra-files!! ---------------------- */

   ACTION_IF GAME_IS ~bgee~ THEN BEGIN
   /*
   * Copies UTF8-encoded tra files over the normal ones, for use with AUTO_TRA
   */
  DEFINE_ACTION_FUNCTION autotra_workaround BEGIN
    COPY ~bg1re/tra/utf8/%LANGUAGE%~  ~bg1re/tra/autotra/%LANGUAGE%~
  END

  LAF autotra_workaround END


    //TRAs declared in LANGUAGE must be reloaded
  LOAD_TRA "bg1re/Tra/autotra/english/setup.tra"
LOAD_TRA "bg1re/Tra/autotra/english/setup_game.tra"
  LOAD_TRA "bg1re/Tra/autotra/%LANGUAGE%/setup.tra"
LOAD_TRA "bg1re/Tra/autotra/%LANGUAGE%/setup_game.tra"

  END ELSE BEGIN

  /*
   * Copies UTF8-encoded tra files over the normal ones, for use with AUTO_TRA
   */
  DEFINE_ACTION_FUNCTION autotra_workaround BEGIN
    COPY ~bg1re/Tra/classic/%LANGUAGE%~  ~bg1re/Tra/autotra/%LANGUAGE%~
  END

  LAF autotra_workaround END


    //TRAs declared in LANGUAGE must be reloaded
  LOAD_TRA "bg1re/Tra/autotra/english/setup.tra"
LOAD_TRA "bg1re/Tra/autotra/english/setup_game.tra"
  LOAD_TRA "bg1re/Tra/autotra/%LANGUAGE%/setup.tra"
LOAD_TRA "bg1re/Tra/autotra/%LANGUAGE%/setup_game.tra"

  END



END //ALWAYS




/* --------------------------------------------- */





AUTO_TRA ~bg1re/tra/autotra/%s~


LANGUAGE ~English~
         ~english~
         ~bg1re/tra/autotra/english/setup.tra~ ~bg1re/tra/autotra/english/setup_game.tra~



BEGIN @5

(...)



Title: Re: Question to LAM bgee_language
Post by: The Imp on March 30, 2014, 10:26:44 AM
...
I think I saw a tutorial where the language files(actually the single .tra file) was replaced if they were needed to be replaced after the language was declared ... that way you don't have to do the above.
Title: Re: Question to LAM bgee_language
Post by: jastey on March 30, 2014, 10:47:04 AM
I have multiple tra files in all my mods, so any solution dealing with only one for the whole mod won't work in my case. And my version also replaces the tra files after the language declaration, but until then, there needs to be at least the setup.tra (with the mod's name in it) to be in the folder specified by the AUTO_TRA path. Or maybe I didn't understand what you where pointing at.

A tutorial of this sort would interest me, do you have a link?
Title: Re: Question to LAM bgee_language
Post by: Isaya on March 31, 2014, 04:55:42 PM
I think the order of ALWAYS and LANGUAGE blocks in the tp2 doesn't matter. From what I understand, WeiDU will always use the original setup.tra file to display at least the first component name before the ALWAYS block is executed. I believe this means the LOAD is always required if you change the setup.tra content.

jastey, I saw you used both the chosen language and english in your set of LOAD commands. You didn't show the language block for another language but I assume it references both english and the languages tra files.
When used that way, WeiDU handles precedence according to the order of the file names in the LANGUAGE command when it comes to reading the various files for a string.
What I wondering is whether loading them afterwards is maintaining the same precedence order as in the LANGUAGE command or if using LOAD is breaking it. Since the description of LOAD says
[/QUOTE]Loads all traName files (variable substitution is done, the file cannot be inlined) to be used in the following tp2 actions, as if you had declared them in LANGUAGE.[/QUOTE]
I'm wondering if the files specified in LOAD come as additions to the list of files to browse and if the order of the LOAD commands will be significant to the browse order.
Title: Re: Question to LAM bgee_language
Post by: jastey on March 31, 2014, 11:48:43 PM
I am not sure i understand your question. As I see it, LOAD does load all tras, overwriting any string numbers that are doubled with the one loaded last. That's how it is done with the setup-bgee.tra CamDawg suggested in his tutorial, where the "setu-bgee.tra" only contains the changed strings e.g. for item descriptions. So, I would assume order is only important for the case of which string you want to show last, and that's the one of the language in question, as English is only the backup.
Title: Re: Question to LAM bgee_language
Post by: Isaya on April 01, 2014, 05:01:39 PM
Oh, I see. Thanks for the explanation.
For some reason I assumed WeiDU was maintaining a list of strings for each file and had to browse them in proper order. But it's way easier to handle if it just maintains a single list of strings and overwrites those that appear in subsequent files. So it's probably what's done.

That means the only thing to ensure when you LOAD files in the ALWAYS block is to handle files in the same order as in LANGUAGE command.
Title: Re: Question to LAM bgee_language
Post by: jastey on April 01, 2014, 11:34:20 PM
Again, it is necessary to match the order of AUTO_TRA in the LANGUAGE block. Because all tras I load there have different string numbers, so I could load them in any order.

But it is important, if loading another language as backup, e.g. English as standard, to load the non-standard (non-backup) language last.
Title: Re: Question to LAM bgee_language
Post by: jastey on April 14, 2014, 01:43:20 AM
(I edited my above code. Paths for the LOAD missed the autotra/ for some instances.)
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 15, 2014, 06:01:22 AM
So... suppose that I would like to update one old mod. Is there an relative "easy" approach for weidu-newbie to archive auto-replace ANSI files with UTF-8 files for BG:EE? Does solution hat are present in BG1NPC Prerelese and Worldmap 10 are currently the "best" solution for dealing with auto-translation ?

Or I miss something and there are plans to handle this by weidu completely ?
Title: Re: Question to LAM bgee_language
Post by: Kulyok on July 15, 2014, 06:25:50 AM
Very good question. I'd like to know, too, please. (Is BG1 NPC approach fully correct and bug/trouble-free in this department, btw?)
Title: Re: Question to LAM bgee_language
Post by: Wisp on July 15, 2014, 01:58:02 PM
I may or may not have something coming down the line. In the mean time, or in the event I don't, I would suggest the methods outlined and/or explained in this thread as the least crummy way of dealing with compatibility with both the regular BGs as well as the EEs. I can't vouch for the implementation in BG1 NPC, as I haven't looked at it. The implementation used in Kelsey (as well as in the Worldmap mod) works. IIRC, Edwin Romance uses an approach that is compatible with AUTO_TRA (something Kelsey does not need).

Edited for 90+ % less commitment.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on July 16, 2014, 02:17:29 AM
Forgive me for keeping pestering you, but here's why I want to know so much whether to adapt BG1 NPC's way or not:

BG1 NPC does not use two sets of files. I looked up Edwin's Romance, and it uses /normal tra files and /utf-8 tra files in two different subfolders. BG1 NPC still utilizes only old files, and includes its own utf-8 converter... if I understand correctly. It doesn't seem to take up much space, either. So I'm wondering - maybe it's so good it could become a part of Weidu and a part of a natural mod-converting process for BG2EE translations?
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 16, 2014, 06:45:02 AM
BG1 NPC does not use two sets of files... BG1 NPC still utilizes only old files, and includes its own utf-8 converter... if I understand correctly.
Exactly. Hawing to deal with two set of files is a nightmare for translators, even if it is only "run my Copy-AND-Convert-ANSI-UTF8.bat file before he will send files to include in official mod archive. And then he just fix missing ~ and forgot to make copy and conversion ...
Title: Re: Question to LAM bgee_language
Post by: Wisp on July 16, 2014, 02:08:11 PM
I can implement something along the lines of a HANDLE_* function for charset conversions. I have my reservations, but I suppose they are inconsequential, and don't let perfect be the enemy of good and all that. There would probably be a list of constraints you would need to operate by, because I can't do magic and this isn't like converting Ogg V. Since it'd be a function, you could use it without having to wait for a new WeiDU.
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 17, 2014, 07:48:20 AM
But even if you provide HANDLE_* function, it would still require to update mod so in this case we can follow BG1NPC and you don't have to code anything. When I asked about "handle this by weidu completely" what I mean is that weidu will convert all ANSI files to UTF-8 on the fly when the game will be XX:EE and without any modification to original mod.

If this this is to much to ask then please forgive me, but you know ... it take years to update mods. And because there are no ANS-UTF8 conversion in 90% of XX:EE mods, it makes BWS completely useless for any non-English installation on XX:EE games :/
Title: Re: Question to LAM bgee_language
Post by: Wisp on July 17, 2014, 12:08:45 PM
Okay, your choice.

When I asked about "handle this by weidu completely" what I mean is that weidu will convert all ANSI files to UTF-8 on the fly when the game will be XX:EE and without any modification to original mod.
I have been looking into adding a more transparent method of dealing with this. I expect I'll present what findings I have at some point (spoiler: the outlook's bleak. Mostly due to licencing.) But mods would still have to be updated. You have to know which encoding to convert from (and no, "ANSI" is not an encoding) and this information cannot be known or inferred from mods in their current state. It has to be added.
Title: Re: Question to LAM bgee_language
Post by: Kaeloree on July 17, 2014, 04:42:07 PM
A HANDLE_* function would be awesome.
Title: Re: Question to LAM bgee_language
Post by: Kulyok on July 18, 2014, 12:30:38 AM
Wish we modders could buy ourselves a license(if it's affordable financially), but given our grey status copyright-wise, that probably isn't going to work. Thank you so much for looking into this!
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 18, 2014, 03:42:55 AM
I have no doubt that HANDLE_* funcion would be useful. But I have question:

It should be easy for WeiDU to read folder name of user selected translation. With this information how about this approach:

1. Setup-oldmod.exe is executed, user selected "Polish" translation,
2. WeiDU detects XX:EE and also read folder with translated files are. Path contain "polish".
3. WeiDU reads "Conversion map:"

IF %tra_folder_name% contains "polish"/"polski" > set encoding to Windows-1250
IF %tra_folder_name% contains "German" > set encoding to ...

and use some sort of open-source character converter or build-in function to convert content of those files on the fly (or eventually make converted copy of them with "_utf8" appendix) and use them for current installation.

This way an old mod can be successfully installed using all translations. Is this approach is possible ?
Title: Re: Question to LAM bgee_language
Post by: Wisp on July 18, 2014, 02:49:12 PM
I expect I'll present what findings I have at some point (spoiler: the outlook's bleak. Mostly due to licencing.)
Okay, that's probably enough suspense.
The thing is, I can't do this because of setup-mymod.exe. WeiDU is licenced under modified GPL terms that permit you to distribute unaltered binaries without any further conditions. There appears to be two possible conversion libraries available (libiconv and Camomile) and they are both licenced under regular LGPL terms, which do not permit the distribution of binaries without additional conditions. I can't very well say "From now on, additional terms and conditions apply to the distribution of WeiDU that did not apply before. I abide by them. Whether you do the same is your business *wink wink, nudge nudge*". Hence, I can't include conversion abilities in WeiDU due to setup-mymod.exe.

However, at the risk of giving things away, perhaps I should mention that I am working on something that will hopefully make setup-mymod.exe obsolete. Once you've gathered around to sing Ding-Dong! The shell menus are dead I'll be able to revisit this problem and most likely implement a nice solution. Until then, I guess I'll try to make the HANDLE_* function user-friendly.

1. Setup-oldmod.exe is executed, user selected "Polish" translation,
2. WeiDU detects XX:EE and also read folder with translated files are. Path contain "polish".
3. WeiDU reads "Conversion map:"

It is possible, but there are a few problems.
1. The name of the language directory is completely arbitrary. It will probably work for most cases, but when it doesn't, it'll get it wrong.
2. There are no guarantees about the encodings (the actual encoding may be different from what we think it is) or the sane-ness of the implementation (e.g., some implementations mix different encodings in the same file.) and WeiDU, by itself, can't know if this is a TRA file that contains strings that should be added to the game, or if it is a TRA file that contains strings that are displayed during the installation. The latter should be converted, the former should not be and if the two kinds of strings exist in the same file, all bets are off.
3. I hate automatic systems that can't be reasonably relied upon to get things right. (In this particular case, the possible treats include unexplained crashes and other problems because no one vetted the compatibility and patch after patch after patch to try to get the automatic-ness right.)
Title: Re: Question to LAM bgee_language
Post by: Wisp on July 19, 2014, 10:32:10 AM
Not that you are likely to miss it anyway, but HANDLE_CHARSETS (as it came to be called) is expounded upon here (http://forums.pocketplane.net/index.php/topic,29193.0.html).
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 19, 2014, 11:40:29 AM
I'm dying today from my first ever allergic reaction. I will reply when I feel better.
Title: Re: Question to LAM bgee_language
Post by: AL|EN on July 21, 2014, 05:22:00 AM
I saw HANDLE_CHARSETS and it's really handy. But you have to face reality that most of the old mods will never be updated due to "you can't host fixed mod archive without signed permission from the author that is absolutely beyond any way of reaching".

Even if I feel that it's a lost case already, I'll show you what it looks like from my perspective. Thats an non-english user who install mods via BWS.

First is the license problems. It's exaggeration of the problem. Period. License for weidu is one thing and license terms for (lib)iconv is another thing and
zipping two tools and two license files into one zip file should be no problem for anyone. especially that is so valuable for this community.

Second:

1. Yes, It can fail in 0.01% cases. This particular mod needs to be updated anyway and renaming one directory is very easy. For the rest of 99.99% it would be fine.
2. Old and not updated mods have ANSI+local codepage character set. I've just checked for eg: Chloe and it's install perfectly fine on BG2:EE (http://prntscr.com/44spr3) when you just convert polish files to UTF-8 using Windows-1250 character set. Same goes for almost all old mods. If there will be a problem, then the mod should get an update. All mods that can be converted automatically deserve it.
3. You can never avoid all of the bugs/crashes. No need to update the old mod is for me something worth every effort and that "something can go wrong sometimes"

Since you already put huge amount of work to wiedu I won't nag you about this anymore.
Title: Re: Question to LAM bgee_language
Post by: Kaeloree on July 21, 2014, 05:52:48 AM
IE modding is already a legal grey area. It does pay to be careful in our case.
Title: Re: Question to LAM bgee_language
Post by: jcompton on July 21, 2014, 08:33:29 AM
I saw HANDLE_CHARSETS and it's really handy. But you have to face reality that most of the old mods will never be updated due to "you can't host fixed mod archive without signed permission from the author that is absolutely beyond any way of reaching".

It's probably not as bad as you think, actually. Although numerically you may be right that a maintainer cannot be feasibly reached for "most" mods (as there are many, many mods out there), I submit that a responsible party can be reached to bless a straight-up EE fix for the vast majority of mods that people actually play.

People can contact me to submit EE fixups of all PPG and WeiDU.org mods, for example. (And I believe at this point that basically everything on PPG which anybody plays has been fixed.) I'd be surprised if other sites didn't have a similar policy among their remaining overlords and maintainers.