WeiDU 247 seems to have broken Generalized Biffing installation

[Gwendolyne_FP]

Hi, wisp.

Everything is said in the title and I can't find in the change-log which WeiDU 247 new feature may cause this. Could you have a look at this please ?

[Wisp]

I am unable to reproduce this problem. Steps were: start from original games, install fixpack, install BGT (I chose not to biff), install gen_biff with 247: biff everything. The game works the same with or without gen_biff. As you have observed, none of the changes for 247 should have impacted MAKE_BIFF (which is not to say none did, and there are also changes in the compile environment).

[Gwendolyne_FP]

Thanks for looking at it.

I also could not reproduce it with a vanilla ToB + ToBEx + BG2 Fixpack game.

[The Imp]

Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.

[Wisp]

Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.

This could likely be correct. Due to changes in OCaml, WeiDU 247 is doing case manipulations in US-ASCII, instead of in Latin-1 (which 246 used), which mangles the file names of many of IA's files. The force is telling me this is going to suuuuck.

[DavidW]

I felt a great disturbance in the force, as if millions of filenames suddenly cried out in terror and were suddenly mangled.

[Gwendolyne_FP]

Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.

This could likely be correct. Due to changes in OCaml, WeiDU 247 is doing case manipulations in US-ASCII, instead of in Latin-1 (which 246 used), which mangles the file names of many of IA's files. The force is telling me this is going to suuuuck.

That means I know what I have to do: hasten AI update.

[jastey]

Guys, you can't do this, one can't browse secretely in the office while having to LOL while reading!

[Miloch]

That means I know what I have to do: hasten AI update.

Once you hasten the AI update, can you use that AI to hasten the IA update?

(Yes, I know I'm necroing; it's part of the irony.)

[Taimon_]

This keeps coming up again and probably needs some fix/workaround.

If I understand this correctly, the OCaml function "String.uppercase" is somehow mangling non-ASCII chars in recent OCaml releases.
I wasn't able to find the change in OCaml changelog, but "String.uppercase" seems to be deprecated and will be removed anyway.
One way would be to replace it with "String.uppercase_ascii". If we only want to uppercase 'a'-'z' this should be sufficient.
However, if some mods use chars from CP1252 that can be uppercased (like 'ä' -> 'Ä') and rely on this transformation, then we probably need a different solution.
(Write a replacement function for "String.uppercase" that behaves similar to old OCaml releases?)

For testing, I assume we need to patch line 355 in tpaction.ml (TP_Biff):

Code: [Select]

find_list := (String.uppercase (directory ^ "/" ^ next)) :: !find_list;
and line 54 in biff.ml:

Code: [Select]

[ (size,file,String.uppercase a,String.uppercase b,tau) ]

Maybe someone can give this a try. Getting my build setup running again would probably take a while.

(On a sidenote: For some reason, the "password forgotten" function of the forum software does not seem to work. Or I haven't been patient enough and it still needs more time to send the mail.)

[Salk]

Always nice to see you, Taimon.

Let's hope Wisp take a look at your suggestion.

[Wisp]

I had assumed this problem was due to case manipulations, but that does not seem to be so. String.*case works as expected on Latin-1 input and String.*case_ascii appears to bounds-check itself to have no effect on non-ASCII input. Since neither takes a Latin-1 codepoint and turns it into a garbled 2-byte sequence, the problem would seem to lie elsewhere, but I have not yet looked into it further.

[Taimon_]

I think it is because OCaml added Unicode support for Windows in 4.06.

That means, Sys.readdir is probably returning UTF-8 encoded strings now, which WeiDU is not prepared to deal with appropriately.

Haven't tried it yet, but I suppose the easiest fix would be to use WINDOWS_UNICODE=0 (or set WINDOWS_UNICODE_MODE to "ansi" in opam) when building OCaml to enable legacy mode.

[Taimon_]

I gave this a try and it looks mostly okay for me:
The BashFor-Loop in Generalized Biffing seems to not mess up the filenames anymore.
But somehow, when I PRINT the filenames, there is still something wrong. (but in a different way than before)

I suggest using at least OCaml 4.08 due to a codepage bug.
If you are using opam, I took the file from ocaml-variants.4.08.1+mingw64 (mingw repo) and added a build-env section (after setenv, before build):

Code: [Select]

build-env: [
  [DEFAULT_STRING = "unsafe"]
  [WINDOWS_UNICODE_MODE = "ansi"]
]
(Please, don't ask how long it took me to figure out how to build a custom compiler ... this was a nightmare.)

[Wisp]

Thanks for looking into this. The XP-compatible build (v249) should, in that case, not have this problem, since it was compiled with OCaml 4.05.

[Taimon]

The XP-compatible build (v249) should, in that case, not have this problem, since it was compiled with OCaml 4.05.

Yes, I don't see the problem with that version.
It behaves very similar to the version I built yesterday. (no mangling of filenames when looping, same PRINT issue)

[AL|EN]

@Taimon Hey, can you share details on how did you build the custom compiler?

[Taimon]

@Taimon Hey, can you share details on how did you build the custom compiler?

Sure, if I can still remember the necessary steps. Some details might be missing.

First I installed cygwin with the necessary packages. I think opam was also in cygwin, otherwise I'm not sure where I got it from.
Then I added the mingw repo as default to opam (opam repository add).
Next step was playing around with the opam switch concept. (Something like a toolchain container.)
To build the custom compiler, I ended up copying the ming repo structure into a local directory, adding it to opam as local repository and then modifying the opam file in the ocaml-variants package I wanted to base my new compiler on. (I think I actually created a new folder in ocaml-variants.)
This new variant can then be used with an opam switch, which in turn is then used to build WeiDU.
(I did some fiddling around with an empty switch and adding the ocaml-variant afterwards, but I don't think this is necessary.)

In the end, it doesn't look so difficult, but the documentation was not very clear on all of this, so I ended up playing around a lot.
Funny thing is, I actually ran into some of your posts on the OCaml board while trying to figure out how to build this.
Didn't look like they were really able to help you.

[AL|EN]

Thanks, I was missing the 'local directory as local opam repository' part.

P.S. Yes, I did try to ask the community but the whole process is a mess. There was one guy who help me try to build OCaml 4.07+ for Windows XP including custom patching etc. It was a nightmare and ultimately we failed

[WanderingScholar]

I know this post is old but is there any update on this? I'm still downgrading WeiDU to v246 every time I need to use gen biffing.

Hopefully it's fixed in v250, if that's released.

[jastey]

Yes, a weidu version that really solves this problem would be really cool. *waves pompons* I know you can do it! <3

[Wisp]

The "legacy" build already does not have this problem. Feel free to use that in the meantime.

[jastey]

That sounds cool! Would that be the "WeiDU-Windows-249-x86-legacy.zip" download?