Author Topic: WeiDU 247 seems to have broken Generalized Biffing installation  (Read 3060 times)

Offline Gwendolyne_FP

  • Planewalker
  • *****
  • Posts: 66
Hi, wisp.

Everything is said in the title and I can't find in the change-log which WeiDU 247 new feature may cause this. Could you have a look at this please ?
« Last Edit: October 14, 2020, 07:07:42 PM by Gwendolyne_FP »

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1169
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #1 on: October 15, 2020, 02:51:44 PM »
I am unable to reproduce this problem. Steps were: start from original games, install fixpack, install BGT (I chose not to biff), install gen_biff with 247: biff everything. The game works the same with or without gen_biff. As you have observed, none of the changes for 247 should have impacted MAKE_BIFF (which is not to say none did, and there are also changes in the compile environment).

Offline Gwendolyne_FP

  • Planewalker
  • *****
  • Posts: 66
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #2 on: October 15, 2020, 03:17:44 PM »
Thanks for looking at it.

I also could not reproduce it with a vanilla ToB + ToBEx + BG2 Fixpack game.

Offline The Imp

  • Planewalker
  • *****
  • Posts: 288
  • Gender: Male
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #3 on: October 17, 2020, 02:16:04 PM »
Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1169
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #4 on: October 17, 2020, 08:24:47 PM »
Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.
This could likely be correct. Due to changes in OCaml, WeiDU 247 is doing case manipulations in US-ASCII, instead of in Latin-1 (which 246 used), which mangles the file names of many of IA's files. The force is telling me this is going to suuuuck.

Offline DavidW

  • Planewalker
  • *****
  • Posts: 314
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #5 on: October 17, 2020, 10:57:02 PM »
I felt a great disturbance in the force, as if millions of filenames suddenly cried out in terror and were suddenly mangled.

Offline Gwendolyne_FP

  • Planewalker
  • *****
  • Posts: 66
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #6 on: October 18, 2020, 07:47:12 AM »
Like I said in the SHS thread, it's very likely that this all involves a non-Latin character usage during the Gen Biff's install, while Infinity Animations was installed during Latin character usage, as then biffing will likely just not find the resources marked in the game, which leads to having animation failure and thus crash.
This could likely be correct. Due to changes in OCaml, WeiDU 247 is doing case manipulations in US-ASCII, instead of in Latin-1 (which 246 used), which mangles the file names of many of IA's files. The force is telling me this is going to suuuuck.

That means I know what I have to do: hasten AI update. ;)

Offline jastey

  • Global Moderator
  • Planewalker
  • *****
  • Posts: 1518
  • Gender: Female
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #7 on: November 01, 2020, 01:49:36 AM »
Guys, you can't do this, one can't browse secretely in the office while having to LOL while reading!

Offline Miloch

  • Barbarian
  • Planewalker
  • *****
  • Posts: 1032
  • Gender: Male
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #8 on: October 10, 2021, 05:27:47 PM »
That means I know what I have to do: hasten AI update.
Once you hasten the AI update, can you use that AI to hasten the IA update?

(Yes, I know I'm necroing; it's part of the irony.)

Taimon_

  • Guest
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #9 on: September 27, 2022, 03:03:01 AM »
This keeps coming up again and probably needs some fix/workaround.

If I understand this correctly, the OCaml function "String.uppercase" is somehow mangling non-ASCII chars in recent OCaml releases.
I wasn't able to find the change in OCaml changelog, but "String.uppercase" seems to be deprecated and will be removed anyway.
One way would be to replace it with "String.uppercase_ascii". If we only want to uppercase 'a'-'z' this should be sufficient.
However, if some mods use chars from CP1252 that can be uppercased (like '' -> '') and rely on this transformation, then we probably need a different solution.
(Write a replacement function for "String.uppercase" that behaves similar to old OCaml releases?)

For testing, I assume we need to patch line 355 in tpaction.ml (TP_Biff):
Code: [Select]
find_list := (String.uppercase (directory ^ "/" ^ next)) :: !find_list;
and line 54 in biff.ml:
Code: [Select]
[ (size,file,String.uppercase a,String.uppercase b,tau) ]

Maybe someone can give this a try. Getting my build setup running again would probably take a while.

(On a sidenote: For some reason, the "password forgotten" function of the forum software does not seem to work. Or I haven't been patient enough and it still needs more time to send the mail.)

Offline Salk

  • Planewalker
  • *****
  • Posts: 873
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #10 on: October 05, 2022, 02:07:11 PM »
Always nice to see you, Taimon.

Let's hope Wisp take a look at your suggestion.

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1169
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #11 on: October 29, 2022, 05:27:00 PM »
I had assumed this problem was due to case manipulations, but that does not seem to be so. String.*case works as expected on Latin-1 input and String.*case_ascii appears to bounds-check itself to have no effect on non-ASCII input. Since neither takes a Latin-1 codepoint and turns it into a garbled 2-byte sequence, the problem would seem to lie elsewhere, but I have not yet looked into it further.

Taimon_

  • Guest
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #12 on: November 19, 2022, 04:10:02 AM »
I think it is because OCaml added Unicode support for Windows in 4.06.

That means, Sys.readdir is probably returning UTF-8 encoded strings now, which WeiDU is not prepared to deal with appropriately.

Haven't tried it yet, but I suppose the easiest fix would be to use WINDOWS_UNICODE=0 (or set WINDOWS_UNICODE_MODE to "ansi" in opam) when building OCaml to enable legacy mode.

Taimon_

  • Guest
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #13 on: November 19, 2022, 02:12:57 PM »
I gave this a try and it looks mostly okay for me:
The BashFor-Loop in Generalized Biffing seems to not mess up the filenames anymore.
But somehow, when I PRINT the filenames, there is still something wrong. (but in a different way than before)

I suggest using at least OCaml 4.08 due to a codepage bug.
If you are using opam, I took the file from ocaml-variants.4.08.1+mingw64 (mingw repo) and added a build-env section (after setenv, before build):
Code: [Select]
build-env: [
  [DEFAULT_STRING = "unsafe"]
  [WINDOWS_UNICODE_MODE = "ansi"]
]
(Please, don't ask how long it took me to figure out how to build a custom compiler ... this was a nightmare.)

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1169
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #14 on: November 20, 2022, 05:14:51 AM »
Thanks for looking into this. The XP-compatible build (v249) should, in that case, not have this problem, since it was compiled with OCaml 4.05.

Offline Taimon

  • Planewalker
  • *****
  • Posts: 328
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #15 on: November 20, 2022, 08:41:54 AM »
The XP-compatible build (v249) should, in that case, not have this problem, since it was compiled with OCaml 4.05.
Yes, I don't see the problem with that version.
It behaves very similar to the version I built yesterday. (no mangling of filenames when looping, same PRINT issue)

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 390
  • Gender: Male
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #16 on: November 27, 2022, 12:47:41 PM »
@Taimon Hey, can you share details on how did you build the custom compiler?
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline Taimon

  • Planewalker
  • *****
  • Posts: 328
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #17 on: November 28, 2022, 09:29:39 AM »
@Taimon Hey, can you share details on how did you build the custom compiler?
Sure, if I can still remember the necessary steps. Some details might be missing.

First I installed cygwin with the necessary packages. I think opam was also in cygwin, otherwise I'm not sure where I got it from.
Then I added the mingw repo as default to opam (opam repository add).
Next step was playing around with the opam switch concept. (Something like a toolchain container.)
To build the custom compiler, I ended up copying the ming repo structure into a local directory, adding it to opam as local repository and then modifying the opam file in the ocaml-variants package I wanted to base my new compiler on. (I think I actually created a new folder in ocaml-variants.)
This new variant can then be used with an opam switch, which in turn is then used to build WeiDU.
(I did some fiddling around with an empty switch and adding the ocaml-variant afterwards, but I don't think this is necessary.)

In the end, it doesn't look so difficult, but the documentation was not very clear on all of this, so I ended up playing around a lot.
Funny thing is, I actually ran into some of your posts on the OCaml board while trying to figure out how to build this. :)
Didn't look like they were really able to help you.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 390
  • Gender: Male
Re: WeiDU 247 seems to have broken Generalized Biffing installation
« Reply #18 on: November 29, 2022, 02:39:50 AM »
Thanks, I was missing the 'local directory as local opam repository' part.

P.S. Yes, I did try to ask the community but the whole process is a mess. There was one guy who help me try to build OCaml 4.07+ for Windows XP including custom patching etc. It was a nightmare and ultimately we failed :P
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: