Post reply

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.
Name:
Email:
Subject:
Message icon:

Verification:
Type the letters shown in the picture
Listen to the letters / Request another image

Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?:

shortcuts: hit alt+s to submit/post or alt+p to preview


Topic Summary

Posted by: FredSRichardson
« on: May 23, 2006, 03:39:24 PM »

Patch for cleanup is attached. Assumes your latest cbif patch is already applied.

Thank you for this! :)  You actually don't need to test for the existance of the cache file:
Code: [Select]
+             if not (file_exists cache_file) then Queue.add cache_file cbifs_to_rem;I added "cache" to the CD search list so that this code isn't reached if one already exists (I tested this out since I had a few cache files left over by the IE).

-Fred
Posted by: the bigg
« on: May 23, 2006, 01:36:09 PM »

Patch for cleanup is attached. Assumes your latest cbif patch is already applied.
Posted by: the bigg
« on: May 23, 2006, 01:16:33 PM »

Patch applied.
Posted by: FredSRichardson
« on: May 23, 2006, 12:43:41 PM »

Okay, this should work a bit better.  If you have IWD1, you can test it with:
Code: [Select]
> ./weidu.asm.exe --biff-get AR3401.WED
[C:\cygwin\usr\src\WeiDU-192.tmp\weidu.asm.exe] WeiDU version 192
[C:\Program Files\Black Isle\Icewind Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
[C:\Program Files\Black Isle\Icewind Dale/dialog.tlk] 34502 string entries
[C:\cygwin\usr\src\WeiDU-192.tmp\weidu.asm.exe] Using scripting style "IWD1"
[C:\Program Files\Black Isle\Icewind Dale\CD2\/data/AR3401.cbf] decompressed bif file 2095712 bytes
[C:\Program Files\Black Isle\Icewind Dale/cache/data/AR3401.bif] 2095712 bytes, 6 files, 1 tilesets
[./AR3401.WED] created from [C:\Program Files\Black Isle\Icewind Dale/data/AR3401.bif]
>

You can see that it reports decompressing the CBF files to the Cache directory.  Note that it will only do this on an operation that grabs resources from CBF files.  I noticed that 'weidu --biff <bif-file>' is a bit broken.  It doesn't report an error if it can't find <bif-file> in the game keys, and <bif-file> must match the key entry exactly.  Here's an xample:
Code: [Select]
> ./weidu.asm.exe --biff 'data\AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
1.TIS at index 1
1.WED at index 0
T.BMP at index 1
M.BMP at index 2
R.BMP at index 3
1.MOS at index 4
A.WAV at index 5
> ./weidu.asm.exe --biff 'data/AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
> ./weidu.asm.exe --biff 'AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
>

Fixing this is a bit of a pain.  I only cared because I was looking for examples to test this new patch with.

Here's that patch:

Code: [Select]
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/Makefile ./WeiDU-192.tmp/Makefile
--- ./WeiDU-192.orig/Makefile 2006-04-24 15:14:58.000000000 -0400
+++ ./WeiDU-192/Makefile 2006-05-23 13:34:23.950168600 -0400
@@ -135,7 +135,7 @@
         baflexer bafparser \
         baflexer_old bafparser_old \
         diff tp dlexer dparser \
-        automate kit
+        automate kit cbif
 ifdef ITEMLIST
 WEIDU_BASE_MODULES  += pretty itemlist
 endif
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/biff.ml ./WeiDU-192/src/biff.ml
--- ./WeiDU-192.orig/src/biff.ml 2006-03-30 16:42:25.000000000 -0500
+++ ./WeiDU-192/src/biff.ml 2006-05-23 13:26:48.106418600 -0400
@@ -9,6 +9,7 @@
 (* Infinity Engine [BIF] *)
 open Util
 open Key
+open Cbif
 
 type biff_file = {
   res_loc       : int ;
@@ -204,10 +205,6 @@
   }
 end
 
-(* comment this out if you don't have zlib *)
-external uncompress : string -> pos:int -> clen:int -> ulen: int -> string
-  = "mlgz_uncompress"
-
 (* reads 'size' bytes that would start at location 'start' in this BIFF
  * if it were not compressed! *)
 let read_compressed_biff_internal fd filename start size chunk_fun =
@@ -215,7 +212,7 @@
   let unc_offset = ref 0 in
 
   (* buffer holds the uncompressed bytes [start_unc,end_unc]  *)
-  let result = Buffer.create size in
+  let (*result*) _ = Buffer.create size in
   let start_unc_offset = ref 0 in
   let end_unc_offset = ref 0 in
 
@@ -248,7 +245,7 @@
       let _ = Unix.lseek fd (!cmp_offset+8) Unix.SEEK_SET in
       let cmp_buff = String.create cmplen in
       my_read cmplen fd cmp_buff filename ;
-      let uncmp = uncompress cmp_buff 0 cmplen uncmplen in
+      let uncmp = Cbif.uncompress cmp_buff 0 cmplen uncmplen in
       (*
       if (String.length uncmp <> uncmplen) then begin
         log_and_print "ERROR: [%s] chunk at offset %d was supposed to have %d bytes of compressed data that expanded to %d, but in reality they expanded to %d"
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/cbif.ml ./WeiDU-192/src/cbif.ml
--- ./WeiDU-192.orig/src/cbif.ml 1969-12-31 19:00:00.000000000 -0500
+++ ./WeiDU-192/src/cbif.ml 2006-05-23 13:26:48.122043600 -0400
@@ -0,0 +1,12 @@
+(* Decompression routines for compressed bif files *)
+
+exception Error of string
+
+let _ = Callback.register_exception "mlgz_exn" (Error "")
+
+external cbf2bif : string -> string -> int
+    = "mlgz_cbf2bif"
+
+external uncompress : string -> pos:int -> clen:int -> ulen: int -> string
+  = "mlgz_uncompress"
+
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/load.ml ./WeiDU-192/src/load.ml
--- ./WeiDU-192.orig/src/load.ml 2006-04-10 16:58:53.000000000 -0400
+++ ./WeiDU-192/src/load.ml 2006-05-23 13:26:48.122043600 -0400
@@ -8,6 +8,7 @@
 It was originally taken from Westley Weimer's WeiDU 185. *)
 
 open Util
+open Cbif
 
 let registry_game_paths () =
   let str_list = "." :: !Arch.registry_paths in
@@ -319,19 +320,33 @@
       Hashtbl.find game.loaded_biffs bif_file (* already here *)
     else begin
       (* we must load the BIF *)
-      let biff_path =
-        let rec trial lst =
+      let biff_path = begin
+        let rec trial f lst =
           match lst with
-            [] -> find_file_in_path game.game_path bif_file
+            [] -> find_file_in_path game.game_path f
           | hd :: tl ->
-            let perhaps = find_file_in_path hd bif_file in
+              let perhaps = find_file_in_path hd f in
             log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
             if file_exists perhaps then
               perhaps
-            else trial tl
-        in
-        trial (game.cd_path_list)
+              else trial f tl
       in
+        (* Check to see if the bif file exists, if it doesn't try for a .CBF file *)
+        let bf = trial bif_file (game.cd_path_list @ [ game.game_path ^ "/cache" ] ) in
+        if file_exists bf then
+          bf
+        else begin
+          let cbf = Filename.chop_extension bif_file ^ ".cbf" in
+          let cbf_file = trial cbf (game.cd_path_list) in
+          if file_exists cbf_file then
+            let cache_file = game.game_path ^ "/cache/" ^ bif_file in
+            let sz = Cbif.cbf2bif cbf_file cache_file in
+            let _ = log_and_print "[%s] decompressed bif file %d bytes\n" cbf_file sz in
+            cache_file
+          else
+            bf
+        end
+      end in
       let the_biff = Biff.load_biff biff_path in
       Hashtbl.add game.loaded_biffs bif_file the_biff ;
       the_biff
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/zlib/zlib.c ./WeiDU-192/zlib/zlib.c
--- ./WeiDU-192.orig/zlib/zlib.c 2003-06-02 06:08:24.000000000 -0400
+++ ./WeiDU-192/zlib/zlib.c 2006-05-23 13:26:48.122043600 -0400
@@ -19,12 +19,13 @@
   static value * exn = NULL;
   if(exn == NULL)
     exn = caml_named_value ("mlgz_exn");
-  raise_with_string(*exn, (char *)msg) ;
+  caml_raise_with_string(*exn, (char *)msg) ;
 }
 
-value mlgz_uncompress(value v_src, value v_pos, value v_len, value unc_len)
+CAMLprim value mlgz_uncompress(value v_src, value v_pos, value v_len, value unc_len)
 {
-  value v_ret;
+  CAMLparam4(v_src, v_pos, v_len, unc_len);
+  CAMLlocal1(v_ret);
   int level, pos, len, out_buf_len, r;
   uLong out_len;
   const char *in_buf;
@@ -42,7 +43,7 @@
    out_buf = malloc(out_buf_len);
    */
 
-  v_ret = alloc_string(Int_val(unc_len));
+  v_ret = caml_alloc_string(Int_val(unc_len));
   out_buf = String_val(v_ret);
 
   if(out_buf == NULL)
@@ -55,7 +56,6 @@
     } else if(r == Z_BUF_ERROR) {
       char *new_buf;
 
-      printf("uncompress 1\n"); fflush(stdout);
       raise_mlgz_exn("uncompress");
       out_buf_len *= 2;
       new_buf = realloc(out_buf, out_buf_len);
@@ -65,11 +65,9 @@
       }
       out_buf = new_buf;
     } else if(r == Z_MEM_ERROR) {
-      printf("uncompress 3\n"); fflush(stdout);
       free(out_buf);
       raise_out_of_memory();
     } else {
-      printf("WeiDU: ZLIB: Warning! Problem decompressing block.\n");
       fflush(stdout);
       out_len = Int_val(unc_len);
       break;
@@ -83,5 +81,224 @@
    memcpy(String_val(v_ret), out_buf, out_len);
    free(out_buf);
    */
-  return v_ret ;
+  CAMLreturn (v_ret) ;
+}
+
+/* zerr() and def() are copied directly from zlib example code. */
+
+#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
+#  include <fcntl.h>
+#  include <io.h>
+#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+#else
+#  define SET_BINARY_MODE(file)
+#endif
+
+#define CHUNK 16384
+
+
+/* Raise an exception for a zlib or i/o error */
+void mlgz_zerr(int ret)
+{
+    switch (ret) {
+    case Z_ERRNO:
+        raise_sys_error(copy_string(strerror(errno))) ;
+        break;
+    case Z_STREAM_ERROR:
+        raise_mlgz_exn("invalid compression level");
+        break;
+    case Z_DATA_ERROR:
+        raise_mlgz_exn("invalid or incomplete deflate data");
+        break;
+    case Z_MEM_ERROR:
+        raise_out_of_memory() ;
+        break;
+    case Z_VERSION_ERROR:
+        raise_mlgz_exn("zlib version mismatch!");
+    }
+}
+
+/* Yes, this is bad practice.  I compensated by using "%.256s" instead of just "%s": */
+static char errstr[1024];
+
+/* This is a bit better for error checking: */
+int fread_check(void* b, size_t sz, size_t cnt, FILE* fp, const char* fn)
+{
+    if (fread(b, sz, cnt, fp) != cnt) {
+        sprintf(errstr, "Failed to read %d bytes from file %.256s", sz*cnt, fn);
+        raise_mlgz_exn(errstr);
+        return 1;
+    }
+    return 0;
+}
+
+int fread_uint(uint32_t* i, FILE* fp, const char* fn)
+{
+    if (fread_check(i, 4, 1, fp, fn))
+        return 1;
+#if defined(__ppc__) || defined(__ppc64__)
+    *i = OSSwapInt32(*i);
+#endif
+    return 0;
+}
+
+
+/* Decompress from file source to file dest until stream ends or EOF.
+   inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
+   allocated for processing, Z_DATA_ERROR if the deflate data is
+   invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
+   the version of the library linked do not match, or Z_ERRNO if there
+   is an error reading or writing the files. */
+int inf(FILE *source, FILE *dest)
+{
+    int ret;
+    unsigned have;
+    z_stream strm;
+    unsigned char in[CHUNK];
+    unsigned char out[CHUNK];
+   
+    /* allocate inflate state */
+    strm.zalloc = Z_NULL;
+    strm.zfree = Z_NULL;
+    strm.opaque = Z_NULL;
+    strm.avail_in = 0;
+    strm.next_in = Z_NULL;
+    ret = inflateInit(&strm);
+    if (ret != Z_OK)
+        return ret;
+    /* decompress until deflate stream ends or end of file */
+    do {
+        strm.avail_in = fread(in, 1, CHUNK, source);
+        if (ferror(source)) {
+            (void)inflateEnd(&strm);
+            return Z_ERRNO;
+        }
+        if (strm.avail_in == 0)
+            break;
+        strm.next_in = in;
+        /* run inflate() on input until output buffer not full */
+        do {
+            strm.avail_out = CHUNK;
+            strm.next_out = out;
+            ret = inflate(&strm, Z_NO_FLUSH);
+            /* assert(ret != Z_STREAM_ERROR); */  /* state not clobbered */ /* no asserts for Ocaml */
+            switch (ret) {
+            case Z_NEED_DICT:
+                ret = Z_DATA_ERROR;     /* and fall through */
+            case Z_DATA_ERROR:
+            case Z_MEM_ERROR:
+                (void)inflateEnd(&strm);
+                return ret;
+            }
+            have = CHUNK - strm.avail_out;
+            if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
+                (void)inflateEnd(&strm);
+                return Z_ERRNO;
+            }
+        } while (strm.avail_out == 0);
+        /* done when inflate() says it's done */
+    } while (ret != Z_STREAM_END);
+    /* clean up and return */
+    (void)inflateEnd(&strm);
+    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
+}
+
+/*
+   Unompresses a CBF files to a specified BIF file.  Return 0 on failure or the
+   number of uncompressed bytes on success.
+ */
+CAMLprim value mlgz_cbf2bif(value _cbf_file, value _bif_file)
+{
+  CAMLparam2(_cbf_file, _bif_file);
+  const char* cbf_file = String_val(_cbf_file);
+  const char* bif_file = String_val(_bif_file);
+  FILE *cbf_fp;
+  FILE *bif_fp;
+  char sigver[9];
+  uint32_t bif_file_len, cmplen, uncmplen;
+  int zret;
+   
+  if (!(cbf_fp = fopen(cbf_file, "rb"))) {
+    sprintf(errstr, "failure opening file: %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (!(bif_fp = fopen(bif_file, "wb"))) {
+    fclose(cbf_fp);
+    sprintf(errstr, "failure opening file: %.256s", bif_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  sigver[8] = 0;
+  if (fread_check(sigver, 1, 8, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (strcmp(sigver, "BIF V1.0")) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "incorrect CBF header for file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fread_uint(&bif_file_len, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (bif_file_len <=0 || bif_file_len > 128) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "corrupt CBF file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  /* Seek ahead past embedded file name, doesn't really matter what it is */
+  if (fseek(cbf_fp, bif_file_len, SEEK_CUR)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "failure seeking %d bytes into file %.256s", bif_file_len, cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fread_uint(&uncmplen, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (fread_uint(&cmplen, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+  /* printf("CBF %s (%ld bytes) -> BIF %s [%ld bytes]", cbf_file, cmplen, bif_file, uncmplen); */
+
+  if ((zret=inf(cbf_fp, bif_fp)) != Z_OK) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    mlgz_zerr(zret);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fclose(cbf_fp)) {
+    fclose(bif_fp);
+    sprintf(errstr, "failure closing file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+  if (fclose(bif_fp)) {
+    sprintf(errstr, "failure closing file %.256s", bif_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+  CAMLreturn(Val_int(uncmplen));
 }
Posted by: FredSRichardson
« on: May 23, 2006, 11:59:10 AM »

WARNING I've found at least one bug in the old patch (also, the exception handling was broken in the old compress code).  I'll get a new patch out soon.
Posted by: FredSRichardson
« on: May 22, 2006, 10:04:55 AM »

Well, file_size is

Code: [Select]
let file_size name =
  try
    let stats = Case_ins.unix_stat name in
    stats.Unix.st_size
  with _ ->  -1

So it's already Case_insed. As for the >=0, it was added by Wes because _ and is't needed by a host of already existing mods, so :(
Yes, I see your point there  and I should've read a bit more of the code. :)

And there's really no reason to fix what's not broken so I'll drop the topic.  Having a test for "exists and is non-zero bytes" is probably safer anway (since a zero-byte file is almost always going to be corrupt for this program).
Posted by: the bigg
« on: May 22, 2006, 09:57:37 AM »

Well, file_size is

Code: [Select]
let file_size name =
  try
    let stats = Case_ins.unix_stat name in
    stats.Unix.st_size
  with _ ->  -1

So it's already Case_insed. As for the >=0, it was added by Wes because _ and is't needed by a host of already existing mods, so :(
Posted by: FredSRichardson
« on: May 22, 2006, 09:51:48 AM »

Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):
Ok, thanks. The cache cleanup code should be straightforward - would you prefer me to code it?
BTW, since I saw that mentioned, you shouldn't do
Code: [Select]
file_exists (String.lowercase (Arch.backslash_to_shash file))
[..]
open_in (String.lowercase (Arch.backslash_to_shash file))
etc.
but use the Case_ins module instead:
Code: [Select]
file_exists file (* already case_insed by Util *)
[..]
Case_ins.perv_open_in file;
etc.
Dev specifically requested that OSX doesn't get the lowercasing  ;)

If you don't mind handling the cache cleanup coding that would be great.  I kind of lost interest when I realized this wasn't quite going to solve my problem with IWD1Tutu (long story).

Do you think it would be a good idea to have a sys_file_exists in case_ins*.ml?  I didn't include this small patch, but this seems like a logical addition:
Code: [Select]
diff -u -r ./WeiDU-192.orig/src/case_ins_linux.ml ./WeiDU-192/src/case_ins_linux.ml
--- ./WeiDU-192.orig/src/case_ins_linux.ml 2006-03-18 19:30:18.000000000 -0500
+++ ./WeiDU-192/src/case_ins_linux.ml 2006-05-20 11:29:09.000000000 -0400
@@ -16,3 +16,5 @@
 let unix_unlink s = Unix.unlink (String.lowercase (backslash_to_slash s)) ;;
 let unix_mkdir s p = Unix.mkdir (String.lowercase (backslash_to_slash s)) p ;;
 let unix_opendir s = Unix.opendir (String.lowercase (backslash_to_slash s)) ;;
+
+let sys_file_exists s = Sys.file_exists (String.lowercase (backslash_to_slash s)) ;;
diff -u -r ./WeiDU-192.orig/src/case_ins_mac.ml ./WeiDU-192/src/case_ins_mac.ml
--- ./WeiDU-192.orig/src/case_ins_mac.ml 2006-03-18 19:30:34.000000000 -0500
+++ ./WeiDU-192/src/case_ins_mac.ml 2006-05-20 11:29:33.187500000 -0400
@@ -16,3 +16,5 @@
 let unix_unlink s = Unix.unlink (backslash_to_slash s) ;;
 let unix_mkdir s p = Unix.mkdir (backslash_to_slash s) p ;;
 let unix_opendir s = Unix.opendir (backslash_to_slash s) ;;
+
+let sys_file_exists s = Sys.file_exists (backslash_to_slash s) ;;
diff -u -r ./WeiDU-192.orig/src/case_ins_win.ml ./WeiDU-192/src/case_ins_win.ml
--- ./WeiDU-192.orig/src/case_ins_win.ml 2006-04-10 17:01:05.000000000 -0400
+++ ./WeiDU-192/src/case_ins_win.ml 2006-05-20 11:28:37.218750000 -0400
@@ -12,3 +12,5 @@
 let unix_unlink s = Unix.unlink s ;;
 let unix_mkdir s p = Unix.mkdir s p ;;
 let unix_opendir s = Unix.opendir s ;;
+
+let sys_file_exists s = Sys.file_exists s ;;

file_exists in util.ml doesn't take any measures to transform the file name (maybe it's important to use the "file_exists and has non-zero bytes" but I can't see why):
Code: [Select]
let file_exists name = (file_size name >= 0)
Posted by: the bigg
« on: May 22, 2006, 09:39:58 AM »

Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):
Ok, thanks. The cache cleanup code should be straightforward - would you prefer me to code it?
BTW, since I saw that mentioned, you shouldn't do
Code: [Select]
file_exists (String.lowercase (Arch.backslash_to_shash file))
[..]
open_in (String.lowercase (Arch.backslash_to_shash file))
etc.
but use the Case_ins module instead:
Code: [Select]
file_exists file (* already case_insed by Util *)
[..]
Case_ins.perv_open_in file;
etc.
Dev specifically requested that OSX doesn't get the lowercasing  ;)
Posted by: FredSRichardson
« on: May 22, 2006, 09:32:38 AM »

Not to be lazy, but, uh, did you update the diff?  :)
I was just getting to that :)

Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):

Code: [Select]
diff -u -r ./WeiDU-192.orig/src/load.ml ./WeiDU-192/src/load.ml
--- ./WeiDU-192.orig/src/load.ml 2006-04-10 16:58:53.000000000 -0400
+++ ./WeiDU-192/src/load.ml 2006-05-21 12:17:58.171875000 -0400
@@ -314,24 +314,41 @@
 
 let skip_next_load_error = ref false
 
+external cbf2bif : string -> string -> int
+    = "mlgz_cbf2bif"
+
 let load_bif_in_game game bif_file =
     if Hashtbl.mem game.loaded_biffs bif_file then
       Hashtbl.find game.loaded_biffs bif_file (* already here *)
     else begin
       (* we must load the BIF *)
-      let biff_path =
-        let rec trial lst =
+      let biff_path = begin
+        let rec trial f lst =
           match lst with
-            [] -> find_file_in_path game.game_path bif_file
+            [] -> find_file_in_path game.game_path f
           | hd :: tl ->
-            let perhaps = find_file_in_path hd bif_file in
-            log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
-            if file_exists perhaps then
-              perhaps
-            else trial tl
+              let perhaps = find_file_in_path hd f in
+              log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
+              if file_exists perhaps then
+                perhaps
+              else trial f tl
         in
-        trial (game.cd_path_list)
-      in
+        (* Check to see if the bif file exists, if it doesn't try for a .CBF file *)
+        let bf = trial bif_file (game.cd_path_list @ [ game.game_path ^ "/cache" ] ) in
+        if file_exists bf then
+          bf
+        else begin
+          let cbf = Filename.chop_extension bif_file ^ ".cbf" in
+          let cbf_file = trial cbf (game.cd_path_list) in
+          if file_exists cbf_file then
+            let cache_file = game.game_path ^ "/cache/" ^ bif_file in
+            let sz = cbf2bif cbf_file cache_file in
+            let _ = log_and_print "[%s] decompressed bif file %d bytes\n" cbf_file sz in
+            cache_file
+          else
+            bf
+        end
+      end in
       let the_biff = Biff.load_biff biff_path in
       Hashtbl.add game.loaded_biffs bif_file the_biff ;
       the_biff
diff -u -r ./WeiDU-192.orig/zlib/zlib.c ./WeiDU-192/zlib/zlib.c
--- ./WeiDU-192.orig/zlib/zlib.c 2003-06-02 06:08:24.000000000 -0400
+++ ./WeiDU-192/zlib/zlib.c 2006-05-22 10:16:54.250000000 -0400
@@ -85,3 +85,221 @@
    */
   return v_ret ;
 }
+
+/* zerr() and def() are copied directly from zlib example code. */
+
+#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
+#  include <fcntl.h>
+#  include <io.h>
+#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+#else
+#  define SET_BINARY_MODE(file)
+#endif
+
+#define CHUNK 16384
+
+
+/* Raise an exception for a zlib or i/o error */
+void mlgz_zerr(int ret)
+{
+    switch (ret) {
+    case Z_ERRNO:
+        raise_sys_error(copy_string(strerror(errno))) ;
+        break;
+    case Z_STREAM_ERROR:
+        raise_mlgz_exn("invalid compression level");
+        break;
+    case Z_DATA_ERROR:
+        raise_mlgz_exn("invalid or incomplete deflate data");
+        break;
+    case Z_MEM_ERROR:
+        raise_out_of_memory() ;
+        break;
+    case Z_VERSION_ERROR:
+        raise_mlgz_exn("zlib version mismatch!");
+    }
+}
+
+/* Yes, this is bad practice.  I compensated by using "%.256s" instead of just "%s": */
+static char errstr[1024];
+
+/* This is a bit better for error checking: */
+int fread_check(void* b, size_t sz, size_t cnt, FILE* fp, const char* fn)
+{
+    if (fread(b, sz, cnt, fp) != cnt) {
+        sprintf(errstr, "Failed to read %d bytes from file %.256s", sz*cnt, fn);
+        raise_mlgz_exn(errstr);
+        return 1;
+    }
+    return 0;
+}
+
+int fread_uint(uint32_t* i, FILE* fp, const char* fn)
+{
+    if (fread_check(&i, 4, 1, fp, fn))
+        return 1;
+#if defined(__ppc__) || defined(__ppc64__)
+    *i = OSSwapInt32(*i);
+#endif
+    return 0;
+}
+
+
+/* Decompress from file source to file dest until stream ends or EOF.
+   inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
+   allocated for processing, Z_DATA_ERROR if the deflate data is
+   invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
+   the version of the library linked do not match, or Z_ERRNO if there
+   is an error reading or writing the files. */
+int inf(FILE *source, FILE *dest)
+{
+    int ret;
+    unsigned have;
+    z_stream strm;
+    unsigned char in[CHUNK];
+    unsigned char out[CHUNK];
+   
+    /* allocate inflate state */
+    strm.zalloc = Z_NULL;
+    strm.zfree = Z_NULL;
+    strm.opaque = Z_NULL;
+    strm.avail_in = 0;
+    strm.next_in = Z_NULL;
+    ret = inflateInit(&strm);
+    if (ret != Z_OK)
+        return ret;
+    /* decompress until deflate stream ends or end of file */
+    do {
+        strm.avail_in = fread(in, 1, CHUNK, source);
+        if (ferror(source)) {
+            (void)inflateEnd(&strm);
+            return Z_ERRNO;
+        }
+        if (strm.avail_in == 0)
+            break;
+        strm.next_in = in;
+        /* run inflate() on input until output buffer not full */
+        do {
+            strm.avail_out = CHUNK;
+            strm.next_out = out;
+            ret = inflate(&strm, Z_NO_FLUSH);
+            /* assert(ret != Z_STREAM_ERROR); */  /* state not clobbered */ /* no asserts for Ocaml */
+            switch (ret) {
+            case Z_NEED_DICT:
+                ret = Z_DATA_ERROR;     /* and fall through */
+            case Z_DATA_ERROR:
+            case Z_MEM_ERROR:
+                (void)inflateEnd(&strm);
+                return ret;
+            }
+            have = CHUNK - strm.avail_out;
+            if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
+                (void)inflateEnd(&strm);
+                return Z_ERRNO;
+            }
+        } while (strm.avail_out == 0);
+        /* done when inflate() says it's done */
+    } while (ret != Z_STREAM_END);
+    /* clean up and return */
+    (void)inflateEnd(&strm);
+    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
+}
+
+/*
+   Unompresses a CBF files to a specified BIF file.  Return 0 on failure or the
+   number of uncompressed bytes on success.
+ */
+value mlgz_cbf2bif(value _cbf_file, value _bif_file)
+{
+    const char* cbf_file = String_val(_cbf_file);
+    const char* bif_file = String_val(_bif_file);
+    FILE *cbf_fp;
+    FILE *bif_fp;
+    char sigver[9];
+    uint32_t bif_file_len, cmplen, uncmplen;
+    int zret;
+   
+
+    if (!(cbf_fp = fopen(cbf_file, "rb"))) {
+        sprintf(errstr, "failure opening file: %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    if (!(bif_fp = fopen(bif_file, "wb"))) {
+        fclose(cbf_fp);
+        sprintf(errstr, "failure opening file: %.256s", bif_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    sigver[8] = 0;
+    if (fread_check(sigver, 1, 8, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (strcmp(sigver, "BIF V1.0")) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "incorrect CBF header for file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    if (fread_uint(&bif_file_len, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (bif_file_len <=0 || bif_file_len > 128) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "corrupt CBF file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    /* Seek ahead past embedded file name, doesn't really matter what it is */
+    if (fseek(cbf_fp, bif_file_len, SEEK_CUR)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "failure seeking %d bytes into file %.256s", bif_file_len, cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    if (fread_uint(&uncmplen, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (fread_uint(&cmplen, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+    /* printf("CBF %s (%ld bytes) -> BIF %s [%ld bytes]", cbf_file, cmplen, bif_file, uncmplen); */
+
+    if ((zret=inf(cbf_fp, bif_fp)) != Z_OK) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        mlgz_zerr(zret);
+        return Val_int(0);
+    }
+
+    if (fclose(cbf_fp)) {
+        fclose(bif_fp);
+        sprintf(errstr, "failure closing file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    if (fclose(bif_fp)) {
+        sprintf(errstr, "failure closing file %.256s", bif_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    return Val_int(uncmplen);
+}
Posted by: the bigg
« on: May 22, 2006, 09:27:59 AM »

Not to be lazy, but, uh, did you update the diff?  :)
Posted by: FredSRichardson
« on: May 22, 2006, 08:55:04 AM »

Quote
I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)
I don't want to look at the tech docs, but I'm fairly sure that Apple's recommendation is that __ppc__ should not be defined on 64-bit systems.

EDIT: What am I thinking? WeiDU will never be compiled as a 64-bit executable. :-/

Yeah, __ppc__ should be sufficient.
Yes, you're right on both counts!  I'll do the right thing and put in both macros anyway.  I'll post another patch here shortly :)

EDIT: I just realized that it's possible that WeiDU will get ported to more OS's with GemRB.  So I'll play it safe and try to do the right thing :)
Posted by: devSin
« on: May 22, 2006, 12:52:20 AM »

OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)
You'd want OSSwapConstInt32(x) and OSSwapConstInt16(x). Mac OS X has macros to swap to little or big endian (generic macros and inline assembly), and limited support for pdp, as well as the posix standard (nh)to(hn)(sl). I use OSSwap* when I'm stealing code from others. :D
Well, since the only big-endian architecture we have to worry about is the Mac, then I'll go ahead and make that change.  I guess the [nh]to[hn][sl] routines can't be used in this case since "net" is always big-endian (and "host" is native).  Now a PDP port of WeiDU, that could be interesting :)
Yes. The order of bytes shall be influenced by the phases of the moon. ;-)

Anyway, if there's ever cause for a Sparc port or somesuch, it'll be easy enough to roll back out into zlib.c.

Quote
I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)
I don't want to look at the tech docs, but I'm fairly sure that Apple's recommendation is that __ppc__ should not be defined on 64-bit systems.

EDIT: What am I thinking? WeiDU will never be compiled as a 64-bit executable. :-/

Yeah, __ppc__ should be sufficient.
Posted by: FredSRichardson
« on: May 22, 2006, 12:49:02 AM »

The only thing left that you may want to do is #ifdef __ppc64__ (I compile the Mac OS X releases on a frumpy G4, though, so no difference to me). I wish I knew for sure if 64-bit PPC also defined __ppc__, sorry.

I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)
Posted by: FredSRichardson
« on: May 22, 2006, 12:44:38 AM »

OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)
You'd want OSSwapConstInt32(x) and OSSwapConstInt16(x). Mac OS X has macros to swap to little or big endian (generic macros and inline assembly), and limited support for pdp, as well as the posix standard (nh)to(hn)(sl). I use OSSwap* when I'm stealing code from others. :D
Well, since the only big-endian architecture we have to worry about is the Mac, then I'll go ahead and make that change.  I guess the [nh]to[hn][sl] routines can't be used in this case since "net" is always big-endian (and "host" is native).  Now a PDP port of WeiDU, that could be interesting :)