Author Topic: iwd1 compressed bif files (cbf)  (Read 9353 times)

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #25 on: May 21, 2006, 09:48:27 PM »
You may want to check both __ppc64__ and __ppc__ (I can't remember if the G5 even defines __ppc__, or just the 64).

I'll try it with one of the CBFs on ID disc 2.

What use for the reference BIFF?

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #26 on: May 21, 2006, 10:02:13 PM »
Code: [Select]
#define BS_4BYTE(a) ((((a)&0xFF000000)>>24)|(((a)&0x00FF0000)>>8)|(((a)&0x0000FF00)<<8)|(((a)&0x000000FF)<<24))
#define BS_2BYTE(a) ((((a)&0xFF00)>>8)|(((a)&0x00FF)<<8)
These will need to be cast. I just hacked
Code: [Select]
#define BS_4BYTE(a) ((uint32_t)(((uint32_t)(a)&0xFF000000)>>24) | \
                   (((uint32_t)(a)&0x00FF0000)>>8)| \
                   (((uint32_t)(a)&0x0000FF00)<<8)| \
                   (((uint32_t)(a)&0x000000FF)<<24))

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #27 on: May 21, 2006, 10:09:21 PM »
Fails with "corrupt CBF file"

I'm using AR200B.CBF from CD2/data.
MD5 (AR200B.CBF) = 89ef1caf7d916024f66d41f14c166fdb

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #28 on: May 21, 2006, 11:45:47 PM »
Try commenting out the byte-swapping code (I only use BS_4BYTE in one place) and see what happens.  Maybe the CBF files on the Mac were created on a Mac.

-Fred
EDIT I take it back.  My file has the same md5sum, so they must be compressed on the same arch.
« Last Edit: May 21, 2006, 11:48:18 PM by FredSRichardson »

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #29 on: May 21, 2006, 11:50:56 PM »
un_sav.c works for them. un_sav requires 3 additional lines to work on PPC. Assuming OSSwapInt32 is more or less identical to the (uint32_t)macro in un_cbf (it is):
Code: [Select]
#include <zlib.h>
#include <stdio.h>
#include <stdlib.h>

int extractFile( FILE *fIn )
{
  unsigned long namelen;
  char namebuf[1024];
  unsigned long cmplen, uncmplen;
  FILE *fOut;
  void *destBuf, *srcBuf;
  unsigned long offset;

  offset = ftell( fIn );
  printf( "un-saving at offset 0x%08lx\n", offset );
  if ( fread( &namelen, 4, 1, fIn ) != 1 ) return 0;
/* BYTE SWAP */
  namelen = OSSwapInt32(namelen); // swap it
  if ( fread( namebuf, 1, namelen, fIn ) != namelen ) return 0;
  if ( fread( &uncmplen, 4, 1, fIn ) != 1 ) return 0;
/* BYTE SWAP */
  uncmplen = OSSwapInt32(uncmplen); // swap it
  if ( fread( &cmplen, 4, 1, fIn ) != 1 ) return 0;
/* BYTE SWAP */
  cmplen = OSSwapInt32(cmplen); // swap it

  fOut = fopen( namebuf, "wb" ); // char array - no swap
 
//  everything else is up to gzip (it will read the little endian chunks and
// spit them out correctly
  if ( fOut == NULL ) return 0;
  srcBuf = malloc( cmplen );
  destBuf = malloc( uncmplen );
  if ( !srcBuf || !destBuf || fread( srcBuf, 1, cmplen, fIn )!=cmplen )
    {
      fclose( fOut );
      if ( destBuf ) free( destBuf );
      if ( srcBuf ) free( srcBuf );
      return 0;
    }
  uncompress( destBuf, &uncmplen, srcBuf, cmplen );
  fwrite( destBuf, 1, uncmplen, fOut );
  fclose( fOut );
  free( destBuf );
  free( srcBuf );
  return 1;
}

void unSav( const char *filename )
{
  char signature[4], version[4];
  FILE *fIn = fopen( filename, "rb" );
  if ( !fIn ) return;
  fread( signature, 1, 4, fIn ); // garbage
  fread( version, 1, 4, fIn );   // read
  while( extractFile( fIn ) ); // nothing Mac specific; let 'er rip
  fclose( fIn );
}

int main( int c, char **v )
{
  unSav( v[1] );
  return 0;
}
This code will successfully decompress the CBF. un_bifc (and probably the bam one) is similarly easy.
« Last Edit: May 22, 2006, 12:36:30 AM by devSin »

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #30 on: May 21, 2006, 11:57:44 PM »
Ooops, I didn't test my code very well.  Try this version:

Code: [Select]
/* cbf2bif.c: Based on jedwin un_bifc, hacked by frichard */

/*
 * Small program the unpacks a CBF file as a BIF file with minimal memory overhead.
 */

#include <zlib.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <strings.h>


int
fread_check(void* b, size_t sz, size_t cnt, FILE* fp, const char* fn)
{
    if (fread(b, sz, cnt, fp) != cnt) {
        fprintf(stderr, "Failed to read %d bytes from %s\n", sz*cnt, fn);
        return 1;
    }
    return 0;
}

/* BS for byte-swap */
#define BS_4BYTE(a) ((uint32_t)(((uint32_t)(a)&0xFF000000)>>24) | \
                   (((uint32_t)(a)&0x00FF0000)>>8)| \
                   (((uint32_t)(a)&0x0000FF00)<<8)| \
                   (((uint32_t)(a)&0x000000FF)<<24))
#define BS_2BYTE(a) ((uint32_t)((((uint32_t)(a)&0xFF00)>>8)|(((uint32_t)(a)&0x00FF)<<8)))

int fread_uint(uint32_t* i, FILE* fp, const char* fn)
{
    if (fread_check(i, 4, 1, fp, fn))
        return 1;
#ifdef __ppc__
    *i = BS_4BYTE(*i);
#endif
    return 0;
}


/* zerr() and def() are copied directly from zlib example code. */

#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
#  include <fcntl.h>
#  include <io.h>
#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
#else
#  define SET_BINARY_MODE(file)
#endif

#define CHUNK 16384


/* report a zlib or i/o error */
void zerr(int ret)
{
    fputs("zpipe: ", stderr);
    switch (ret) {
    case Z_ERRNO:
        if (ferror(stdin))
            fputs("error reading stdin\n", stderr);
        if (ferror(stdout))
            fputs("error writing stdout\n", stderr);
        break;
    case Z_STREAM_ERROR:
        fputs("invalid compression level\n", stderr);
        break;
    case Z_DATA_ERROR:
        fputs("invalid or incomplete deflate data\n", stderr);
        break;
    case Z_MEM_ERROR:
        fputs("out of memory\n", stderr);
        break;
    case Z_VERSION_ERROR:
        fputs("zlib version mismatch!\n", stderr);
    }
}

/* Decompress from file source to file dest until stream ends or EOF.
   inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
   allocated for processing, Z_DATA_ERROR if the deflate data is
   invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
   the version of the library linked do not match, or Z_ERRNO if there
   is an error reading or writing the files. */
int inf(FILE *source, FILE *dest)
{
    int ret;
    unsigned have;
    z_stream strm;
    unsigned char in[CHUNK];
    unsigned char out[CHUNK];
   
    /* allocate inflate state */
    strm.zalloc = Z_NULL;
    strm.zfree = Z_NULL;
    strm.opaque = Z_NULL;
    strm.avail_in = 0;
    strm.next_in = Z_NULL;
    ret = inflateInit(&strm);
    if (ret != Z_OK)
        return ret;
    /* decompress until deflate stream ends or end of file */
    do {
        strm.avail_in = fread(in, 1, CHUNK, source);
        if (ferror(source)) {
            (void)inflateEnd(&strm);
            return Z_ERRNO;
        }
        if (strm.avail_in == 0)
            break;
        strm.next_in = in;
        /* run inflate() on input until output buffer not full */
        do {
            strm.avail_out = CHUNK;
            strm.next_out = out;
            ret = inflate(&strm, Z_NO_FLUSH);
            assert(ret != Z_STREAM_ERROR);  /* state not clobbered */
            switch (ret) {
            case Z_NEED_DICT:
                ret = Z_DATA_ERROR;     /* and fall through */
            case Z_DATA_ERROR:
            case Z_MEM_ERROR:
                (void)inflateEnd(&strm);
                return ret;
            }
            have = CHUNK - strm.avail_out;
            if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
                (void)inflateEnd(&strm);
                return Z_ERRNO;
            }
        } while (strm.avail_out == 0);
        /* done when inflate() says it's done */
    } while (ret != Z_STREAM_END);
    /* clean up and return */
    (void)inflateEnd(&strm);
    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
}

int
cbf2bif(const char* cbf_file, const char* bif_file)
{
    FILE *cbf_fp;
    FILE *bif_fp;
    char sigver[9];
    uint32_t bif_file_len, cmplen, uncmplen;
    int zret;
   
    if (!(cbf_fp = fopen(cbf_file, "rb"))) {
        fprintf(stderr, "Failure opening file: %s\n", cbf_file);
        return 1;
    }
    if (!(bif_fp = fopen(bif_file, "wb"))) {
        fclose(cbf_fp);
        fprintf(stderr, "Failure opening file: %s\n", bif_file);
        return 1;
    }

    sigver[8] = 0;
    if (fread_check(sigver, 1, 8, cbf_fp, cbf_file)) {
        fclose(cbf_fp);
        fclose(bif_fp);
        return 1;
    }
   
    if (strcmp(sigver, "BIF V1.0")) {
        fprintf(stderr, "Incorrect CBF header for file %s\n", cbf_file);
        fclose(cbf_fp);
        fclose(bif_fp);
        return 1;
    }

    if (fread_uint(&bif_file_len, cbf_fp, cbf_file)) {
        fclose(cbf_fp);
        fclose(bif_fp);
        return 1;
    }
   
    if (bif_file_len <=0 || bif_file_len > 128) {
        fclose(cbf_fp);
        fclose(bif_fp);
        fprintf(stderr, "Corrupt CBF file %s\n", cbf_file);
        return 1;
    }

    if (fseek(cbf_fp, bif_file_len, SEEK_CUR)) {
        fclose(cbf_fp);
        fclose(bif_fp);
        fprintf(stderr, "Failed to seek ahead in file %s\n", cbf_file);
        return 1;
    }

    if (fread_uint(&uncmplen, cbf_fp, cbf_file)) {
        fclose(cbf_fp);
        fclose(bif_fp);
        fprintf(stderr, "Failed to seek ahead in file %s\n", cbf_file);
        return 1;
    }

    if (fread_uint(&cmplen, cbf_fp, cbf_file)) {
        fclose(cbf_fp);
        fclose(bif_fp);
        fprintf(stderr, "Failed to seek ahead in file %s\n", cbf_file);
        return 1;
    }

    printf("CBF %s (%d bytes) -> BIF %s [%d bytes]\n", cbf_file, (int)cmplen, bif_file, (int)uncmplen);


    if ((zret=inf(cbf_fp, bif_fp)) != Z_OK) {
        fclose(cbf_fp);
        fclose(bif_fp);
        zerr(zret);
        return 1;
    }

    if (fclose(cbf_fp)) {
        fclose(bif_fp);
        fprintf(stderr, "Failure closing file %s\n", cbf_file);
        return 1;
    }
    if (fclose(bif_fp)) {
        fprintf(stderr, "Failure closing file %s\n", bif_file);
        return 1;
    }
   
    return 0;
}


int main( int argc, char *argv[] )
{
    if (argc != 3) {
        fprintf(stderr, "Usage: %s <in-cbf-file> <out-bif-file>\n", argv[0]);
        exit(1);
    }
   
    if (cbf2bif(argv[1], argv[2])) {
        fprintf(stderr, "Conversion failed.\n");
        exit(1);
    }
    return 0;
}

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #31 on: May 22, 2006, 12:05:11 AM »
un_sav.c works for them. un_sav requires 3 additional lines to work on PPC. Assuming OSSwapInt32 is more or less identical to the (uint32_t)macro in un_cbf (it is):

Yes un_sav (and un_bifc) have the problem that they use "uncompress" which loads the whole file into memory.  Some of these CBF files can be quite big so this is a problem for WeiDU.  Try out the newer version I just posted.

OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)

-Fred

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #32 on: May 22, 2006, 12:08:40 AM »
Ooops, I didn't test my code very well.  Try this version:
The result
Code: [Select]
Mainframe:~/Desktop akay$ ./a.out AR200B.CBF AR200B.CBF.uncompressed
CBF AR200B.CBF (1145074 bytes) -> BIF AR200B.CBF.uncompressed [1939843 bytes]
Mainframe:~/Desktop akay$ MD5 /Users/akay/Desktop/AR200B.CBF.uncompressed
MD5 (/Users/akay/Desktop/AR200B.CBF.uncompressed) = 59470dd75d3914b052ecf6dc956ccd30
I'd change the following to uint16_t
Code: [Select]
#define BS_2BYTE(a) ((uint32_t)((((uint32_t)(a)&0xFF00)>>8)|(((uint32_t)(a)&0x00FF)<<8)))not least because it's such a vital part of the program. ;-)

Assuming our MD5s match, it looks to work OK here!

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #33 on: May 22, 2006, 12:13:56 AM »
OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)
You'd want OSSwapConstInt32(x) and OSSwapConstInt16(x). Mac OS X has macros to swap to little or big endian (generic macros and inline assembly), and limited support for pdp, as well as the posix standard (nh)to(hn)(sl). I use OSSwap* when I'm stealing code from others. :D

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #34 on: May 22, 2006, 12:15:31 AM »
Ooops, I didn't test my code very well.  Try this version:
The result
Code: [Select]
Mainframe:~/Desktop akay$ ./a.out AR200B.CBF AR200B.CBF.uncompressed
CBF AR200B.CBF (1145074 bytes) -> BIF AR200B.CBF.uncompressed [1939843 bytes]
Mainframe:~/Desktop akay$ MD5 /Users/akay/Desktop/AR200B.CBF.uncompressed
MD5 (/Users/akay/Desktop/AR200B.CBF.uncompressed) = 59470dd75d3914b052ecf6dc956ccd30
I'd change the following to uint16_t
Code: [Select]
#define BS_2BYTE(a) ((uint32_t)((((uint32_t)(a)&0xFF00)>>8)|(((uint32_t)(a)&0x00FF)<<8)))not least because it's such a vital part of the program. ;-)

Assuming our MD5s match, it looks to work OK here!
Ooops, good catch!  More sloppy coding on my part (I probably shouldn't write code I don't use :D).

BTW: I've found that the way to be happiest with IWD1 is to unpack all the CBF files to BIF files and put them into the "data" directory.  That is, if disk space is no problem.

For IWD1Tutu, I have to make sure the CBF files are all un-packed.  I'm referencing resources directly in the IWD1 BIF files from IWD2 (by making a mess of the Chitin.key).  I haven't checked to see if IWD2 can handle CBF files.

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #35 on: May 22, 2006, 12:24:47 AM »
The only thing left that you may want to do is #ifdef __ppc64__ (I compile the Mac OS X releases on a frumpy G4, though, so no difference to me). I wish I knew for sure if 64-bit PPC also defined __ppc__, sorry.

I'd toyed with decompressing all the CBFs and BIFCs, but during gameplay, ID isn't intensive enough to really need it, and BG2 can decompress the BIFC archive faster than a slow machine could ever load the required resources, so I just put up with any overhead the unzip requires. For editing, though, I'd have to agree, but luckily most mortals don't ever need to deal with the compressed resources.

It would probably be most beneficial if you do some magic to prevent the game trying to cache the data (the increased copy would kill off some of the benefit), but I haven't checked to see if any solutions work on Mac OS X (I always assumed that the game recreates cache/data if it doesn't exist, but maybe not).
« Last Edit: May 22, 2006, 12:30:14 AM by devSin »

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #36 on: May 22, 2006, 12:44:38 AM »
OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)
You'd want OSSwapConstInt32(x) and OSSwapConstInt16(x). Mac OS X has macros to swap to little or big endian (generic macros and inline assembly), and limited support for pdp, as well as the posix standard (nh)to(hn)(sl). I use OSSwap* when I'm stealing code from others. :D
Well, since the only big-endian architecture we have to worry about is the Mac, then I'll go ahead and make that change.  I guess the [nh]to[hn][sl] routines can't be used in this case since "net" is always big-endian (and "host" is native).  Now a PDP port of WeiDU, that could be interesting :)

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #37 on: May 22, 2006, 12:49:02 AM »
The only thing left that you may want to do is #ifdef __ppc64__ (I compile the Mac OS X releases on a frumpy G4, though, so no difference to me). I wish I knew for sure if 64-bit PPC also defined __ppc__, sorry.

I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)

Offline devSin

  • Planewalker
  • *****
  • Posts: 1632
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #38 on: May 22, 2006, 12:52:20 AM »
OSSwapInt32?  I probably should've used that instead of rolling my own.  I'm glad to see at least one vendor is including standard byte-swapping macros :)
You'd want OSSwapConstInt32(x) and OSSwapConstInt16(x). Mac OS X has macros to swap to little or big endian (generic macros and inline assembly), and limited support for pdp, as well as the posix standard (nh)to(hn)(sl). I use OSSwap* when I'm stealing code from others. :D
Well, since the only big-endian architecture we have to worry about is the Mac, then I'll go ahead and make that change.  I guess the [nh]to[hn][sl] routines can't be used in this case since "net" is always big-endian (and "host" is native).  Now a PDP port of WeiDU, that could be interesting :)
Yes. The order of bytes shall be influenced by the phases of the moon. ;-)

Anyway, if there's ever cause for a Sparc port or somesuch, it'll be easy enough to roll back out into zlib.c.

Quote
I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)
I don't want to look at the tech docs, but I'm fairly sure that Apple's recommendation is that __ppc__ should not be defined on 64-bit systems.

EDIT: What am I thinking? WeiDU will never be compiled as a 64-bit executable. :-/

Yeah, __ppc__ should be sufficient.
« Last Edit: May 22, 2006, 12:57:28 AM by devSin »

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #39 on: May 22, 2006, 08:55:04 AM »
Quote
I think it's safe to say that __ppc__ is defined on all Mac's unless you're running an Intel Mac.  I think it's equivalent to __x86__ (even Cygwin/GCC on my dual-core Athalon X2 defines this.)
I don't want to look at the tech docs, but I'm fairly sure that Apple's recommendation is that __ppc__ should not be defined on 64-bit systems.

EDIT: What am I thinking? WeiDU will never be compiled as a 64-bit executable. :-/

Yeah, __ppc__ should be sufficient.
Yes, you're right on both counts!  I'll do the right thing and put in both macros anyway.  I'll post another patch here shortly :)

EDIT: I just realized that it's possible that WeiDU will get ported to more OS's with GemRB.  So I'll play it safe and try to do the right thing :)
« Last Edit: May 22, 2006, 09:04:16 AM by FredSRichardson »

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #40 on: May 22, 2006, 09:27:59 AM »
Not to be lazy, but, uh, did you update the diff?  :)
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #41 on: May 22, 2006, 09:32:38 AM »
Not to be lazy, but, uh, did you update the diff?  :)
I was just getting to that :)

Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):

Code: [Select]
diff -u -r ./WeiDU-192.orig/src/load.ml ./WeiDU-192/src/load.ml
--- ./WeiDU-192.orig/src/load.ml 2006-04-10 16:58:53.000000000 -0400
+++ ./WeiDU-192/src/load.ml 2006-05-21 12:17:58.171875000 -0400
@@ -314,24 +314,41 @@
 
 let skip_next_load_error = ref false
 
+external cbf2bif : string -> string -> int
+    = "mlgz_cbf2bif"
+
 let load_bif_in_game game bif_file =
     if Hashtbl.mem game.loaded_biffs bif_file then
       Hashtbl.find game.loaded_biffs bif_file (* already here *)
     else begin
       (* we must load the BIF *)
-      let biff_path =
-        let rec trial lst =
+      let biff_path = begin
+        let rec trial f lst =
           match lst with
-            [] -> find_file_in_path game.game_path bif_file
+            [] -> find_file_in_path game.game_path f
           | hd :: tl ->
-            let perhaps = find_file_in_path hd bif_file in
-            log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
-            if file_exists perhaps then
-              perhaps
-            else trial tl
+              let perhaps = find_file_in_path hd f in
+              log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
+              if file_exists perhaps then
+                perhaps
+              else trial f tl
         in
-        trial (game.cd_path_list)
-      in
+        (* Check to see if the bif file exists, if it doesn't try for a .CBF file *)
+        let bf = trial bif_file (game.cd_path_list @ [ game.game_path ^ "/cache" ] ) in
+        if file_exists bf then
+          bf
+        else begin
+          let cbf = Filename.chop_extension bif_file ^ ".cbf" in
+          let cbf_file = trial cbf (game.cd_path_list) in
+          if file_exists cbf_file then
+            let cache_file = game.game_path ^ "/cache/" ^ bif_file in
+            let sz = cbf2bif cbf_file cache_file in
+            let _ = log_and_print "[%s] decompressed bif file %d bytes\n" cbf_file sz in
+            cache_file
+          else
+            bf
+        end
+      end in
       let the_biff = Biff.load_biff biff_path in
       Hashtbl.add game.loaded_biffs bif_file the_biff ;
       the_biff
diff -u -r ./WeiDU-192.orig/zlib/zlib.c ./WeiDU-192/zlib/zlib.c
--- ./WeiDU-192.orig/zlib/zlib.c 2003-06-02 06:08:24.000000000 -0400
+++ ./WeiDU-192/zlib/zlib.c 2006-05-22 10:16:54.250000000 -0400
@@ -85,3 +85,221 @@
    */
   return v_ret ;
 }
+
+/* zerr() and def() are copied directly from zlib example code. */
+
+#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
+#  include <fcntl.h>
+#  include <io.h>
+#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+#else
+#  define SET_BINARY_MODE(file)
+#endif
+
+#define CHUNK 16384
+
+
+/* Raise an exception for a zlib or i/o error */
+void mlgz_zerr(int ret)
+{
+    switch (ret) {
+    case Z_ERRNO:
+        raise_sys_error(copy_string(strerror(errno))) ;
+        break;
+    case Z_STREAM_ERROR:
+        raise_mlgz_exn("invalid compression level");
+        break;
+    case Z_DATA_ERROR:
+        raise_mlgz_exn("invalid or incomplete deflate data");
+        break;
+    case Z_MEM_ERROR:
+        raise_out_of_memory() ;
+        break;
+    case Z_VERSION_ERROR:
+        raise_mlgz_exn("zlib version mismatch!");
+    }
+}
+
+/* Yes, this is bad practice.  I compensated by using "%.256s" instead of just "%s": */
+static char errstr[1024];
+
+/* This is a bit better for error checking: */
+int fread_check(void* b, size_t sz, size_t cnt, FILE* fp, const char* fn)
+{
+    if (fread(b, sz, cnt, fp) != cnt) {
+        sprintf(errstr, "Failed to read %d bytes from file %.256s", sz*cnt, fn);
+        raise_mlgz_exn(errstr);
+        return 1;
+    }
+    return 0;
+}
+
+int fread_uint(uint32_t* i, FILE* fp, const char* fn)
+{
+    if (fread_check(&i, 4, 1, fp, fn))
+        return 1;
+#if defined(__ppc__) || defined(__ppc64__)
+    *i = OSSwapInt32(*i);
+#endif
+    return 0;
+}
+
+
+/* Decompress from file source to file dest until stream ends or EOF.
+   inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
+   allocated for processing, Z_DATA_ERROR if the deflate data is
+   invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
+   the version of the library linked do not match, or Z_ERRNO if there
+   is an error reading or writing the files. */
+int inf(FILE *source, FILE *dest)
+{
+    int ret;
+    unsigned have;
+    z_stream strm;
+    unsigned char in[CHUNK];
+    unsigned char out[CHUNK];
+   
+    /* allocate inflate state */
+    strm.zalloc = Z_NULL;
+    strm.zfree = Z_NULL;
+    strm.opaque = Z_NULL;
+    strm.avail_in = 0;
+    strm.next_in = Z_NULL;
+    ret = inflateInit(&strm);
+    if (ret != Z_OK)
+        return ret;
+    /* decompress until deflate stream ends or end of file */
+    do {
+        strm.avail_in = fread(in, 1, CHUNK, source);
+        if (ferror(source)) {
+            (void)inflateEnd(&strm);
+            return Z_ERRNO;
+        }
+        if (strm.avail_in == 0)
+            break;
+        strm.next_in = in;
+        /* run inflate() on input until output buffer not full */
+        do {
+            strm.avail_out = CHUNK;
+            strm.next_out = out;
+            ret = inflate(&strm, Z_NO_FLUSH);
+            /* assert(ret != Z_STREAM_ERROR); */  /* state not clobbered */ /* no asserts for Ocaml */
+            switch (ret) {
+            case Z_NEED_DICT:
+                ret = Z_DATA_ERROR;     /* and fall through */
+            case Z_DATA_ERROR:
+            case Z_MEM_ERROR:
+                (void)inflateEnd(&strm);
+                return ret;
+            }
+            have = CHUNK - strm.avail_out;
+            if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
+                (void)inflateEnd(&strm);
+                return Z_ERRNO;
+            }
+        } while (strm.avail_out == 0);
+        /* done when inflate() says it's done */
+    } while (ret != Z_STREAM_END);
+    /* clean up and return */
+    (void)inflateEnd(&strm);
+    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
+}
+
+/*
+   Unompresses a CBF files to a specified BIF file.  Return 0 on failure or the
+   number of uncompressed bytes on success.
+ */
+value mlgz_cbf2bif(value _cbf_file, value _bif_file)
+{
+    const char* cbf_file = String_val(_cbf_file);
+    const char* bif_file = String_val(_bif_file);
+    FILE *cbf_fp;
+    FILE *bif_fp;
+    char sigver[9];
+    uint32_t bif_file_len, cmplen, uncmplen;
+    int zret;
+   
+
+    if (!(cbf_fp = fopen(cbf_file, "rb"))) {
+        sprintf(errstr, "failure opening file: %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    if (!(bif_fp = fopen(bif_file, "wb"))) {
+        fclose(cbf_fp);
+        sprintf(errstr, "failure opening file: %.256s", bif_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    sigver[8] = 0;
+    if (fread_check(sigver, 1, 8, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (strcmp(sigver, "BIF V1.0")) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "incorrect CBF header for file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    if (fread_uint(&bif_file_len, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (bif_file_len <=0 || bif_file_len > 128) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "corrupt CBF file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    /* Seek ahead past embedded file name, doesn't really matter what it is */
+    if (fseek(cbf_fp, bif_file_len, SEEK_CUR)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        sprintf(errstr, "failure seeking %d bytes into file %.256s", bif_file_len, cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+
+    if (fread_uint(&uncmplen, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+   
+    if (fread_uint(&cmplen, cbf_fp, cbf_file)) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        return Val_int(0);
+    }
+    /* printf("CBF %s (%ld bytes) -> BIF %s [%ld bytes]", cbf_file, cmplen, bif_file, uncmplen); */
+
+    if ((zret=inf(cbf_fp, bif_fp)) != Z_OK) {
+        fclose(cbf_fp);
+        fclose(bif_fp);
+        mlgz_zerr(zret);
+        return Val_int(0);
+    }
+
+    if (fclose(cbf_fp)) {
+        fclose(bif_fp);
+        sprintf(errstr, "failure closing file %.256s", cbf_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    if (fclose(bif_fp)) {
+        sprintf(errstr, "failure closing file %.256s", bif_file);
+        raise_mlgz_exn(errstr);
+        return Val_int(0);
+    }
+    return Val_int(uncmplen);
+}

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #42 on: May 22, 2006, 09:39:58 AM »
Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):
Ok, thanks. The cache cleanup code should be straightforward - would you prefer me to code it?
BTW, since I saw that mentioned, you shouldn't do
Code: [Select]
file_exists (String.lowercase (Arch.backslash_to_shash file))
[..]
open_in (String.lowercase (Arch.backslash_to_shash file))
etc.
but use the Case_ins module instead:
Code: [Select]
file_exists file (* already case_insed by Util *)
[..]
Case_ins.perv_open_in file;
etc.
Dev specifically requested that OSX doesn't get the lowercasing  ;)
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #43 on: May 22, 2006, 09:51:48 AM »
Still no cleanup code, and the load.ml patch is the same.  This just adds MacOS big-endian support (and a few minor cleanups):
Ok, thanks. The cache cleanup code should be straightforward - would you prefer me to code it?
BTW, since I saw that mentioned, you shouldn't do
Code: [Select]
file_exists (String.lowercase (Arch.backslash_to_shash file))
[..]
open_in (String.lowercase (Arch.backslash_to_shash file))
etc.
but use the Case_ins module instead:
Code: [Select]
file_exists file (* already case_insed by Util *)
[..]
Case_ins.perv_open_in file;
etc.
Dev specifically requested that OSX doesn't get the lowercasing  ;)

If you don't mind handling the cache cleanup coding that would be great.  I kind of lost interest when I realized this wasn't quite going to solve my problem with IWD1Tutu (long story).

Do you think it would be a good idea to have a sys_file_exists in case_ins*.ml?  I didn't include this small patch, but this seems like a logical addition:
Code: [Select]
diff -u -r ./WeiDU-192.orig/src/case_ins_linux.ml ./WeiDU-192/src/case_ins_linux.ml
--- ./WeiDU-192.orig/src/case_ins_linux.ml 2006-03-18 19:30:18.000000000 -0500
+++ ./WeiDU-192/src/case_ins_linux.ml 2006-05-20 11:29:09.000000000 -0400
@@ -16,3 +16,5 @@
 let unix_unlink s = Unix.unlink (String.lowercase (backslash_to_slash s)) ;;
 let unix_mkdir s p = Unix.mkdir (String.lowercase (backslash_to_slash s)) p ;;
 let unix_opendir s = Unix.opendir (String.lowercase (backslash_to_slash s)) ;;
+
+let sys_file_exists s = Sys.file_exists (String.lowercase (backslash_to_slash s)) ;;
diff -u -r ./WeiDU-192.orig/src/case_ins_mac.ml ./WeiDU-192/src/case_ins_mac.ml
--- ./WeiDU-192.orig/src/case_ins_mac.ml 2006-03-18 19:30:34.000000000 -0500
+++ ./WeiDU-192/src/case_ins_mac.ml 2006-05-20 11:29:33.187500000 -0400
@@ -16,3 +16,5 @@
 let unix_unlink s = Unix.unlink (backslash_to_slash s) ;;
 let unix_mkdir s p = Unix.mkdir (backslash_to_slash s) p ;;
 let unix_opendir s = Unix.opendir (backslash_to_slash s) ;;
+
+let sys_file_exists s = Sys.file_exists (backslash_to_slash s) ;;
diff -u -r ./WeiDU-192.orig/src/case_ins_win.ml ./WeiDU-192/src/case_ins_win.ml
--- ./WeiDU-192.orig/src/case_ins_win.ml 2006-04-10 17:01:05.000000000 -0400
+++ ./WeiDU-192/src/case_ins_win.ml 2006-05-20 11:28:37.218750000 -0400
@@ -12,3 +12,5 @@
 let unix_unlink s = Unix.unlink s ;;
 let unix_mkdir s p = Unix.mkdir s p ;;
 let unix_opendir s = Unix.opendir s ;;
+
+let sys_file_exists s = Sys.file_exists s ;;

file_exists in util.ml doesn't take any measures to transform the file name (maybe it's important to use the "file_exists and has non-zero bytes" but I can't see why):
Code: [Select]
let file_exists name = (file_size name >= 0)

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #44 on: May 22, 2006, 09:57:37 AM »
Well, file_size is

Code: [Select]
let file_size name =
  try
    let stats = Case_ins.unix_stat name in
    stats.Unix.st_size
  with _ ->  -1

So it's already Case_insed. As for the >=0, it was added by Wes because _ and is't needed by a host of already existing mods, so :(
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #45 on: May 22, 2006, 10:04:55 AM »
Well, file_size is

Code: [Select]
let file_size name =
  try
    let stats = Case_ins.unix_stat name in
    stats.Unix.st_size
  with _ ->  -1

So it's already Case_insed. As for the >=0, it was added by Wes because _ and is't needed by a host of already existing mods, so :(
Yes, I see your point there  and I should've read a bit more of the code. :)

And there's really no reason to fix what's not broken so I'll drop the topic.  Having a test for "exists and is non-zero bytes" is probably safer anway (since a zero-byte file is almost always going to be corrupt for this program).

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #46 on: May 23, 2006, 11:59:10 AM »
WARNING I've found at least one bug in the old patch (also, the exception handling was broken in the old compress code).  I'll get a new patch out soon.

Offline FredSRichardson

  • Planewalker
  • *****
  • Posts: 190
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #47 on: May 23, 2006, 12:43:41 PM »
Okay, this should work a bit better.  If you have IWD1, you can test it with:
Code: [Select]
> ./weidu.asm.exe --biff-get AR3401.WED
[C:\cygwin\usr\src\WeiDU-192.tmp\weidu.asm.exe] WeiDU version 192
[C:\Program Files\Black Isle\Icewind Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
[C:\Program Files\Black Isle\Icewind Dale/dialog.tlk] 34502 string entries
[C:\cygwin\usr\src\WeiDU-192.tmp\weidu.asm.exe] Using scripting style "IWD1"
[C:\Program Files\Black Isle\Icewind Dale\CD2\/data/AR3401.cbf] decompressed bif file 2095712 bytes
[C:\Program Files\Black Isle\Icewind Dale/cache/data/AR3401.bif] 2095712 bytes, 6 files, 1 tilesets
[./AR3401.WED] created from [C:\Program Files\Black Isle\Icewind Dale/data/AR3401.bif]
>

You can see that it reports decompressing the CBF files to the Cache directory.  Note that it will only do this on an operation that grabs resources from CBF files.  I noticed that 'weidu --biff <bif-file>' is a bit broken.  It doesn't report an error if it can't find <bif-file> in the game keys, and <bif-file> must match the key entry exactly.  Here's an xample:
Code: [Select]
> ./weidu.asm.exe --biff 'data\AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
1.TIS at index 1
1.WED at index 0
T.BMP at index 1
M.BMP at index 2
R.BMP at index 3
1.MOS at index 4
A.WAV at index 5
> ./weidu.asm.exe --biff 'data/AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
> ./weidu.asm.exe --biff 'AR3401.bif'
eidu.asm.exe] WeiDU version 192
nd Dale/CHITIN.KEY] 268 BIFFs, 19992 resources
nd Dale/dialog.tlk] 34502 string entries
eidu.asm.exe] Using scripting style "IWD1"
>

Fixing this is a bit of a pain.  I only cared because I was looking for examples to test this new patch with.

Here's that patch:

Code: [Select]
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/Makefile ./WeiDU-192.tmp/Makefile
--- ./WeiDU-192.orig/Makefile 2006-04-24 15:14:58.000000000 -0400
+++ ./WeiDU-192/Makefile 2006-05-23 13:34:23.950168600 -0400
@@ -135,7 +135,7 @@
         baflexer bafparser \
         baflexer_old bafparser_old \
         diff tp dlexer dparser \
-        automate kit
+        automate kit cbif
 ifdef ITEMLIST
 WEIDU_BASE_MODULES  += pretty itemlist
 endif
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/biff.ml ./WeiDU-192/src/biff.ml
--- ./WeiDU-192.orig/src/biff.ml 2006-03-30 16:42:25.000000000 -0500
+++ ./WeiDU-192/src/biff.ml 2006-05-23 13:26:48.106418600 -0400
@@ -9,6 +9,7 @@
 (* Infinity Engine [BIF] *)
 open Util
 open Key
+open Cbif
 
 type biff_file = {
   res_loc       : int ;
@@ -204,10 +205,6 @@
   }
 end
 
-(* comment this out if you don't have zlib *)
-external uncompress : string -> pos:int -> clen:int -> ulen: int -> string
-  = "mlgz_uncompress"
-
 (* reads 'size' bytes that would start at location 'start' in this BIFF
  * if it were not compressed! *)
 let read_compressed_biff_internal fd filename start size chunk_fun =
@@ -215,7 +212,7 @@
   let unc_offset = ref 0 in
 
   (* buffer holds the uncompressed bytes [start_unc,end_unc]  *)
-  let result = Buffer.create size in
+  let (*result*) _ = Buffer.create size in
   let start_unc_offset = ref 0 in
   let end_unc_offset = ref 0 in
 
@@ -248,7 +245,7 @@
       let _ = Unix.lseek fd (!cmp_offset+8) Unix.SEEK_SET in
       let cmp_buff = String.create cmplen in
       my_read cmplen fd cmp_buff filename ;
-      let uncmp = uncompress cmp_buff 0 cmplen uncmplen in
+      let uncmp = Cbif.uncompress cmp_buff 0 cmplen uncmplen in
       (*
       if (String.length uncmp <> uncmplen) then begin
         log_and_print "ERROR: [%s] chunk at offset %d was supposed to have %d bytes of compressed data that expanded to %d, but in reality they expanded to %d"
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/cbif.ml ./WeiDU-192/src/cbif.ml
--- ./WeiDU-192.orig/src/cbif.ml 1969-12-31 19:00:00.000000000 -0500
+++ ./WeiDU-192/src/cbif.ml 2006-05-23 13:26:48.122043600 -0400
@@ -0,0 +1,12 @@
+(* Decompression routines for compressed bif files *)
+
+exception Error of string
+
+let _ = Callback.register_exception "mlgz_exn" (Error "")
+
+external cbf2bif : string -> string -> int
+    = "mlgz_cbf2bif"
+
+external uncompress : string -> pos:int -> clen:int -> ulen: int -> string
+  = "mlgz_uncompress"
+
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/src/load.ml ./WeiDU-192/src/load.ml
--- ./WeiDU-192.orig/src/load.ml 2006-04-10 16:58:53.000000000 -0400
+++ ./WeiDU-192/src/load.ml 2006-05-23 13:26:48.122043600 -0400
@@ -8,6 +8,7 @@
 It was originally taken from Westley Weimer's WeiDU 185. *)
 
 open Util
+open Cbif
 
 let registry_game_paths () =
   let str_list = "." :: !Arch.registry_paths in
@@ -319,19 +320,33 @@
       Hashtbl.find game.loaded_biffs bif_file (* already here *)
     else begin
       (* we must load the BIF *)
-      let biff_path =
-        let rec trial lst =
+      let biff_path = begin
+        let rec trial f lst =
           match lst with
-            [] -> find_file_in_path game.game_path bif_file
+            [] -> find_file_in_path game.game_path f
           | hd :: tl ->
-            let perhaps = find_file_in_path hd bif_file in
+              let perhaps = find_file_in_path hd f in
             log_only "BIFF may be in hard-drive CD-path [%s]\n" perhaps ;
             if file_exists perhaps then
               perhaps
-            else trial tl
-        in
-        trial (game.cd_path_list)
+              else trial f tl
       in
+        (* Check to see if the bif file exists, if it doesn't try for a .CBF file *)
+        let bf = trial bif_file (game.cd_path_list @ [ game.game_path ^ "/cache" ] ) in
+        if file_exists bf then
+          bf
+        else begin
+          let cbf = Filename.chop_extension bif_file ^ ".cbf" in
+          let cbf_file = trial cbf (game.cd_path_list) in
+          if file_exists cbf_file then
+            let cache_file = game.game_path ^ "/cache/" ^ bif_file in
+            let sz = Cbif.cbf2bif cbf_file cache_file in
+            let _ = log_and_print "[%s] decompressed bif file %d bytes\n" cbf_file sz in
+            cache_file
+          else
+            bf
+        end
+      end in
       let the_biff = Biff.load_biff biff_path in
       Hashtbl.add game.loaded_biffs bif_file the_biff ;
       the_biff
diff -b -w -x '*.exe' -x obj -x '*.txt' -x 'iw12*' -x '*~' -x  -Naur ./WeiDU-192.orig/zlib/zlib.c ./WeiDU-192/zlib/zlib.c
--- ./WeiDU-192.orig/zlib/zlib.c 2003-06-02 06:08:24.000000000 -0400
+++ ./WeiDU-192/zlib/zlib.c 2006-05-23 13:26:48.122043600 -0400
@@ -19,12 +19,13 @@
   static value * exn = NULL;
   if(exn == NULL)
     exn = caml_named_value ("mlgz_exn");
-  raise_with_string(*exn, (char *)msg) ;
+  caml_raise_with_string(*exn, (char *)msg) ;
 }
 
-value mlgz_uncompress(value v_src, value v_pos, value v_len, value unc_len)
+CAMLprim value mlgz_uncompress(value v_src, value v_pos, value v_len, value unc_len)
 {
-  value v_ret;
+  CAMLparam4(v_src, v_pos, v_len, unc_len);
+  CAMLlocal1(v_ret);
   int level, pos, len, out_buf_len, r;
   uLong out_len;
   const char *in_buf;
@@ -42,7 +43,7 @@
    out_buf = malloc(out_buf_len);
    */
 
-  v_ret = alloc_string(Int_val(unc_len));
+  v_ret = caml_alloc_string(Int_val(unc_len));
   out_buf = String_val(v_ret);
 
   if(out_buf == NULL)
@@ -55,7 +56,6 @@
     } else if(r == Z_BUF_ERROR) {
       char *new_buf;
 
-      printf("uncompress 1\n"); fflush(stdout);
       raise_mlgz_exn("uncompress");
       out_buf_len *= 2;
       new_buf = realloc(out_buf, out_buf_len);
@@ -65,11 +65,9 @@
       }
       out_buf = new_buf;
     } else if(r == Z_MEM_ERROR) {
-      printf("uncompress 3\n"); fflush(stdout);
       free(out_buf);
       raise_out_of_memory();
     } else {
-      printf("WeiDU: ZLIB: Warning! Problem decompressing block.\n");
       fflush(stdout);
       out_len = Int_val(unc_len);
       break;
@@ -83,5 +81,224 @@
    memcpy(String_val(v_ret), out_buf, out_len);
    free(out_buf);
    */
-  return v_ret ;
+  CAMLreturn (v_ret) ;
+}
+
+/* zerr() and def() are copied directly from zlib example code. */
+
+#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(__CYGWIN__)
+#  include <fcntl.h>
+#  include <io.h>
+#  define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+#else
+#  define SET_BINARY_MODE(file)
+#endif
+
+#define CHUNK 16384
+
+
+/* Raise an exception for a zlib or i/o error */
+void mlgz_zerr(int ret)
+{
+    switch (ret) {
+    case Z_ERRNO:
+        raise_sys_error(copy_string(strerror(errno))) ;
+        break;
+    case Z_STREAM_ERROR:
+        raise_mlgz_exn("invalid compression level");
+        break;
+    case Z_DATA_ERROR:
+        raise_mlgz_exn("invalid or incomplete deflate data");
+        break;
+    case Z_MEM_ERROR:
+        raise_out_of_memory() ;
+        break;
+    case Z_VERSION_ERROR:
+        raise_mlgz_exn("zlib version mismatch!");
+    }
+}
+
+/* Yes, this is bad practice.  I compensated by using "%.256s" instead of just "%s": */
+static char errstr[1024];
+
+/* This is a bit better for error checking: */
+int fread_check(void* b, size_t sz, size_t cnt, FILE* fp, const char* fn)
+{
+    if (fread(b, sz, cnt, fp) != cnt) {
+        sprintf(errstr, "Failed to read %d bytes from file %.256s", sz*cnt, fn);
+        raise_mlgz_exn(errstr);
+        return 1;
+    }
+    return 0;
+}
+
+int fread_uint(uint32_t* i, FILE* fp, const char* fn)
+{
+    if (fread_check(i, 4, 1, fp, fn))
+        return 1;
+#if defined(__ppc__) || defined(__ppc64__)
+    *i = OSSwapInt32(*i);
+#endif
+    return 0;
+}
+
+
+/* Decompress from file source to file dest until stream ends or EOF.
+   inf() returns Z_OK on success, Z_MEM_ERROR if memory could not be
+   allocated for processing, Z_DATA_ERROR if the deflate data is
+   invalid or incomplete, Z_VERSION_ERROR if the version of zlib.h and
+   the version of the library linked do not match, or Z_ERRNO if there
+   is an error reading or writing the files. */
+int inf(FILE *source, FILE *dest)
+{
+    int ret;
+    unsigned have;
+    z_stream strm;
+    unsigned char in[CHUNK];
+    unsigned char out[CHUNK];
+   
+    /* allocate inflate state */
+    strm.zalloc = Z_NULL;
+    strm.zfree = Z_NULL;
+    strm.opaque = Z_NULL;
+    strm.avail_in = 0;
+    strm.next_in = Z_NULL;
+    ret = inflateInit(&strm);
+    if (ret != Z_OK)
+        return ret;
+    /* decompress until deflate stream ends or end of file */
+    do {
+        strm.avail_in = fread(in, 1, CHUNK, source);
+        if (ferror(source)) {
+            (void)inflateEnd(&strm);
+            return Z_ERRNO;
+        }
+        if (strm.avail_in == 0)
+            break;
+        strm.next_in = in;
+        /* run inflate() on input until output buffer not full */
+        do {
+            strm.avail_out = CHUNK;
+            strm.next_out = out;
+            ret = inflate(&strm, Z_NO_FLUSH);
+            /* assert(ret != Z_STREAM_ERROR); */  /* state not clobbered */ /* no asserts for Ocaml */
+            switch (ret) {
+            case Z_NEED_DICT:
+                ret = Z_DATA_ERROR;     /* and fall through */
+            case Z_DATA_ERROR:
+            case Z_MEM_ERROR:
+                (void)inflateEnd(&strm);
+                return ret;
+            }
+            have = CHUNK - strm.avail_out;
+            if (fwrite(out, 1, have, dest) != have || ferror(dest)) {
+                (void)inflateEnd(&strm);
+                return Z_ERRNO;
+            }
+        } while (strm.avail_out == 0);
+        /* done when inflate() says it's done */
+    } while (ret != Z_STREAM_END);
+    /* clean up and return */
+    (void)inflateEnd(&strm);
+    return ret == Z_STREAM_END ? Z_OK : Z_DATA_ERROR;
+}
+
+/*
+   Unompresses a CBF files to a specified BIF file.  Return 0 on failure or the
+   number of uncompressed bytes on success.
+ */
+CAMLprim value mlgz_cbf2bif(value _cbf_file, value _bif_file)
+{
+  CAMLparam2(_cbf_file, _bif_file);
+  const char* cbf_file = String_val(_cbf_file);
+  const char* bif_file = String_val(_bif_file);
+  FILE *cbf_fp;
+  FILE *bif_fp;
+  char sigver[9];
+  uint32_t bif_file_len, cmplen, uncmplen;
+  int zret;
+   
+  if (!(cbf_fp = fopen(cbf_file, "rb"))) {
+    sprintf(errstr, "failure opening file: %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (!(bif_fp = fopen(bif_file, "wb"))) {
+    fclose(cbf_fp);
+    sprintf(errstr, "failure opening file: %.256s", bif_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  sigver[8] = 0;
+  if (fread_check(sigver, 1, 8, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (strcmp(sigver, "BIF V1.0")) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "incorrect CBF header for file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fread_uint(&bif_file_len, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (bif_file_len <=0 || bif_file_len > 128) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "corrupt CBF file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  /* Seek ahead past embedded file name, doesn't really matter what it is */
+  if (fseek(cbf_fp, bif_file_len, SEEK_CUR)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    sprintf(errstr, "failure seeking %d bytes into file %.256s", bif_file_len, cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fread_uint(&uncmplen, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+   
+  if (fread_uint(&cmplen, cbf_fp, cbf_file)) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    CAMLreturn(Val_int(0));
+  }
+  /* printf("CBF %s (%ld bytes) -> BIF %s [%ld bytes]", cbf_file, cmplen, bif_file, uncmplen); */
+
+  if ((zret=inf(cbf_fp, bif_fp)) != Z_OK) {
+    fclose(cbf_fp);
+    fclose(bif_fp);
+    mlgz_zerr(zret);
+    CAMLreturn(Val_int(0));
+  }
+
+  if (fclose(cbf_fp)) {
+    fclose(bif_fp);
+    sprintf(errstr, "failure closing file %.256s", cbf_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+  if (fclose(bif_fp)) {
+    sprintf(errstr, "failure closing file %.256s", bif_file);
+    raise_mlgz_exn(errstr);
+    CAMLreturn(Val_int(0));
+  }
+  CAMLreturn(Val_int(uncmplen));
 }

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #48 on: May 23, 2006, 01:16:33 PM »
Patch applied.
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: iwd1 compressed bif files (cbf)
« Reply #49 on: May 23, 2006, 01:36:09 PM »
Patch for cleanup is attached. Assumes your latest cbif patch is already applied.
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: