Author Topic: Manipulating sequences and frames within .bams  (Read 1361 times)

Offline Galactygon

  • Modding since 2002
  • Planewalker
  • *****
  • Posts: 378
  • Gender: Male
  • Creator of spells
Manipulating sequences and frames within .bams
« on: November 26, 2018, 03:41:41 PM »
Are there any existing .bam manipulation macros or functions? It might sound a bit crazy, but I am looking into adding and removing existing frames to sequences as well as adding and removing sequences themselves.

My method is as follows:
(1) generate an array of sequences in $BAM_V10_CYCLES_FRAMES as $BAM_V10_CYCLES_FRAMES("0") for sequence0, and so on until $BAM_V10_CYCLES_FRAMES("n") for sequence n.
(2) The entries within the array are strings in the form of frame numbers separated by commas i.e. if $BAM_V10_CYCLES_FRAMES("0") is set to "0,1,2" then that means the first sequence uses frames 0, 1, and 2.
(3) If I wanted to manipulate the frames within the sequences, I would have to modify the strings within the 1d array. i.e. by INNER_PATCH_SAVE and REPLACE_TEXTUALLY.

So I've got 2 macros (not functions). One to generate $BAM_V10_CYCLES_FRAMES, and the other to apply $BAM_V10_CYCLES_FRAMES back into the .bam file.

All works fine until I add or delete frames in a sequence. I haven't even tried adding new sequences yet. I get corrupted .bams whenever I try changing the size of the frame lookup table and all the offsets of the frame data, which is the only thing occouring after the frame lookup table.

I've tried saving a copy of the .bam with the extra frame added in DLTCEP to see how they compare in a hex editor, but the offsets for the frame data seem to be updated correctly, as well as the cycle->frame table. So I'm really, really stumped.

I'll post the code here:
Quote

//\\ Get cycle and frame data of a .bam file
// cycles are stored as strings within $BAM_V10_CYCLES_FRAMES
// as $BAM_V10_CYCLES_FRAMES("0") for cycle 0 and so on until $BAM_V10_CYCLES_FRAMES("n") for cycle n
// strings within the array are stored in the form of frame0,frame1,...,framen
DEFINE_PATCH_MACRO "GET_BAM_V10_CYCLES_FRAMES" BEGIN
   PATCH_IF ("%SOURCE_FILE%" STRING_MATCHES_REGEXP "^.+\.bam" = 0 OR "%SOURCE_FILE%" STRING_MATCHES_REGEXP "^.+\.bamc" = 0) BEGIN
      READ_ASCII 0x0 sg (4) //Signature
      PATCH_IF (~%sg%~ STRING_EQUAL_CASE ~BAMC~ = 1) BEGIN
         READ_LONG 0x8 dl //Uncompressed data length
         DECOMPRESS_REPLACE_FILE 0xc (SOURCE_SIZE - 0xc) dl
      END ELSE BEGIN
         PATCH_PRINT ~%SOURCE_FILE% is not a compressed BAM.~
      END
      // frames and cycles number
      READ_SHORT 0x0008 frames_num
      READ_BYTE 0x000a cycles_num
      // Get frames offset
      READ_LONG 0x000c frames_off
      // Get cycles offset
      SET cycles_off = (LONG_AT (0x000c) + frames_num * 12)
      // Get frame lookup offset (frames in cycles)
      READ_LONG 0x0014 framelookup_off           // 00,01,02,03
      // Get frame data offset
      SET framedata_off = (LONG_AT (frames_off + 0x8) BAND (BNOT BIT31)) // disable compressed bit
      // Reset array
      PATCH_CLEAR_ARRAY BAM_V10_CYCLES_FRAMES
      // Build array of cycles with number of frames in cycle for each cycle entry
      FOR (cycle_index = 0; cycle_index < cycles_num; cycle_index += 1) BEGIN // Generate cycles & number of frames
         READ_SHORT (cycles_off + cycle_index * 4) $BAM_V10_CYCLES_FRAMES("%cycle_index%") // look at number of frames for current cycle
      END
      // Replace number of frames in each cycle entry with string containing frames
      // in the form of "framenumber:frame0,frame1,frame2,..."
      SET cycle_index = 0
      SET current_frame_in_cycle = 0
      SPRINT cycles_frames ""
                FOR (at_framelookup_off = framelookup_off; at_framelookup_off < framedata_off; at_framelookup_off += 2) BEGIN
         READ_SHORT at_framelookup_off current_frame // get current frame from frame lookup table
         // Reset counter of current frame in cycle if index jumps a cycle
         PATCH_IF current_frame_in_cycle > $BAM_V10_CYCLES_FRAMES("%cycle_index%") - 1 BEGIN
            SET current_frame_in_cycle = 0 // reset current frame count
            INNER_PATCH_SAVE cycles_frames "%cycles_frames%" BEGIN // Remove comma in beginning of string
               REPLACE_TEXTUALLY "^," ""
            END
            SPRINT $BAM_V10_CYCLES_FRAMES("%cycle_index%") "%cycles_frames%" // store frames string for previous cycle
            SPRINT cycles_frames "" // reset frames string for next cycle
            SET cycle_index += 1 // increment cycle index
         END
         SPRINT cycles_frames "%cycles_frames%,%current_frame%" // add current frame to frames string
         SET current_frame_in_cycle += 1
                END
                // Store frames of last cycle
      INNER_PATCH_SAVE cycles_frames "%cycles_frames%" BEGIN // Remove comma in beginning of string
         REPLACE_TEXTUALLY "^," ""
      END
      SPRINT $BAM_V10_CYCLES_FRAMES("%cycle_index%") "%cycles_frames%"
   END
END

//\\ Apply cycle and frame data of a .bam file
// Writes cycle information into the .bam file from array $BAM_V10_CYCLES_FRAMES
DEFINE_PATCH_MACRO "APPLY_BAM_V10_CYCLES_FRAMES" BEGIN
   PATCH_IF ("%SOURCE_FILE%" STRING_MATCHES_REGEXP "^.+\.bam" = 0 OR "%SOURCE_FILE%" STRING_MATCHES_REGEXP "^.+\.bamc" = 0) BEGIN
      // get frames and cycles number
      READ_SHORT 0x0008 frames_num
      READ_BYTE 0x000a old_cycles_num
      // Get frames offset
      READ_LONG 0x000c old_frames_off
      // Get cycles offset
      SET old_cycles_off = (LONG_AT (0x000c) + frames_num * 12)
      // Get palette offset
      READ_LONG 0x0010 old_palette_off
      // Get frame data offset of first frame
      SET old_framedata_off = (LONG_AT (frames_off + 0x8) BAND (BNOT BIT31)) // disable compressed bit

      READ_LONG 0x0014 old_framelookup_off           // 00,01,02,03

                //\\ Check & remove invalid frame references first
      PATCH_PHP_EACH "BAM_V10_CYCLES_FRAMES" AS array_index => entry BEGIN
         INNER_PATCH_SAVE entry "%entry%" BEGIN
            REPLACE_EVALUATE "\([^,; %TAB%]+\)" BEGIN
               PATCH_IF NOT IS_AN_INT MATCH1 OR MATCH1 >= frames_num OR MATCH1 < 0 BEGIN
                  SPRINT MATCH1 "-1" // mark as invalid
               END
            END "%MATCH1%"
            // Remove invalid frames from string
            REPLACE_TEXTUALLY "-1," ""
            REPLACE_TEXTUALLY ",-1" ""
            REPLACE_TEXTUALLY "-1" ""
         END
         SPRINT $BAM_V10_CYCLES_FRAMES("%array_index%") "%entry%" // Reapply fixed changes
      END
      
      //\\ Write data

      //\\ Generate cycles table & frame lookup table
      SPRINT cycles_table ""
      SPRINT frame_lookup_table ""
      SET tot_entry_num = 0 // reset count
      PATCH_PHP_EACH "BAM_V10_CYCLES_FRAMES" AS array_index => entry BEGIN
         SET cycle_entry_num = 0 // reset count in current cycle
         // Generate cycles table
         INNER_PATCH_SAVE cycles_table "%cycles_table%" BEGIN
            INSERT_BYTES (array_index * 4) (4)
            WRITE_SHORT (array_index * 4 + 0x2) tot_entry_num
         END
         // Generate frame lookup table
         INNER_PATCH "%entry%" BEGIN
            REPLACE_EVALUATE "\([0-9]+\)" BEGIN
               INNER_PATCH_SAVE frame_lookup_table "%frame_lookup_table%" BEGIN
                  INSERT_BYTES (tot_entry_num * 2) (2)
                  WRITE_SHORT (tot_entry_num * 2) MATCH1
               END
               SET tot_entry_num += 1
               SET cycle_entry_num += 1
            END ""
         END
         // Generate cycles table
         INNER_PATCH_SAVE cycles_table "%cycles_table%" BEGIN
            WRITE_SHORT (array_index * 4 + 0x0) cycle_entry_num
         END
      END

      // Get cycles number
      /*SET new_cycles_num = array_index + 1
      SET new_cycles_diff = (new_cycles_num - old_cycles_num)
      // If insertion or deletion occours
      PATCH_IF new_cycles_num != old_cycles_num BEGIN
         WRITE_BYTE 0xa new_cycles_num // update number of cycles
         // Update offsets of all frame data in frame entries list
         FOR (current_frame = 0; current_frame < tot_entry_num; current_frame += 1) BEGIN
            // increment/decrement offset to frame data of current frame while ignoring BIT31 (compressed/uncompressed)
            WRITE_LONG (LONG_AT (old_frames_off) + 0x12 * current_frame + 0x8) ((THIS BAND BNOT BIT31) + (new_cycles_diff) * 4) BOR (THIS BOR BIT31)
         END
         // Update offset of palette table
         WRITE_LONG 0x10 (THIS + ((new_cycles_num - old_cycles_num) * 4)) // increment palette lookup table offset
         // Update offset of frame lookup table
         WRITE_LONG 0x14 (THIS + ((new_cycles_num - old_cycles_num) * 4)) // increment frame lookup table offset
      END
      // do actual insertion & deletion
      PATCH_IF new_cycles_num > old_cycles_num BEGIN
         INSERT_BYTES old_cycles_off ((new_cycles_num - old_cycles_num) * 4)
      END ELSE
      PATCH_IF new_cycles_num < old_cycles_num BEGIN
         DELETE_BYTES old_cycles_off ((old_cycles_num - new_cycles_num) * 4)
      END*/
               

      // Get new size of frame lookup table
      READ_LONG 0x0014 old_framelookup_off           // 00,01,02,03
      SET old_frame_lookup_size = (old_framedata_off - old_framelookup_off)
      SET new_frame_lookup_size = STRING_LENGTH "%frame_lookup_table%"
      SET new_frame_lookup_diff = (new_frame_lookup_size - old_frame_lookup_size)
      // Increment offsets of frame data in frame entries list
      PATCH_IF new_frame_lookup_size != old_frame_lookup_size BEGIN
         // Update offsets of all frame data in frame entries list
         FOR (current_frame = 0; current_frame < frames_num; current_frame += 1) BEGIN
            // increment/decrement offset to frame data of current frame while ignoring BIT31 (compressed/uncompressed)
            WRITE_LONG (old_frames_off + 12 * current_frame + 0x8) ((THIS BAND BNOT BIT31) + new_frame_lookup_diff) BOR (THIS BOR BIT31)
         END
      END
      PATCH_IF new_frame_lookup_size > old_frame_lookup_size BEGIN
         INSERT_BYTES old_framelookup_off ((new_frame_lookup_size - old_frame_lookup_size) * 2)
      END ELSE
      PATCH_IF new_frame_lookup_size < old_frame_lookup_size BEGIN
         DELETE_BYTES old_framelookup_off ((old_frame_lookup_size - new_frame_lookup_size) * 2)
      END

      // Write new frame lookup data & update cycles info
      WRITE_ASCIIE old_cycles_off "%cycles_table%"
      WRITE_ASCIIE old_framelookup_off "%frame_lookup_table%"
   END
END

COPY_EXISTING "MMUMSP.bam" "override"
   LAUNCH_PATCH_MACRO "GET_BAM_V10_CYCLES_FRAMES"

   SPRINT la $BAM_V10_CYCLES_FRAMES("0")
   INNER_PATCH_SAVE la "%la%" BEGIN
      REPLACE_TEXTUALLY "0,1,2" "0,0,1,2"
   END
   SPRINT $BAM_V10_CYCLES_FRAMES("0") "%la%"

   LAUNCH_PATCH_MACRO "APPLY_BAM_V10_CYCLES_FRAMES"
BUT_ONLY


Offline DavidW

  • Planewalker
  • *****
  • Posts: 316
Re: Manipulating sequences and frames within .bams
« Reply #1 on: November 27, 2018, 01:19:06 PM »
I played a lot with this in early incarnations of IWD-in-BG2, but it’s been ages. Looking back at my code from them, the only suggestion I have is: are you allowing for the fact that the same frame often gets used in multiple cycles?

Offline Galactygon

  • Modding since 2002
  • Planewalker
  • *****
  • Posts: 378
  • Gender: Male
  • Creator of spells
Re: Manipulating sequences and frames within .bams
« Reply #2 on: November 27, 2018, 01:52:41 PM »
David, thanks for your reply, great to see you active here! My underlying goal is to lengthen the casting animations for creatures that have their casting animations at 30 or so frames. IWD2 did it for a large number, but not all creature animations by repeating the last 16 frames over and over again until you got 100+ frames in a sequence. i.e. compare iwd2's cornugon animation to the one present in bg2 today.

The way I see it in the .bam format is that you've got general info coming first in the header with the number of frames & cycles defined present in the .bam file as well as the transparent index used in the palette. Next come the offsets defining where to look for the list of frame entries (and the location of the list of cycles based on number of frames). After that comes the list of frame entries with basic info i.e. x,y,size as well as the important offset to where the frame data is stored.

So far, none of the previous info needs to be patched if I alter the frame lookup table except for the frame data offset for each frame which I think I did correctly in the code above and testing with PATCH_PRINTs and comparing it to a file manually changed in DLTCEP. The cycle entries that come next I made sure I patch correctly with the proper increment in all cycles when adding a frame to the first cycle (when comparing to a file manually changed in DLTCEP). I did not change the offset for the palette because I didn't add new cycles yet. Then I changed the frame lookup table with the new info which matches that of a file manually changed in DLTCEP. I think I did everything correctly yet the .bam seems corrupted.

In the above code example all I did was add a new instance of frame 0 in sequence 0 at the beginning.

Offline Galactygon

  • Modding since 2002
  • Planewalker
  • *****
  • Posts: 378
  • Gender: Male
  • Creator of spells
Re: Manipulating sequences and frames within .bams
« Reply #3 on: November 29, 2018, 03:37:16 PM »
I now found out that the code works flawlessly whenever the frames are not compressed (rle). Most likely there are referenced offsets within the rle compression inside the frame data that are not updated. All of that is way beyond me. Does WeiDU allow compression and decompression of RLE data in frames?

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
Re: Manipulating sequences and frames within .bams
« Reply #4 on: December 01, 2018, 07:06:58 PM »
I now found out that the code works flawlessly whenever the frames are not compressed (rle). Most likely there are referenced offsets within the rle compression inside the frame data that are not updated. All of that is way beyond me. Does WeiDU allow compression and decompression of RLE data in frames?
You could take a look at how BAM Batcher does it.
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Galactygon

  • Modding since 2002
  • Planewalker
  • *****
  • Posts: 378
  • Gender: Male
  • Creator of spells
Re: Manipulating sequences and frames within .bams
« Reply #5 on: December 02, 2018, 04:20:32 PM »
Thanks for your link, Sam. They are really useful and I will take a look at them when I get to it.

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: