Author Topic: Feature request: Allow access to partially buffered content with COPY_LARGE  (Read 1413 times)

Offline Argent77

  • Planewalker
  • *****
  • Posts: 187
It is currently not possible to use COPY action with files that are larger than 16 MB. We have to use COPY_LARGE instead, which doesn't allow patching files. To circumvent this restriction I would like to suggest a functionality that allows you to request partial buffers for read and write access. That way it will be possible to patch files even if they exceed the file size of 16MB.

I thought of something like this:

PATCH_BUFFER offset size BEGIN patch-list END when-clause

where:
- offset indicates start offset within file.
- size indicates buffer size in bytes. Cannot exceed maximum buffer size (around 16MB).
- patch-list defines zero, one or more patch commands.
- when-clause controls whether the buffer is written back to the destination. It should include "BUT_ONLY", and possibly "IF regexp", "UNLESS regexp" and/or "IF_SIZE_IS bufferSize". I don't think "IF_EXISTS" would make sense though.

Patch functions that can modify buffer size, such as INSERT_BYTES or DELETE_BYTES, should also be supported.

Example:
Code: [Select]
COPY_LARGE ~%MOD_FOLDER%/tis/largefile.tis~ ~override~
  // reading buffer of fixed size
  PATCH_BUFFER 0 0x18 BEGIN
    READ_LONG 0x08 numTiles
    READ_LONG 0x0c tileSize
    READ_LONG 0x10 startOfs
  END BUT_ONLY

  // reading sequence of buffers with dynamic size
  SET ofs = startOfs
  FOR (idx = 0; idx < numTiles; ++idx) BEGIN
    PATCH_BUFFER ofs tileSize BEGIN
      READ_ASCII 0 palette (1024)
      // ...
      WRITE_ASCIIE 0 ~%palette%~ (1024)
    END BUT_ONLY
    SET ofs += tileSize
  END

  // appending data
  PATCH_BUFFER ofs 0 BEGIN
    INSERT_BYTES 0 tileSize
    // ...
    WRITE_ASCIIE 0 ~%palette%~ (1024)
    WRITE_ASCIIE 1024 ~%pixels%~ (4096)
  END BUT_ONLY

I don't know whether it is even possible to add such a feature, but it would greatly help modding if it could be implemented.
« Last Edit: July 31, 2017, 07:39:19 AM by Argent77 »

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1176
My own thought-experiments along these lines have been that the large-file problem could be solved by changing the implementation of file buffers from being simple strings, to being arrays of strings. You'd then be able to load arbitrarily large files into memory with the standard actions and COPY_LARGE would become obsolete. Technically, I can't see a real obstacle. You need some code for figuring out what position in which string corresponds to a given offset, and you need some code for handling byte insertion and deletion, and reads that cross from one string to the next, but those should be solvable problems. But there might be artificial limitations I'm not aware of, and, of course, there is the actual (and not inconsiderable) work involved.

Do you want me to look into PATCH_BUFFER, as well? I'm afraid I can't give an ETA for either.

Offline Argent77

  • Planewalker
  • *****
  • Posts: 187
Your approach sounds even better. There is no need to look further into PATCH_BUFFER if your idea can be realized.

Although I can imagine that implementation of string arrays would be non-trivial. In addition to the points you have outlined, there are also the problems of reshuffling data in case INSERT_BYTES exceeds buffer limit or DELETE_BYTES removes data from consecutive buffers to solve.

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
I've been working on some functions to add/delete/replace tiles in a tileset on-the-fly (instead of shipping the entire modified tileset with the mod just to append a few door tiles, making the mod incompatible with any other mod trying to do the same).  TIS V1 is fully supported with limited TIS V2 support in the works.  The major obstacle for my effort being particularly useful is WeiDU's filesize limit.  If you need a functional example illustrating why support for large files would be helpful to motivate you, I offer this one.  Here is another:
Quote
The new implementation needs to COPY BG:EE TLK file in order to read 0xa offset (total amount of strings). It's possible that weidu won't be able to do it if BG:EE TLK is very large (there is COPY size limit, although there's still about 8 MB left when it comes to vanilla file). If someone will report that in heavily modded BG:EE the size is reached we will switch to Lua code in order to read this value.
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1176
Your approach sounds even better. There is no need to look further into PATCH_BUFFER if your idea can be realized.
Then I shall consider this and COPY_EXISTING_LARGE a possible plan B.

aqrit

  • Guest
This issue prevents the widescreen mod from operating at 4K resolutions.

OCaml allows much larger string sizes in a 64-bit process, which may be another possibility to resolve this issue.
The last attempt at that failed though:
http://forums.pocketplane.net/index.php/topic,24120.msg338275.html#msg338275

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: