Pocket Plane Group
Friends and Neighbors => Weimer Republic (WeiDU.org) => WeiDU => Topic started by: Argent77 on July 30, 2017, 02:59:01 PM
-
It is currently not possible to use COPY action with files that are larger than 16 MB. We have to use COPY_LARGE instead, which doesn't allow patching files. To circumvent this restriction I would like to suggest a functionality that allows you to request partial buffers for read and write access. That way it will be possible to patch files even if they exceed the file size of 16MB.
I thought of something like this:
PATCH_BUFFER offset size BEGIN patch-list END when-clause
where:
- offset indicates start offset within file.
- size indicates buffer size in bytes. Cannot exceed maximum buffer size (around 16MB).
- patch-list defines zero, one or more patch commands.
- when-clause controls whether the buffer is written back to the destination. It should include "BUT_ONLY", and possibly "IF regexp", "UNLESS regexp" and/or "IF_SIZE_IS bufferSize". I don't think "IF_EXISTS" would make sense though.
Patch functions that can modify buffer size, such as INSERT_BYTES or DELETE_BYTES, should also be supported.
Example:
COPY_LARGE ~%MOD_FOLDER%/tis/largefile.tis~ ~override~
// reading buffer of fixed size
PATCH_BUFFER 0 0x18 BEGIN
READ_LONG 0x08 numTiles
READ_LONG 0x0c tileSize
READ_LONG 0x10 startOfs
END BUT_ONLY
// reading sequence of buffers with dynamic size
SET ofs = startOfs
FOR (idx = 0; idx < numTiles; ++idx) BEGIN
PATCH_BUFFER ofs tileSize BEGIN
READ_ASCII 0 palette (1024)
// ...
WRITE_ASCIIE 0 ~%palette%~ (1024)
END BUT_ONLY
SET ofs += tileSize
END
// appending data
PATCH_BUFFER ofs 0 BEGIN
INSERT_BYTES 0 tileSize
// ...
WRITE_ASCIIE 0 ~%palette%~ (1024)
WRITE_ASCIIE 1024 ~%pixels%~ (4096)
END BUT_ONLY
I don't know whether it is even possible to add such a feature, but it would greatly help modding if it could be implemented.
-
My own thought-experiments along these lines have been that the large-file problem could be solved by changing the implementation of file buffers from being simple strings, to being arrays of strings. You'd then be able to load arbitrarily large files into memory with the standard actions and COPY_LARGE would become obsolete. Technically, I can't see a real obstacle. You need some code for figuring out what position in which string corresponds to a given offset, and you need some code for handling byte insertion and deletion, and reads that cross from one string to the next, but those should be solvable problems. But there might be artificial limitations I'm not aware of, and, of course, there is the actual (and not inconsiderable) work involved.
Do you want me to look into PATCH_BUFFER, as well? I'm afraid I can't give an ETA for either.
-
Your approach sounds even better. There is no need to look further into PATCH_BUFFER if your idea can be realized.
Although I can imagine that implementation of string arrays would be non-trivial. In addition to the points you have outlined, there are also the problems of reshuffling data in case INSERT_BYTES exceeds buffer limit or DELETE_BYTES removes data from consecutive buffers to solve.
-
I've been working on some functions to add/delete/replace tiles in a tileset on-the-fly (instead of shipping the entire modified tileset with the mod just to append a few door tiles, making the mod incompatible with any other mod trying to do the same). TIS V1 is fully supported with limited TIS V2 support in the works. The major obstacle for my effort being particularly useful is WeiDU's filesize limit. If you need a functional example illustrating why support for large files would be helpful to motivate you, I offer this one. Here (http://gibberlings3.net/forums/index.php?showtopic=28161&page=64#entry255390) is another:
The new implementation needs to COPY BG:EE TLK file in order to read 0xa offset (total amount of strings). It's possible that weidu won't be able to do it if BG:EE TLK is very large (there is COPY size limit, although there's still about 8 MB left when it comes to vanilla file). If someone will report that in heavily modded BG:EE the size is reached we will switch to Lua code in order to read this value.
-
Your approach sounds even better. There is no need to look further into PATCH_BUFFER if your idea can be realized.
Then I shall consider this and COPY_EXISTING_LARGE a possible plan B.
-
This issue prevents the widescreen mod from operating at 4K resolutions.
OCaml allows much larger string sizes in a 64-bit process, which may be another possibility to resolve this issue.
The last attempt at that failed though:
http://forums.pocketplane.net/index.php/topic,24120.msg338275.html#msg338275