SET_2DA_ENTRY thoughts

[devSin]

I can't think of any obvious, easy way to do it. Loosely, when I parse the 2DA file into a two dimensional array (or list of lines, where each line is an array of words) I throw away the original position/whitespace information. SET_2DA_ENTRY gets around this by re-reading the file, searching for the cell you changed, splitting the file (as a string) into strings before and after that cell, and then pasting the three strings together (the before-bit, your new value, and the after-bit). The searching and the string concatenation both take time, and doing that whole process once per update would get you back to SET_2DA_ENTRY's performance, more or less.

Does this adversely affect normal SET_2DA_ENTRY performance then, as well? I never looked closely at the 2DA code (one of the more complex TP actions), but I was always under the impression that the reformatting was done explicitly (basically 'widest column' = 'new uniform column width') and was just a convenience feature. If you're pushing out a new 2DA using only the old data (or at least only the data that serves any purpose), how much of a hit is it to have to go back and try and fit the new data into the existing 2DA (vs. just creating a new 2DA with your own formatting)? Even a minor speed hit would be worth reverting to the old behavior (I know you always protested that white space greater than a single space between elements means absolutely nothing to the game, and I was even louder in protesting that it was just too ugly :-)), as SET_2DA_ENTRY really is quite common these days (I have 427 in my Tp2, although it's not quite the definition of common). I admit to not really following the differences in the new "fast" Patch2DA vs. the old, slower implementation (which still does the reformatting), but if any additional speed gains are there for the taking, it seems like an awful waste to choose the prettier solution just to look pretty.

... and shave off a bit more time. You don't need a separate variable for each different value stored.

I was wondering about this, but wasn't sure if it was allowed in the implementation. I was looking at it in terms of the loop (only getting one shot for each run through), rather than just the patch actions getting executed individually within the loop (it makes sense this would work, as it can track the changing value of ~%row%~ and whatnot); I got stuck thinking each successive invocation would overwrite the previous PatchLater actions for each single run of the loop. Much thanks for pointing this out.

I'm glad to hear that it seems to be working performance-wise (10% isn't bad).

Indeed. There's still a tiny hit for the WEAPPROF patch, but WeiDU muscles through it in record time, now (once I get out of the obsessive copies and compiles that occupy the first 2/3 of the Tp2, v180 more or less screams through the rest without any substantial delay for any single patch). Thanks for the great work on v178+!

[weimer]

Does this adversely affect normal SET_2DA_ENTRY performance then, as well?

I'm afraid that I don't really understand your question. When I added SET_2DA_ENTRY_LATER I didn't change anything to make vanilla SET_2DA_ENTRY "worse" or "slower". One of the reasons that I was able to code up _LATER so quickly was that I just copied the "2DA output" code from vanilla SET_2DA_ENTRY. Vanilla SET_2DA_ENTRY actually has two possible ways to output a 2DA file -- one way preserves the original whitespace and one way reconstructs the whole file (with wide columns). I was unable to think of a way to adapt the whitespace-preserving output approach to _LATER (basically, the whitespace-preserving approach requires that only one entry cell be changed), so I copied the reconstruct-the-whole-thing code and used that. Since that output code is the same, _LATER and vanilla-when-it-uses-the-wide-column-output-approach have the same performance issues. Vanilla-when-it-preserves-whitespace is different.

how much of a hit is it to have to go back and try and fit the new data into the existing 2DA

There are two ways in which this can be a hit. One way is algorithm running time -- this could be done in linear time (= quickly) by keeping track of the makeup of the whitespace or somesuch with extra tables. Unfortunately, that would either require trickery or custom parsing/lexing routines (the default library ones all strip/ignore whitespace). The important hit is thus the programmer thinking time -- in that regard it's "too much of a hit" because it's not worth it to me. I'll accept patches for it, but it's not a 15-minute job and I don't believe in it, so it's unlikely that I'll just whip it out in my spare time.

obsessive copies and compiles

By the way, is there any reason you do this:

Code: [Select]

COPY ~FIXPACK/RESOURCES/ARE/AR6005.ARE~ ~OVERRIDE~
~FIXPACK/RESOURCES/BAM/MBASG1.BAM~ ~OVERRIDE~
~FIXPACK/RESOURCES/BAM/MBASG1E.BAM~ ~OVERRIDE~
~FIXPACK/RESOURCES/BAM/MBASG2.BAM~ ~OVERRIDE~
~FIXPACK/RESOURCES/BAM/MBASG2E.BAM~ ~OVERRIDE~
...

Instead of just:

Code: [Select]

COPY ~FIXPACK/RESOURCES/ARE/AR6005.ARE~ ~OVERRIDE~
~FIXPACK/RESOURCES/BAM/~ ~OVERRIDE~ // copy all BAM files to override

It would cut your TP2 parsing time and the TP_COPY time a bit. 

[devSin]

I'm afraid that I don't really understand your question.

I'm speaking of the "vanilla" SET_2DA_ENTRY. At the behest of myself and others, you changed the behavior (around the same time the new "fast" Patch2DA was added). Unless you specify an invalid row or column or required column count (in which case WeiDU falls back to the old "slow" Patch2DA code), the output has no formatting changes (I'm guessing this means you switched to the find/split/paste/write routine). Even though the new "fast" Patch2DA is faster than the old implementation, I'm wondering if there isn't room to be faster still by ditching the new "preserve the original formatting" behavior and just dumping out the newly constructed 2DA.

Since that output code is the same, _LATER and vanilla-when-it-uses-the-wide-column-output-approach have the same performance issues. Vanilla-when-it-preserves-whitespace is different.

I may have read it wrong, but my thought was that preserving the original layout (actually patching the file, vs. just the relevant data) was the performance drain. As it is now, the default (for valid Patch2DA actions) is to preserve the original whitespace (from what I've seen, this is done automatically, but only with the new "fast" implementation).

SET_2DA_ENTRY gets around this by re-reading the file, searching for the cell you changed, splitting the file (as a string) into strings before and after that cell, and then pasting the three strings together (the before-bit, your new value, and the after-bit).

I wonder how this approach effects general processing time? Some of the current patch actions involving buffering the file, creating a new "patch" buffer, breaking the file apart somewhere and stitching everything together once the "patch" buffer has been fully constructed. Is there a difference here? It's undoubtedly a little more complex with SET_2DA_ENTRY, since you can't really know in advance where you're going to split the file; is the majority of the time in the process spent searching for the split position, or just the combination of all the actions involved (load->construct->reload->find->split->stitch->write)?

This isn't a not-so-subtle hint to start making fundamental architectural changes; I know you've stated in the past that handling everything as a string makes it really easy to add new stuff, and I'm simply curious as to how the approach is holding up as WeiDU gets more complex.

By the way, is there any reason you do this:

I can never decide if I want the explicit list, or just the directory. It depends on what day of the week I post a revision to the Tp2, I guess. It used to be that my copy lists weren't that long, so it was OK (and kept WeiDU from backing up stupid .DS_Store for each component), but it is getting a little out of hand. The list is easy enough to compile, though, so it's not too much of an issue. I think that once I'm done "for sure" (as if), I'll finally clean stuff up to be more efficient (and less of an eye sore); for now, it's just a personal quirk.

I do have to admit to wanting some sort of feature allowing me to specify "virtual" directories (basically, just a way to group a bunch of inlined files into a single virtual directory, so I could do something like COMPILE ~.../my/virtual/folder~ instead of the current COMPILE ~everyInlinedScriptName~ list). I tried passing the fake directory name (back when I followed the "unique inlined file name" rule), with no success (so, if I defined all inlined scripts as .../creaturebafs/*.baf, I tried COMPILE ~.../creaturebafs~).

Anyway, thanks for indulging my contemplations.

[the bigg]

Just a quick note for those interested / the readme writer:

Code: [Select]

COPY_EXISTING ~weapprof.2da~ ~override~
  SET_2DA_ENTRY_LATER ~_#_#_#weapprof~ 10 2 1

COPY_EXISTING ~sw1h01.itm~ ~override~
[other 234621342986412387 unrelated actions]

COPY_EXISTING ~weapprof.2da~ ~override~
  SET_2DA_ENTRY_LATER ~_#_#_#weapprof~ 10 3 2
  SET_2DA_ENTRIES_NOW ~_#_#_#weapprof~ 1
Will work. Yes, I use extensively this splitting of the 2DA handling in Refinements 

Also, you can easily shoot yourself in the foot if you try the following:

Code: [Select]

COPY_EXISTING ~kitlist.2da~ ~override~
  SET_2DA_ENTRY_LATER ~_#_#_#weapprof~ 10 2 1

COPY_EXISTING ~sw1h01.itm~ ~override~
[other 234621342986412387 unrelated actions]

COPY_EXISTING ~weapprof.2da~ ~override~
  SET_2DA_ENTRY_LATER ~_#_#_#weapprof~ 10 3 2
  SET_2DA_ENTRIES_NOW ~_#_#_#weapprof~ 1As this will apply both SET_2DA_ENTRY_LATER to weapprof (surprise, surprise!)

[devSin]

I guess since normal variables copy over (if you read or set a value in COPY_EXISTING ~src1~ ~dest~, you can access it in COPY_EXISTING ~src2~ ~dest~), this is the intended behavior (and, like normal variables, WeiDU will start all over if you initialize it in the new TP2 action).

EDIT: ah, I just saw what you're referring to. I guess the moral of the story is to make sure you use PatchNow before you try to use the same name to do something else.

[the bigg]

I guess since normal variables copy over (if you read or set a value in COPY_EXISTING ~src1~ ~dest~, you can access it in COPY_EXISTING ~src2~ ~dest~), this is the intended behavior (and, like normal variables, WeiDU will start all over if you initialize it in the new TP2 action).

EDIT: ah, I just saw what you're referring to. I guess the moral of the story is to make sure you use PatchNow before you try to use the same name to do something else.

Also, I believe that probably not everybody knows that variables carry over between different actions, so it may be helpful to be noted.

[weimer]

Even though the new "fast" Patch2DA is faster than the old implementation, I'm wondering if there isn't room to be faster still by ditching the new "preserve the original formatting" behavior and just dumping out the newly constructed 2DA.

I don't want to spend too much time going over this (and I don't want to bore you if you know all of this). String concatenation is expensive. In the FastPreserve approach we do some trickery involving counting the number of newlines and whitespace groups, then searching forward that many from the beginning of the file to find the exact byte offset of the cell you want to change. We then cut out the string before that and the string after that, and do two concatenations: before + newCell + after. Assume each string concat takes 1 time unit. This approach takes 2 time units per update (say).

The "slower" approach destroys the formatting by rebuilding the entire file. This takes loosely one string append per cell entry, regardless of whether you changed it or not. Actually, the special buffer structure used means that it takes time equal to the log of the number of entries. In weapprof.2da, that's 2914 entries, so the time is log_2(2914) = 11.

So now you do the math. If N is the number of updates you want to do, then option A takes 2N time and option B takes 11 time, so if N>5 you want to use option B.

Before _NOW and _LATER, N was always 1, so the whitespace-preserving approach (option A) always looked better. However, if you're actually using _NOW and _LATER to buffer a non-trivial number of updates, option B becomes the winner.

Now, the numbers 2N and 11 and whatnot aren't exact, but that's the upshot.

Unless you specify an invalid row or column or required column count (in which case WeiDU falls back to the old "slow" Patch2DA code)

Actually, that try-catch bit there mostly catches problems with the string searching. If for some reason I'm unable to count the right number of newlines and find you cell there (possibly because you gave the wrong count or something, but also possibly because the regexp decides not to work on a Mac or whatever) we fall back.

Some of the current patch actions involving buffering the file, creating a new "patch" buffer, breaking the file apart somewhere and stitching everything together once the "patch" buffer has been fully constructed. Is there a difference here?

I don't know exactly which actions you're thinking of (I guess ADD_CRE_ITEM is one). If they do either "a^b^c" or "Buffer.add", they're fine. The first case is just two appends not in a loop, so that's fine, and the second case uses balanced trees for O(log N) append time. If you see an action that's doing "a^b^c" inside a loop or "a^b^c^d^...^z", that would be bad.

This isn't a not-so-subtle hint to start making fundamental architectural changes; I know you've stated in the past that handling everything as a string makes it really easy to add new stuff, and I'm simply curious as to how the approach is holding up as WeiDU gets more complex.

Fine from my perspective.

I do have to admit to wanting some sort of feature allowing me to specify "virtual" directories

Wouldn't specifying the virtual directory take just as long as making the COMPILE statement? As in:

Code: [Select]

SPECIFY_VIRTUAL_DIR ~mydir~
   ~.../yada/1.d~
   ~.../yada/2.d~
   ~.../yada/3.d~
COMPILE ~mydir~


// versus

COMPILE ~.../yada/1.d~
   ~.../yada/2.d~
   ~.../yada/3.d~


Yes, I use extensively this splitting of the 2DA handling in Refinements

Don't shoot yourself in the foot, bigg.

I'm in the office today waiting around to get an official offer letter from Virginia, so if you guys want something coded while I'm here, let me know. I'll post WeiDU later today with the column-count thing in _LATER fixed.

[the bigg]

Don't shoot yourself in the foot, bigg.

Has been working correctly for me, so it's fine as long as I don't use multiple variables per file, don't use the same variable in different files or try to READ_2DA_ENTRY on a to-be-patched value. Or am I missing something structural?

[weimer]

Or am I missing something structural?

No, it works fine as long as you're paying attention. :-)

[devSin]

Before _NOW and _LATER, N was always 1, so the whitespace-preserving approach (option A) always looked better. However, if you're actually using _NOW and _LATER to buffer a non-trivial number of updates, option B becomes the winner.

Once I finally figured out how _LATER was working, I consider SET_2DA_ENTRY more or less dead. The only places I'm not able to swap it out are files where there's just a single Patch2DA action (that can't be easily excised and REPLACE_TEXTUALLYed). The only sticky part is that dumb required column count -- I switched over to Later/Now without modifying the original code, so I have to go back and figure out all the rows and columns so I only have to PatchNow once. Boo.

Wouldn't specifying the virtual directory take just as long as making the COMPILE statement? As in:

Not if WeiDU automatically included inlined files specified from a certain directory (.../myDir/file.baf) in the virtual directory (say myDir). I wasn't thinking what you're thinking, but my approach is probably a tedious exercise in futility that we can live without.

I don't know exactly which actions you're thinking of

All of Japh's code, and any of the _FILE or DELETE_ or INSERT_ actions. I think PatchGam was the only one to overdo it a bit, though. (EDIT: I just checked, and it's not too horrible- a^b^c^d^e; nobody uses it, anyway.)

Fine from my perspective.

Yeah, the only place where WeiDU gets stuck in the mud anymore is the initial parse and compiling all those scripts. Everything else just runs straight through to the end.

I'm in the office today waiting around to get an official offer letter from Virginia, so if you guys want something coded while I'm here, let me know. I'll post WeiDU later today with the column-count thing in _LATER fixed.

Do later and now do variable substitution? I don't know that I would ever actually do this, but I was thinking of something evil like SET_2DA_ENTRY_LATER ~%SOURCE_RES%~, letting WeiDU chew up free RAM, and then mass dumping it out with a single SET_2DA_ENTRIES_NOW ~%SOURCE_RES%~ patch from a single copy action (for all the files).

[weimer]

The only sticky part is that dumb required column count

I'm not really sure what the problem is here -- both _ENTRY and _LATER require the "dumb" column count. Presumably you're saying that you changed everything to be bug-for-bug compatible with the current WeiDU version and now you'll have to change it again. Sorry.

Yeah, the only place where WeiDU gets stuck in the mud anymore is the initial parse and compiling all those scripts. Everything else just runs straight through to the end.

Can you throw the timing info from the DEBUG file at me again? I might be able to do something about TRA parsing. Include 10 lines from your TRA file, or the words "stop now" if you ever use forced string refs (e.g., @200 = !100 ~This Exact String Becomes TLK Entry #100~).

Do later and now do variable substitution?

Yes (for the row, column, value, and column count).

[devSin]

Presumably you're saying that you changed everything to be bug-for-bug compatible with the current WeiDU version and now you'll have to change it again.

I never bothered to use consistent required column counts (instead using the actual required count). So, to avoid having multiple SET_2DA_ENTRIES_NOW in a single patch, I have to go back and work out the new row/column for the new required count where it was different. So I'd have something like Patch2DA 0 1 2 ~~, Patch2DA 0 1 3 ~~, etc., in the same TP2 action, which means I'd have to PatchLater ~~, PatchNow ~~ 2, PatchLater ~~, PatchNow ~~ 3; nasty. I'm just whining, here, though (it's not a bug or anything).

Can you throw the timing info from the DEBUG file at me again? I might be able to do something about TRA parsing.

Code: [Select]

WeiDU Timings
load TLK                         0.000
process copy_trans               0.000
parsing .log files               0.000
preprocess APPEND_EARLY          0.000
process .D files                 0.017
postprocess .D actions           0.017
resolve strings                  0.033
preprocess .D actions            0.033
process .D actions               0.033
unmarshal BIFF                   0.033
unmarshal compressed BIFF        0.067
resolve labels                   0.083
parsing .ids files               0.150
BUT_ONLY                         0.167
unmarshal KEY                    0.567
find local string                0.600
marshal and save TLK             0.750
unmarshal DLG                    0.933
marshal DLG                      0.950
unmarshal TLK                    1.200
parsing .d files                 1.600
READ_*                           2.300
eval_pe                          4.050
TP_Copy                          4.500
parsing .baf files               5.450
loading files                    9.083
process_patch2                   9.417
saving files                    23.600
parsing .tp2 files              27.117
parsing .tra files              29.983
process_action                  90.333 // probably a third of this is compilation :(
TOTAL                          213.067

Include 10 lines from your TRA file, or the words "stop now" if you ever use forced string refs (e.g., @200 = !100 ~This Exact String Becomes TLK Entry #100~).

I admit to not having a clue what you're trying to say here (I'm not current on TRA lingo, so I'm not sure of this "forced" strref).

Yes (for the row, column, value, and column count).

That's not going to help me PatchNow ~%SOURCE_RES%~ n, though. :)

[weimer]

process_action                  90.333 // probably a third of this is compilation

The reported time as all "exclusive" or "non-intersecting", so ".D actions" and whatnot already covers compiling D files. At any rate, next version will differentiate by action so you can be sure.

I admit to not having a clue what you're trying to say here (I'm not current on TRA lingo, so I'm not sure of this "forced" strref).

That's good, it means you're probably not using features that would prevent me from making TRA parsing go faster. Paste in 10 indicate lines from your TRA files (and/or just link your TRA file) and I'll take a look.

That's not going to help me PatchNow ~%SOURCE_RES%~ n, though.

I can't really imagine that helping you.

Code: [Select]

COPY_EXISTING_REGEXP ~*.2DA~ override
  PATCH_LATER ~%SOURCE_RES%~ 1 2 3
  PATCH_NOW ~%SOURCE_RES%~

... is the same as using ~foo~ instead of ~%SOURCE_RES%. Note that _LATER doesn't store which file it was working on and _NOW always applies to the current file, so unless you yourself make sure that they match up you're just shooting yourself in the foot.

I can't see what you're imagining, and any way I can think of to use variables in the variable name just ends up hurting you or pointing at some confusion in how the command works. You'd have to show a real example to convince me of what you're trying for.

[devSin]

Here's a random TRA sampling

Code: [Select]

@219 = ~(sniffle)~ []
@340 = ~~ [2AMERC05]
@375 = ~ I'm not sticking around for this!~ [GOLDAN24]
@1000001 = ~No, I'm sorry, none of them sound familiar.~ []
@1000078 = ~The gem that Mekrath took from me and hid within his altar! Please, it is imperative that the jewel makes it safely into our hands. Go, and don't return until you hold it in your hands.~ []
@1010607 = ~Black Talon Elite~ []
@1024443 = ~Yeah, whatever. Go away. Surly's got a bad hangover.~ []
@1036897 = ~Today we have fought well and lived to tell, yes <SIRMAAM>. Now old Grady will pull some wisdom out of his tail end, and young Elotte will be wide-eyed for days.~ []
@1041422 = ~Shvanana can stave off de oldest one's anger for a while, to save you mummers, but dusts is costly, an' so is Shvanana's time. A thousand golders, an' I do it.~ []
@1046566 = ~Now... go and kill that rebel, my little surface <PRO_RACE>. All this bloodletting has only served to whet my appetite. (hee hee!)~ []
@1051948 = ~Chaos~ []If you want something more concrete, I'll get it up, or you can just download the whole TRA file.

I can't see what you're imagining, and any way I can think of to use variables in the variable name just ends up hurting you or pointing at some confusion in how the command works. You'd have to show a real example to convince me of what you're trying for.

COPY_EXISTING_REGEXP is so yesterday. Take this:

Code: [Select]

COPY_EXISTING ~HPMONK.2DA~ ~OVERRIDE~
~HPPRS.2DA~ ~OVERRIDE~
~HPROG.2DA~ ~OVERRIDE~
~HPWAR.2DA~ ~OVERRIDE~
~HPWIZ.2DA~ ~OVERRIDE~
  READ_2DA_ENTRY 0 1 4 ~sides~
  FOR (row = 0; row < 10; row = row + 1) BEGIN
    READ_2DA_ENTRY ~%row%~ 2 4 ~rolls~
    PATCH_IF (~%rolls%~ = 1) BEGIN
      SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 2 ~0~
      SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 3 ~%sides%~
    END
  END
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~INTMOD.2DA~ ~OVERRIDE~
  FOR (row = 0; row < 24; row = row + 1) BEGIN
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 1 ~100~
  END
  FOR (row = 0; row < 3; row = row + 1) BEGIN
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 4 ~2~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ + 3 4 ~1~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ + 6 4 ~5~
  END
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~LUABBR.2DA~ ~OVERRIDE~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 12 1 ~FMT~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 17 1 ~FMC~
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~LUFMC.2DA~ ~OVERRIDE~
  FOR (row = 19; row < 22; row = row + 1) BEGIN
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 4 ~1~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 5 ~99~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 6 ~1~
  END
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 19 1 ~AP_SPCL928~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 20 1 ~AP_SPCL929~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 20 7 ~AP_SPCL928~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 21 1 ~AP_SPCL930~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 21 7 ~AP_SPCL929~
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~LUFMT.2DA~ ~OVERRIDE~
  FOR (row = 20; row < 23; row = row + 1) BEGIN
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 4 ~1~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 5 ~99~
    SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ ~%row%~ 6 ~1~
  END
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 20 1 ~AP_SPCL928~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 21 1 ~AP_SPCL929~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 21 7 ~AP_SPCL928~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 22 1 ~AP_SPCL930~
  SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ 22 7 ~AP_SPCL929~
BUT_ONLY_IF_IT_CHANGESLet's say, rather than having me try to come up with some name, I could just SET_2DA_ENTRY_LATER ~%SOURCE_RES%~ (the name of a file is always guaranteed to be unique). None of the TP2 actions would have SET_2DA_ENTRIES_NOW (it would just be a bunch of LATERs). Then, at some appropriate point in the future, I do:

Code: [Select]

COPY_EXISTING ~HPMONK.2DA~ ~OVERRIDE~
~HPPRS.2DA~ ~OVERRIDE~
~HPROG.2DA~ ~OVERRIDE~
~HPWAR.2DA~ ~OVERRIDE~
~HPWIZ.2DA~ ~OVERRIDE~
~INTMOD.2DA~ ~OVERRIDE~
~LUABBR.2DA~ ~OVERRIDE~
~LUFMC.2DA~ ~OVERRIDE~
~LUFMT.2DA~ ~OVERRIDE~
  SET_2DA_ENTRIES_NOW ~%SOURCE_RES%~ n
  // n here is of course the lowest common requiredColumnCount, so
  // all the original SET_LATERs would have to be updated for the
  // new column count
  // but still...
BUT_ONLY_IF_IT_CHANGESSince all of the data is retained across multiple actions, and the list name = filename is guaranteed to be unique, there shouldn't be any chance for conflict. Note that I said I'd probably never do this (actually, I probably would, but I still wouldn't recommend it), I was just making sure it couldn't be done.

[weimer]

Here's a random TRA sampling

OK, whatever. We now have the "TRA binary" format -- TRB. "weidu --trbify foo.tra" creates "foo.trb", a TRB file can be used almost anywhere a TRB file is expected. Do not use TRB files. Here are some performance comparisons for your huge TRA file:

Code: [Select]

[dev.tra] parsed
[dev.tra] has 74333 translation strings
parsing .tra files               1.092

[dev.trb] has 74333 translation strings
parsing .trb files               0.070

Do not use this feature. You must provide the documentation for this feature. If the WeiDU internal representation of TLK strings changes (very unlikely, has never happened AFAIK) or if the OCaml compiler guys decide to change their marshal code (possible) your mod will stop working and weidu auto-update won't fix it. But if you're just developinging on your home machine and will ship with the TRA, it's fine.

Let's say, rather than having me try to come up with some name, I could just SET_2DA_ENTRY_LATER ~%SOURCE_RES%~

In your example you can just use ~modifier~ every time and the behavior is the same.

I was just making sure it couldn't be done.

It can't be done and I don't see it being added anytime soon.

[weimer]

Blah blah, 181 up, you can try out TRB files and see what TP2 actions are actually soaking up your time (from WeiDU's perspective).

[devSin]

We now have the "TRA binary" format

Nice.

your mod will stop working and weidu auto-update won't fix it

I assume this means that if I have a mod X that includes a binary TRA, and the Setup-MyMod executable is automatically updated for Windows users, and your string code or their marshal code changed, then the mod with the existing TRB will fail? So, on Mac OS X, I first have to update OCaml, update and/or re-compile WeiDU, and ship it with the old TRB to break anything?

Does your ominous note mean that TRB is dead at birth (i.e., if it breaks, well, sorry, you shouldn't have been using it anyway), or that if something under-the-hood changes, the TRB file (either how it is read in or written) will change and all your TRB files created with earlier versions of WeiDU are bust?

In your example you can just use ~modifier~ every time and the behavior is the same.

Even across multiple TP2 actions? Wouldn't there be conflicts (since it's storing up everything in ~modifier~, wouldn't it try to make all the modifications to each file)?

[weimer]

Does your ominous note mean that TRB is dead at birth (i.e., if it breaks, well, sorry, you shouldn't have been using it anyway), or that if something under-the-hood changes, the TRB file (either how it is read in or written) will change and all your TRB files created with earlier versions of WeiDU are bust?

Both, I guess. I wouldn't ship one unless you're on the ball and warn your users that they may have to come back to you for a mod update if things go wrong. I saw it more as a way to keep the develop-test cycle cheap on your local machine.

If you and others discover that you can't live your life without shipping this feature I can include some sort of support for shipping both the TRA and the TRB (compressed?) and parsing the TRB first if possible. But that would take five or ten minutes, and until you try it out on your machine and see whether it makes a difference or not there's no reason to do that.

Even across multiple TP2 actions? Wouldn't there be conflicts (since it's storing up everything in ~modifier~, wouldn't it try to make all the modifications to each file)?

No: PATCH_NOW clears ~myvar~ every time.

[devSin]

So if I deferred PatchNow, opting to have some special action to do it all at once, it would fail after the first patch, right?

Code: [Select]

COPY_EXISTING ~HPMONK.2DA~ ~OVERRIDE~
... (every other file where I stored up PatchLater actions in ~modifier~
  SET_2DA_ENTRIES_NOW ~modifier~ n
  // after patching HPMONK, ~modifier~ doesn't point to any items = no patches = fail or just do nothing to everything else in file list?
BUT_ONLY_IF_IT_CHANGESI probably should have taken out the PatchNows in the "what if" example to better illustrate that the magic trick was just to actually patch the files all at once later (since there's no deallocation until the PatchNow). Anyway, "it doesn't work" is good enough for me.

An actual question I had is what the expected behavior is for patches to the same entry from the same list:

Code: [Select]

COPY_EXISTING ~some.2DA~ ~somewhere~
  FOR ( i=10; i>0; i-=1 ) BEGIN
    SET_2DA_ENTRY_LATER ~foo~ 0 1 ~%i%~
  END
  SET_2DA_ENTRIES_NOW ~foo~ 2Will this patch the same entry 10 times (just as using a series of SET_2DA_ENTRY actions would), or does it start overwriting the same variable (the list index isn't updated, I guess)? I imagine the former, but I wanted to check (to help out some subset of { thebigg : CamDawg } in writing documentation, or something :-D).

[devSin]

I finally got a minute to check this. I haven't done any concrete tests yet, but here's the stats for a simple parse/load/do nothing:

Code: [Select]

WeiDU Timings
unmarshal BIFF                   0.000
load TLK                         0.000
parsing .log files               0.000
loading files                    0.050
eval_pe                          0.417
parsing .trb files               0.550
unmarshal KEY                    0.567
unmarshal TLK                    1.217
parsing .tp2 files              27.433
TOTAL                           30.233More to come when I get a chance to do a full run.

[weimer]

Will this patch the same entry 10 times (just as using a series of SET_2DA_ENTRY actions would), or does it start overwriting the same variable

... neither or both. It will continue to add entries to the list of deferred updates, but since the time taken by _NOW is "11" and not "2N", it won't take any more time. Last _LATER update to (x,y,) wins.

So if I deferred PatchNow, opting to have some special action to do it all at once, it would fail after the first patch, right?

Not necessarily. I reallly can't understand what you're talking about or what you're trying to do. _LATER ~myvar~ appends an update to a linked list called ~myvar~. _NOW ~myvar~ applies all of the updates in the list ~myvar~ to the current file and then sets ~myvar~ to NULL. You should be able to derive all of the behavior from that.

[devSin]

I reallly can't understand what you're talking about or what you're trying to do.

Given the following TP2 code:

Code: [Select]

COPY_EXISTING ~foo.2da~ ~override~
  FOR ( i=10; i>0; i-=1 ) BEGIN
    SET_2DA_ENTRY_LATER ~theStuff~ ~%i%~ 1 ~%i%HeyNow~
  END
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~bar.2da~ ~override~
  FOR ( i=8; i>2; i-=1 ) BEGIN
    SET_2DA_ENTRY_LATER ~theStuff~ 2 ~%i%~ ~whatever~
  END
BUT_ONLY_IF_IT_CHANGES

COPY_EXISTING ~foo.2da~ ~override~
~bar.2da~ ~override~
  SET_2DA_ENTRIES_NOW ~theStuff~ 2
BUT_ONLY_IF_IT_CHANGESIf PatchLater doesn't associate the list entries with the file being patched, how is it going to know which patches were for foo and which were for bar? And, if PatchNow chucks the list after execution, how's there going to be anything left for bar? Basically, I was thinking of having all the PatchLaters use ~%SOURCE_RES%~, and then using PatchNow in a single copy action afterward (since %SOURCE_RES% will be unique to each file, and contain the changes for only that file). [EDIT: I modified the original example to clarify what the code would look like in this situation.]

I couldn't have been more wrong about compile vs. actual process action times (COMPILE doesn't take up 1/3 of it, it takes up *all* of it)

Code: [Select]

WeiDU Timings
MKDIR                            0.000 // lots of these, but WeiDU doesn't actually have to do anything
PATCH_PRINT                      0.000 // not too many of these
AT_*                             0.000 // 2 of these
load TLK                         0.000 // only 1 (no localized F) with default strings
process copy_trans               0.000 // none
APPEND_COL                       0.000 // 2 of these
process .D files                 0.000 // a huge inlined .D file (~ 500 dialogues)
parsing .log files               0.000 // no mods installed
ADD_PROJECTILE                   0.000 // 2 of these
preprocess APPEND_EARLY          0.000 // none
postprocess .D actions           0.017 // from the giant .D
unmarshal compressed BIFF        0.033 // default BIFFs only
resolve labels                   0.033 // I'm guessing more .D stuff
process .D actions               0.033 // from the giant .D
unmarshal BIFF                   0.050 // the Understudy
<<<<<<                           0.050 // at least a thousand of these
preprocess .D actions            0.083 // from the giant .D
resolve strings                  0.083 // from whatever
parsing .ids files               0.117 // n/a
APPEND                           0.167 // dialogue or Tp2? (100 or so TP_Append)
BUT_ONLY                         0.217 // for every single Tp2 action!
parsing .trb files               0.500 // Why do we have to parse TRA files, too, then?
unmarshal KEY                    0.550 // yeah
find local string                0.667 // huh?
marshal and save TLK             0.783 // yeah
marshal DLG                      0.933 // from the giant .D file, I guess
unmarshal DLG                    1.033 // from .D
unmarshal TLK                    1.200 // whatever
parsing .d files                 1.650 // from the 500+ dialogue updates
READ_*                           2.150 // lots of these
eval_pe                          3.800 // tons of expressions
STRING_SET                       4.617 // every single string
COPY                             4.950 // about 6,000 files
parsing .baf files               5.850 // about 1,000 of these
loading files                    9.167 // yeah
parsing .tra files               9.583 // Where did this come from?
process_patch2                  13.233 // actually doing some important stuff here
saving files                    19.883 // go disk go
parsing .tp2 files              26.617 // yeah
COMPILE                         86.600 // ouch! I know what to comment out for normal dev runs, now
TOTAL                          194.650I had other things going on the installation machine, so the timings aren't really optimal, but should be close enough. Under 4 minutes now, at least. The ball is still up in the air for binary TRAs (I want to do a "clean" run here with only WeiDU running), but cutting the parse time in half isn't giving a bad initial impression.

[weimer]

Given the following TP2 code:

That code won't work (as you note). I'm not sure why you want to defer the _LATER for so long and then do them all at once, but I can see where having variables in the _NOW/_LATER variable name would help with that.

and then using PatchNow in a single copy action

Why not put one _NOW at the end of the each action where you do > 0 _LATERs?

I had other things going on the installation machine, so the timings aren't really optimal, but should be close enough. Under 4 minutes now, at least. The ball is still up in the air for binary TRAs (I want to do a "clean" run here with only WeiDU running), but cutting the parse time in half isn't giving a bad initial impression.

You didn't convert all of your TRAs over, so it's hard to tell. Are those other TRAs for that mod or some other mod?

The COMPILE time may actually be for processing translation strings, marshaling BCS files, ... ah, I have a theory. The cost is probably from push_copy_trans. Do you load a huge generic/universal TRA file at the beginning of your TP2? I'm guessing that you do!

[weimer]

You could check fairly easily (well, five minutes later) by changing the definition of push_copy_trans on line 94 of src/dc.ml to:

Code: [Select]

let push_copy_trans () =
  Stats.time "push_copy_trans" (fun () ->
  let now = match !trans_strings with
    hd :: tl -> hd
  | [] -> Hashtbl.create 511
  in 
  trans_strings := (Hashtbl.copy now) :: !trans_strings
  ) ()

... and then checking the timing. If my guess is write, most of the time from COMPILE will go to pct.

[devSin]

Why not put one _NOW at the end of the each action where you do > 0 _LATERs?

I currently do (and am fine with this). I was just thinking about the way PatchLater works, and about exploiting the fact that WeiDU retains all the data indefinitely (either the first PatchNow ~patchLaterList~ or WeiDU just finishes installing). It was really just a consequence of having to re-examine all the ancient 2DA code in the TP2 here, and thinking about how I could streamline it by changing everything to requiredColumnCount 1 and then just shoving all the data out in one universal push at the end.

Like I said, it's not terribly practical, nor very safe to rely on user-controlled data floating around in the ether. I was just wondering if variable substitution was done, and if the suggested approach would be possible if it was, not that I think it's something anybody should or would ever actually do.

The COMPILE time may actually be for processing translation strings, marshaling BCS files, ... ah, I have a theory. The cost is probably from push_copy_trans. Do you load a huge generic/universal TRA file at the beginning of your TP2? I'm guessing that you do!

The TRA you downloaded is the single TRA for the mod. I load it (via the LANGUAGE command at the head of TP2). It's only actually required for the STRING_SETs, some textual REPLACE in 2DA files, a slew of SAY patches, and about 5-10 scripts (and then using REPLACE, not any AUTO_TRA or anything). I simply don't know how to defer loading all 5M of a TRA (I was hoping STRING_SET x USING would help, but it turned out to be egregiously slow (since it reloads for each action)), when it's only needed for the STRING_SETs, a couple hundred simple patches, and 10 scripts (using AUTO_TRA would be dumb, as I'd have a single TRA for patching all of one string in a script).

You could check fairly easily (well, five minutes later) by changing the definition of push_copy_trans on line 94

I'm back home so I'll do this now.