Author Topic: This code combination is too slow...  (Read 1171 times)

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
This code combination is too slow...
« on: August 27, 2005, 11:00:15 AM »
Complete package (without the weidu executable): http://poisson.dm.unipi.it/~bigiani/files/test.rar.

Basically, I'm using READ_2DA_ENTRY on a 220 Kb file, which is composed of two coloumns: the first short (a file name), the second long (an item description, coded to remove spaces and newlines), eg
Code: [Select]
BRUENPLA Mithral___SPACE___Field___SPACE___Plate___SPACE___+2:___SPACE______NL___This___SPACE___is___SPACE___the___SPACE___personal___SPACE___armor___SPACE___of___SPACE___Bruenor___SPACE___Battlehammer,___SPACE___and___SPACE___never___SPACE___a___SPACE___finer___SPACE___kit___SPACE___have___SPACE___you___SPACE___seen.___SPACE______SPACE___It___SPACE___is___SPACE___truly___SPACE___a___SPACE___marvel___SPACE___of___SPACE___battlefield___SPACE___craftsmanship.___SPACE______NL___STATISTICS:___NL___Resistance___SPACE___to___SPACE___slashing___SPACE___damage:___SPACE___30___NL___Resistance___SPACE___to___SPACE___piercing___SPACE___damage:___SPACE___15___NL___Resistance___SPACE___to___SPACE___blunt___SPACE___damage:___SPACE___5___NL___Resistance___SPACE___to___SPACE___missile___SPACE___damage:___SPACE___35___NL___Dexterity___SPACE___penalty:___SPACE___5___NL___Penalty___SPACE___to___SPACE___movement:___SPACE___25___NL___Spell___SPACE___failure:___SPACE___45___NL___Penalty___SPACE___to___SPACE___pick___SPACE___pocket:___SPACE___90___NL___Penalty___SPACE___to___SPACE___lock___SPACE___picking:___SPACE___60___NL___Penalty___SPACE___to___SPACE___trap___SPACE___finding___SPACE___and___SPACE___disarming:___SPACE___65___NL___Penalty___SPACE___to___SPACE___Move___SPACE___Silently:___SPACE___95___NL___Penalty___SPACE___to___SPACE___hide___SPACE___in___SPACE___shadows:___SPACE___90___NL___Armor___SPACE___Class:___SPACE___-1___NL___Weight:___SPACE___45___NL___Not___SPACE___Usable___SPACE___By:___NL______SPACE___Bard___NL______SPACE___Druid___NL______SPACE___Mage___NL______SPACE___Thief___NL______SPACE___Archer___NL______SPACE___Stalker___NL______SPACE___Barbarian___NL______SPACE___Beast___SPACE___Master___NL______SPACE___Kensai___NL______SPACE___Monk
(there are 110 lines, totalling 220 kb of text).

This in order to automate those lines of
Code: [Select]
COPY_EXISTING ~a.itm~ ~override~
SAY DESC @1
SAY UNIDENTIFIED_DESC @2
we all so like typing.

However, using READ_2DA_ENTRY on this beast is really slow.

This code (which only reads the filenames until it finds the correct one) takes one minute and an half, while the rest of the whole component (including READ_2DA_ENTRY on the real description, turning it into human-readable form, SAYing it, and otherwise patching the file) doesn't slow more than 10 seconds.

If you pay attention to file you're operating on (via the PATCH_PRINT I added), you'll see that items not on the 2da listing (non-armor and non-shield items) are processed istantly, while items at the top of the list (eg CHAN*) are decently quick, and the ones at the bottom of the list (SHLD*, WA*) are the slowest, with even a second per one for the very last ones.

So, I was wondering if it's possible to find a way to speed this up, maybe with something alike to the SET_2DA_ENTRY_LATER/SET_2DA_ENTRIES_NOW combo, via READ_2DA_ENTRIES_NOW/READ_2DA_ENTRY_FORMER: while successive S_2_E_L memorize the changes to be applied in the RAM rather than in the file and S_2_E_N flushes all changes to the file in one go, R_2_E_N will read all entries in the RAM in one go, and then R_2_E_F will return the value from the RAM, rather than from the file.
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline Avenger_teambg

  • Planewalker
  • *****
  • Posts: 399
Re: This code combination is too slow...
« Reply #1 on: August 30, 2005, 02:24:53 AM »
Guess you hit the limits of weidu :)
All purpose non-interpreted scripts can't beat the speed of stuff specifically written for a given problem.
Maybe weidu needs another command designed exactly for this purpose, or you need another tool :)

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: This code combination is too slow...
« Reply #2 on: August 30, 2005, 03:37:11 AM »
Guess you hit the limits of weidu :)
[snip]
Maybe weidu needs another command designed exactly for this purpose[etc]
That's, uh, the purpose of a feature request  :P
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

Offline the bigg

  • The Avatar of Fighter / Thieves
  • Moderator
  • Planewalker
  • *****
  • Posts: 3804
  • Gender: Male
Re: This code combination is too slow...
« Reply #3 on: August 30, 2005, 10:26:47 AM »
I was able to speed this code up by adding array functionality to tp2 (without ever altering the source of weidu.exe, obviously). self-told times went from 1:20 min to 19 secs (and I was installing the full mod, not only that snippet I posted)  :hooray: !

example code of array functionality: array is an int array, stringarray is a array of strings. whereas in other languages you'd write array, here you have to do a lot of workarounds via EVALUATE_BUFFER, underscores and temp variable.
Code: [Select]
COPY_EXISTING ~sw1h01.itm~ ~override~
FOR (i = 0; i < 10; i += 1)
BEGIN
SPRINT loop ~%i%~

                // setting values into an int array
SET EVALUATE_BUFFER ~array_%loop%~ = 2 * i // SET array_0 = 2 * 0

                // setting values into an array of strings
SPRINT EVALUATE_BUFFER ~stringarray_%loop%~ ~I am %loop%~ // SPRINT stringarray_0 ~I am 0~
END

FOR (i = 0; i < 10; i += 1)
BEGIN
SPRINT loop ~%i%~

                // reading from an array of ints
SET thisread = EVALUATE_BUFFER ~array_%loop%~

                // reading from an array of strings, notice the double SPRINT directive
SPRINT thisstring EVALUATE_BUFFER ~stringarray_%loop%~
SPRINT thisstring EVALUATE_BUFFER ~%%thisstring%%~ // notice the double %% quoting

                // PATCH_PRINT doesn't support EVAL_BUFFER, yet.
PATCH_PRINT ~%i% * 2 = %thisread%; %thisstring%~
END
Notice that the array index I used is a string (it's created via SPRINT loop ~%i%~), so you can even do an array with string index (EG SPRINT loop ~%SOURCE_REF%~).

In a nutshell: to write in an array, you need the index to be a string (so, if it's a number, use SPRINT). Once that is done, you write in it by SET/SPRINT EVALUATE_BUFFER ~array.name_%loop.variable.name%~.
To read ints you do the same thing (the index becomes a string, then you use EVALUATE_BUFFER ~array.name_%loop.variable.name%~ to access its value).
To read from an array of strings, you need to firstly SPRINT temporary.variable EVALUATE_BUFFER ~array.name_%loop.variable.name%~, then to SPRINT real.variable EVALUATE_BUFFER ~%%temporary.variable%%~, with double '%' signs.

Since SPRINT and EVAL_BUFFER aren't speed-optimized functions, you are advised to read all your arrays into normal variables and operate on these copies, rather than working onto the in-array variables.



PS: 2,3,N-dimensional matrixes are left as an exercise to the reader  :)
« Last Edit: August 31, 2005, 04:07:04 AM by the bigg »
Author or Co-Author: WeiDU (http://j.mp/bLtjOn) - Widescreen (http://j.mp/aKAiqG) - Generalized Biffing (http://j.mp/aVgw3U) - Refinements (http://j.mp/bLHoCc) - TB#Tweaks (http://j.mp/ba02Eg) - IWD2Tweaks (http://j.mp/98OFYY) - TB#Characters (http://j.mp/ak8J55) - Traify Tool (http://j.mp/g1Ry9A) - Some mods that I won't mention in public
Maintainer: Semi-Multi Clerics (http://j.mp/9UeIwB) - Nalia Mod (http://j.mp/dng9l0) - Nvidia Fix (http://j.mp/aRWjjg)
Code dumps: Detect custom secondary types (http://j.mp/hVzzXG) - Stutter Investigator (http://j.mp/gdtBn8)

If possible, send diffs, translations and other contributions using Git (http://j.mp/aBZFrq).

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: