Author Topic: Help creating/using a simple array (for use as lookup table)  (Read 1664 times)

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
Help creating/using a simple array (for use as lookup table)
« on: September 18, 2017, 02:37:42 PM »
Background:
I'm writing a function (for a testing/debugging tool, not a mod) that takes a large file, reads it a byte at a time, formats that byte, then SPRINTs the formatted string to the end of a string to be PRINTed or otherwise saved to a log file for later examination.

The code looks something like this (but looped for every byte in the file):
Code: [Select]
READ_BYTE 0x0 mybyte
SPRINT mybyte ~   %mybyte%~
INNER_PATCH_SAVE mybyte~%mybyte%~ BEGIN // Format the current byte to be exactly 3 characters, padded with spaces
  DELETE_BYTES 0 (BUFFER_LENGTH - 3)
END
SPRINT info ~%info%%mybyte% ~ // String containing every byte in the file, converted to an integer (0-255) and formatted with spaces
This is a simplified version and while what I have written works, it takes 40 minutes (reported by ACTION_TIME, thanks Wisp!) to process a ~1MB file which is absurdly ridiculous.

What I'd like to do:
I've got a couple thoughts on how to make this faster (but I'm certainly open to suggestions!).  The primary one being instead of having to format all 1,000,000 bytes of the file one at a time, I'd like to format all 256 possible integers, store them in an array, and use the array as a lookup table so that
Code: [Select]
SPRINT mybyte~   %mybyte%~
INNER_PATCH_SAVE mybyte~%mybyte%~ BEGIN // Format the current byte to be exactly 3 characters, padded with spaces
  DELETE_BYTES 0 (BUFFER_LENGTH - 3)
END
doesn't have to be executed for every single byte in the file.

My problem is that the array syntax WeiDU uses is vastly different than any array/object syntax I'm used to, and even after reading the documentation and included array construct "tutorial" I don't understand how to dynamically assign key, value pairs (or retrieve them).  Any help or suggestions would be appreciated.

-Sam.
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Magus_BGforge

  • Planewalker
  • *****
  • Posts: 75
Re: Help creating/using a simple array (for use as lookup table)
« Reply #1 on: September 20, 2017, 05:36:48 AM »
I'm sure there's a more efficient way for this, but as for arrays, here's the syntax:
Code: [Select]
ACTION_DEFINE_ASSOCIATIVE_ARRAY bytemap BEGIN
  1 => ~001~
  2 => ~002~
END
OUTER_SET x = 1
OUTER_SPRINT y $bytemap(~%x%~)
PRINT ~%y%~

Offline Argent77

  • Planewalker
  • *****
  • Posts: 187
Re: Help creating/using a simple array (for use as lookup table)
« Reply #2 on: September 20, 2017, 10:20:44 AM »
Arrays are quite tricky in WeiDU when you have to construct them dynamically. I'm never certain which notation ( $array(idx) or array_%idx% ) works in the respective context.

This code seems to work correctly though:
Code: [Select]
// create fixed length value strings for byte range
OUTER_FOR (idx = 0; idx < 256; ++idx) BEGIN
  ACTION_IF (idx < 10) BEGIN
    OUTER_TEXT_SPRINT value ~  %idx%~
  END ELSE ACTION_IF (idx < 100) BEGIN
    OUTER_TEXT_SPRINT value ~ %idx%~
  END ELSE BEGIN
    OUTER_TEXT_SPRINT value ~%idx%~
  END
  OUTER_TEXT_SPRINT $EVAL ~array~(~%idx%~) ~%value%~
END

// test run: print 20 randomly chosen values
OUTER_FOR (idx = 0; idx < 20; ++idx) BEGIN
  OUTER_SET curVal = RANDOM(0 255)
  OUTER_TEXT_SPRINT result $EVAL ~array~(~%curVal%~)
  PRINT ~Pass #%idx%: "%result%"~
END

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
Re: Help creating/using a simple array (for use as lookup table)
« Reply #3 on: September 20, 2017, 12:40:38 PM »
I'm sure there's a more efficient way for this, but as for arrays, here's the syntax:
Code: [Select]
ACTION_DEFINE_ASSOCIATIVE_ARRAY bytemap BEGIN
  1 => ~001~
  2 => ~002~
END
OUTER_SET x = 1
OUTER_SPRINT y $bytemap(~%x%~)
PRINT ~%y%~
Thank you.  Although a bit unwieldy, I have found
Code: [Select]
ACTION_DEFINE_ARRAY array BEGIN "  0" "  1" "  2" "  3" "  4" "  5" "  6" "  7" "  8" "  9" " 10" " 11" " 12" " 13" " 14" " 15" " 16" " 17" " 18" " 19" " 20" " 21" " 22" " 23" " 24" " 25" " 26" " 27" " 28" " 29" " 30" " 31" " 32" " 33" " 34" " 35" " 36" " 37" " 38" " 39" " 40" " 41" " 42" " 43" " 44" " 45" " 46" " 47" " 48" " 49" " 50" " 51" " 52" " 53" " 54" " 55" " 56" " 57" " 58" " 59" " 60" " 61" " 62" " 63" " 64" " 65" " 66" " 67" " 68" " 69" " 70" " 71" " 72" " 73" " 74" " 75" " 76" " 77" " 78" " 79" " 80" " 81" " 82" " 83" " 84" " 85" " 86" " 87" " 88" " 89" " 90" " 91" " 92" " 93" " 94" " 95" " 96" " 97" " 98" " 99" "100" "101" "102" "103" "104" "105" "106" "107" "108" "109" "110" "111" "112" "113" "114" "115" "116" "117" "118" "119" "120" "121" "122" "123" "124" "125" "126" "127" "128" "129" "130" "131" "132" "133" "134" "135" "136" "137" "138" "139" "140" "141" "142" "143" "144" "145" "146" "147" "148" "149" "150" "151" "152" "153" "154" "155" "156" "157" "158" "159" "160" "161" "162" "163" "164" "165" "166" "167" "168" "169" "170" "171" "172" "173" "174" "175" "176" "177" "178" "179" "180" "181" "182" "183" "184" "185" "186" "187" "188" "189" "190" "191" "192" "193" "194" "195" "196" "197" "198" "199" "200" "201" "202" "203" "204" "205" "206" "207" "208" "209" "210" "211" "212" "213" "214" "215" "216" "217" "218" "219" "220" "221" "222" "223" "224" "225" "226" "227" "228" "229" "230" "231" "232" "233" "234" "235" "236" "237" "238" "239" "240" "241" "242" "243" "244" "245" "246" "247" "248" "249" "250" "251" "252" "253" "254" "255" ENDworks well enough and resolves quickly with minimal computations.

I'm sure there's a more efficient way for this
Agreed.  My next bottleneck seems to be the large number of string concatenations.  I'm thinking calculating the total amount of space needed, allocating that space via INSERT_BYTES, and then many WRITE_ASCIIs should be quicker.  The question is whether having to be in an INNER_PATCH_SAVE requites enough overhead to negate the speed increase from not having to repeatedly increase the string buffer size.  I'll have to test this.

@Argent77
Thanks for this example!  I believe I understand how
Quote
$array(idx)
is used, but what is
Quote
array_%idx%
supposed to do and how/in what context would WeiDU know that this should be resolved into a value in an array?

Is
Quote
$EVAL
documented somewhere?  Presumably this is EVALUATE_BUFFER, but it evaluates array syntax instead of (or in addition to?) variables surrounded with %s?  When can it be used, and is it possible to have an array expression evaluated within a string (such as for PRINT) instead of going through OUTER_TEXT_SPRINT first?
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Argent77

  • Planewalker
  • *****
  • Posts: 187
Re: Help creating/using a simple array (for use as lookup table)
« Reply #4 on: September 20, 2017, 01:47:36 PM »
EVAL (or EVALUATE_BUFFER) can be used in this context to evaluate variable names.

Example:
Code: [Select]
OUTER_TEXT_SPRINT var_name ~first_var~
OUTER_SET EVAL ~%var_name%~ = 10  // resolves into: OUTER_SET first_var = 10
OUTER_TEXT_SPRINT var_name ~second_var~
OUTER_SET EVAL ~%var_name%~ = 10  // resolves into: OUTER_SET second_var = 10

Its basically the same with array variables. The '$' in front of EVAL is simply the array variable prefix of $array(index). array_0 appears to be just another notation of $array(0) in WeiDU. In any case you can use one or the other variant to set or read from the same variable.
« Last Edit: September 20, 2017, 05:10:35 PM by Argent77 »

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
Re: Help creating/using a simple array (for use as lookup table)
« Reply #5 on: September 22, 2017, 12:36:22 PM »
In this example
Code: [Select]
OUTER_SPRINT var1 ~ps~
OUTER_SET var2 = 0
OUTER_SPRINT EVAL ~%var1%%var2%~ ~%var2%~
OUTER_SPRINT var3 ~%ps0%~
PRINT ~var3 = %var3%~
the third line resolves into OUTER_SPRINT ps0 ~0~, so in line 4 the variable var3 gets set to "0" so line 5 prints "var3 = 0".  How do I achieve the same result if I don't know that the variable in line 3 resolves to ps0, I just know that it is ~%var1%%var2%~?  Substituting line 4 for OUTER_SPRINT var3 EVAL ~%var1%%var2%~ doesn't work like I thought it would.  What am I missing?
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1176
Re: Help creating/using a simple array (for use as lookup table)
« Reply #6 on: September 22, 2017, 01:13:58 PM »
Code: [Select]
OUTER_SPRINT var3 EVAL "%%var1%%var2%%"
[code]

$EVAL was made redundant some past version ago. The array construct automatically evaluates variables in the array name.

Offline Sam.

  • The moose man
  • Planewalker
  • *****
  • Posts: 86
  • Gender: Male
    • Classic Adventures Homepage
Re: Help creating/using a simple array (for use as lookup table)
« Reply #7 on: September 29, 2017, 01:40:07 PM »
Code: [Select]
OUTER_SPRINT var3 EVAL "%%var1%%var2%%"
[code]

$EVAL was made redundant some past version ago. The array construct automatically evaluates variables in the array name.
That's some fun devil's magic.  Can I use something like %%var%% to perform a direct double-deref?

It turns out that it takes significantly longer to store a large number of values in an array than it does to store them in a large number of independent variables.

After quite a few iterations of testing, I've been able to get that initial ~40 minutes down to around 27 seconds.  My goal was under 30 seconds, so I'm happy.  This makes me wonder about all those large mods that take "hours" to install.  Makes me wonder if they were coded poorly not coded with efficiency/install time in mind.
"Ok, I've just about had my FILL of riddle asking, quest assigning, insult throwing, pun hurling, hostage taking, iron mongering, smart-arsed fools, freaks, and felons that continually test my will, mettle, strength, intelligence, and most of all, patience! If you've got a straight answer ANYWHERE in that bent little head of yours, I want to hear it pretty damn quick or I'm going to take a large blunt object roughly the size of Elminster AND his hat, and stuff it lengthwise into a crevice of your being so seldom seen that even the denizens of the nine hells themselves wouldn't touch it with a twenty-foot rusty halberd! Have I MADE myself perfectly CLEAR?!"
-- <CHARNAME> to Portalbendarwinden

Offline Argent77

  • Planewalker
  • *****
  • Posts: 187
Re: Help creating/using a simple array (for use as lookup table)
« Reply #8 on: September 29, 2017, 03:04:05 PM »
Code: [Select]
OUTER_SPRINT var3 EVAL "%%var1%%var2%%"
[code]

$EVAL was made redundant some past version ago. The array construct automatically evaluates variables in the array name.
That's some fun devil's magic.  Can I use something like %%var%% to perform a direct double-deref?
That's indeed some useful info. It helps tremendously in the script parser library I'm currently working on.

After a bit of experimenting I got direct double dereferencing working. It looks like you need two EVAL directives in front of the string.

Example:
Code: [Select]
OUTER_TEXT_SPRINT var ~real content~
OUTER_TEXT_SPRINT outer ~var~
OUTER_TEXT_SPRINT inner ~outer~

// double dereferencing in two steps
OUTER_TEXT_SPRINT temp EVAL ~%%inner%%~
OUTER_TEXT_SPRINT result EVAL ~%%temp%%~

// direct double dereferencing
OUTER_TEXT_SPRINT direct EVAL EVAL ~%%%inner%%%~

PRINT ~result = %result%, direct = %direct%~

Both variables evaluate to "real content".

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: