v1.33b20 Issues / Suggestions

[Sam.]

These refer to NearInfinity v1.33b20 as of 2009-09-03 found on GitHub.

Issues/Problems:

Unless the "Keyfile Cleanup" tool has gotten a complete make-over since v1.33b19, it can still completely destroy your game.  I used it on my ToB game and despite my having made a backup of my chitin.key, I had to completely reinstall my game to get it working again (but thin again I was dumb enough to install a few mods after doing the Keyfile Cleanup but before realizing that the game was broken).

Suggestions:

• Add a R-Click Cut/Copy/Paste dialog box to things like script editing.
• Add a "Search" function for file names, especially the file types that aren't identified by a string reference such as items and creatures.
• Add some real support for .PLT files (the solid color selections for the whole body aren't even close to accurate).  It might be worth looking into how WizWom made his BMP<->PLT converters.
• Add support for exporting .PLT files as .BMP
• I think it would be amazingly useful to enable "Store Purchases" info as checks (like how the "Flags" work), instead of having to add an individual entry for every type of item purchased.
• It would be great if there was an option under "Options" to set the memory usable by NI.  As it it, the "File->View Area (Experimental)" frequently runs out of memory.
• Could Ctrl+F be made to where it sends to the main window whenever it is pressed when any NI window is active?  (If that didn't make sense I can try to explain further.)
• Could more text fonts be added, such as Time New Roman?
• Add support to export .TIS files as .BMPs and be prompted for the dimensions.
• Add ability to open more that 1 external file at a time (Drag & Drop supports it but "Browse..." does not).
• Check to make sure the same game (same path to chitin.key) is not found under Game tab more that once.  (I currently have three different entries for the same Icewind Dale - Heart of Winter game that I can't make go away.)
• Also, and I know it's complicated, but can you add a check for potions equipped in a creature's inventory that are unusable by that creature?

TIA,
Sam.

[Jarno Mikkola]

Suggestion...
-Could somebody export/compile the v1.33b20 as a .jar file so it could be executed in PC too.

[Taimon]

I won't be working on NI for a while, but I'll keep things in mind.
Fixing bugs has a much higher priority for me, though.

[Sam.]

I have noticed that even after a child window of NI is closed, it doesn't seem to be removed from memory.  So when NI's allocated memory is full, it kind of freezes and will not open any more windows.  For me, it freezes up after opening and then closing around 15 or 16 child windows.  NI must be completely restarted for it to start working again.  I recall other people commenting about this in earlier versions, so I don't think it's just me.  Is this a problem with NI or java, and can it be fixed?  It gets really frustrating having to restart NI every couple of minutes while I'm trying to work on something.

TIA,
Sam.

[Jarno Mikkola]

...

Using the BP-BGT_Worldmap's NearInfinity Patches... Link?

[Sam.]

...

Using the BP-BGT_Worldmap's NearInfinity Patches... Link?

I did know about these, and while they increase the amount of memory available for NI to use, they don't address the underlying problem.  The child windows aren't being released from memory when they close.  I did a test to make sure it wasn't just my imagination, and the child windows really aren't being released from memory when they close.

[Jarno Mikkola]

I did know about these, and while they increase the amount of memory available for NI to use, they don't address the underlying problem.  The child windows aren't being released from memory when they close.  I did a test to make sure it wasn't just my imagination, and the child windows really aren't being released from memory when they close.

Yes, because the info in them hasn't been re-saved as of yet... is that a good or bad thing, I do not know.

[devSin]

It's fairly common to hold on to windows after their initial creation (I'm not sure any NI windows are truly disposed in code). I didn't know anybody would open 15 windows at the same time (unless you're talking about 15 open/close sessions, but that shouldn't be killing NI for you). I suppose it can be looked at, but there aren't any places I'm aware of where memory is handled manually (is it even possible with Java?).

Resource filename search sounds OK, but it would require changes to the interface, and I can't think of a svelte way to do it. Note you can search for "No such index" and similar to pick up no-string files. Attribute reference searcher also allows you to search for the current selected field across all resources of the same type. (So, if you already know the filename, I'm not sure why you're needing to search for it.) I believe you can also use the Open File command to select the filename from a list of available resources.

I believe we currently load a PLT file by itself without any creature association (meaning, there are no colors to pick from). There could be color pickers for every single location, I suppose, but I don't know how useful that would be. I agree that putting the ToB animation palette BMPs there is totally pointless.

As with window management, I don't think we can manually adjust memory at runtime, so no preference to set the memory level. Yay, Java! You can start the VM with -Xms (start; try 64M) and -Xmx (max; try 128M) to make Java use more memory.

We don't do anything special with the fonts (you get a list of fonts that are accessible to Java by default). I don't know how we can call out to fonts across all three platforms, but it doesn't sound like something that would be impossible.

Yeah, the game history menu is pretty bonky.

Probably no for the potion checker (it's easy enough to do this in WeiDU). NI supports all the different Infinity games, and they all use different usability rules, so it would be a bunch more code than just creating a checker with WeiDU for exactly what you want.

[Sam.]

(unless you're talking about 15 open/close sessions, but that shouldn't be killing NI for you)

That's exactly what I'm talking about.

Resource filename search sounds OK, but it would require changes to the interface, and I can't think of a svelte way to do it.  (So, if you already know the filename, I'm not sure why you're needing to search for it.)

For instance, if you wanted to find all .BAM inventory paperdolls, you could search all of the .BAM file names for "INV".

Yeah, the game history menu is pretty bonky.

Where are the game paths stored.  In the registry?  In a dynamic variable in the .JAR?  Maybe I can clean mine out manually.

TIA,
Sam.

[devSin]

That's exactly what I'm talking about.

Yuck. I'll let somebody with more Windows Java experience isolate this, but we should be getting that memory back eventually (certainly, I can open and close windows all day here without issue).

Where are the game paths stored.  In the registry?  In a dynamic variable in the .JAR?  Maybe I can clean mine out manually.

On Windows, they're probably in the registry. Try searching for "LastGameID1" - "LastGameID5" and "LastGamePath1" - "LastGamePath5" keys.

[ScuD]

OK, here's what you need to do to build NearInfinity from the source code.
1. Download and install the latest version of JDK 6 (not JRE!!!). It is available at java.sun.com downloads. For example, to z:\java. It will install JDK and JRE in separate folders.
2. Download and unpack the latest Ant binary from http://ant.apache.org/bindownload.cgi. For, example, to z:\. It will create a folder named apache-ant1.7.1.
3. Add 2 environment variables
JAVA_HOME=<your path to JDK folder>, like z:\java\jdk1.6.0_16
ANT_HOME=<our path to Ant folder>, like z:\apache-ant1.7.1
4. Add %JAVA_HOME%\bin;%ANT_HOME%\bin to your environment PATH variable.
5. Download and unpack Taimon's sources from http://github.com/taimon/NearInfinity. For example, to z:\NI. (Folder \infinity and build.xml file should be there)
6. Go to the source folder (z:\NI in our case), start command line and simply type ant. It should compile the source and create NearInfinity.jar.

[Taimon]

I have noticed that even after a child window of NI is closed, it doesn't seem to be removed from memory.  So when NI's allocated memory is full, it kind of freezes and will not open any more windows.  For me, it freezes up after opening and then closing around 15 or 16 child windows.  NI must be completely restarted for it to start working again.

Sounds like something is wrong with your JVM. I can open/close as many windows as I like, no freezing.

To actually determine how much memory the application uses, you have to ask the JVM directly after a garbage collection.
(Try the jconsole application in the JRE/JDK - you can trigger a garbage collection cycle manually there.)

[Sam.]

I have noticed that even after a child window of NI is closed, it doesn't seem to be removed from memory.  So when NI's allocated memory is full, it kind of freezes and will not open any more windows.  For me, it freezes up after opening and then closing around 15 or 16 child windows.  NI must be completely restarted for it to start working again.

Sounds like something is wrong with your JVM. I can open/close as many windows as I like, no freezing.

To actually determine how much memory the application uses, you have to ask the JVM directly after a garbage collection.
(Try the jconsole application in the JRE/JDK - you can trigger a garbage collection cycle manually there.)

So, here is NI before opening any child windows:

Code: [Select]

VM Summary
Thursday, September 24, 2009 9:47:17 AM CDT
________________________________________
Connection name:  pid: 2596 C:\Documents and Settings\Sam Schmitz\Desktop\Baldur's Gate documents\NearInfinity\NearInfinity beta 20.jar
Virtual Machine:  Java HotSpot(TM) Client VM version 14.2-b01
Vendor:  Sun Microsystems Inc.
Name:  2596@Sammoose
Uptime:  4 minutes
Process CPU time:  4.500 seconds
JIT compiler:  HotSpot Client Compiler
Total compile time:  0.343 seconds

________________________________________
Live threads:  16
Peak:  17
Daemon threads:  13
Total threads started:  25
Current classes loaded:  2,289
Total classes loaded:  2,318
Total classes unloaded:     29


________________________________________
Current heap size:  12,925 kbytes
Maximum heap size:  65,088 kbytes
Committed memory:  22,612 kbytes
Pending finalization:  0 objects

Garbage collector:  Name = 'Copy', Collections = 42, Total time spent = 0.205 seconds
Garbage collector:  Name = 'MarkSweepCompact', Collections = 6, Total time spent = 2.053 seconds

________________________________________
Operating System:  Windows XP 5.1
Architecture:  x86
Number of processors:  1
Committed virtual memory:  49,188 kbytes
Total physical memory:  2,088,316 kbytes
Free physical memory:    682,792 kbytes
Total swap space:  2,717,652 kbytes
Free swap space:  1,349,680 kbytes

________________________________________
VM arguments: 
Class path:  C:\Documents and Settings\Sam Schmitz\Desktop\Baldur's Gate documents\NearInfinity\NearInfinity beta 20.jar
Library path:  E:\Program Files\Java\jre6\bin;.;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Common Files\DivX Shared\;C:\Program Files\QuickTime\QTSystem\
Boot class path:  E:\Program Files\Java\jre6\lib\resources.jar;E:\Program Files\Java\jre6\lib\rt.jar;E:\Program Files\Java\jre6\lib\sunrsasign.jar;E:\Program Files\Java\jre6\lib\jsse.jar;E:\Program Files\Java\jre6\lib\jce.jar;E:\Program Files\Java\jre6\lib\charsets.jar;E:\Program Files\Java\jre6\classes


Code: [Select]

Time:  2009-09-24 10:02:28
Used:      13,196 kbytes
Committed:      15,504 kbytes
Max:      65,088 kbytes
GC time:  0.159 seconds on Copy (37 collections)
0.332 seconds on MarkSweepCompact (3 collections)

and here is NI after opening and then closing child windows untill NI freezes:

Code: [Select]

VM Summary
Thursday, September 24, 2009 9:55:01 AM CDT
________________________________________
Connection name:  pid: 2596 C:\Documents and Settings\Sam Schmitz\Desktop\Baldur's Gate documents\NearInfinity\NearInfinity beta 20.jar
Virtual Machine:  Java HotSpot(TM) Client VM version 14.2-b01
Vendor:  Sun Microsystems Inc.
Name:  2596@Sammoose
Uptime:  11 minutes
Process CPU time:  17.453 seconds
JIT compiler:  HotSpot Client Compiler
Total compile time:  0.921 seconds

________________________________________
Live threads:  16
Peak:  19
Daemon threads:  13
Total threads started:  39
Current classes loaded:  2,492
Total classes loaded:  2,632
Total classes unloaded:    140


________________________________________
Current heap size:  64,944 kbytes
Maximum heap size:  65,088 kbytes
Committed memory:  65,088 kbytes
Pending finalization:  0 objects

Garbage collector:  Name = 'Copy', Collections = 74, Total time spent = 0.413 seconds
Garbage collector:  Name = 'MarkSweepCompact', Collections = 26, Total time spent = 6.443 seconds

________________________________________
Operating System:  Windows XP 5.1
Architecture:  x86
Number of processors:  1
Committed virtual memory:  95,560 kbytes
Total physical memory:  2,088,316 kbytes
Free physical memory:    619,096 kbytes
Total swap space:  2,717,652 kbytes
Free swap space:  1,285,024 kbytes

________________________________________
VM arguments: 
Class path:  C:\Documents and Settings\Sam Schmitz\Desktop\Baldur's Gate documents\NearInfinity\NearInfinity beta 20.jar
Library path:  E:\Program Files\Java\jre6\bin;.;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Common Files\DivX Shared\;C:\Program Files\QuickTime\QTSystem\
Boot class path:  E:\Program Files\Java\jre6\lib\resources.jar;E:\Program Files\Java\jre6\lib\rt.jar;E:\Program Files\Java\jre6\lib\sunrsasign.jar;E:\Program Files\Java\jre6\lib\jsse.jar;E:\Program Files\Java\jre6\lib\jce.jar;E:\Program Files\Java\jre6\lib\charsets.jar;E:\Program Files\Java\jre6\classes


Code: [Select]

Time:  2009-09-24 09:58:54
Used:      64,967 kbytes
Committed:      65,088 kbytes
Max:      65,088 kbytes
GC time:  0.413 seconds on Copy (74 collections)
8.537 seconds on MarkSweepCompact (34 collections)

When I choose the option to do a Garbage Collect, nothing appears to happen.  Does this mean that there is never any Garbage Collection going on at all?

The data points on the graph of memory usage are:

Code: [Select]

Time Used Committed Max
40080.41826 14185328 15876096 66650112
40080.4183 14368568 15876096 66650112
40080.41834 13512840 15876096 66650112
40080.41839 13696672 15876096 66650112
40080.41844 13901256 15876096 66650112
40080.41848 14076288 15876096 66650112
40080.41853 14247648 15876096 66650112
40080.41858 13350424 15876096 66650112
40080.41862 13544480 15876096 66650112
40080.41867 13729200 15876096 66650112
40080.41872 13905736 15876096 66650112
40080.41877 14095776 15876096 66650112
40080.41881 14288792 15876096 66650112
40080.41908 13576064 15876096 66650112
40080.41915 13792616 15876096 66650112
40080.41921 14101408 15876096 66650112
40080.41925 14214064 15876096 66650112
40080.4193 14373968 15876096 66650112
40080.41934 13457744 15876096 66650112
40080.41939 13624408 15876096 66650112
40080.41944 13789496 15876096 66650112
40080.41948 13955424 15876096 66650112
40080.41953 14125000 15876096 66650112
40080.41957 14301848 15876096 66650112
40080.41962 14467080 15876096 66650112
40080.41967 13511088 15876096 66650112
40080.41971 13667496 15876096 66650112
40080.41976 13841008 15876096 66650112
40080.41981 13998328 15876096 66650112
40080.41985 14171376 15876096 66650112
40080.4199 14327760 15876096 66650112
40080.41994 13383632 15876096 66650112
40080.41999 13558200 15876096 66650112
40080.42004 13711896 15876096 66650112
40080.42008 13902080 15876096 66650112
40080.42013 14068512 15876096 66650112
40080.42018 14230352 15876096 66650112
40080.42022 14402280 15876096 66650112
40080.42027 13475576 15876096 66650112
40080.42031 13645944 15876096 66650112
40080.42036 13830232 15876096 66650112
40080.42041 13998400 15876096 66650112
40080.42045 14164760 15876096 66650112
40080.4205 14330856 15876096 66650112
40080.42055 14496936 15876096 66650112
40080.42059 14292016 15876096 66650112
40080.42064 16884144 24203264 66650112
40080.42069 20383360 24203264 66650112
40080.42073 23514944 35840000 66650112
40080.42078 30231968 35840000 66650112
40080.42082 34638728 35840000 66650112
40080.42087 36257216 58884096 66650112
40080.42092 39446320 58884096 66650112
40080.42096 42646456 58884096 66650112
40080.42101 48805728 58884096 66650112
40080.42106 52159864 58884096 66650112
40080.4211 55378792 58884096 66650112
40080.42115 58381184 66650112 66650112
40080.4212 64676880 66650112 66650112
40080.42125 64553832 66650112 66650112
40080.42129 65091680 66650112 66650112
40080.42134 65364696 66650112 66650112
40080.42139 65523728 66650112 66650112
40080.42143 65688952 66650112 66650112
40080.42148 65854176 66650112 66650112
40080.42153 66019352 66650112 66650112
40080.42157 66184000 66650112 66650112
40080.42162 66363016 66650112 66650112
40080.42167 66527568 66650112 66650112
40080.42171 64747712 66650112 66650112
40080.42178 65024608 66650112 66650112
40080.42181 65250320 66650112 66650112
40080.42186 65250320 66650112 66650112
40080.4219 65381048 66650112 66650112
40080.42195 65527976 66650112 66650112
40080.42199 65727168 66650112 66650112
40080.42204 65897440 66650112 66650112
40080.42208 66062664 66650112 66650112
40080.42213 66222840 66650112 66650112
40080.42218 66393112 66650112 66650112
40080.42222 66558336 66650112 66650112
40080.42227 66191336 66650112 66650112
Not as useful as the real graph, but oh well....

[Taimon]

GC is running, you see it in these lines:

Code: [Select]

GC time:  0.413 seconds on Copy (74 collections)
8.537 seconds on MarkSweepCompact (34 collections)

What happens if you start the JVM with increased maximum heap size, let's say 128 MiB. Exactly the same behavior?
The working set size (after all the stuff has been initialized) might be around 60 MiB, so you wouldn't really see an effect of the GC. But with 128 MiB max. heap it should be observable.

And what type of resources are you opening in those child windows? Does it even matter at all?

[Sam.]

GC is running, you see it in these lines:

Code: [Select]

GC time:  0.413 seconds on Copy (74 collections)
8.537 seconds on MarkSweepCompact (34 collections)

What happens if you start the JVM with increased maximum heap size, let's say 128 MiB. Exactly the same behavior?
The working set size (after all the stuff has been initialized) might be around 60 MiB, so you wouldn't really see an effect of the GC. But with 128 MiB max. heap it should be observable.

Exactly the same behavior except it took 37 open/close windows before NI froze.

Code: [Select]

Time:       2009-09-25 12:13:40
Used:       127,654 kbytes
Committed:  130,112 kbytes
Max:        130,112 kbytes
GC time:    0.484 seconds on Copy (80 collections)
            3.698 seconds on MarkSweepCompact (16 collections)

And what type of resources are you opening in those child windows? Does it even matter at all?

I have been using items an effects, but I just tried with MOS files and it froze after 80 windows with pretty much the same used, committed, max, and GC time.

[Taimon]

I've no idea what is happening on your system.
Does the same thing happen with older versions of NI?

[Sam.]

I've no idea what is happening on your system.
Does the same thing happen with older versions of NI?

Back through at least Jon's v1.32, yeah.

[Taimon]

Might be something with your JVM installation.
Do you have a chance to test on a different computer (with the most recent Java RE)?

By the way, what OS/Java version are you running?

[Sam.]

Might be something with your JVM installation.
Do you have a chance to test on a different computer (with the most recent Java RE)?

By the way, what OS/Java version are you running?

I just installed the latest java on my brother's new laptop and tested NI, and it still froze up.  Both of our computers are:

Your Java configuration is as follows:
Vendor: Sun Microsystems Inc.
Version: Java 6 Update 16
Operating System: Windows XP 5.1
Architecture: x86

[ScuD]

I looked at the NI with YourKit. It seems the application does not free the memory for the first opened window with a game resource. But that's OK, it is reusable for next opened window. The problem is each new opened and closed window increases the memory usage slightly. Full GC does not help. So the memory leak takes place.
I suspect some GUI elements are being recreated each time. Particularly, infinity.gui.viewframe adds a hashmap through infinity.resource.cre.CreResource (and through other resources as well, as I understand). It is possible this hashmap is not removed.

[Taimon]

You can't explicitly free memory in Java. If the ViewFrame isn't referenced any more, the GC should take care of it.
Besides, you should certainly get a java.lang.OutOfMemoryError, if there is a real problem with memory leakage.

Hm, maybe that's the case and you don't see the Exception.
Can someone else reproduce the issue?

[ScuD]

You can run GC manually from the profiler.
If you get +6K heap memory usage each time you open a window, it will take a lot of time to use all the heap by opening/closing windows, especially if you set -Xmx512m. So basically it is possible to get an OutOfMemoryError, but it will take some time. In my case after about half an hour all text turns into black rectangles - this is the sign that app is lacking the memory.
Anyway, the memory usage really increases during each window open -> the memory leak exists. Here is the link to GC log. I just opened and closed some windows randomly. You can see that the overall memory usage only increases.
I am positively sure the issue can be reproduced by anyone who opens NI and works with it for some time.
Of course, simply opening to edit some stuff and closing NI won't bring any problems to a user.

[Sam.]

Besides, you should certainly get a java.lang.OutOfMemoryError, if there is a real problem with memory leakage.

I get those when I use the "View Area (Experimental)" feature on more than one or two maps....  But not when NI freezes after the open/close window thing.  It probably doesn't matter, but I will go ahead and note that when NI "freezes", it won't open any new child windows, but it will still open files in the main window.

[Taimon]

If you get +6K heap memory usage each time you open a window, it will take a lot of time to use all the heap by opening/closing windows

Exactly, this is why I don't think it's related to Sam's problem here.

I did try to force the issue by giving the VM only 12 MiB heap size. Even then I was able to open/close 50+ windows, before the VM finally decided to kill the program. I guess the "freezing" is when the GC thread desperately tries to find some unreferenced objects and runs all the time.

[temujin.]

easiest way to reproduce the "freezing up" is to open & close several windows of SCS' 10000 line scripts, and/or run Tools --> Check Scripts (which, by the way, is very inaccurate and gives random results each time you run, unless you use Kung Fu Man's version).

if you set -Xmx1024m, things will last longer. maybe an hour or so... but eventually, everything becomes slower than molasses (in all versions of NI)