Author Topic: How I can programmatically copy an extracted tp2-defined wiedu mod?  (Read 1533 times)

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
First of all, this topic try to cover already occurred challenges from the perspective of mod manager which I'm facing during actual development.

@wisp
I'm fine with the fact that nothing will be changed for tp2, it's not the reason why I'm binging this topic. I will try to present my question as simple as it is possible but first, I need to present typical user case of mod installation just to be sure that we are on the same page:

Let's look at the example of the perfectly valid tp2 definition of the weidu mod:

Distribution format: zip file (the name dosen't matter)

Filenames and folders:
TheFolderWithModDataCanAlsoBeCalledAnything5345s
Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.tp2
Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.exe

Mod code:
Quote
BACKUP ~AnyRandomPath354887fdg/NotbodyWillNotTellMeHowIShouldWriteMyMods!/#LetsAddYetAnotherDir#/backup~
SUPPORT ~mail@mail.net~
VERSION ~0.0.0~

BEGIN "" NO_LOG_RECORD
    COPY ~TheFolderWithModDataCanAlsoBeCalledAnything5345s/Items/potn52.itm~ ~override/potn52.itm~
    PRINT "This is working weidu mod!"

This mod can be installed without problems and I don't think that I need to put a zipped example to provie it.

Definitions and Conditions:
D:\MyAllWeiduMods - a directory where player has extracted all of his mods, it contains many extracted mods, example: https://prnt.sc/m3coq6

Player extract zip file into D:\MyAllWeiduMods so the outcome after such operation will be:

Quote
D:\MyAllWeiduMods\TheFolderWithModDataCanAlsoBeCalledAnything5345s
D:\MyAllWeiduMods\TheFolderWithModDataCanAlsoBeCalledAnything5345s\Items\potn52.itm
D:\MyAllWeiduMods\Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.tp2
D:\MyAllWeiduMods\Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.exe

Now, let's look at the every step which is required to install only this mod from the player perspective after extracion occurs:

player needs to copy:
- a folder called D:\MyAllWeiduMods\TheFolderWithModDataCanAlsoBeCalledAnything5345s to GAMEDIR
- a file called D:\MyAllWeiduMods\Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.tp2 to GAMEDIR
- a file called D:\MyAllWeiduMods\Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.exe to GAMEDIR
- then execute GAMEDIR\Setup-MyVeryNiceModAnywayTP2canBeCalledAnything.exe and follow installation

It is not a problem, it's a simple select & copy operation. So what's the catch? It's done by human who can determine files and folders of the particular mod. But it's impossible for code to determine the "TheFolderWithModDataCanAlsoBeCalledAnything5345s" folder name in order to copy it along with tp2 and exe. I know that it sounds weird but it's true. No, coping all conten of the D:\MyAllWeiduMods dir is not a solution because there might me 2GB of other mods extracted and the player want's to install only one. The extra top-level folder also isn't a solution eithier.

@wisp
Are you seeing this? Do you agree? If I miss some magical thing which allows me to determine such random folder name please tell me. Again, It's ony to be sure that I didin't missin anything oblivious.
« Last Edit: January 05, 2019, 08:46:28 AM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline DavidW

  • Planewalker
  • *****
  • Posts: 316
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #1 on: January 05, 2019, 09:35:31 AM »
I don't see how there could be. A WEIDU mod consists of a copy of weidu.exe, a tp2, and a bunch of files. It's pure convention that the files are all gathered into a single folder, let alone that the folder has any particular name.

But this is a problem you're creating for yourself by putting all the files into one directory. The information about which files are associated with which mod is encoded in the way the mods are distributed, but you're erasing that information. Instead, store everything you get from (say) SCS in a folder called "stratagems" (matching the tp2 name) within MyAllWeiduMods rather than directly in MyAllWeiduMods. Then all you need to do to install it is to copy everything in the "stratagems" folder into GAMEDIR and execute (setup-)stratagems.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #2 on: January 05, 2019, 11:38:10 AM »
I don't see how there could be. A WEIDU mod consists of a copy of weidu.exe, a tp2, and a bunch of files. It's pure convention that the files are all gathered into a single folder, let alone that the folder has any particular name.
Yep, I also don't see how it could be possible. One thing, it's weidu tp2-defined mod, let's not ommit that.

But this is a problem you're creating for yourself by putting all the files into one directory. The information about which files are associated with which mod is encoded in the way the mods are distributed, but you're erasing that information. Instead, store everything you get from (say) SCS in a folder called "stratagems" (matching the tp2 name) within MyAllWeiduMods rather than directly in MyAllWeiduMods. Then all you need to do to install it is to copy everything in the "stratagems" folder into GAMEDIR and execute (setup-)stratagems.

What you say about 'information about which files (and folders) are associated with which mod' is pure convention, used by modder, not a 'tp2 weidu mod specification', right?

If I would follow you suggestion about "extracting everything into to-level folder" it woudl mean that the only way for a "mod manager" to include a mod is to:
a)
- ask user to provide a path to zip archive (or iemod package)
- extract it to some random folder (since I don't have access to tp2 name before extraction)
- save the custom folder name as a source of the mod files
b)
- ask user to provide a path to top-level folder
- save the custom folder name as a source of the mod files

I've already tryed it. It's a nightmare which is very prone tomistakes from both modders and players: extra folders inside zip, wrongly selected top-level folder, custom archive format etc. We already have mod archives with 'blablamymod\mymod' extra dir and there is nothing which would prevent people to select "cdtweaks" as top-level directory after extracting "Tweaks Anthology". It's a road to hell.

Lastly, can we not try to blame player because he just wanted to have "D:\Mods\AI\", "D:\Mods\NPC" etc folder structure with mods and instead look for a solution which could be free from problems?

How about a solution which:
- standarize weidu mod package (.iemod)
- standarize mod data folder names so mods can be extracted anywhere once and then copy to multiple games ( such feature also allows for delta updates and localy added translations!)
- it would be possible to scan one folder for specific file in order to know everything which is needed for copy operation

If you ask me, It's very important for "mod manager" to provide such basic "copy mod to game" feature.

So the next question follows:
@wisp
Do you think that being able to copy all mod required files and folders (notice that I'm not using 'tp2' or even weidu here) into a game directory for installation, is a crucial feature of the mod manager?
« Last Edit: January 05, 2019, 11:49:55 AM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline DavidW

  • Planewalker
  • *****
  • Posts: 316
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #3 on: January 05, 2019, 12:05:36 PM »
Perhaps I'm misunderstanding the context. But as far as I can see, one way or another any management program has to get hold of the actual content of the mod. It's then just a question of whether that program (a) saves the content into one big folder shared by every mod, or (b) saves the content into a bespoke folder. If you don't have access to the mod's tp2 name in advance, just get your tool to rename the folder after opening it, or else make a lookup table with entries like 'setup-stratagems.tp2=folder0009' (it doesn't need to be sophisticated, plain text will do).

As for "not blaming player", I don't really know what you mean, but (I take it) we're talking about how an automated program handles its data organization. If so, it doesn't need to do it in any very human-readable way. (If we're just talking about how someone manually organizes their own mod collection, there's no problem in any case.)

On one specific thing:
Quote
What you say about 'information about which files (and folders) are associated with which mod' is pure convention, used by modder, not a 'tp2 weidu mod specification', right?
Not really. By definition, a mod is a bunch of files distributed as a whole. Files are in the mod iff they're in the distribution. (If someone else writes a mod called "spear castle stratagems" which also has a "stratagems" folder, what determines whether a given file in 'stratagems' is part of my mod or part of theirs is: was it in my distribution, or theirs?) That distribution information is erased if multiple distributions are combined into a single folder.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #4 on: January 05, 2019, 01:30:55 PM »
On one specific thing:
Quote
What you say about 'information about which files (and folders) are associated with which mod' is pure convention, used by modder, not a 'tp2 weidu mod specification', right?
Not really. By definition, a mod is a bunch of files distributed as a whole. Files are in the mod iff they're in the distribution. (If someone else writes a mod called "spear castle stratagems" which also has a "stratagems" folder, what determines whether a given file in 'stratagems' is part of my mod or part of theirs is: was it in my distribution, or theirs?) That distribution information is erased if multiple distributions are combined into a single folder.

This is extreme case when two authors and each of them created a mod which contains: 'setup-spells.tp2' and 'spells' directory with 'spell.spl' file. It is still possible. Yet, even if I will extract two identical mods from different authors into two separate directory, it won't prevent overwriting them after coping to the gamedir. Even sequencing them (copy modA, install modA, copy modA from another dir, install modA) won't prevent overwriting the same files inside 'override' directory. I was under impression that the community agree to use "reserved prefixes" in order to avoid such situation. Are you implicating that a mod manager should have a design which prevent such situation?

Perhaps I'm misunderstanding the context. But as far as I can see, one way or another any management program has to get hold of the actual content of the mod. It's then just a question of whether that program (a) saves the content into one big folder shared by every mod, or (b) saves the content into a bespoke folder. If you don't have access to the mod's tp2 name in advance, just get your tool to rename the folder after opening it, or else make a lookup table with entries like 'setup-stratagems.tp2=folder0009' (it doesn't need to be sophisticated, plain text will do).
I'm greatfull for you feedback so let me share some more info:

At the begging, my app was simply scanning tp2 files, parse and save the special 'basename' without 'setup-' and extension and use it as a value as a folder name which is required to copy (if it's directly or one level above). So I'm associating "mymod.tp2" with "mymod" folder but I'm not relaying on the data from player (pointing to a top-level dir) or even a mod package (extraction to custom dir, then renaming). I'm relaying on the tp2 data from extracted mods. The life was nice and easy.
Then the "Worldmap" mod comes with "bp-bgt_worldmap" instead of "bp-bgt-worldmap" and break it into pieces. I didn't wanted to even try asking about 'adopting a mod to mod manager' so I use BACKUP from tp2 which solve this problem. The life was nice and easy, I could focus on other things. Then, you came and introduce "immutable" mod design which change "BACKUP" to some random path, which I can't process/relay anymore. I can't blame you, you are doing what's best for you mod, I didn't even try to ask you about changing it. And adding custom code to cover a mod (like BWS did) is not an option. So here we are.

You suggested solution relays on either player pointing to a correct top-level dir (after he extracted mod package) or the assumption that the mod package will not have any extra top-level directories. It's too prone to mistakes from both modders and players.

As for "not blaming player", I don't really know what you mean, but (I take it) we're talking about how an automated program handles its data organization. If so, it doesn't need to do it in any very human-readable way. (If we're just talking about how someone manually organizes their own mod collection, there's no problem in any case.)
My goal was to:
-not relay on extracting archives (instead copy mod files and folders) in order to allow players to add translations/local fixes and delta updates (long story for other time) because extraction overwrite files
-not impose any special place where player extracts all mods (let's assume that they don't overwrite each other by following community rules about prefixes)

It should be perfectly fine for a player to have such folder structure:
D:\MyAllWeiduMods\AI\<extractedmods>
D:\MyAllWeiduMods\NPC\<extractedmods>
D:\MyAllWeiduMods\Quests\<extractedmods>
D:\MyAllWeiduMods\Tweaks\<extractedmods>
D:\MyAllWeiduMods\SomeRandomFolderName\AndAnotherOne\<extractedmods>

-at the same time, allow a mod manager to copy only selected mods to gamedir and install it

All what I wanted was working fine, when I relay on BACKUP keyword data. But now, when even BACKUP can be anything, I can't even relay on BACKUP anymore. So it's impossible to cover 100% of mods and allow such flexibility for the players. Either I will require some special folder design or filling some info. And I don't want to mainly because I believe that we can create a new design which will allow for all of this. I'm pointing at this because I want people to be aware and I want to look for a solution which will allows for what I want and be 100% foolproof. Or atleast better than current tp2.
« Last Edit: January 07, 2019, 06:44:13 AM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline DavidW

  • Planewalker
  • *****
  • Posts: 316
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #5 on: January 05, 2019, 02:30:36 PM »
I was under impression that the community agree to use "reserved prefixes" in order to avoid such situation. Are you implicating that a mod manager should have a design which prevent such situation?
No; I was just illustrating a logical point. (Actually, people mostly don't use modder prefixes in naming their folders, but probably they should; I think it's historical coincidence that prefixes took off for game files but not for mod folders).

Quote
At the begging, my app was simply scanning tp2 files, save the special 'basename' without 'setup-' and extension and use it as a value of the folder which is required to copy (if it's directly or one level above). So I'm associating "mymod.tp2" with "mymod" folder but I'm not relaying on the data from player (pointing to a top-level dir) or even a mod package(extraction to custom dir, then renaming). I'm relaying on the tp2 data from extracted mods. The life was nice and easy.
Then the "Worldmap" mod comes with "bp-bgt_worldmap" instead of "bp-bgt-worldmap" and break it into pieces. I didin't wanted to even try asking about 'adopting a mod to mod manager' so I use BACKUP from tp2 which solve this problem. The life was nice and easy, I could focus on other things. Then, you came and introduce "immutable" mod design which change "BACKUP" to some random path, which I can't process/relay anymore. I can't blame you, you are doing what's best for you mod, I didn't even try to ask you about changing it. And adding custom code to cover a mod (like BWS did) is not an option. So here we are.
Again, and for the record, I have been structuring mods that way for nearly six years, well before you started on this tool. The post I wrote on immutability is recent, but the concept isn't: SCS has been built that way ever since early 2013.

On the broader issue: if I understand rightly, you want to let the player manually collect the mods, organise them in whatever folder structure they like, and then have your tool parse the file structure and automatically install the mods. In that case, no: there is no automatic, 100% reliable way to do it. However, you can heuristically do it in a way that I think will work for practically any mod actually in existence. Do it like this:

1) Start with the executable, setup-mymod.exe.
2) Check if either mymod.tp2 or setup-mymod.tp2 exists in the same folder as the executable.
3a) If it doesn't exist, then the mod folder must be called mymod. (Look in mymod for the tp2; if it's not there, the mod isn't working or wasn't properly downloaded.)
3b) If it does exist, parse the tp2 to extract BACKUP, find the root folder of BACKUP, and see if that folder exists.
4a) If it does, that's the mod folder.
4b) If it doesn't, you're stuck. (You might as well try guessing that mymod is the mod folder anyway.)

I'd be surprised if *any* mod gets to 4b; I'd be amazed if very many do. Standard practice for at least ten years has been to include the tp2 file in the mod folder (which requires the mod folder name to be fixed by the executable name). And virtually everyone uses the "backup directory in mod folder" coding format. The only mods that won't be caught by my heuristic are those that use the old tp2 convention and do something very exotic with their backup folder. (And insisting that everyone either (i) include the tp2 file in their mod folder or (ii) use the usual backup convention is a pretty unintrusive requirement compared to most alternatives.)

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: How I can programmatically copy an extracted tp2-defined wiedu mod?
« Reply #6 on: January 07, 2019, 05:56:23 AM »
Again, and for the record, I have been structuring mods that way for nearly six years, well before you started on this tool. The post I wrote on immutability is recent, but the concept isn't: SCS has been built that way ever since early 2013.
Understood. It's not that I'm not seeing that you mods have "MyMod\MyMod.tp2" stricture which I can use. I was simply trying one algorithm, then another in order to solve all other cases, like worldmap-like and tp2-only mods etc.

On the broader issue: if I understand rightly, you want to let the player manually collect the mods, organise them in whatever folder structure they like, and then have your tool parse the file structure and automatically install the mods. In that case, no: there is no automatic, 100% reliable way to do it.
Yep, atleast allow them to have categories like D:\Mods\Quests\<>. I was pretty much sure that it's impossible for such weird&unusual case, but I had to ask in case I miss anything. One thing for the record, we both speaking about tp2-design. When there will be a chance to be free from the old concepts and , I believe everything is possible :)

Now, about you proposition:
Algorithm which is based on exe have advantage: you don't need to scan one-level above for "MyMod" folder. For tp2 scanning algorithm you have to, it can give you incorrect results but at the same time it won't match "non-weidu" exe. I started with weidu setup-xxx.exe during first alpha. But I have to ditch .exe as first pass of the scanning algorithm. The reason was that some mods miss weidu exe and I could quickly solve this problem by coping exe with setup-'tp2name' pattern. Relaying on tp2 location instead of exe gives 'weidu relative execution' problem but I think I have handled it.

After reading you proposition again, I come to an idea how to avoid "mymod\mymod.exe" + "mymod\mymod.tp2" + "mymod\my_mod" as "mod data folder" problem, when using tp2 location as foundation for scanning algorithm. Now I think I finally know how to handle all possible cases, including tp2-only mods, without relaying on exe (tp2 offers less problems later), without worrying about "BACKUP" when mod has "MyMod\MyMod.tp2" structure. The 4-eyes rule works!

Finally, the case from first post is unsolvable but lucky for players, there is no mod which use such weird structure. My only wish would be so that "new design" will cover it too somehow but that's another story...
« Last Edit: January 07, 2019, 07:07:34 AM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: