Author Topic: Zeitgeist: install order of mods  (Read 2929 times)

Offline subtledoctor

  • Planewalker
  • *****
  • Posts: 131
Zeitgeist: install order of mods
« on: June 03, 2017, 02:28:31 PM »
Just as a preface, anyone who was privy to the way I worked with Grammarsalad on the Faiths & Powers mod would see that I am sort of obsessive about the structure and skeleton of these kinds of things.  I had DESIGNATED components, matching up to string ranges in the .tra files, before any of the components were completed, and before most were even created.  So excuse me if this suggestion seems basic or obvious.  This is just where my mind goes when I think of this stuff.  (Thus, for example, my suggestion to create a component flag that doesn't do anything.)

Anyway I was reading some of the comments about the state of BWS over at G3, and it occurred to me that there are really 4 main functions of that program:
1) downloading mods
2) establishing install order
3) resolution of conflicts
4) triggering Weidu install of everything

Right now, Zeitgeist encompasses #4.  I think it's not too much to ask for players to handle #1, and my idea for a compatibility component flag would shift #3 to modders. Leaving the question of how to deal with #2.  here again, as with conflict resolution, I think it is inefficient, and really asking too much of one person, to have whoever is maintaining the mod manager deal with install order issues.  I think this could be kicked out to be handled by the community.

My basic idea is, players could have a text document, with a basic prescribed format - something as simple as
Code: [Select]
setup-cdtweaks.tp2     100
setup-cdtweaks.tp2     200
setup-NPC_EE.tp2     100

It's totally arbitrary.  The important part is to simply set up the required format, and have it be something that Zeitgeist can read.  Have players drop a file like this into their directory, and have Zeitgeist a) display mods in this order in the UI; and 2) install mods in this order when the player hits "go."

Then, let players figure out the best order.  Let people make different versions of this file and pass them around on forums.  All they will need to do is drop it into their game folder, and then run Zeitgeist.  And hopefully the formatting will be simple enough that rank noobs would be able to edit it or insert new entries if they want to.

Basically, if not exactly this, something like this.  This way Zeitgeist itself will only need to be concerned with #4, but it will support mechanisms for players and modders to deal with #1, #2, and #3.

I am aware that both of my suggestions rely heavily on component numbers, and that can be problematic if modders don't use DESIGNATED.  But a) life isn't perfect; b) component numbering is really perfect for these kinds of applications (in addition to REQUIRE_COMPONENT, FORBID_COMPONENT etc.) and c) modders should be encouraged to start using DESIGNATED if they don't already, because it is extremely useful.

There's my random thoughts for the day.  Cheers

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: Zeitgeist: install order of mods
« Reply #1 on: June 04, 2017, 03:49:20 PM »
Quote
My basic idea is, players could have a text document, with a basic prescribed format - something as simple as
It already exist: weidu.log, Zeitgeist needs only to import weidu.log and sort present mods using it's mods+component and install sequence. It won't work.
When you bring BWS exported configs/compilations experience at the table, even one single change to one single mod will break every weidu(installation).log which was posted before the change. The DESIGNATED is not something which can be reliable. What we need is GUID-like system for components but i have't think about it much.
« Last Edit: August 03, 2017, 01:09:00 PM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline subtledoctor

  • Planewalker
  • *****
  • Posts: 131
Re: Zeitgeist: install order of mods
« Reply #2 on: June 04, 2017, 07:19:55 PM »
Well, right, but weidu.log is an important file which is needed to show after the fact which mods/components were installed. Maybe just use that formatting, but a different file extension... so I could share my successful weidu.log with you, and you could simply rename it to weidu.ord (or whatever) and then Zeitgeist would use it as a guide for display order/installation order of whatever mods I have in my game folder when I start Zeitgeist.

BUT the reason I posted this is because Wisp is developing Zeitgeist and he would know best what kind of formatting of the document could be parsed by Zeitgeist. That's what should(/must) control.

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1176
Re: Zeitgeist: install order of mods
« Reply #3 on: June 09, 2017, 04:06:22 PM »
There may or may not be one or more features for assisting with install orders under consideration, but it the event such was the case, it'd be for after the currently planned things have been taken care of.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: Zeitgeist: install order of mods
« Reply #4 on: August 03, 2017, 01:18:11 PM »
Quote
so I could share my successful weidu.log with you, and you could simply rename it to weidu.ord (or whatever) and then Zeitgeist would use it as a guide for display order/installation order of whatever mods I have in my game folder when I start Zeitgeist.

BWS have similar feature. And all problems with them are also applied to other GUI. If it's based on DESIGNATED, it won't work. You could ofc implement "Import from weidu/log" button and see all the problems which BWS Team has seen in the past 3 years. This includes:

- build-in mod/component compilations being totally worthless because of the DESIGNATED changes
- exported custom mod/component list being totally worthless also because of the DESIGNATED changes
- install order being totally worthless again because of the DESIGNATED changes

The install order sequence of the mod/components cannot relay on DESIGNATED. We need GUID/UUID keyword for every component http://forums.pocketplane.net/index.php?topic=29676.msg338758#msg338758
« Last Edit: August 03, 2017, 01:23:55 PM by AL|EN »
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

Offline subtledoctor

  • Planewalker
  • *****
  • Posts: 131
Re: Zeitgeist: install order of mods
« Reply #5 on: August 03, 2017, 03:38:13 PM »
What do you mean "it won't work if it's based on DESIGNATED?"  If I do a successful install and then give you my weidu.log, and you use that as a template for your install, the only way it will fail is if a mod changes component numbers within those 10 seconds.  The window for failure is very small.  And in the event of failure, someone will do an install and then there will be a valid template again and people can pass it around etc. until the next failure.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 391
  • Gender: Male
Re: Zeitgeist: install order of mods
« Reply #6 on: August 03, 2017, 03:46:43 PM »
You basically describe the reasons why a system which is based on the DESIGNATED won't work. It might work in most of the cases until someone will use outdated "best mod compilation" exported list and he will get wrong components and wrong install order. The failure is inevitable and the "until the next failure" is no acceptable solution. I've encounter this issue countless of times during BWS maintenance: it can'e even properly store current selection of the mods which you have selected a month ago. Image ongoing frustration when you have all you desired tweaks and suddenly it all go to hell during installation of after two weeks of progress in the game. I don't want to see such bullshit anymore.
Project Infinity public BETA - mod manager for Infinity Engine games
Modder's Guide to Github - you cannot have progress without committing changes

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #7 on: August 10, 2017, 09:03:51 PM »
Quote
1) downloading mods
2) establishing install order
3) resolution of conflicts
4) triggering Weidu install of everything

Right now, Zeitgeist encompasses #4.  I think it's not too much to ask for players to handle #1, and my idea for a compatibility component flag would shift #3 to modders

1) is good not only to players but to modders because that way they can feel confortable in depending on another mod stable features. But it's really really good for players obviously. It's not even that hard, wget will get you 90% of the way there, github repositories the other 10%. If zeigeist wants to create a 'file format', something like a the zip download of a github page being accepted by it as long as everything else is correct (manifest, tp2 on the right dir etc) would be very very good for auto updated mods.

2) Pretty difficult to do 'in general'. Mlox tries to this 'flying with no breaks' that is, there isn't a single install order, but a emergent one from a collection of precedences. It gets pretty messy. My preference for IE, that has much less mods might be 'user defined' install orders with import and exports. The obvious 'depends on' dependencies would be handled by the mods (so if that changes the mod order acquires a error), but things like this first , then this one , then that by the absolute order. I think a community of curated mod orders could arise for 'thematics' with import/export.

3) pretty obvious that this should *not* be left to modders, or not only. No modder has the time to check that his mod has no conflicts with hundreds of other mods, or a modder can *easily* abandon a mod after a few years. A user finding a conflict, can report it fast or edit it right on the github page if it's good enough (it opens a pull request). Unlike learning WEIDU, this is actually possible for barbarian lusers. There is also the obvious point that modders like to delay things, and a conflict rule update download can be much faster than a new release.

4) This is good.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #8 on: August 10, 2017, 09:09:57 PM »
4) This is good, but i can't help but think that if Zeitgeist is going to punt on 1-3 it should have a easy way to be fed the output of a tool that does 3) because it's really uh.... kind of pointless to seperate this in 1-4 otherwise.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #9 on: August 10, 2017, 09:15:15 PM »
But, playing devil advocate, assuming everything about conflicts and dependencies is centralized in .tp2 (centralization is a virtue i like), a way to 'mitigate' the loss of 'wisdom of the crowds' is to have conflict modifications on a 'fixpack'.

Problem is, it's both less effective (more specialized user knowledge) and i think it might bork how Zeitgeist checks. Fixpacks operate on diff patches, Zeitgeist is (presumably) not going to treat fixpacks 'special' and install them prior to checking for conflicts to uninstall the pack after.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #10 on: August 10, 2017, 09:39:29 PM »
Though if it would it would treat fixpacks special... it sort of solve the issue, until the diff file gets accepted by upstream (if ever). The people doing the actual conflicts would need to know WEIDU, but other random people could report issues on the github page.

I have no opinion on the component numbers, designated uiid discussion.


Version numbers for mods... assuming the dependencies and conflicts are specified on the tp2... are not very important except for the fixpacks ? To make sure it doesn't get applied to where it's fixed already. Though i suppose the diff would fail then, so no problem.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #11 on: August 10, 2017, 09:47:27 PM »
Ok i'm pretty convinced: centralizing conflicts and dependencies on the mod itself is the right approach, because it allows the 'fixes' not to care about version numbers, only that the diff applies or not, and if it doesn't, presumably things are fixed (the maintainer checks obviously).

I have to request then, that Zeitgeist, please please allow a preprocessing step for a mod list before conflict checking (install a fixpack weidu or something completely other). Megamods lists *need* updates like this and responsiveness between mods releases or forceful updates to dead mods to work.

Offline subtledoctor

  • Planewalker
  • *****
  • Posts: 131
Re: Zeitgeist: install order of mods
« Reply #12 on: August 11, 2017, 12:03:44 AM »
I think a community of curated mod orders could arise for 'thematics' with import/export.

Re: install order(s), that is precisely what I had in mind.

Regarding conflicts reporting: it doesn't need to be an either/or proposition. Right now there's nothing modders themselves can do to mark conflicts. A component flag could make that trivially easy - every time a player drops a note on a forums like "hey when I use your sphere system, the Sandrah NPC has no spells" I could quickly add this to that component:
Code: [Select]
CONFLICT ~sandrahnpc.tp2~ ~100~ ~Sandrah has no spells~
That would of course not be exhaustive. But it would be bi-directional, so it could help even with mods no longer being maintained. And it could be parsed and added to some metadata, supplemented by more metadata, which would then be processed by Zeitgeist whenever a player checks two boxes that shouldn't be checked at the same time.

(Of course I have no idea how Zeitgeist actually works, so I am very much speaking in the abstract.)
« Last Edit: August 12, 2017, 04:01:34 PM by subtledoctor »

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #13 on: August 11, 2017, 09:27:08 AM »
No, i'm pretty much convinced the central (without extra metadata) is better now (as long as the conflict language of weidu is rich enough, ie: oneof requirments, conflicts etc). This is because unlike metadata, updates don't have to care about mod versions.

Because if updates follow the same scheme as the fixpack; with diff files, they either apply, and the mod was meant to have those new conflicts or requirements, they're either already applied, and everything is fine, or they fail and the mod should be disabled from the list.


Now this doesn't necessarily mean that it must be Zeitgeist applying fixpacks before checking, just that they 'should' before running the check and in order to run the check we want at least the main mod tp2.


Fortunately, github pages allows to get not only the archive, but the latest version of a file easily. We could do a scheme where a plugin or a script apart takes a mod order list and a list of mod download places, filters the second with the first to get the required mods links, downloads the tp2 from those places, and makes Zeitgeist check *only* the tp2. Then when the check is done, the mods chosen, and the 'install' button pressed another plugin 'downloads for real', applies the fixpack to *all* the files, not just the tp2, makes zeitgeist recheck (because timing) and if it doesn't work stops, if works, actually start the install.

This could also be a (more complicated for the user) script song and dance.

A open question in my mind is EET and BGT. These need to apply mods to *both* games, and either need to combine files before or after mods install. I'm actually rather confused on if or how Zeitgeist will support these. I suspect it wont, and more hackery will be needed.


In the end, it might end up that the only real interesting part of zeitgeist we would use is a command line --check-conflicts [listoftp2] singletp2 to report conflict and script the rest for convenience sake.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #14 on: August 11, 2017, 05:16:50 PM »
Umm, with github and a standard form for the repositories, this could be done already as long as WEIDU had some conflicts, depends, depends-one-of (to components hopefully) and we don't need to care about compression and uncompression.

Git allows download a single file or the 'repository' at depth 1 (that is just the last version). Everything comes zipped and unzips into a directory automagically. It even checks consistency.

All that it's required is that WEIDU gains those conflicts, depends, depends-one-of rules, and a

--check-requirements [list of tp2] single-tp2 [optional component number]

This would allow external tools to get all this info and do the rest, if mod order is specified with curated ordered lists of components (like today really).

It just needs doing that, getting the fixpack to put in all the conflict and requirements knowledge they already have on the tp2 - by means of the diff files, and this could be solved relatively quickly, except translating all the code necessary for BGT, EET and IWD-in-EET (but this should be relatively well documented i presume).

Zeitgeist will still provide a great impetus for 'sane' packaging with its new format, but i think the great advantage is if we can get those same sane folder structures in github pages.[/list]

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #15 on: August 11, 2017, 05:23:24 PM »
Though i have to say i'm unsure about not having ordering. I think you should have a ordering rule, even if it's not used for sorting. Some mods *have* to be installed before others, for consistency or conflict reasons, and work perfectly fine in the opposite order. As such, it's another check that the provided list is sane.

Of course, some mods have to be (optionally, nearly) first or  (optionally, nearly) last, but i think it's not absolutely necessary to care about those absolutes, since every joe and jane would figure that out quickly when sharing lists. Random mod #23 having to go before random mod #54 is another matter.

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #16 on: August 11, 2017, 07:14:23 PM »
Github apparently disables git archive so the 'simple' way to get single files - for the tp2 - is disabled, but there is a alternative:

https://stackoverflow.com/a/4605068/214260

for example, to get the mod 'tales of Anegh' tp2, you can do

https://raw.githubusercontent.com/whiteagnus/ToA/master/ToA/setup-toa.TP2

and to get the whole dir, unziped you can either do

git clone --depth 1 https://github.com/whiteagnus/ToA.git /path_of_game (doesn't need unzip, but has .git gunk)

There is a command to download a zip, but that leaves out the possibility to update the repo, which is valuable not to download again, if we're reinstalling or updating (right?). It would be better if WEIDU supported having the mods be outside of the game prior to install/uninstall.

Anyway, so i think a 'mod list' would have at least the relative path it expects to the tp2, in this case:
ToA/setup-toa.TP2

From this, another list for the download/checker tool would have
path                        user+repo            branch
ToA/setup-toa.TP2   whiteagnus/ToA    master

this would permit to build both urls above and still be flexible to change branches.

Offline K4thos

  • Planewalker
  • *****
  • Posts: 110
Re: Zeitgeist: install order of mods
« Reply #17 on: August 12, 2017, 01:13:28 PM »
Quote
A open question in my mind is EET and BGT. These need to apply mods to *both* games, and either need to combine files before or after mods install. I'm actually rather confused on if or how Zeitgeist will support these. I suspect it wont, and more hackery will be needed.
I don't think BGT allows mods to be installed on BG1 before conversion. In EET this feature is more a workaround for the fact that many BG:EE mods don't have EET support. Since they work just fine when installed on BG:EE before EET starts conversion (so they are treated like vanilla game files) I added such option because why not. Ideally the list of mods installed on BG:EE (section 1 of this post) will (hopefully) shrink over time. This was meant as a temporary solution, so designing Zeitgeist around it sounds a bit excessive. There's no that much mods which still need this approach and biggest ones like BG1 NPC Project already uses cpmvars so adding native EET support is not much of a problem (there is some progress on it actually last time I heard). The situation is worse for mods designed exclusively with BG:EE in mind since those don't use multi-platform specific code but still I don't think EET compatibility state with other mods should dictate the way weidu is developed. As for normal compatibility checks there is GAME_IS ~eet~ parameter already.

edit: I think I misunderstood your post, sorry. You're probably talking about detecting game installation as either BGT or EET before any mod is installed on BG2/BG2:EE. This indeed may need some additional hackery to implement conflict resolve feature directly in Zeitgeist.
« Last Edit: August 12, 2017, 02:04:46 PM by K4thos »

SCO

  • Guest
Re: Zeitgeist: install order of mods
« Reply #18 on: August 12, 2017, 01:35:02 PM »
Hey, @K4thos, you're a maintainer of big world setup (or at least a person that occasionally modifies code there) right?

Because i think i know why it's not working on wine.

Many times the program will start a external process, get its standard output into a string a check if that string has a substring (to check for success). This happens to check for successful extraction checking for "Everything is Ok" from 7z and i think it also does when it needs to copy inner folders out into the root of the game (fails finding the tp2 file).

Anyway, this fails because for some reason autoit (not wine, i tested this by using the 7z.exe bundled on BWS on a mod archive) is corrupting the inputstream, probably by assuming a different codepage or charset.

"Everything is Ok" turns into "øvørythingàisàOk" on the BWS text frame (every output of a external process is corrupted).
I could hack it for extraction by either ignoring the check or trying to compare against the corrupted string, but that won't work for the tp2 I think.
Do you have any insights how to fix this? LANG=C wine ... didn't work.

Offline K4thos

  • Planewalker
  • *****
  • Posts: 110
Re: Zeitgeist: install order of mods
« Reply #19 on: August 12, 2017, 01:59:04 PM »
Quote
Hey, @K4thos, you're a maintainer of big world setup (or at least a person that occasionally modifies code there) right?
not really, I have no idea about the way BWS works. I'm part of the "BiGWorldProject organization" on Github just to have full access to mods hosted there, not to mess with BWS (which I don't think is even hosted on Github but on bitbucket.org to which I don't have access). Currently BWS is not in active development (when it comes to new functionality) anymore since there is no one left to maintain it. Maybe AL|EN will accept a commit if you know how to fix it on Wine. I can't help with this task.
« Last Edit: August 12, 2017, 01:59:54 PM by K4thos »

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: