Author Topic: Roadmap 2  (Read 406 times)

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1035
Roadmap 2
« on: March 16, 2019, 03:43:50 PM »
So I figure I should post a new one of these.

But let's start with an evaluation of roadmap 1:

Zeitgeist

Less has happened than I would have wished. I've fixed a significant
graphical glitch and tinkered around a bit. I've also implemented a
suggestion from CamDawg that allows for oggdec, et al. to be
distributed with WeiDU/Zeitgeist instead of as part of the mod
itself. There's also the addition of archives; see below.

Replacement for READLN

This has been postponed and folded into another of my zany
schemes. See "TP3".

Cross-platform distribution format

In October I added the ability to create and extract ZIP archives to
Zeitgeist. I choose ZIP because it's a very widely supported format
and more advanced compression algorithms and formats offer very
limited advantages over ZIP since most mods consist of text, which
compresses well regardless, and already compressed resources (images,
audio), which further compress poorly, regardless. I tested this with a
few mods and between ZIP, 7Z and TAR.XZ the biggest difference I saw
was on the order of 100 kBi.

There are still some UX concerns (notably, no prompt before
overwriting) and it's a few bits shy of a full-baked feature.


Onto the new roadmap:

Stack management is all grown up and now it's moving out

I'm working on adding stack management to Zeitgeist. The slightly
longer plan is to allow mods the possibility of pinning their WeiDU
version. Since version-pinning is incompatible with WeiDU's stack
management, this feature can only be implemented if something else
takes over the duty.

For example, suppose the mod stack looks like this:

foo - WeiDU 300
bar - WeiDU 299
baz - WeiDU 301

The listed WeiDU version (fictitious) are pinned by the respective
mod. One WeiDU cannot handle all three mods, because maybe 301 broke
something that worked in 299 and maybe 299 does not support some
feature that exists in 301, and both of these hypotheticals apply to
300.

When the user requests to uninstall bar, Zeitgeist would call on WeiDU
301 to uninstall baz, WeiDU 299 to uninstall bar and then WeiDU 301
again to reinstall baz. Each mod can be handled with its version of
WeiDU.

In stack management today, all three mods are handled by the same and
most recent locally available version of WeiDU.

I guess we'll have to wait and see what the implications of this will
be. One possibility is a new release model, but that's the stuff of
future maps of the road.

"TP3"

I posted about the idea some time ago, here.

A config system to replace READLN would simply be an object in the
larger JSON file. This way you avoid the question of where it should
be located and what it should be called. It can be CoWed
(copy-on-write) if you want immutability. It's available to WeiDU and
other programs alike.

Pinning your mod's WeiDU version would be easy.

The install-time predicates that give non-WeiDU install drivers such a
hard time today could be simplified into predicates that are fully
knowable in advance. This would, of course, mean that some TP2
predicates that are supported today would not have "TP3" analogues,
but on the whole I think there's more to gain than there is to lose.

It should be possible to write a conversion function that takes TP2 as
input and emits "TP3".


There's possibly also some stuff about tests, the build system and the
documentation, but it's not really roadmap stuff and I'm even less
certain about when and if these things will get done.


Feel free to comment.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 307
  • Gender: Male
Re: Roadmap 2
« Reply #1 on: March 25, 2019, 06:49:22 AM »
Stack management and pinning specific WeiDU version - wonderful news indeed, finally weidu improvement won't be limited at all \o/

Config system - one of the possible candidates is 'Minimum Stats Cheat' component of TA mod. Can you confirm that it will cover 'immutable design' aka user-edited configuration won't be overwritten via extracting new mod version? How about instead storing mod configs locally for single game, you will use user-directory/follow EE location for storing game config?

Cross-platform distribution format - the zip is good choice, but what are requirements for the file structure of the .zip/.iemod archives? I mean, it can't be ANY structure because of possible extra top-level directories. Those need to be established and verified because as soon as you skip it, people will create non-standard .zip/.iemod files which will cause problems during extraction/downloading etc. What about manifest requirements? This is affecting every 3rd party tool because they can't require additional constrains except the ones which are established requirements.

Install-time predicates - this is something which worries me much. If this is one of the TP3 feature, did you given up on providing those predicates form old mods which use tp2? Given the fact that old mods can't be updated, would be possible to call such 'TP3-conversion' function for extracted mod and use output as a source of the "install-time predicates"?

Keep up good work!
« Last Edit: March 25, 2019, 12:54:11 PM by AL|EN »
You cannot have progress without committing changes...A Basic Modder's Guide to Git and Github

Offline Wisp

  • Moderator
  • Planewalker
  • *****
  • Posts: 1035
Re: Roadmap 2
« Reply #2 on: March 31, 2019, 05:53:04 PM »
Config system - one of the possible candidates is 'Minimum Stats Cheat' component of TA mod. Can you confirm that it will cover 'immutable design' aka user-edited configuration won't be overwritten via extracting new mod version? How about instead storing mod configs locally for single game, you will use user-directory/follow EE location for storing game config?

Well, I've been thinking you could version-stamp the copy and
invalidate copies that do not match the version in the "TP3". And to
be clear, with version, I mean something distinct from the mod
VERSION. Generally, the problem with preserving copies and using them
together with future versions of the mod is that you do not know if
options have been added, or have had their semantic meaning
changed. Versioning the copy puts the modder, who is probably the best
person to make the call, in charge of when it is safe to reuse old
copies and when they should be invalidated.

I haven't thought about global copies. I suppose there could be an
option of either saving the copy locally or globally.

Quote
Cross-platform distribution format - the zip is good choice, but what are requirements for the file structure of the .zip/.iemod archives? I mean, it can't be ANY structure because of possible extra top-level directories. Those need to be established and verified because as soon as you skip it, people will create non-standard .zip/.iemod files which will cause problems during extraction/downloading etc. What about manifest requirements? This is affecting every 3rd party tool because they can't require additional constrains except the ones which are established requirements.

I haven't given much thought to what people include in their mod
archives. I figured the manifest was obsoleted by the "TP3" file. I
don't really see the need. If it matters to something other than the
mod itself, it should be in the "TP3" or manifest file. Can you
elaborate on your concerns? Maybe I'm missing something.

Quote
Install-time predicates - this is something which worries me much. If this is one of the TP3 feature, did you given up on providing those predicates form old mods which use tp2? Given the fact that old mods can't be updated, would be possible to call such 'TP3-conversion' function for extracted mod and use output as a source of the "install-time predicates"?

Probably not exclusive to "TP3", but perhaps implemented there
first. With TP2 I'm more or less only able to support a well-formed
subset of all possible predicates. My thoughts were running more along
the lines of "TP3" not needing any translation into a standard format,
like TP2 needs, and support of the full "TP3" set.

Offline AL|EN

  • Planewalker
  • *****
  • Posts: 307
  • Gender: Male
Re: Roadmap 2
« Reply #3 on: April 04, 2019, 05:50:37 AM »
Well, I've been thinking you could version-stamp the copy and
invalidate copies that do not match the version in the "TP3". And to
be clear, with version, I mean something distinct from the mod
VERSION. Generally, the problem with preserving copies and using them
together with future versions of the mod is that you do not know if
options have been added, or have had their semantic meaning
changed. Versioning the copy puts the modder, who is probably the best
person to make the call, in charge of when it is safe to reuse old
copies and when they should be invalidated.
Changing semantic meaning of the options should not be allowed and the solution shouldn't be limited because someone would do crazy things. If a modder is facing such case, he should use new config key. New default config keys can be easy compared vs existing ones and can be copied to global config.

I haven't thought about global copies. I suppose there could be an
option of either saving the copy locally or globally.
If only global config exist, read values from it, if global and local config exists, read from local. It's a matter of providing standard location for reading values from config.

I haven't given much thought to what people include in their mod
archives. I figured the manifest was obsoleted by the "TP3" file. I
don't really see the need. If it matters to something other than the
mod itself, it should be in the "TP3" or manifest file. Can you
elaborate on your concerns? Maybe I'm missing something.
Ok, manifest isn't really needed if the tp3 has all the data from manifest.
Regarding concerns: zipped executable's, zipped rar's, double and tipple extra top-level directories, override&backup folders included etc. So how about:
- determine standards for 'infinity engine mod package' folder structure and content
- create tools to verify it: similar to dotnet new / puppet validate it would be 'iemp new' for creating new mod from template / 'iemp validate' to validate mod against standard

BTW: The .iemod file extension is not the good choice, it's "Infinity Engine Mod Package" , not mod definition file. So how about .iemp as file extention?

Probably not exclusive to "TP3", but perhaps implemented there
first. With TP2 I'm more or less only able to support a well-formed
subset of all possible predicates. My thoughts were running more along
the lines of "TP3" not needing any translation into a standard format,
like TP2 needs, and support of the full "TP3" set.


I'm all for tp3 but man, when? Creating new tp3 will take a lot of time, switching to the new standard will take even more time.
Even supporting subset of predicates would be better than nothing. But you still can list all of them, right? I don't understand what is so difficult to simply emit them it the way how internal parser validate correctness of tp2 file in terms of OR/AND keywords/round brackets which determine logic. Just start with including component keywords with values as a yet another JSON key and let see outcome.
You cannot have progress without committing changes...A Basic Modder's Guide to Git and Github

 

With Quick-Reply you can write a post when viewing a topic without loading a new page. You can still use bulletin board code and smileys as you would in a normal post.

Name: Email:
Verification:
Type the letters shown in the picture
Listen to the letters / Request another image
Type the letters shown in the picture:
What color is grass?:
What is the seventh word in this sentence?:
What is five minus two (use the full word)?: