weidu next generation - next version should abandon auto-overwrite design

[AL|EN]

If there is one thing which I consider a major weidu flaw it would be weidu auto-overwrite design. I cannot stress enough how irrational it looks to me, compared to every other SDK's.

Let's look at the historical example:
- when Westley Weimer stopped weidu developing, Valerio Bigiani, AKA The Bigg took his place
- new weidu versions above 185 have changed the way how regexp works
- to this day, some modders refuse/forbidden to use new weidu versions because of the changed behavior and even if they did it because they are lazy, they still did it
- yet, many players will extract more than one single mod into game directory and weidu will overwrite old version of the setup-mod.exe > mod code won't be processed exactly as the author wanted
- this might lead to inconsistency between the state of the game which player has vs state of the game which mod author expects

this might look like not so important problem, but there is recent example: weidu 241 break EET and BWS at the worst time when it might happen - when EET in on hold and BWS is out of support. How mod authors can work with a framework/sdk which has a chance to break their mods in the future?

But all of this aren't even the main reasons to abandon auto-overwrite update. I believe it's the basic design principals of how to create stable and consistent system/framework/sdk.


How is even possible to have consistency when:
OCalm might change
weidu code can change
game itself might change

Combine it all and you have huge probability to have inconsistent installation of the same mods but for the different time because the weidu version which was used to create and release mod is not the last one. When you have one simple mod, it might work for 10 years but EE have some very complicated mods like T&B/F&P.

Also when you have 100+ mods, removing auto-overwrite design can have positive impact at uninstallation consistency. Mods will be installed and uninstalled using exactly the same version which was used by author at the time of their creation.

Wisp, by putting huge effort into testing and code regression testing, you have saved modders community from some major breaking changes disaster. But at the same time, taking care about "does changing one single byte can cause old mods to breakup?" is major drawback of the improvements.
When you abandon this auto-overwrite scheme, you will be free to do everything you always wanted. You can introduce breaking changes without caring about 10+ years of the historic past. And that might be biggest impact for weidu progress. You cannot have progress without changes.


tl;dr: weidu should not auto-update itself, allowing mods to be installed using always the same version of the weidu. Also it will allow for breaking changes/improvements but changes will not affect the previous mods

[The Imp]

When you abandon this auto-overwrite scheme, you will be free to do everything you always wanted. You can introduce breaking changes without caring about 10+ years of the historic past. And that might be biggest impact for weidu progress. You cannot have progress without changes.

The thing is, are you sure this can be done without affecting say billions of things. As in, the crap is already in the pants as the auto update is in every version of weidu since at least v202 from what I remember.
And you can prevent it locally anyways with the "--noautoupdate" -command if you want, as you are aware of ... but the weidu.log'ing will be difficult, let alone all the other things a I have no concept of.
Have you tried to get the BWS to run without the update command it has build in it, at the start ?
Yeah, I am sure is it will be a cheese show.

- when Westley Weimer stopped weidu developing, Valerio Bigiani, AKA The Bigg took his place
- new weidu versions above 185 have changed the way how regexp works
- to this day, some modders refuse/forbidden to use new weidu versions because of the changed behavior and even if they did it because they are lazy, they still did it

Yeah, sure, and the update was actually the solution for the unregistered edits of the weidu.exe a few programmers made their mods under, but that was a decade ago. Aka v186 ... v202.

weidu 241 break EET and BWS at the worst time ...

The weidu.exe's need testing too, and there might not have been a EET tester at their post.

But all of this aren't even the main reasons to abandon auto-overwrite update. I believe it's the basic design principals of how to create stable and consistent system/framework/sdk.

The weidu has had the make--biff command that made it impossible to stack uninstall, but that was a feature that has been undone.
Is there any other that has a similar feature, I have no idea... do you ?

[Wisp]

Removing old versions of WeiDU is necessary for the way WeiDU does stack operations. Consider this:

Mod A uses WeiDU xyy.
Mod B uses WeiDU xyz.
WeiDU xyz has some new feature/fix/whatever that does not exist in older versions and which is relevant to mod B.
Mod A is installed before mod B.
The user attempts to uninstall mod A using its WeiDU xyy.
What happens?

It's certainly possible to do stack operations in other ways, but none of those ways are as simple as just not always using latest WeDU.

[AL|EN]

And you can prevent it locally anyways with the "--noautoupdate" -command if you want, as you are aware of ...
Have you tried to get the BWS to run without the update command it has build in it, at the start ?

BWS is using  "--noautoupdate" command but even single execution of the weidu which was initiated by player  will auto update all weidu thus making  "--noautoupdate" option not providing desired results.

Consider this:

Mod A uses WeiDU xyy.
Mod B uses WeiDU xyz.
WeiDU xyz has some new feature/fix/whatever that does not exist in older versions and which is relevant to mod B.
Mod A is installed before mod B.
The user attempts to uninstall mod A using its WeiDU xyy.
What happens?

Step 0: Player launch setup-A.exe in order to uninstall mod A, choose to (U)ninstall
Step 1: Mod B is uninstalled using WeiDU xyz
(because setup-A.exe will call setup-B.exe to uninstall modB instead of processing uninstallation of modB)
Step 2: Mod A is uninstalled using WeiDU xyy - exactly the same version of the WeiDU which it was shipped and installed
Step 3: Mod B is reinstalled using WeiDU xyz which has all new features which are necessary/relevant to mod B

What's the problem?

EDIT: More details added.

[GeN1e]

- to this day, some modders refuse/forbidden to use new weidu versions because of the changed behavior and even if they did it because they are lazy, they still did it

You mean the same modders who make a point to be incompatible with the "mainstream" modding and check for megamods to prevent their own mods from installing on such games, and throw tantrums when an unsuspecting player dares to ask them a question if their mods have internal issues unrelated to compatibility? Are you serious?

this might look like not so important problem, but there is recent example: weidu 241 break EET and BWS at the worst time when it might happen - when EET in on hold and BWS is out of support. How mod authors can work with a framework/sdk which has a chance to break their mods in the future?

No idea what happened with 241, but wasn't a new version released just a week later?

[AL|EN]

GeN1e, we are at the same side when it comes to what you describe. What I posted was an example of the possible influence of the "weidu framework" changes and the fact that it's auto-updating feature is a blocking factor for any significant improvement. It doesn't matter that the next weidu can fix something, it's a flaw in design. I asked 3 senior developers about this scheme and the most gentle opinion was "   
irresponsible". If weidu will abandon such scheme, there won't be any problem with implementing new features for eg: http://forums.pocketplane.net/index.php/topic,29663.0.html - even this single change can have huge impact for "BWS successor". That is why I'm insistent at this subject.

[ikki]

GeN1e, we are at the same side when it comes to what you describe

That's old stories, i don't know of one mod that doesn't install fine (now) with the new version of weidu

[Wisp]

Step 0: Player launch setup-A.exe in order to uninstall mod A, choose to (U)ninstall
Step 1: Mod B is uninstalled using WeiDU xyz
(because setup-A.exe will call setup-B.exe to uninstall modB instead of processing uninstallation of modB)
Step 2: Mod A is uninstalled using WeiDU xyy - exactly the same version of the WeiDU which it was shipped and installed
Step 3: Mod B is reinstalled using WeiDU xyz which has all new features which are necessary/relevant to mod B

What's the problem?

That's not what happens, nor could it happen with what's available in a modded game today (partly because setup-mymod.exe is not required to exist; nor will it ever be).

I'm open to implementing a new system for managing versions. It would dove-tail nicely with another couple of things, but it would require moving the stack management outside WeiDU, because WeiDU does not have the resources or breath of scope to handle running and managing an inferior WeiDU process and the environment this would require.

[AL|EN]

How about putting all 'weidu framework' inside library (.dll) and on top of that, having a master 'weidu-worker' executable which will handle installing/uninstalling mods by using setup-MyMod.dll as a source of all functions?

[c4_angel]

What will happen if put a read-only flag to old setup-xxx.exe(s)?

[AL|EN]

On Windows it won't interrupt overwriting. Linux/OSX probably the same but I can't check.

[Creepin]

Ok, so the mods can't be uninstalled properly if they have different versions of WeiDU. I don't understand the reasoning behind that, but let's play it is so indeed. Still, why not make this autoupdate optional? Opt-in, preferrably. Something you DO if you want to peruse uninstall feature. After all I bet there's 4 times more people who just installs mods than those who uninstalls them.

[AL|EN]

Example of weidu.log:

Ascension - installed with weidu 246
modX - installed with weidu 240
modY - installed with weidu 241
modZ - installed with weidu 242

It won't make difference even if assuming that weidu is not autoupdating itself. When player will try to uninstall "Ascension", it's "weidu 246" which will process all operations of the re-installation of all remaining mods like modX,modY,modZ. So it would be literally 'installing all mods using Ascension weidu".

[me]

modX may install fine without errors with weidu 240, but there might be some function or macro that is bugged in 240 which might have been fixed in 242. Having all mods autoupdate to 246 would ensure all mods use the updated (fixed) macro.

Autoupdate is one of those things where the benefits outweigh the drawbacks IMHO.

[AL|EN]

Autoupdate is one of those things where the benefits outweigh the drawbacks IMHO.

Really? I wonder why no other SDK/Framework/Software is using such scheme? So let's summarize drawbacks and benefits:

Drawbacks:
- community split
- inconsistent mod installations across years ( we will never know how many bugs was introduced by this )
- it break mods in the past
- breaking mods in the future
- weidu 'hotfix' release
- it slows introducing any kind of changes which might 'break something for all other mods which can't be updated', lots of manual tests are needed
- it is one of the reason which makes almost impossible for other developers to contribute into weidu codebase "because something might break for old mods'
- it prevents introducing breaking changes which are needed for external systems, without affecting older mods

Benefits:

modX may install fine without errors with weidu 240, but there might be some function or macro that is bugged in 240 which might have been fixed in 242. Having all mods autoupdate to 246 would ensure all mods use the updated (fixed) macro.

And the 'fixed' version will produce different results for eg. regexp, which will create bugged installation and the bug reports from players won't make sense for mod author because he will not be able to reproduce results. So it's not a benefit but a drawback.

The only benefit i see is some kind of false sense of "my mod will use fixed functions in the future' with a hope that the 'fixed' version won't change how the mod works.

[The Imp]

Alien, have you actually read a weidu.log that has been installed with a variety of weidu.exe's not in consecutive order ?
And the "community split" happened when the Westley Weimer went away, and there was multiple versions of weidu.exe's... with different marco'es etc as part of them, cause they literally had a single author, each. Which eventually resulted in the fact that the beta's were assigned the xx values from the 2zyxx as up to that point there was only 185 versions and lower, but after a simple 4 digit number was not enough to recursively set it straight. This design was there from where it began.
If you need to make a marco/regexp that works with the old weidu mods, you can go ahead and make those out ... and replace them in the mods own .tp2 files.

[me]

And the 'fixed' version will produce different results for eg. regexp, which will create bugged installation and the bug reports from players won't make sense for mod author because he will not be able to reproduce results.

Sure he will, when he looks at the player's debug log that shows the version number.

Anyway, you keep mentioning this regexp thing. Is there a clear example where the old (v185 or earlier) implementation is somehow more accurate/effective/superior in certain circumstances ?

Maybe the reason regexp was changed was because the old way had some limitations. For example, I'm looking at one of the changes that involve regexp:

Version 232:
  * Fixed a stack overflow when using COPY_EXISTING_REGEXP and you have more
    than ~60000 files between your override and key.

Can you do this with the old (v185) WeiDU? Serious question. (don't have the game at the moment).

[The Imp]

Can you do this with the old (v185) WeiDU? Serious question. (don't have the game at the moment).

Really, nobody cares, even the Improved Anvil today uses the latest weidu.exe (well it was v23900 back then, but the point is still there).

[me]

Obviously Alien does, since he claims the version that came with the mod would always work without issues. My point is even that may not necessarily be true. If v185 cannot handle 60000+ files (certainly possible in a mega Big World install), then you would have no choice but to use an updated Weidu.

Which is why it's a safe bet to have autoupdate on (in the context of IE modding).  You can always make things backwards-compatible, but you can't always make 'em future-proof.

[Wisp]

Still, why not make this autoupdate optional? Opt-in, preferrably. Something you DO if you want to peruse uninstall feature. After all I bet there's 4 times more people who just installs mods than those who uninstalls them.

I hope you can appreciate the irony of asking me to break a minority use-case.

[subtledoctor]

Obviously Alien does, since he claims the version that came with the mod would always work without issues.

I don't think AL|EN ever claimed that.  The point is a simple one: we at least know that mod has been tested and released with that Weidu version.  Does it still have bugs?  Sure, maybe, depending on the mod.  But if Weidu changes and is updated, there will not have been any opportunity to test the result.

Best practice, in the absence of any other concerns (!) would pretty clearly be to have each mod use the version of Weidu that it last shipped with.

I'm open to implementing a new system for managing versions. It would dove-tail nicely with another couple of things, but it would require moving the stack management outside WeiDU

Also pretty clearly, this is the ideal solution.  Something like, have the executable be a container that contains weidu.  As each setup-___.exe file gets executed, copy it inside the container as "weidu202.exe," "weidu239.exe," etc.  And keep a log of which mod used which version.  Then for stack operations, simply use each version sequentially, in order of the log. 

That might be a bit pie-in-the-sky, since it would involve making a whole new software tool and I don't know who's about to spare the time for that.  Plus, I don't know how easy it would be to make something like this cross-platform (I have to say the suggestion of using .dll files made me throw up a little bit )  But from an abstract design perspective this would be ideal.

Frankly I'm not sure stack operations should be prized so highly anyway.  They can be prone to failure if a mod in the middle of the stack donesn't uninstall cleanly (and that's not something Weidu can control), and it's not really worth doing if the player has a 3+-hour SCS install at the end of their order.  And for really big installs that cannot easily be undone/redone by hand, the stack operations can be moved out to a tool like BWS.

[The Imp]

The point is a simple one: we at least know that mod has been tested and released with that Weidu version. 

The problem with this is that ... see this ... no idea where the problem is, but it HAS TO BE IN THE FACT THAT THE WEIDU.EXE WAS NOT UPDATED !!!!! I would assume that the weidu.exe in the SCS's .exe doesn't know what the eet in the game_is function is, in the cdtweaks .tpa -file is... but that's just my assumption. Feel free to reproduce and verify your own theories and show how NOT updating the weidu would be so much better than doing it. When it FAILS in a regular mod install when the OS doesn't allow the update.

[CamDawg]

The point is a simple one: we at least know that mod has been tested and released with that Weidu version. 

The problem with this is that ... see this ... no idea where the problem is, but it HAS TO BE IN THE FACT THAT THE WEIDU.EXE WAS NOT UPDATED !!!!! I would assume that the weidu.exe in the SCS's .exe doesn't know what the eet in the game_is function is, in the cdtweaks .tpa -file is... but that's just my assumption. Feel free to reproduce and verify your own theories and show how NOT updating the weidu would be so much better than doing it. When it FAILS in a regular mod install when the OS doesn't allow the update.

To be fair we don't know that yet. I suspect this has more to do with SCS' easy-install script which IIRC invokes WeiDU directly instead of setup-scs.

[AL|EN]

@Imp:
SCS 30 contains weidu 23700 and doesn't have any kind of problems during installation
Tweaks-Anthology v4 contains weidu 24200 and doesn't have any kind of problems during installation, same for v3, v2, v1, all betas , I've used/tested them by myself during development
Both mods have no problems coexisting to each other, nor depends on the weidu version which isn't included inside archive. Lack of auto-update is definitely not the cause of the problem, nor it would be in any other case unless mod archive doesn't contain proper weidu version.

I bevile SCS has nothing to do with it, it's some kind of weird Tweaks-Anthology installation>updating operations during which setup-cdtweaks.exe was not properly extracted from the archive while all other mod files were replaced.

[AL|EN]

Frankly I'm not sure stack operations should be prized so highly anyway.  They can be prone to failure if a mod in the middle of the stack donesn't uninstall cleanly (and that's not something Weidu can control), and it's not really worth doing if the player has a 3+-hour SCS install at the end of their order.  And for really big installs that cannot easily be undone/redone by hand, the stack operations can be moved out to a tool like BWS.

Finally something to add to a install/uninstall plate. Thanks for pointing the fact that currently, preventing re/uninstallation failures (user case), have higher priority than solid/unchanging and future bullet-proof installation. That's bad no matter how you would look at it.


@wisp, any news about "implementing a new system for managing versions/call stack operations"?