Logic World Modding Roadmap

by @pipe01Developer4 months ago (edited4 months ago)

The following is a rough plan of our ideas for the modding system in Logic World and the logicworld.net mod sharing system. None of it is set in stone yet, so all input and ideas on how to improve it are more than welcome, as well as any things we may have forgotten about.

Code distribution

Mod code will be distributed and loaded as binary DLL files, users upload the compiled DLL files themselves.

  • Pros to distributing source code:
    • Users can easily see how the mod works since the code is available in plain text.
    • Can check if the mod does something malicious just by looking at the code, making manual review easier.
    • Modder can just upload the code as-is, not worrying about compilation issues.
  • Cons to distributing source code:
    • The game needs a lot of extra code and resources to compile the code, like the C# compiler.
    • Harder to handle mod to mod dependencies, optional dependencies are impossible/very hard.
    • The game has to compile mods at startup, increasing loading times (this can be mitigated with caching, but still affects first launch).
    • Source code is usually bigger than compiled binary.
    • A compiled DLL of a mod should be identical across all game installations so it’s less efficient to make every user compile it.

Modifying the base game’s code

We will distribute assemblies that facilitate hooking alongside the game’s binaries, generated by MonoMod.

This system will not guarantee backwards and forwards compatibility, the responsibility for ensuring a mod works with the current game version falls on the mod author.

  • Pros:
    • Generated assemblies will be strongly typed, allowing for easy and idiomatic modifications.
    • No need to distribute assemblies like Harmony with each mod.
    • Can still use MonoMod Hooks manually if more customization is needed.
  • Cons:
    • Harmony is very popular, and people are probably more used to it.

Modifying base game mechanics

A hook system (name TBD, events?) will expose certain points of the game to modders in a stable interface to allow easy modification of core mechanics.

GUI system

Mods can use a simple C# (or XML?) API for creating interfaces, intended only for editing custom components’ custom data. In the future we will integrate Unity’s UXML system to create any kind of interface.

Dependencies and loading

Mods can specify two types of dependencies in their manifest: required and optional. A required dependency mod must be present in the mods list, whereas optional dependencies will not be enforced.

A dependency tree will be automatically built to determine the mod loading order, cyclical dependencies are not allowed and will cause mod loading to fail.

(Maybe?) Mods will optionally be able to specify a priority number between 0 and 100, which will only be considered when multiple mods are located at the same depth in the dependency tree.

Each dependency of a mod can specify a version selector using a simplified version of the syntax that e.g. NPM uses.

If two versions of the same mod are present, mod loading will fail.

Mod hub in logicworld.net

There will be a section in logicworld.net dedicated to mods. Mods packaged as .lwmod files (TBD, maybe just .zip?) will be uploaded either through a form on the website or through a mod making CLI (TBD, maybe through the game itself).

Each mod will be identified by an ID that follows the pattern AUTHOR.MODNAME, where AUTHOR is the logicworld.net username of the uploader and MODNAME is an arbitrary string set by the author.

Mods can have multiple versions uploaded on the same ID, and they will all be available for download. Versions are immutable, they cannot be replaced after uploading. Version numbers will be of the form “v1.2.3”, ideally following semver but it’s ultimately up to the modder’s discretion. Any mod version uploaded must come sequentially after the latest version, e.g. you cannot upload v1.0.0 after uploading v2.0.0.

Auto updating

Before being loaded, the game will check if a mod has any new updates and warn the user if there are. The user will then have the following choices:

  • “Install new version”: Download the latest version and load it
  • “Remind me later”: Ignore update, but show this dialog again the next time the game starts.
  • “Skip version”: Ignores this specific update and doesn’t warn the user again, but warn the user when a newer version is released.

If the danger level of a mod has increased from version to version, it will be included on the update warning and extra confirmation will be required.

These dialogs can be disabled through a setting, in which case the user will update each mod individually through the mods menu.

Code scanning

Uploaded assemblies will be statically analyzed and a “danger level” will be ascertained. For example, if an assembly has code that accesses the file system the danger level will be high, and if it spawns processes it will be higher. The level will be indicated in the mod listing on the website, accompanied by a warning if the level is higher than a (TBD) threshold.

Connecting to a server with mods

When a client connects to a server that has one or more client and server-side mods installed, the connection will be blocked by the server. The connecting client will then be prompted to download and install any missing mods from logicworld.net, provided that they are available there. (Perhaps only allow mods with a low danger level to be automatically installed?)


10 comments
@Ecconia4 months ago

Nice roadmap so far!
I have things to note to the existing points, and new points to bring up.
I fully agree to what @GHXX did write in Discord btw (like every point).

Below you will find the two sections of responses to the here proposed roadmap:

@Ecconia4 months ago (edited4 months ago)

Reponse to the proposed road map:

DLL Distribution:

Delivering DLL mods is basically game industry standard, although I like the charm of distributing source code.
It won’t be impossible to optionally depend on mods, it is just indeed hard (all interaction via reflection).
With delivering DLLs, there is no motivation for “some” modders to upload their source-code. The smallest form of motivation is a link-field for source code. Cause with public source it is easier to let others contribute and increases trust etc.
Btw, if you want to improve LW file size, have a look at the shipped Unity assets, there are a lot of unused resources.
I also claim, that the compilation of mods on game-start would not be that big of an issue, if players can see what is currently being done - there has to be a loading bar showing which mod is getting compiled. Nothing is more annoying than loading without indicator or descriptions and with window freezing.

Modifying the base game:

Small correction: Harmony is only delivered by a single mod, all other mods depend on that mod.

For Unity games BepInEx is the industry modding standard. It provides a lot of tools including MonoMod and Harmony. A real power-horse. However it is made for Unity and hence not working on the LW server (as is).
The biggest advantage of Harmony is that it has an internal state, to keep track of everything that got patched, this is used to prevent collisions and issues, when multiple parties are patching the same methods - it was designed to be used for this purpose. Ofc Harmony also provides a set of tools to make the process of patching super easy and straight forward - and why not take luxury, if you can?

I like the idea to be provided with the tool to hijack the game #363, as currently my modding is heavily relying on it.
Currently the modding community is only using Harmony - successfully. The downside is, it is only about 2-4 modders that use Harmony (as most are only adding simple components). So this number is not quite representative, of which tool is the best for hijacking the game.

With only this few modders, it is fine to (for now), not provide any framework, as it is not clear which should be used.
As soon as the community is a bit bigger, and we got enough experience with modding frameworks, a decision should be made - not before. But going against the industry standard is difficult :P
Also we are having the major issue that .net7 is not yet supported by MonoMod.Common (at least, maybe the subsystems too). For now, please keep distributing the .net6 until we have a reliable way to take care of that. The community will take care of the rest - somehow.
I am getting pretty frustrated, if I cannot remove unused components from save-files and hence have incompatibilities (I need Harmony on the server for that).

Hook system:

I cannot say much about this. Sounds nice, but I would probably just stick to using Harmony to hook into the game directly.
You can do us a favor by separating focuses of concerns, so that we actually have enough hook points, without re-coding bigger parts of the game.
As it is currently very unclear what modders actually want to hook into, it is difficult to say which hooks should exist. (Hint: One to notice specific cluster content changes).
Maybe sample what the community needs and implement that then? I assume this system would be similar to the Bukkit’s event system? There exists something like that already in LW, but only for placing and destroying of components - not quite useful yet.
For easier hooking, maybe use less internal keywords, as these only give our IDE’s less ability to check our references to classes and their content => more bugs.

GUI System:

Currently Akasus has already started cooking a framework (noice).
But as you want to change GUI at some point anyway, yes please add a framework (#316 - Modular editing window).

And indeed, there is currently a big demand for adding custom components edit windows.
We are able to generate them currently, although it takes a huge effort to recreate the UI elements and window frames from LW ourself. It would be nicer to be able to use the prefabs, that you are using too (and just instantiate them), that way the GUI stays up to date and it is much less work for us.

On the other hand, we could make our own framework for that, with own Unity components. The problem here is, that EditComponent expects a ConfigurableMenu which hard limits us to use LW’s system, without being able to improve/adjust it to our needs.
This hard dependency should be lowered as quick as possible (for example, require a minimal but compatible interface instead).

But editing custom data is not the only use-case for windows. CustomWirePlacer uses a window to provide quick access settings - it is standalone though and could use a custom framework.

Dependencies:

No, sadly optional and required are not sufficient to describe the mod’s server<->client relationship. I will go into more details, in the “new to roadmap” sections later.

I absolutely support having a dependency graph over the current system (the WIP mod loader that we are working on, will have that too).
The current system priorities is bad for obvious reasons, I have a long list. Due to its disadvantage I would completely leave them out, unless it is trivial to add them as tie breaker. But even then - they will never be a reliable tool to make good decisions!
I would say, lets leave them out until there is a good usecase, and then think about how to solve the issue better.

Having dependencies with version restrictions is “nice to have”, as it can result in more pretty mod loading error messages. But that is all to it. Modders new to versioning, are not that good with it, so there might be complications too.
I am not familiar with dependency version requirements though. Sounds like it can get funny easily.

Mod hub:

Uploading mods to the website is a MUST, it is the easiest most obvious and reliable approach. There should also be a decent public (Rest) API, so that third party tools and scripts as well as the game itself can upload and interact with mods. That way everyone can be happy and have it their way.

.lwmod, these files are currently (almost) impossible to be used, as the game does not fully support them #447 - .lwmod file fully ignored & #348 - Sound clip not found. Try zipping MHG and enjoy :)
I like the concept though. Especially as the data is compressed then (oh no, more load time). Bundled files are much easier to manage and less likely to corrupt.
I do like the .lwmod file extension, as it makes distinguishing the usecase more easy. For editing the content, it takes an extra renaming step though (before extraction).

Adding the author or rather the current maintainer of a mod into the mod ID is nice for managing mods and identifiying modders in charge.
This system works well, if every user has its own mod, that it is in charge of. However mod ownerships do change, also forks of mods might happen. These versions should be hot-swappable with their originals.
This is a problem, because currently components in the save file, include the mod ID into them. Now one sure can write save-file converters, but that breaks backwards compatibility and causes further headache.
In these cases a mod must be able to pretend to be a different mod, as in mod XYZ wants to claim to be mod ABC, in such cases there must only be no other mod ABC installed. I would call this: “imposter-feature”. That way compatibility will be preserved and collisions avoided.
I do not know (yet), if there are further issues with enforcing the author in the mod ID.

For version handling, I am not 100% sure. Yes your concept works for the perfect modder/user.
However, I do agree with @GHXX that at least one should be able to add versions in any order, to be able to provide fixes for older concepts of one mod.
Disallowing replacing an existing version makes me feel weird though. Humans do mistakes, so it might happen, that the wrong or incomplete files get uploaded. These things can be noticed within seconds or 2 hours and fixed right away. But “simply” updating the version might not always be trivial and can cause even more grieving.
Also, this might cause modders to be forced to skip versions, as one version is suddenly unusable on the official mod portal - but fine on other mod portals. Nah the bad feeling does not go away.
If you allow overwriting versions, then you can easily solve the immutability issue by also providing an upload timestamp, that the client can consider, when checking for mod updates. If you really are against replacing, you might also provide a simple history of when a mod file was replaced. But an overwritten mod version, is as good as deemed “deleted” by the author and should no longer be provided for download.

Keep CDN caching in mind for download URLs. I saw a website that added a hash of the file, in the filename (causes ugly downloads), please put hashes into the URL and not in the file-name.
^But with a hashing system, modders can fully decide for themself what to name their mod files (might not be the best idea though, TBD).

Auto updating:

(Response to pre-editing):
Auto updating in general is a big fat NO.
The player MUST always confirm, before any automated download / version change of mods.

And sadly I must also heavily raise my voice to the new proposal (post-editing):

  • In general, one does not update mods, unless they have bug-fixes or needed features (a reason to update).
  • Updating mods, can and will result in save corruption and broken/missing features (modders are also only humans).
  • If you start the game, you just want to straight up join SP or MP, without first caring about mods. (You set up your mods, you don’t touch it, unless needed or curiosity rises).
  • Remote MP servers normally lag behind with updating mods. There is no need to care for update when intending to join such a server. It is highly possible that one even has to downgrade a mod, to join a server.
  • Checking for mod updates before loading them basically means sending network requests before reaching the main menu - please don’t ever do that. This delays the launch of the game by seconds in the worst case.

It is good to (opt-out) check for update and show an indicator in the main menu, that updates are available. Once the user opens the mod menu, any updates to this point are marked as read - until new updates are available. This should be enough, to give the user a reason to check the mod menu, if it cares. But TBD how the indicator looks like or behaves.

The mod menu, may contain an Update all button and in general allow choosing the versions of each mod.

It can be an opt-in feature to automatically update all or specific mods.
But I strongly advice you to not add this feature. Until you are able to deal with the consequences. Just bear in mind, that you will need to restart the game for most updates (not necessarily for new mods) and that it is not clear as to when in game launch the update should happen - a user should be able to skip this step too.

Code Scanning:

Sounds fun, besides that I am pretty sure that it will flag all of my mods as dangerous, as I am using Reflection even for component adding mods (to do CustomData hackery).
It is also very good to notify the user, if the type of level of danger has changed/increased.
But besides a “level” there should always be a list of threats - as a number is not really representive.
It might be nice, to allow moderators to review specific mod versions and declare them as harmless.

Multiplayer with mods:

On joining, the client should be able to choose which optional mods to install besides the required mods.
Do not automatically install any mod, the client has to confirm the actions (downloads and version changes) that are about to be made - always.
Warning about threat level should also happen in this mod switch list (ofc).

I really recommend you (MHG) to try Factorio. Try to use the modding system (as in one hosts a server with mods, and the other person joins). Play with it, it is amazing how smooth this game solved mod support and multiplayer! They even have optional Steam integration - account linking - buying on their website. You can learn a lot from this game! You can also grab a standalone version of the game and see how easy it is compatible with the bought game. Mad respect to these developers.

@GHXX4 months ago (edited4 months ago)

possibly it would work to not allow modifying existing versions, but allowing deleting bad ones. So if you mess something up, you delete the current one and make a new one, which then has a different version number for immutability reasons, but nobody can get the broken one anymore

@Ecconia4 months ago (edited4 months ago)

There are some new steps that I would like to be changed/improved for the modding roadmap, that are not yet part of the current roadmap.
As each of them is not that trivial, I will respond to this message with the new points. As each is quite complex, it will take me some time to write all of them down.

So far I have the following points:

  • Dynamic component/packet ID mapping (see below).
  • Custom Data handling API should be more powerful (add CD update packets + decouple server/client/savefile further to reduce overhead).
  • More powerful Manifest file (+ Dependencies on Client/Server only + More control over server-client relationship).
  • Ability to disable/snooze mods (allow to disable mods without restarting the game).
@Ecconia4 months ago (edited2 months ago)

Manifest

Mod type

Previously this was clientOnly or not. This proposal suggests required and optional, but either is simply not powerful enough.

This is not yet fully done, mod types with 'imply' property are still missing

How it should be:

The server first sends its mod-list to the client (including versions), each mod will be typed with required or optional.
The client then can adjust its mods to match the server and send its mod-list back.

There are three core questions to be asked:

  • Will the client/server even attempt to load the mod?
  • Will client/server send the existence of the mod to the other side?
  • Will client/server refuse a connection from the other side, if the mod is not present on other side?

With that I have started to collect a few mods types:

  • Both, must be installed on client and server (adds components). [LW-Vanilla 0.91]
  • Server, only installed on server, no sharing of presence (fixes server).
  • Client, only installed on client, no sharing of presence (adds tools, fixes client).
  • Either, can be installed on either side, but no sharing of presence (either it does what the ones above do, or its a library for mods).
  • Server+, must be installed on the server, can optionally be installed on the client. Server exposes itself.
  • Server+!, must be installed on the server, can optionally be installed on the client. Server keeps quiet, client exposes itself.
  • Client+, must be installed on the client, can optional be installed on the server. Server exposes itself. (Client Tool, that needs server aid WireTracer).
  • Client+!, must be installed on the client, can optional be installed on the server. Server keeps quiet, client exposes itself. [LW-Vanilla 0.91]
  • Either+, can be installed on either side, but sharing of presence? Server exposes itself.
  • Either+!, can be installed on either side, but sharing of presence? Server keeps quiet, client exposes itself.

Main types:

The first four entries are straight forward, and there is plenty of usage for these. They are basically set in stone.
The advantage of Either is, that libraries can now stay hidden or a performance mod behaves fully different on client/server but is not required anymore.
The Server/Client types could be Either, but why waste resources to loading something on one side, that is not meant for it.

Optional types:

As for the optional types, these can dictate where they must be installed.
Due to the server maintaining the packet-ID and component-ID lists and serving different clients, the client needs to be more adaptive:

A mod that is Server+, must be on the server, can be on the client. This means, that a client must uninstall/disable the mod, if it is not installed on the server.
Point here is, it must be on the server if the client wants to use it.

A mod that is Client+, must be installed on the client, can be on the server. But the server cannot just uninstall/disable it, as another client might have it.
Point here is, the client can use it, without the server having it, but the server can have it anyway.

As for Either+, there are no constrains. It can be installed as standalone, but the two parts can interact with each other.

Special presence sharing types:

Which leaves the ! in the mod types and presence sharing.

As for Both, this will always expose itself, the mod is mandatory required on both sides.
But for every optional type, it somehow must expose itself, else it would use one of the non-sharing mod types.
But it is not clear, which side shares its presence first. By default, the server will share all its mods. And the client then can answer its mod list to make the server aware of which mods are actually enabled (of the optional ones, primarily).

And one side has to share its presence. Currently there is the missing possibility of the server not sharing the presence of a mod at all, yet the client sharing it.
For those cases the ! is used.

  • Server+! makes no sense, as the client is required to unload a mod, if its presence is not shared.
  • Client+! a client mod, that shares its presence to the server (current default btw), the server can then treat this client differently somehow - while staying stealthy (the mod is not exposed by default).
  • Either+! a mod that can be standalone on either side, announced presence by the client. Here the server can act differently on that too.

With this, all of the 3 questions are answers for each mod type, and every possible mod type should exist and can be expressed by a single string/type.

Other Manifest changes:

Dependency-Graph:

A Dependency means, that the mod will load after a certain other mod. Which ensures, that the resources of that mod are available at compile time and when the mod is being used.

A dependency is hard, as in the referenced mod is required to be installed. But there also should be soft dependencies, where it is optional for the mod to be installed. The mod then can use the API, to check if the mod is actually installed.
This is useful to for example detect different variants of possible permission mods (MC example).

Another thing to add would be the ability to load before another mod - useful to intercept its loading or prepare some resources. A good example would be to load before MHG and then exchange some of its loaded resources or inject some.

But too many variants can cause difficulty (see mod types). Hence I would suggest, to have the loading before other mods to be soft-depending too. If mod is installed, load before it.

The variants LoadBefore and LoadAfter would be added, to make full use of the dependency graph. While both do not require a referenced mod to be installed.

Client and Server are not the same!

The API and architecture on server and client are very different. And mods that are running on server and client don’t necessarily do the same things on both sides or even cannot do so.

Lets take for example Harmony, there exists a mod lwharmony which only works on clients. So someone else had to make a mod that adds Harmony to the server, but that was named HarmonyForServers.
A mod needing Harmony would have to depend on both. But it could not do that, as then the server would require clients to install HarmonyForServers which is totally pointless there. Further the server would try to depend on lwharmony which is client only and does not load there.

The solution is quite simple, let Dependencies be an alias for SharedDependencies and add ClientDependencies and ServerDependencies.

The same applies to LoadBefore and LoadAfter, there has to be Server and Client variants.

I do not mind it much, if in SUCC there is a Server or Client parent for LoadAfter LoadBefore and Dependencies or if they all are in there with prefix.

@Ecconia4 months ago

Snoozing (disable) mods

Not enough refactoring and loading madness yet? Then hear this idea!

It is common for many games to restart fully, if the active mods are being changed (Oxygen not Included; Factorio; Minecraft…).
And this makes fully sense, if the mod support is rather half baked and there are no APIs whatsoever and everything has to be hacked into the game.
But with LogicWorld there exists small subsystems and APIs to interact with each part of the game.

So the idea is, to not require restarting, but instead perform unloading of loaded mod changes. This would reduce the amount of restarts needed.
This however does not reload a mod, nor fully disable it. It basically just “snoozes” the thing that the mod would normally do.
For LogicWorld, it would look like the mod does nothing then - besides adding one or more assemblies.

The main use case of this change would be to allow quick changing between MP servers with different mod set.
But one could also take advantage of manual snoozing of mods, to prevent corrupting the component-ID-map of an SP save.


The biggest issue with snoozing mods would be that the current LW does not support it. There are two primary challenges:

  • As currently most sub-systems are loading all data from GameData at once on boot, these need to be changed to allow loading data “per mod” and also unloading data “per mod”.
  • To not always scan the assembly or parse the GameData files when loading or unloading data into/from the system, parsed or found data should be stored in a cache before eventually being added to systems.

If these two changes would be done, it is trivial to snooze mods that only add (e.g. languages / components / themes / Commands / Packets etc).
But mods can do much more, especially hijacking the game. But Harmony allows unpatching methods, so if mods get an API callback for snoozing, they can restore the original method state and hence be able to snooze. Same if mods interact with Unity, that action has to be undone manually by the mod on snoozing.

So there is a lot of trust in the modders hand, that such a system is possible. But modders should be able to choose for their mod if they can be snoozed and how.
Following logical snooze types should be available:

  • No snooze: The mod cannot currently be snoozed, as it either is doing irreversible changes (loading assemblies) or simply does not support it by lack of implementation.
  • Full snooze: If the mod has a mod class, snooze will be called on it, to let the mod unload what needs to be unloaded. Further LW will unload everything it loaded for the mod.
  • Partial snooze: The mod does not fully support snoozing, but nevertheless users can snooze it, without expecting damage to the game.

But in the end it boils down to: Snoozing? Yes/No!


This is a typical “nice to have” feature, but it is not necessarily required.
It would improve the life of the players a little bit.
But has a high implementation cost on LW’s side.
On top, it also has some responsibility on the modders side, to properly declare and optionally handle the snoozing.

@Ecconia4 months ago (edited4 months ago)

Another thing worth mentioning, addresses the actual mod loading.

Currently LogicWorld does not update its window until all mod loading and more is finished.

This is a big problem on multiple layers:

  • Humans do need to see that a program is doing something. There must be at least a loading screen with a status, saying what is currently being loaded. A simple one, may have a progress bar too. => In unity this would be its own scene, that is switched once done.
  • LogicWorld just fails to start if something goes wrong, for users and modders this means a black screen without further explanation. If something goes wrong while starting, there should be an error screen showing the issue caused by most likely a mod compilation or initialization error.

So ideally, before loading a single mod or any files of it (including MHG), a loading scene is shown, that has a progress bar, showing roughly what is being initialized.
When it comes to mods, it can show how many there are in total and if it is currently being compiled or loaded (with details on what exactly).

Anything goes really, as long as it has some basic information on what is happening - and is able to show errors as text in case something goes wrong.

Now I said “before loading anything from GameData”, the problem here ofc are language files, the loading screen will have text and that needs to be loaded somehow. But I think we can be creative here TBD.

Currently error messages of mod loading are not very helpful. They should be improved to for example print compilation errors on screen. Show that if a mod misses a dependency, why that dependency was not loaded yet. Which mod’s files could not be loaded? And whatever else might occur.

Related issues: #339 #310

@Ecconia4 months ago (edited4 months ago)

Current Custom Data handling:

Client and Server store the latest seen CD of a component in a cache to always use it.
If a component has a custom implementation, that will also additionally store the parsed CD.

Client:

  • When receiving a CD update, it is applied (cache + custom-impl).
  • When editing CD, it is only (optionally) applied to the custom-impl and directly sent as BuildRequest.

Server:

  • When cloning and saving, persistent data is saved first.
  • When the server changes CD, no extra steps are taken.
  • When the server receives CD, it validates that the component exists.

Every above action on the server, will forward the CD (serialized or received) to the clients and then “apply” it locally (including deserialization).


LogicWorld’s CustomData has a simple concept:

  • CD describes the whole state of a component.
  • CD is the same in save file, server and client.
  • Persistent data are parts of the CD, that are not nessecary to be transmitted to the client immediately. It can be updated in CD, for saving or cloning of a component.

This concept has a few issues that affects vanilla components already:

  • Many components have previousXXX variables, to detect changes to specific CD parts.
  • It is not possible to undo changes to specific parts of CD, only the whole component state can be reset.

With memory components (added by modders), CD becomes larger.
And LW has the habit of sending the full CD more often than it has to.
A combination resulting in network overhead. And the client receiving CD parts it cannot use or process.

To my knowledge, having the full CD on the client present serves the purpose of the client downloading a world from any server. But the persistent data concept already slightly invalidates this, as the client cannot yet request the server to save persistent data. (The client could save directly after the server saves, to get the whole world).


LW uses a simple and generic Custom Data concept, that makes it easy to maintain the game and to create components.
The downside is overhead on server, client and network. And no way to undo components partially.

Fix these issues, the concept (and even more the implementation) of Custom Data has to be reworked.
But I cannot write down a better concept, but rather things that I would change to the whole handling:

1) [This suggested change is for my modding usage mandatory.]

One major improvement to the API would be the ability to register “listeners” to CD, looking like this: sizeXChanged(int previous, int current) (could have no or just one argument). The component designer would register these methods in a CD manager that LW provides.
The other major improvement should be the ability to add single parts of CD to the undo histroy.

The advantage is much less overhead on primary the client, as custom data does not have to be parsed and processed as whole, and there is no need to detect changes manually - which makes components a bit more robust.

To implement these two features, I would add a new BuildRequest+WorldUpdate type. This type would allow a component to update its other client/server side with messages/events/updates (from now on named event) in form of a byte array (to allow the component designer maximum posibilities).
To make this compatible with the history system, it should be possible to generate inverted events that can be added to the clients history. This however strictly requires a component to have custom code on the server. As without, the data cannot be interpreted.
As I would like to see this integrated in the LW’s CD manager, it might be a good solution, to have two versions of the CD manager wrapper, one that does not require custom server code and one that does. [The custom server code would be as simple as generating an empty component class with the wrapper as parent class].

Another great thing about sending custom events is, that components can send data that is unrelated to its component state (CD) - allowing to stream data from server to client or the other way round.

Using the event system to transfer CD however, means that all the data is persistent. And needs to be serialized before cloning or saving (only when needed).

2) [This suggested change, is difficult to implement, heavily TBD - but maybe kind of requried]

If we ignore the fact that LW clients potentially may want to download a world or find a different solution to implement this, we can consider the following:

With increasing size of CD for modded components. It might be good to fully exclude some sections of CD from being sent to the client ever, which reduces overhead.

In general there is no need for persistent data to be sent to the client. I see no reason to not treat persistent data as excluded data.

The disatvantages of this system are plenty and all on the implementation side of the server. Here a list:

  • The CD cache of a component may never contain excluded data. This means, that when saving the CD with excluded data must be stored temporary or elsewhere.
  • When cloning a component, CD has to be serialized twice for client and server.
  • The serialization on a server component needs to know if it is with or without excluded data.

3) [This is not really a suggestion, but in general a call to remove overhead]

LogicWorld server should “simply” stop broadcasting CD when it is not required.

Currently it is broadcasting CD when saving and whenever something is copied. This is redundant, as the client does not need the persistent data and client CD does not simply corrupt, hence does not need a refresher.

The LogicWorld client is at serval places in code serializing and immediately deserializing CD to “refresh” itself (according to method naming). I am convinced, that this is pure redundency, and if not, that there are significantly neater ways to take care of this. This should simply not happen.

This is mostly done when resizing things (or drawing boards). It should be sufficient, for the resizing code, to remember the XYZ values and only change/reset these.
-> But there are a bunch of caveats here with when data is to be broadcasted to the server and the CD wrapper/manager. TBD.


Previous ideas by me: #344 #345

@Ecconia4 months ago (edited4 months ago)

Dynamic component ID mapping

This is imho a mandatory change.

The currently most commonly existing mod type are component addition mods.
These basically don’t change anything in the game, except adding components to the client/server.

There currently exists one major downside to this. Different servers have different set of mods installed, commonly MHG and MHG + Cheese.
So now you got MHG + Cheese installed, and you actually can join vanilla servers, but if someone performs undo you will crash #346 as the component-ID maps are not equal on client and server. (Funnily this is not checked on join).

A modded component can exist without disturbing vanilla functionality, or any other component.
So there should be an improvement to the game, that allows users to have as many components installed as they please, but when joining a server, those not required, will be suspended/hidden (removed from the component menu, removed from the component ID map).
This change ramps up the ease of use of modding as well as robustness and reduces required restart amounts.

Technically this is easy to implement. The server already sends its component-ID mapping to the client.
All the client has to do, is to confirm, that the component in that mapping are loaded locally (else disconnect) and finally enable/disable the components and set their local IDs - instead of doing that while mod loading.
This is also fully compatible with SP.

Having multiple component mods installed, will no longer be an issue, as the client disables the ones not relevant on the current server.

Another major upside of this is, that it is significantly more robust, as if any OS/Runtime sorts the files of folders differently, the components would be loaded in a different order. With this change, no sorting is required, as only the server’s order is deciding on the order of components.
-> LW is probably sorting components, but this won’t be required then.

Dynamic packet ID mapping

Less urgently, but should also be done.

Currently no mod is adding packets to the game.
Same as with components, client and server need exactly the same packet ID mapping (except that this is never validated, one just crashes if there is an error). And this request would never be an issue, if the mapping is always the same. But it should be possible for OPTIONAL mods to provide packets too.

Server and client mods might be optional on the other installation side. And this should be possible to be configured in the Manifest file.
A server mandatory mod, might be a region manager, which when optionally installed on the client shows region outlines.
A client mandatory mod, might be one to show statistics, which when optionally installed on the server can show additional statistics.
Both of these mods can run on its primary side, but both also use custom packets to communicate with the other side, if it has a mod installed.

This is currently not possible, at all. Hence I would suggest to apply pretty much exactly the same mechanism as with component IDs.
Let the server send over a list of all packets, and tag them with required and optional.
If a required packet is not installed on the client, it disconnects.
The client will then assign the IDs sent by the server to its packets, if an optional packet is not installed, it will be ignored upon receiving.

Same as with components, one could require the server to just send the mod list, but that again is not as fail-safe and requires version and sorting of packets to be precise.

With this change, mods can go mad with packet installing, ignoring if the other side has that mod. But it must define if the mod is required or optional on the server side, which will be applied to the packets.

This change requires LW to assign packet IDs on join instead of on mod loading.
Practically, it would remove the requirement of loading MHG packets first or at all (ignoring the discovery packets).

Oh right, the biggest issue here is, that the server has to send a new packet to the client, right after its discovery request, containing all the packets required.
In this packet other meta information could be sent too - like a mod list.

In general discovery and the packet-index packet should not be part of the dynamic packet list, they might have hardcoded IDs or better get parsed specifically in a different context (like MC is doing it - much better - but TCP - with Lidgren/UDP the whole joining is already unsafe).

@pipe01Developer4 months ago

Responding to some comments @GHXX made on Discord here for posterity:

1) does monomod include ALL featurs of harmony? e.g. - accessing the instance of class of a patched method - modifying the return value of a method - Patching by name / when stuff is private

AFAIK there’s nothing that Harmony does that can’t be done with MonoMod.

2) monomod makes on-disk changes as it generates a dll. Does that play well with steam in the context of updates and validating game files?

3) what is the advantage of using this over harmony?

4) >Generated assemblies will be strongly typed, allowing for easygoing and idiomatic modifications. Isnt all of c#, excluding ‘dynamic’ strongly typed? Did you mean “strongly named assemblies”?

No need to distribute assemblies like Harmony with each mod. Thats also not required if the game distributes a copy of harmony OR if there is a single mod that offers harmony (which was the case in 0.90.3 for example)

Can still use MonoMod Hooks manually if more customization is needed.

Is the idea to provide preexisting hooks? If that’s the case then i would suggest making such hooks just a public method which is listed on some api docs and can then be pre or postfixed using harmony which does the same to my understanding but is much more intuitive.

TLDR for 4) if at all possible id argue that it likely is better to stick with harmony unless monomod includes ALL harmony features AND harmony cant/wont be updated for net7

It doesn’t actually modify the DLL files, it works pretty much the same way Harmony does. The difference is that MonoMod allows you to create a separate DLL file that contains method stubs for every method, property, etc that the game assemblies have, regardless of accessibility.

This is what I meant with strong typing, you don’t need to hardcode class and method names to access internal stuff, you can use the generated hooks to make sure you don’t make mistakes. Check the example code to take a closer look at what I mean.

5) a hook system could definitely be useful, e.g. when someone wants to run code when a simulation graph was modified i.e a wire was added (i think the bundled wire mod could have made use of this)

Yeah, although honestly I’m still not sure what this will look like. If MonoMod’s hook generation works well, there may be no need for this, as it would be basically the same thing. Maybe we will only provide a very reduced amount of event points for common functionality, and leave everything else up to MonoMod hooks.

6) gui system would be great, but it may be worth not limiting it to custom data. One idea would be allowing to place sliders and other components into the ui, similar to RimWorld, although id opt for a setup/use style of doing it rather than having a single call that adds it if its not there and also updates it

In the future we will provide a powerful UI system powered by Unity’s UXML, but in the meantime we want to give you something to address the most immediate need, which I think is editing custom components’ data. The details aren’t clear yet so I’m not sure how customizable it will be, but I do intend on at the very least letting you choose the kind of widgets that will show on the dialog.

8) Mod uploading .lwmod may work pretty well, especially if its just a zip file I think uploading it via website and cli would be best, that way stuff can be automated and you dont have to boot up the game to upload your mod. (this would also allow people to tell their gitserver to automatically compile the mod and push it to LW if a commit was made on the publish-branch for example. 5d.ghxx.dev’s GitHub actions do this for instance, minus the uploading it to lw part obviously)

I definitely want to make some kind of CLI utility for mod making and uploading, there are many things that could be automated like project and components scaffolding.

9) mod id and versions AUTHOR.MODNAME is a good choice imo, and so is version immutability. As far as uploading older major versions go: Id argue it does make sense to upload an older major version. As an example: Look at Minecraft’s Industrial craft: I believe there was a time when IC2 and IC1 coexisted, this possibility would be taken away with the enforcement of this rule. Thus I think it would make sense to allow people to upload any major version but enforce that major and patch must not be smaller or equal to what the current values for the newest release with the currently selected major version are (i.e revision or patch has to increment if the major version stays the same)

That’s a good point, I think we will probably end up doing this.

10) Auto updating on boot: Id argue that this should not be done. Some modders might not follow conventions regarding when to push what version numbers. so you might end up with a modder making a breaking change which then causes havoc for everyone. What would be better imo is having an ingame list of the installed mods, letting you see what the current and new versions are (factorio style) and letting you update specific ones, or all, with the click of a button. Additionally, there isnt really a benefit to pushing out a mod update quickly imo; in fact, it only makes it easier to push devastating or malicious code to more people.

Oops, I changed my mind about this but forgot to update it. We settled on something like what you said, I have updated the post.

11) code scanning: i believe this is a first in terms of modding, while definitely good. id not sum it all up to a numerical score, but instead list it on the mod’s page: Reads Files On the local system Peforms http requests Spawns new processes Whats a bit tricky though is that it might be easy to get around this detection (dm for specific details) Additionally, the game client could, upon installing the mod, ask the user if thats fine that the mod does that, adding an extra layer of protection. Such that its not possible to make a mod, get people to install it, then add code which flags it as Reads-system-files and then have people update it without noticing

I agree that it would be easy to circumvent detection in some cases, but the goal is to identify that such circumvention has ocurred by, for example, detecting the use of reflection. At the end of the day though, it is impossible to know exactly what every piece of code does, but this will at least give users a heads up on most cases. We don’t intend to build an antivirus, after all.

What i also wanted to point out is, that mod signing (by lw.net) is not on the list and i feel like it should be added as the prerequisites are all met and the benefits could be quite good. With mod signing, players could actually download mods from multiplayer servers directly (possibly only asking lw.net if a certain mod version has been taken down due to containing malicious code) Additionally this also severely mitigates the re-distribution problems of mods. E.g. think of the minecraft-mod-redistributors such as 9minecraft.net and similar ones. You never know what they put into the mod files.

This is actually already implemented in-game, although I didn’t realize that, since lwmods aren’t actually being used, nobody knows about it. You brought up great points, that is exactly why we’re going to sign every mod that is downloaded from logicworld.net.

Deleting mod versions If a mod does contain malicious code i think it would be worth for staff to be able to delete it off of the lw-mod-website and to also force all clients to disable the mod once locally (but allowing people to re-enable it maybe?) Such a deleted mod version also shouldnt be auto-downloadable from a gameserver (even if the signature is good)

And another topic: Id suggest requiring a lw accountt linked to a game purchase for uploading mods on lw.net, that way malicious people can be banned off the platform and making malicious mods becomes expensive. I do however hope that nobody innocent gets negatively affected by this. But i think overall this should be beneficial

Those are also great ideas, we will definitely implement them in the future. Especially the second one, that might come on launch.


Thank you for such a thorough response!