Improve bevy ECS integration

[ErnWong]

Right now, if we were to use the provided bevy plugin, we would need to write most of the game logic outside of bevy's ECS. It would be good to integrate crystalorb with bevy's ECS.

See other similar plugins for examples

• https://github.com/gschup/bevy_ggrs
• https://github.com/HouraiTeahouse/backroll-rs/tree/main/bevy_backroll

Also investigate any new or upcoming bevy ECS features that would make the plugin more ergonomic.

[Shatur]

Yes, It would be great to decouple world somehow. Because right now I should put all my networking logic into a single data structure.

[Shatur]

@ErnWong, do you have any design comes to mind? I would like to help.

[ErnWong]

Thanks for your interest! I've got some ideas that might be cool for us to try out. I'll see if I can do bit of investigation this weekend to flesh them out and will get back to you. (Feel free to ping me if I don't get back).

Ideally, it'll be nice to see if we can leave the core CrystalOrb crate more or less the same, and have the changes mostly reside on our bevy plugin crate side of things. CrystalOrb is designed around simulating two instances of the World at any given moment, which, if I understand correctly, is quite different to how backroll and ggrs does it, so it may be cleaner to structure our bevy integration approach differently to theirs.

The Bevy Sandwich Idea

I might as well share what I have in mind, but feel free to suggest other ideas too!

If we went with CrystalOrb's current style of doing things, then we might try and implement the crystalorb::world::World trait for an entire bevy_app::App (or bevy_ecs::world::World) and we'll end up with a delicious bevy sandwich (maybe we should call the plugin bevy_sandwich 😂):

• We have an outer bevy_app::App, consisting of
  • e.g. the users' game rendering systems
  • other game logic systems that doesn't need to be networked
  • entities and components to be displayed to the screen
  • the bevy_crystalorb plugin, which supplies:
    • a crystalorb::client::Client<bevy_crystalorb::World> bevy resource, which:
      • internally spawns two instances of bevy_crystalorb::World, each of which consists of:
        • an inner bevy_app::App, which consists of
          • entities and components to be networked
          • update systems and setup systems for running your networked game simulation logic
    • a bevy_crystalorb::update_system, which invokes crystalorb::client::Client::update and syncs crystalorb::client::stage::Ready::display_state with the outer bevy app's components.

The game developer would probably set up their game code the usual way, and tag/register their components and systems that they want CrystalOrb to network. Then, the CrystalOrb bevy plugin would hook the inner bevy apps to the correct components and systems, and act as a bridge between the inner and outer bevy apps.

Hmm, that might be a bit confusing. Maybe I should try and draw it out:

Yikes, that looks even more confusing. This approach might sound a bit heavy-weight on paper, but so far, CrystalOrb's design decisions has been based around choosing the "cleaner" option over a more performant option. ("Clean" is very subjective though 😛 )

Bevy Subworlds

Having bevy apps inside a bevy app sounds a bit like the bevy subworlds RFC (bevyengine/rfcs#16). I haven't explored that RFC in too much detail, but I'm not sure if it'll be easy to make our ownership hierarchy clean using that approach (since I'm guessing we'll need some sort of circular reference between bevy and our crystalorb client?).

[Shatur]

Sounds interesting :) But why we need to spawn two worlds?

You may also find it interesting that the Naia network engine is also trying to implement integration for Bevy: naia-lib/naia#22
I thought you might be interested in looking at this implementation. I talked a bit with the author on Discord, he plans to complete the integration in a few weeks. Naia advantage is that it doesn't need a nightly compiler.

[ErnWong]

We simulate two worlds because whenever we want to perform a rollback:

1. It can take multiple render frames to fast forward the world back to the current timestamp, so we need a second world to simulate and display during this time.
2. Once rollback and fast-forward has completed, we don't want to abruptly show the corrected world to the player or it will be very noticeable. Instead, we want to gradually transition into the corrected world state, so we'd need to keep both worlds simulating and linearly interpolate the display state between the two worlds.

At least, that's the approach taken by CrystalOrb. It's like operating in some kind of higher level "World Algebra" where we treat the world as a black box. (I'm making these terms up btw)

[Shatur]

We simulate two worlds because whenever we want to perform a rollback:

Got it, sounds reasonable!

[ErnWong]

Hmm, looking at the top right quadrant of the diagram, using an outer bevy stage to mark which systems we want to run in the inner bevy app might not be a good idea:

1. Low impact: Extra things we need to pass down
2. Medium impact: Misleading design because the user would register systems in the outer bevy app even though they're only run in the inner bevy apps.
3. High-ish impact: Users can't use other bevy plugins (like bevy_rapier) and incorperate it into their networked simulations because the user doesn't have control over what stage these plugins puts their systems into. It is also restrictive that we're only assuming that the user's custom systems should all fit into a single stage.

Rather than give the illusion that the user is adding systems to the outer bevy app, it might be better to expose the inner bevy app directly to the user to add things into, or accept an app bundle (or a set of app bundles) [EDIT: typo, should be PluginGroup or set of PluginGroups] that we use to initialise the inner bevy app.

[jamescarterbell]

For anyone who comes across this, I'm currently working on a bevy plugin that makes the bevy sandwich.

https://github.com/jamescarterbell/crystalorb/tree/feature/bevy_plugin

[ErnWong]

@jamescarterbell
Continuing on a discord discussion, if I understand correctly, there are two open questions. (There are probably more to come, but here are two for now).

1. How do we pass in the necessary information to initialise the inner bevy worlds?
2. How do we add or refer to the desired network resources?

I'll throw in some ideas - see if they help.

Regarding 1. Initialising the Client's inner bevy worlds

Initialising the Server is easy because it only needs on instance, but the Client requires two instances.

This might not be an exhaustive list, but I can think of two approaches:

Option A: Passing in a bevy app as a blueprint

The game developer passes in a bevy app when instantiating a BevyCrystalOrbClientPlugin, and we use this bevy app as a blueprint for creating our two instances of the inner bevy apps.

I'm not sure how easy it is to clone the bevy app this way. Systems might be fine, but it might not be possible for resources and components? (Feel free to disprove me - I'm not too familiar with the inner workings of bevy to answer this).

Option B: Passing in some sort of "recipe" for initialising the bevy app

The game developer passes in some sort of "recipe" for setting up the bevy app. For example, this "recipe" could be a closure, a struct implementing some sort of factory trait.

Bevy's Plugin trait is a good example, and I think it might be a good fit for our purposes because in my previous bevy game, I've already wrapped all my game systems in a plugin for organisational purposes anyway (I'm not a game dev though so I can't back up my claim that it's a good idea).

One possible downside of using Bevy's Plugin trait is that Bevy might want to go for a more structured Plugin management system in the future that makes it unsuitable for us, but when that time comes, it should be easy to swap out their Plugin trait with our own InnerWorldFactory trait.

The crystalorb client/server currently initialises their World/Worlds using the World's Default::default implementation, but that might be too restrictive. I'm happy to change that to something a little more flexible, for example, maybe something that takes a world factory lambda:

//////////////////////////////////////////////////////////////////////////////
// Dummy representation of a modified client.rs from the core crate:

// W no longer needs to impl Default
pub struct Client<W>(W, W);

impl<W> Client<W> {
    pub fn new<WorldFactory:Fn() -> W>(world_factory: WorldFactory) -> Self {
        // I can then create as many worlds as I like!
        let world1 = world_factory();
        let world2 = world_factory();
        Self(world1, world2)
    }
    
    // Rename the existing Client::new to Client::new_with_default_world
    pub fn new_with_default_world() -> Self
    where
        W: Default,
    {
        // Convenient but not necessary - what existing Client::new API does at the moment.
        let world1 = Default::default();
        let world2 = Default::default();
        Self(world1, world2)
    }
}

I hacked together a small proof of concept as an example: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=261239bf8a121a0bb37669661300b76c

But feel free to modify it to suit what you're doing, or completely disregard it if it doesn't play well with your code.

Regarding 2. Referring to the desired network resources?

I'll get back to you about this question later... 😄

[jamescarterbell]

So there's two big issues at play:

1. Schedules are non trivial to clone properly (and it may just be impossible)
2. The builder idea is fairly unergonomic.

The core issue is that bevy schedules can't run on multiple worlds because they do some caching stuff apparently, so because of that we need two schedules and dedicated worlds for each schedule, which makes it hard to integrate with the two world architecture of crystal orb.

[ErnWong]

Hmm, good point. The builder idea might not be the most ergonomic, although I'm not too sure I'm fully aware of why that is and would be curious to understand it better to help make better decisions.

I'm guessing you're referring to the extra boilerplate code needed to wrap around the systems we want for the inner worlds? (Which I agree is a valid concern btw). Would you happen to know if there are other concerns about ergonomic other than the extra boilerplate? E.g. Perhaps, does it restrict the game developer from using certain game designs? Perhaps, does it prevent certain existing games from being easily ported over to use crystalorb?