OK, this is awesome. As a fellow game developer who learned to navigate the complexities of matchmaking and server orchestration the hard way (and begrudgingly), seeing Rivet open source a matchmaker and server orchestrator is exciting and honestly refreshing. For those outside of games, the use of modern webdev / infra tooling is frustratingly nascent, and production-ready OSS usable by smaller-than-AAA-sized studios is even more rare, just beginning to come online (the efforts behind https://game.ci are a recent shining example in game infra).

I'm particularly encouraged here by the tools and approaches chosen — Redis for matchmakers, ClickHouse for analytics/logs, Rust for speedy backend operations were tools I eventually came upon as best for the job after a long and painful process of iteration with inappropriate cloud dev tooling (with basically any datastore other than Redis you will run in to issues matchmaking).

High traffic/concurrency in matchmaking is a hard problem when you need fast, consistent matches. Existing industrial-sized matchmakers (Google's OSS OpenMatch, AWS GameLift, etc.) require locking in to a very specific format or spinning up an entire K8S cluster, which for game devops teams already stretched thing is a tall ask). And for all that effort, you may discover the performance characteristics of their approaches aren't what you need, or (as has happened to us with different game web service/API/middleware providers) they may choose to abruptly deprecate the service upon which you built.

Appreciate the dedication to making this available and the focus on helping gamedevs get up and running as fast as possible. Looking forward to playing around with Rivet and seeing where it might fit in our stack — & would love to talk shop on this with anyone since I've been tortured by / interested in these problems while trying to simultaneously run a game studio / make new games / keep everything online the last few years!

> https://game.ci

I don't recommend it.

Game CI has been in development for a long time. It only really supports GitHub Actions. It can't correctly build il2cpp Unity projects. The licenses needed to run it are more expensive than Unity Build Automation / Unity Cloud Build. If you want to automate Unity builds and you don't want to learn the Windows Containers ecosystem, Jenkins and/or Tekton, you should use Unity's service.

CI/CD is a bad choice for most developers, on most platforms, for most clients. That said, it makes sense to do for your backends.

> or (as has happened to us with different game web service/API/middleware providers) they may choose to abruptly deprecate the service upon which you built.

Which service was that? GameSparks? That sucks.

> with inappropriate cloud dev tooling

Well everyone takes their own journey to discover how shitty Lambda, Cognito, CloudFormation and related are.

> with basically any datastore other than Redis you will run in to issues matchmaking

Matchmakers do not have to be complicated.

> CI/CD is a bad choice for most developers, on most platforms, for most clients. That said, it makes sense to do for your backends.

Are you making this statement about game devs, or more broadly? I don’t know much about game dev, but if more broadly that’s a pretty spicy take.

Only in my experience, automated building of the client side of a game is usually low ROI. In broader contexts yeah you should be doing CI/CD.

I'm having a good experience with some automation build automation with Councourse CI. Of course, I self host, and I am doing some real trivial stuff right now.

This echos my experiences, especially with the move to “live services” model for games. Often the teams working on the services for multiplayer games are very small so it’s all about leverage. They’re also embedded within a culture that doesn’t value or understand the practices and tools that are taken for granted in webdev, mostly because they’re hard to transfer to the other domains in gamedev.

I’m glad to see the open source development of these tools. Personally, there’s a lot of stuff colleagues and I have built that I would love to see open sourced. Otherwise we just end up implementing the same services over and over again.

Another example is patch delivery. It’s a solved problem yet everyone keeps rolling their own, or only shipping to Steam. Of course, all managed with some Jenkins scripts written by someone that’s no longer at the company. I want Fastlane for games that makes it easy to target the various distribution channels across desktop/mobile/console.

The lack of a standardized patch system for games is mind boggling! Sure there’s some for desktop software exes which just ship the new version every time but those apps are rather small and games can be several GB! We had to roll our own for our game but I couldn’t believe there wasn’t a universal standard when I went searching for one last year. Butler from itch.io looked interesting but I couldn’t decipher how to use it in the end. Also rolling patches manually seemed crazy to me and we ended up using a library similar to rsync that works pretty well and makes deploying patches easier but still not perfect.

Thanks for sharing your experience and kind words — we would always be happy to talk shop. I think it’s a shame that current infra solutions prevent a lot of hobbyists and studios from ever launching a multiplayer games, I would love to see it as prolific as single player experiences.

The fact that this solution is based on nomad is a big red flag:

- no updates after December

- you probably won't have the right to use nomad after that

- every providers offer Kubernetes managed solution, by using nomad now you're pushing that burden to host and manage infra since no one is offering nomad services

Using Kubernetes is a huge advantage since all the dev ops tool can run on it, it's tested and everyone is using it.

As for more flexible than Agones I highly doubt that, do you have example?

Counterpoint: Providers offer k8s managed solutions because it's such a pain in the ass to run yourself. Nomad is far simpler to setup, operate and upgrade. And lastly the folks that are independent-minded enough to go with the less-popular option (Nomad) are also the folks more likely to host it themselves no matter what.

Kuberentes is such an overengineered disaster. In 10 years, everyone is going to wake up with a hangover and pretend they never heard of it. Anyone here built a SOAP system? Of course not, you would never.

What exactly is over-engineered about it? A lot of the seeming complexity is due to solving things that completing systems didn't even try to tackle or did so incompletely. For the base case of I want to run a service and expose an http API it's pretty easy, and then you get to coexist with other services.

> For the base case of I want to run a service and expose an http API it's pretty easy

Are you kidding me?

Before you can do anything, you need...

1. API Server

2. Scheduler

3. Kubernetes Controller Manager

4. etcd

5. Cloud Controller Manager

6. Nodes

7. Pods

8. Container Runtime Engine

9. kubelet

10. kube-proxy

And then before you can even expose it on HTTP you need an ingress, kube-dns, load balancer...

Oh, don't forget that you'll actually want Helm and all of its components...

Meanwhile, Nomad is a single binary, and it can even run scripts and other non-containerized workloads.

As if you dont need nodes, pods and a runtime for Nomad.

An ex-colleague “sold” us developers nomad once as an easier, better alternative to k8s. They used exact same arguments, and we ended up with Nomad for our test and staging environments. It was such a mess, while it covered basic “run this” scenarios fine, nothing like kustomize or helm for nomad existed back then (I doubt it exists now). The tooling was non-existent, neither were good practices or recommended setups - nobody seemed to know how to tie it into a ci/cd setup without a bunch of bash scripts to get basic stuff like templating working. Implementing any kind of edge case or a tricky requirement felt like we are doing breakthrough work since no documentation, guide or SO page covered anything but “simple setup of your nomad cluster in two steps” - nomad is simply not popular enough. That experience made me swear off any unpopular but “trust-me-bro-its-actually-good” software.

Sorry for the rant, your post made me remember how pissed I was at our tech stack back then.

Meanwile no one offers Nomad managed service, so who create those clusters, who operate them, who keep security / upgrade on...?

You go to any cloud provider they all offer managed Kubernetes solution.

If you're running it on your own setup it gets complex. However half the concepts you listed exist in Nomad also, like the container runtime engine.

It's true everyone doing anything with infrastructure should be using Kubernetes, it's over, it has won.

But it's complicated.

Game development uses all sorts of red flags throughout all of history.

The people making successful games are talented enough to deal with the downsides of anything.

Nomad specifically is used by e.g. Roblox to orchestrate. Do I think that's a good idea? No. But it doesn't mean it can't work.

With regards to your specific criticism I don't think that stuff really matters, there are tons of people using tons of totally unsupported / unsupportable game engines, open source libraries, etc.

I believe the it's over because tech X has thinking is at best lazy and at worst dangerous. It implies that the everyone's requirements are the same and static and that solution X has reached all of those requirements.

If people blindly accepted X has won we'd be still using Perl, XML, CGI-Bin, Apache, Angular, etc. As another comment has said, I believe years from now, folks will be looking back at in disbelief at how anyone could have considered k8s to be the best solution without room for improvement or alternative approaches.

> I believe the it's over because tech X has thinking is at best lazy and at worst dangerous. It implies that the everyone's requirements are the same and static and that solution X has reached all of those requirements.

I'm somewhat inclined to agree, because iterating on what we have and developing new tech and practices seems like a good idea (the concept of 12 Factor Apps comes to mind).

However, I can't help but to wonder what this would look like, when taken to the extreme. Perhaps something along the lines of: "We are only going to use Ubuntu for the servers and Java for all of our back end services, as well as React for the front end. Nothing else, no matter how many different projects there are."

And on a certain level, it makes sense: consistency and skills that carry over seem like a nice thing to have, there's even terms out there like a company being "a Java shop" and so on.

That said, such inflexibility might not always be nice (e.g. trying to do machine learning in Java) but at the same time if a specific technology is in demand in the job market in most companies, you don't really get much of a choice.

To return from the analogy, in the particular case of Kubernetes, you'll have to learn and use it and there's only so much free time for exploring other options.

> If people blindly accepted X has won we'd be still using Perl, XML, CGI-Bin, Apache, Angular, etc

For many folks, some of those might indeed be good choices.

Some backup software that I use is written in Perl and is actually pretty good (BackupPC).

XML can be a nice format to use and has lots of tooling available. Even SOAP wasn't necessarily the end of the world thanks to WSDL - being able to plug a file into SoapUI and getting an API that you can play around with, something that took Swagger/OpenAPI years to reproduce for REST.

There's something nice to be said about the simplicity of CGI and FastCGI still makes sense with something like PHP (PHP-FPM).

I actually still use Apache for hosting my personal stuff - if has most of the features you might want and with mod_md it's a bit closer to Caddy than Nginx in getting LetsEncrypt certs (ACME). If you disable .htaccess, reasoning about the config also becomes about as easy as with Nginx (nginx.conf).

I've also seen Angular be used reasonably well and the "batteries included" feel of it was nice, especially when React had about 1500+ dependencies for a new project. I do personally lean towards Vue, though (oh and migrating away from AngularJS after the EOL was a pain, when that was relevant).

My point is that if something is good enough for your needs and helps you put food on the table, then you can just go with that choice.

Let those who prefer to experiment do so (hopefully not with your livelihood), celebrate their efforts and successes and learn from their failures, though. Kubernetes is established, Nomad is interesting, even Docker Swarm or Compose is okay.

Last Nomad release was 1.6.1 and were released July 21, 2023.

Really cool! I'll consider it when launching my godot game. Is there any plan to support "relayed multi-player", with the actual game server being one of the players, and rivet in charge of match making and making the bridge between the remote players. Like nakama : https://heroiclabs.com/docs/nakama/concepts/multiplayer/rela...

Cool project (based on me skimming the documentation). I’m a hobbyist game dev whose been attempting a deploy of a simple multiplayer game on AWS, specifically gamelift, for about four months.

Turns out I thought I knew more than I really did thus I’ve been spending much of my hobbyist time learning more AWS. I’m pretty stubborn about a local dev experience which takes time to establish an appropriate mental model to do. So it goes. Pro tip: find your application “interfaces” and discover what tooling can be used instead of a cloud resource (e.g. a mongoDB compatible document database is the interface to which AWS DocumentDB and MongoDB are concrete implementations). I have a docker compose file I use for local dev. It’s the best setup I’ve found for myself but I am open for advice.

Anyways I’d just like to voice myself as a possible target audience. Being able to develop locally is important to me and the hardest parts with AWS is having little support on how exactly to do this. Assuming this is even a fair ask to make…

P.S. another pro tip; I use Unity + Mirror networking for my multiplayer games. GameLift requires code integrated into your game code for it to function. I use nested prefabs to handle two cases: (1) when I want gamelift code to execute and (2) when I just want to focus on writing the multiplayer code. My production code uses (1) and I usually do development in (2). (2) is a child of (1).

P.P.S. GameLift has a lag of about 2-3 minutes when registering my running game server in Unity editor with a “gamelift anywhere fleet” before it can receive traffic. I didn’t know about this for a while and stressed out over the inconsistency. Make sure to measure timings on this so you don’t go crazy.

Thanks for the kind words and tips!

> Anyways I’d just like to voice myself as a possible target audience

Local development is tricky, and we have plenty of room for improvement here.

Currently we provide a special developer token [1] which will cause API endpoints to return mock responses when used locally. This way, you can use the exact same API with no difference in production & locally, so you don't have to mess with nested prefabs.

However, this doesn't help much with exposing your local machine for testing. We've toyed around with the idea of something similar ngrok built in to our Rivet CLI and integrated in to the API.

[1]: https://rivet.gg/docs/general/concepts/dev-tokens#mock-respo...

To be honest I do wonder if my fixation with local development is even justifiable. Looking around at projects and documentation I do get the sense that local dev is not a priority, and maybe that isn’t a bad thing. Amazing projects exist. Perhaps other game devs are more resourceful than I am and simply build, working around limitations.

I do enjoy learning and this constraint requires I deeply learn. However there is an opportunity cost.

When you say open source, do you mean that I can self host everything and not use your rivet.gg SaSS for anything?

What are the features only available in SaSS? Do you anticipate to have new features only for SaSS users? Is there a policy for releasing SaSS features to open source users?

I’m curious about this as well. Seems similar to Nakama [1] and they’ve got some gotchas that aren’t right up front about payment.

1. https://heroiclabs.com/nakama/

What are the gotchas?

fwiw re [9] and nomad w/ HashiCorp license changes, the referenced sbom https://github.com/rivet-gg/rivet/blob/main/docs/infrastruct...

lists nomad as apache 2.0, when its either MPL or BUSL depending on version.

also fwiw, your ci is failing, because your secrets detection ci hook is barfing on recently updated click house sha 256 in your nix file.

Wow, this is super cool. As a hobbyist game dev, I really hope to see bevy support soonish, as it has a dedicated community and is also Rust-based. Or rather, a rust crate for whatever rust-based engine wants to use it. Rust-based game engines show a lot of promise so I think the crate would be a worthwhile endeavor

We plan to at some point — being engine agnostic is one of our goals.

This is so cool! Out of curiosity, could you walk through your decisionmaking around using Nix and how it fits in with Rivet?

> Apache Traffic Server is under-appreciated as a large file cache. Used as an edge Docker pull-through cache to improve cold starts & as a CDN cache to lower our S3 bill.

I would love to know more!

Very cool, I have an abandoned web game side project and may try to run it on this!

One question, am I correct in understanding you bake in the assumption that 1 lobby = 1 container? I can see how that’d be desirable for the nice properties it gives you architecturally but if your game isn’t very demanding (mine’s turn based) might that end up stranding a lot of resources?

So glad to see open source. Bold step.

Good stuff & great that you've gone ahead and open sourced it. Been following for a while and you seem to be steadily shipping and improving the product. Keep it up.

wow, this is really cool, i remember krunker going viral when it was launched and how smooth it was. definitely one of the top IO games. so really cool that you're opensourcing this

Plausible as a log store confuses me. We don't aggregate logs. Why use a columnar database for storing logs?

I would be interested in knowing how this compares to Epic Online Services (EOS). We use EOS for connecting P2P lobbies and doing server browsing. When using unreal that service is basically free. Managing game servers is of course not part of the EOS package but it's a big question if your game really needs dedicated servers or could do with p2p lobbies. Running your own servers is quite expensive after all. Also console support is also always a concern for us so whether they check all the boxes so we can use them inside the console ecosystem would be very important to know.

Nice to see some more good press for Nomad. IMHO it's superior to Kubernetes for most use cases.

Open sourced - nice!

given nomad is under business license now, can Hashi come after you?

they address this in the original post:

>> Are you affected by the recent Nomad BSL relicensing [8]?

> Maybe, see [9].

And then

> [9]: https://news.ycombinator.com/item?id=37084825

so, they won't receive security updates after Dec.

Nice to see this open source :)