My name is Matheus and I am the technical director at Game Smithing. Hopefully you have read our other post about the plans we have for Rogue. If not, you can go check it out here.

It will still be a while longer until we have something like a trailer or a demo to show you, however, something we’ve learned while developing Soulstone Survivors is that there’s a lot of value in communicating and showcasing our work with the community.

As we have resumed development on Rogue Soulstone, we’ve decided to start this devlog series to document the development and progress of the game. Spoiler alert, since we are currently working on the core architecture for the project, this post is quite a bit technical, but if you are curious about how the sausage is made, read on! As the game progresses we will showcase levels, characters, mechanics and so on, but for now… netcode!

To start, let’s take it back to the beginning….

CCTV footage of me after 2 weeks

As soon as we’ve decided Rogue would support a multiplayer experience, we knew we had some digging to do. Although some of our team, including myself, has had some professional experience developing multiplayer games in the past, we are far from experts. Recognizing that, I knew I’d have to do some deep research into networking in general as well as game netcode.

I went on a couple of weeks of reading hundreds of pages of documents, specifications, blogs and forums, as well as watching dozens of hours of video on topics ranging from internet protocols (IP, UDP/TCP), services such as NAT (and how to traverse it), SSL/TLS and so on.

The idea here was just to get the lay of the land on general networking concepts such that I would have a strong foundation to build upon. A lot of it was purely theorical, but quite a bit was practical information that I have already put to use when developing the foundation for the game — such as NAT traversal. Once I felt comfortable with the more low-level general networking concepts, I moved on to game-specific netcode.

I wanted to learn about the whole spectrum of techniques and architectures games currently use and have used in the past. There’s a lot of content out there, but surprisingly enough, a lot of it is from the late 90s to early 00s. There seems to be fewer resources on more recent games… maybe the technology has stabilized by now such that we are no longer struggling with these issues as much… anyway, I’ve learned a lot from this process too and it has informed some of the decisions I’ve made in the general architecture of the game code.

After a few weeks of purely researching the topic, I’ve decided it was about time I wrote some code. I was quite confident that I had enough of an idea on how to go about it that I wouldn’t need to throw away any code… ah, how naive I was… I first started by implementing the core, lower-level systems: data serialization and a reliable UDP socket layer.

Data serialization is a simple concept: you have a bunch of separate items you want to send to the other player (the character position, its health, its weapon) and you need to put that into a series of bytes to send over the network. One of my initial objectives was to avoid having to write specific code to serialize a particular piece of data; I wanted the system to “just work”. That was a mistake. The system turned out super complicated and very slow. When compared to a straight-forward way of serializing data, it was 10x slower. So I scrapped that and wrote a much simpler system that, although it requires us to write specific code to serialize something, it is so simple to use that it really isn’t a big deal. We now have a pretty simple, but very efficient system (in both runtime speed and data compression) to serialize pretty much any data we want. I wrote a lot of automated tests for it too, so I’m pretty confident it’s robust enough to use in production.

I then moved on to writing a custom “reliable UDP protocol”. The UDP protocol is pretty barebones, which is what makes it a great fit for real-time multiplayer, but it doesn’t have one important feature: a guarantee that your message will reach its destination. This is actually fine in most cases — for example, if I’m sending the player position 30 times a second, it doesn’t matter if one of them gets lost because we will send a new one just a few milliseconds later. However, if you click a button on your game, and this message gets lost in the internet, the friend you are playing with might never know that the button was actually pressed. In short, there are many cases in which we’d like to ensure that a message will eventually arrive, such as sending “I equipped this item” or “I pressed this button”.

Here too, my first implementation was thrown out. It was way too complicated, even though it worked. Sometimes you have to just write something to realised how much simpler it could be… maybe I should rewrite this blog post too… hmmm… anyway. My second implementation works even better while being a whole lot simpler. We can now send messages to any known machine, reliably or unreliably, and the system will batch several messages together as long as they fit in 1 internet packet or, if the message is too large, it will split it into multiple packets, making for a very effective use of bandwidth and processing power.

Up until this point, between the entire socket and serialization code, I must’ve written about 10000 lines of code… and after re-evaluating and simplifying, while at the same time making it much easier to work with, we are now at about ~3100 lines of code for the 2 systems. These low-level systems are the basis in which the game will be built upon, so the time spent making sure these systems are as simple and easy to use as possible, will pay tons of dividends across the development of the full game.

gameplay automated tests summed up

Automated tests are not very common in indie game development, and for good reason. It’s extremely hard to write automated tests for gameplay code, and because it’s so difficult, it’s usually simpler and more effective to test the game manually. Also, you are typically playing the game every day, so you’ll most likely run into most issues as soon as they pop up.

Unlike gameplay code, low-level systems such as sockets or serialization libraries are much easier to write automated tests for, but much harder to test manually. These automated tests allows you to make changes and have a high degree of confidence that you didn’t break anything. And believe me, the last thing you want is to have a bug in the game and wonder if the issue is with the gameplay code or the networking code; you really need to be confident that your low-level systems are working because everything is built upon them — if they don’t work, nothing will work.

To add insult to injury, we are talking about network code here, which by its very nature is unreliable. So if you run into something unexpected, you just won’t know if it happens because a packet got lost on the internet or if it’s because your socket implementation is broken. Of you if you see a 5 instead of a 10 for an enemy health, you won’t know if it’s because the damage calculation is broken or your serialization system is broken.

So, tl;dr: if you ever write a low-level system such as these, please save yourself a ton of time and write a lot of automated tests.

simple match-making process we use daily

Networking is like sending letters… if you don’t know the destination address, you’ll never be able to communicate. This is where a match-making server comes in. A match-making server is just a computer on the internet with a pre-specified known address. The game will just ship with a list of possible addresses for these servers. This server is responsible for matching players together, which just means something like: “Hey Paul, Jon lives on address XX. Hey Jon, Paul lives on address YY” — now Jon and Paul know each other’s addresses and can send letters back and forth.

I wrote a very simple match-making server in which a player can create a lobby and other players can join that lobby. The server will share the addresses of everyone in the lobby so they can communicate directly with one another.

Here’s how it looks like in the game:

After getting two computers to talk to each other (which you would think would be easier in this day and age), it was time to come up with a gameplay architecture that would be suitable for a multiplayer game. In here, “architecture” just means starting a project from scratch and deciding how the code-base will look like. I have “architectured” quite a few games in the past, but never a multiplayer one.

There’s ways to architecture the code such that things happen “automatically”. The idea is that you create 1 complicated system which handles a lot of the work behind the scenes. Although that sounds amazing, it never really works out like that. What often happens is that you have to work around the system to do anything the system wasn’t designed for.

This is why so many games (and software in general) end up evolving into what’s known as spaghetti code. You have this massive system that orchestrates 80% of your game… but the other 20%, the parts that are probably responsible for the fun stuff in the game, the stuff that is different, outside of the norm… ends up being a bit hacky, stitched together around the limitations you create for yourself when you made these major decisions at the start of the project. We don’t want to go down that rabbit hole.

We’ve chosen flexibility. We believe the challenges this game will bring are quite complex and we want to have the flexibility to tackle these challenges in any way we want. The trade-off we are making is having to pay a higher upfront cost whenever we need to develop a new system. This is not particularly great for prototyping, but in the long run we believe it will pay off.

Let me give you an example to make things concrete:

We aren’t going to have a system that automatically creates “networked objects”. In many networking solutions, particularly the ones used in Unity games, if you create an object on your local computer (say you throw an item on the ground), the networking system will automatically detect this new object, and inform all of the players that this object was created behind the curtains, so that they will also create a copy of this object in the same position as you did in your machine. It just works “magically”, from the point of view of your game code, you just believe it is there and that it will work and trust the system with your life.

In our system, however, the code that creates the object would also have to purposely send a message such as “I just created this object” to all other players. If there are multiple places in which objects are created, all of those places would need to send that message as well.

This may sound a bit dumb and a lot harder to develop, after all, who doesn’t like “MAGIC”? But in reality what this gives us is full control of what is happening in the game and although it is more code, it’s not particularly complicated code, so it doesn’t end up costing much. So we pay a small premium when developing a new system, but we gain complete control of what and how the system will work, which will be invaluable when creating sophisticated systems that are particularly difficult to develop.

Basically: it will be easier to develop complicated systems, but harder to develop simpler systems. That’s a trade-off we are willing to make.

Here’s how the ““game”” looked like a couple of weeks ago:

this footage was captured with a 300ms ping

This is obviously very crude and early, designed purely to test out the concepts explained above, such as how we create projectiles and sync them over the network, how do we keep player position consistent across multiple computers, how can we make the movement smooth and not janky due to network variability, etc, so it might look ugly, but it proves the foundations upon which we can start to build the real game.

That’s it for our first devlog entry. I tried my best to keep it as simple as possible, but if you folks want to get into the weeds of the technical details (or the opposite), do let us know in the comments! For now we don’t have a specific timeline for these devlogs, as work in these initial stages varies a lot, where one week there is something super cool to show and then you go a couple of months working purely on core systems, but if there is anything you would like to see do let us know as well!

If you have any questions or comments, please post them below. I’ll try my best to reply to them here or in a future devlog. I’ve anticipated a question which I can answer now:

There are some great off-the-shelf solutions out there, so to understand our decision we need to talk about the concept of “allowable tolerances”. It is pretty simple: how precise do you need to be when doing something? The stopwatch used to track how long your egg has been boiling on the stove is a lot less precise than the stopwatch used to track a potential world record at the Olympics. Even though they both track time, they are not interchangeable.

For a simpler project, using an off-the-shelf solution is more than adequate because you don’t need much control over how things are done. As long as it gets done, it’s all good. When you have a sophisticated project with a lot of moving pieces and you want everything to feel just right, you need control. And when you need control, an off-the-shelf solution is hardly the best option, for a few reasons:

Because it’s a generic tool that was not build in the context of your game, your game has to adapt to its context. There’s usually the “right” way to use it and doing it any other way is like swimming against the tide. Even the ones that give you control, if you want to exercise that control, you’ll end up doing everything yourself — after all, that’s why you needed the control in the first place. But now you are doing it yourself, but within their framework, which may or may not be conducive to what you are trying to build, so you get the worse of both worlds.

Game development is very difficult and iteration is key. During the development of Soulstone Survivors, we created dozens of tools to help us develop the game; anything from automatically downloading localization files, to build systems, to scene management, shader conversions and many more.

Using an off-the-shelf solution means you are stuck with their tooling. It may be that you need to press a button every time you change some code so the system knows about your changes, or it may be that if this process is automatic, it takes too long such that now you have to wait 10 seconds longer every time you change some code.

We could build tools on top of the framework to alleviate the problem, but that is unlikely to result in something optimal. And we are aiming for optimal here.

I don’t necessarily mean financial cost, although that’s also a factor in these decisions. I mean the ongoing cost of using and developing atop of a third-party framework vs building our own “framework”.

This game will likely take a couple of years to develop and, hopefully, will be maintained for many more years after that. If it takes us a couple of months of 1 person to build a networking solution, that’s a small fraction of the development time. So using an off-the-shelf tool would save us a fraction of the development time, but would be an ongoing cost during development for however many years the game is still alive. The cost here comes in the form of working around the framework whenever we need to do something that was not particularly suitable for it, or having to use its sub-optimal tooling as mentioned above, or the financial cost associated with it. It’s all friction points that we’d like to eliminate for a long and large project such Rogue.

If we can, we plan to continue making games for the rest of our lives, and we want co-op and multiplayer to be an important part of our future projects, and building our own tech will allow us to build expertise in those areas. If it costs us a few months to build that expertise now, it’s a price we are willing to pay, as it will pay dividends for as long as the company is still going, which will hopefully will be many decades.

Given these factors, we are extremely confident in our decision to build our own network tech.