I joined Container Solutions a year ago as a Senior Software Engineer (my work Cake Day was April 1st). I focus on Cloud Native Applications: micro-service oriented architectures, containerization and Infrastructure Automation. Lately I have been developing and deploying Prometheus exporters, plus Grafana dashboards.
In my teenage years, I pursued a computer science education as an excuse to spend time playing video games on a PC. Video games were my entry point into the fascinating world of programming. I learned basic programming skills in high school where my final project was a two player, graphical Tic Tac Toe video game!
Some time ago, I was listening to the Life Well Wasted podcast Work episode. It told the surreal story of how a developer went from a blue collar job to working at Bioware, the studio behind hugely popular games like Dragon Age and the Mass Effect series. As far as I knew, video game programming requires a lot of expertise. I got fascinated by his step-by-step learning process. He got started by developing a series of small simple video games. I had an epiphany! Why not invest some time learning game programming as well?
I decided to learn video game programming as a hobby, so I bought some books and Udemy courses to get started. (I admit I have been spending a lot more time enjoying just playing video games but I want to pick up some steam again soon).
Besides design and programming, there is an operational side in video games I want to explore. At Container Solutions, as a learning organisation, we value research and development. We have an Innovation Office to encourage every employee to spend time experimenting and learning on a project of their own choosing. My personal Innovation Project is researching Cloud Native Applications in the video game industry. I am interested in learning about development tools, CI/CD pipelines and containerisation. (One outcome of my research is the publishing of blog posts, so keep an eye on this blog!)
As part of my learning efforts I joined developer communities, Amsterdam Indie Game Developers and Unreal Engine Users NL. Some colleagues from these meetups mentioned they were attending UnrealFest 2019. Unreal Engine is a suite of integrated tools for game developers to design and build games. It is also utilised in academia for developing scientific and educational simulations.
I decided to attend UnrealFest since I have never visited Prague or attended a video game conference. I wanted to get an insight of the operations side of the industry, learn some game development. Meet people. Have fun!
Lessons learned at UnrealFest
I'm inexperienced with Unreal Engine and expected most of the talks to be quite advanced. So I handpicked some talks where I could learn as much as possible about development. "100 Unreal Engine 4 Tips and Tricks" by the @epicgames evangelist team was a fun talk. 
I memorised one tip in particular, disabling "ticks" by default in Blueprints. Enabling them then becomes an intentional action by the programmers.Blueprints are Unreal Engines visual scripting language. They are an alternative to pure source code for rendering and physics. Blueprints are also utilised for executing continuous functions on every frame. A use case could be to check if a player interacted with a certain object, for example a door. This approach is intensive as more processing happens at each frame render time. It's optimal to react to events, a common pattern in software development.
As a side node, so far I'm quite interested in the Unreal Engine support for visual scripting. In general programming I have come across visual scripting editors in learnings environments. Scratch seems popular, Yahoo Pipes was big amongst my tech friends. I'm not aware of these tools being mainstream in developer workflows. Although, there are visual editors for BPM and rules engines.
I found Blueprints particularly interesting because lately I have been implementing Prometheus exporters. Exporters are simple web applications that query and filter some specific AWS metrics. They then export metrics which Prometheus can ingest. The tech industry is moving in the direction of micro-services and Functions As A Service. Smaller, modular applications. Easier to represent graphically. Is a visual scripting tool as powerful as Blueprints effective for general programming ? Food for thought!
Another UnrealFest session I attended was Jessica Baker’s "Automated Testing at Scale in Sea of Thieves" talk. I wanted to understand how game developers approach unit testing. Unreal Engine offers support for local unit test development and execution. My main takeaway from Jessica’s talk was that "Sea of Thieves" game team took full advantage of automated testing. In previous titles, and we've all been there, there was only manual testing. Automated testing reduced "bug count over time" in Sea Of Thieves, as expected.
Running video game tests on CI is a practice, as expected. I talked with people from different video game studios and Teamcity seems to be the CI system of choice.Some other teams I met utilise Jenkins, though. Licensing costs might influence some of the choices here, particularly for smaller studios ?
I have never used Teamcity myself but I learned something new. It features pre-tested delayed commits. Pre-commits are only merged into a target branch if CI job passes. It is useful for preventing committing defective code into a build! The common pattern at projects i contribute to is one feature branch per developer. Pre-commits are useful for teams collaborating on the same branches. Many teams do Trunk Based Development, where developers all work on a single source code branch.
A Few Other Awesome Things
Between conference talks I mingled with other attendees. Met a video editor from Slovenia learning Unreal Engine as a hobby too. Also met fellow countrymen video game programmers from a small Portuguese video game studio. I am surprised most attendants were quite social and proactive in networking. It was very busy in the corridors. There were many interesting people to talk to!
- Valentin Galea gave a talk about how Splash Damage approaches development. His slides are available here. It gave me an insight into the best practices of the development process of video games. I also had a glimpse of the challenges. "Development Process" is another way of saying "Collaborating With Other People". Source version control of game projects is a core ongoing issue ?
- Unreal Engine project repositories can consist of source code and Blueprints. Source code files are text files but Blueprints are a binary format. Diff merging binary files is, impractical to say the least. Good file locking a "must have" feature of the utilised source control systems. Even with file locking, issues can arise if same files are being modified by other team members.
- Ben Marsh (Epic Games) talked about "Workflow on Fortnite". He noted that Unreal Editor features a notification system for modifications across branches.
- Perforce appears to have widespread utilisation in the industry. Perforce Streams get mentioned a few times, both at conference and casual talks. At a previous meetup I attended, some was utilising a file locking plugin for Git LFS. He mentioned it was possible for developers to circumvent the file locking mechanism. Might not be resilient enough for larger studios ?
Cloud Native Connections
There is considerable Cloud Native alignment between game development and traditional projects. Games are multi-year projects. Teams must scale to the hundreds for larger projects. There are technical challenges with source control project file count and size. Only heard of Source Control Contention when i started taking a look at these types of projects. I have not heard about build times in optimised pipelines taking less than 10/15 minutes. Strong project layout can have a large impact on team workflow. My understanding is some studios take use a outsourcing often. This and project characteristics imply splitting larger projects into sub-projects. Then there are external teams, for which security must also be a consideration.
These challenges must have a direct impact when designing CI/CD pipelines. There is a lot of value for game projects in Cloud Native best practices. Cloud upscaling and downscaling, resiliency. Automation and faster feedback loops.
Final thoughts
I wanted more insights into the usage of Container Orchestration in build pipelines. There were no references to it in the talks I attended. I will watch conference videos on Epic Games Youtube channel when they're published.
Had a brief talk with a Build Tools Engineer of a popular studio. He mentioned they utilise Windows Containers (up to 5GB in size). He also noted that utilising Linux containers in Unreal projects is challenging. Linux containers are an area that needs more research. I found a lot of information about them on Adam Rehn's website and I intend to explore Unreal Engine Linux containers in follow up blog posts.
I had a blast at UnrealFest! Met studios that developed video games I played together with my friends in the past -- there were some memory lane trips for sure. Also had glimpses of the future with good things on the horizon in this exciting area. I'm looking forward to next year's edition!
A big Thank You to Michelle Gienow, our content lead editor, for helping me edit the blog post.
A special thanks as well to Jessica Baker and Valentin Galea for their feedback on the early drafts of this blog post. Their studios are hiring here and here ;-)
Looking for a new challenge? We're hiring!
Previous article
