Quick Progress Reports - Engineering

Battlescape
4

January 2016 Summary

This last month, our focus has been on launching the game prototype to our Dev-access backers.

To that end, we have been working on our company infrastructure, in particular the web servers and database systems. This is not something we can rush, because it would open the door to serious problems down the line. @INovaeKeith is more eloquent than me on that subject:

“While it isn’t particularly sexy everything that comes next is dependent on account management, authorization, deployment, and patching. None of this code existed outside of our website prior to this month. On top of that since this code is critical it has to be done right. If we screw something up here we’re going to be getting bogged down by support requests due to issues with payments and being able to play the game someone just purchased. Even worse we could have a security breach that leaks user information or allows their account to be compromised. The amount of testing involved in making sure this code is solid is significant.”

Here’s a basic list of things we have been working on in January:

  • server authentication (players will be able to host their own dedicated servers in the future, it has to be secure)
  • client authentication
  • access rights verification (who has access to the prototype/game)
  • security/encryption of requests and tokens
  • automatic bug and crash reporting
  • sessions logging (both for client and server)
  • kickstarter database importing
  • indiegogo database polling and importing (because people can keep pledging)
  • rewards claiming / matching databases (lots of people used a different email to pledge)
  • versionning and patching ( full or incremental, nobody wants to download a 4 GB patch for every single minor update )
  • stats, error logs and backups
  • available servers list displayed in the launcher
  • connection limitations / logging queue
  • launcher, account management
  • performance and ability to scaling up
  • and probably a lot of other things I’m forgetting

The features are highlighted in bold are the most critical ones. I’ll quickly review them:

  • client authentication is critical because we need to verify the identity of the client, that he does indeed have Dev-access rights to the game prototype, and that hasn’t been banned for whatever reason.

  • security/encryption is critical because as you know, hackers usually come pretty early into play and we do not want to compromise on security. We’ve already done that mistake in the past, never again. In addition, a lot of the early code will probably have serious vulnerabilities, so we need to be able to log in and track incorrect requests to verify that somebody isn’t trying to reverse-engineer our code.

  • KS/IGG database importing: both databases use a different format and pledge levels, so we need to merge everything into our own database. IGG is currently causing us trouble because it is an on-going campaign. Which means that people can continue pledging even after the release of the game prototype. Obviously, people who are going to pledge on IndieGoGo will be expecting to be able to download and play the game within minutes, which means we need a mechanism that will “poll” for any new pledge and inject/merge that stuff into our database in real-time.

  • I should also mention that we needed a solution for those people who pledged with a different e-mail address than the account they registered on our forums/website in the past. Keith has been working on that in the past days and added the ability to link multiple e-mail addresses to a single account.

  • versionning and patching is critical because we need to ensure that everybody has the same, latest version before they try to join a game server. We are expecting frequent patches both for the client and server, so the version numbers will change all the time. In addition we wanted patching to be incremental and automatic, to avoid asking people for downloading and installing a 4 GB patch every day. That would get tiring very quickly. It is IMO one of the most frustrating things with Star Citizen’s launcher, and the reason why I believe a lot of people are not testing the game as often as they wish. A 30 GB patch is not something to be taken lightly, especially for people with not-ultra-fast connections. Fortunately we aren’t anticipating 30 GB patches any time soon in Battlescape, but even a couple GBs patches can become quickly annoying if they aren’t done incrementally.

At the moment, from this entire list, many of these systems are either done, or well advanced. The part that probably still needs the most work is patching. Patches are already automatically downloaded and extracted in the launcher, but the patching procedure itself ( copying the files, registering them, etc… ) still requires a bit of work. One of the main challenges we still need to address are third-party dependencies, such as the CRT and other libs we are using, which themselves have various dependencies.

Once all these systems are done, we still need to do a round of testing. This should go pretty fast, but at the moment I am incapable of saying if this will take a day, or two, or a week. We are using Azure to host our web environment and virtual machines, and the entire thing has to work with different machines all collaborating together. The really good part about Azure is that scaling up is as simple as pressing a button. If we see a flood of new players, we can easily add new game servers, increase the performance of the servers or raise the size of the database. This means that all the work that we’ve been doing on the infrastructure so far is expected to be a robust foundation for the future, and we won’t have to redo it by the final release. That is, excluding the website. The current website is a bit old and inflexible, which is the reason why posting news on it isn’t too easy. There is no content management system, so any news that we want to post on the front-page requires to upload the entire website to the Azure portal, entirely, every single time. Not very practical.

34 Likes
5

Monday, 1st Feb 2016

Yesterday the 1st was the day two of our artists ( Kristian and Jan ) started to work on full time on Battlescape. This was the perfect timing to move our entire management infrastructure to Slack and Jixee. For real-time communication and meetings we were using Hipchat before, but recently it seemed to have become pretty unresponsive ( my Hipchat was randomly freezing, sometimes for up to a minute or two ) and Keith keeps saying that their mobile app isn’t too good. Last week-end, we purchased and moved on to Slack, and I must say it’s ( no pun intended… err, actually, pun intended… ) pretty Slick. The interface is clean and it looks responsive ( I hope it stays that way ), so we are pretty happy so far.

As for Jixee, we will use it as our task management system. We aren’t too familiar with it but we heard that it was created by a company that is developer-friendly so it is more developer-focused than other solutions. So far my experience with it has been very positive, everything has been pretty straightforward and I’m looking forward to using it daily. We were using an open-source tracker solution ( Mantis ) before, but it obviously wasn’t as good as Jixee. The only thing that I dislike so far with Jixee is the lack of a task category. They have this concept of a Project, and each Project contains a certain number of Boards, which you can think of as categories. But even so, being able to assign a sub-category to a given task would be very helpful. Also, I would have liked to see the task creator name to appear when viewing a task’s form.

While I was getting used to Jixee, I had a look at their web API interface and tried to figure out how to link it to our own infrastructure. We want players to be able to report bugs in the game launcher, and those will automatically be sent and authenticated to our servers. Then the reports will be forwarded to Jixee and placed into an automatic queue, awaiting reviewing and sorting. So I wrote a little program to experiment and submit a task through Jixee’s web API, and it didn’t work.

I decided to use Fiddler ( a software that lets you compose and send HTTP messages manually ) to create an HTTP request, attaching a JSON object containing the task data. Jixee’s servers kept returning an HTTP 200 OK response, without any error, but I couldn’t see the task appear in the live project. Supposedly the API was working in real-time, so I was pretty sure I was doing something wrong.

After two hours of trying everything I could think of ( trying various strings for the project name, tasks, requestor etc… ) I was about to give up and contact their support, when I had a revelation. When I was composing my HTTP message in Fiddler I was making a pretty big assumption that the JSON object ( whose syntax, at this point, I was pretty sure was correct ) was correctly, automatically deserialized by Jixee’s server as a JSON object. In fact, it wasn’t. I was missing a simple content-type header to specify that the incoming data was in json format. Surely, after I fixed it, everything started to work perfectly and the launcher’s bug report system became functional.

All of this reminds me that my desktop, which is my main work computer, is still running on Windows 7. I’m planning to upgrade to Windows 10 in the coming months, but I haven’t decided when just yet. Probably before the free upgrade offer from Microsoft expires in summer. The only thing worrying me is that Keith has already moved on to Windows 10, which means that if I move on to it too, testing and developping on Windows 7 will become more infrequent. But I don’t think it will cause any problem, as we haven’t seen any compatibility issue between 7 and 10 so far.

Tomorrow, I will continue my work on game sessions authentication, which I am hoping will be functional soon.

40 Likes
6

Wednesday, 3 Feb 2016

Today, I read this analysis about video game projects in 2015 that were successfully funded. As expected, Infinity: Battlescape made it to the top 20. More than $40M were raised in total, not bad at all, and that’s only for Kickstarter. I wonder how the numbers look like on other platforms like IndieGoGo. Probably lower.

At some point in the future, I want to write a post-morterm about our Kickstarter campaign. What went right, what went as expected, and where we got caught on guard. It might take a while to write though so I won’t do that in this progress report.

Yesterday I pretty much finished my work on authentication. Everything is now functional: you can run the launcher, log in to the game ( speaking of which, we had to support external login through Google or Facebook too ), launch the client, which itself logs in to the game server, which itself verifies your identity and access rights in to our database and replies with a “yay!” or “nay!” to the server.

I say “pretty much” because there’s still a bunch of things to do before it can be called “complete”. I’m only testing it with a single test user in the database, as I do not have the production DB on my local drive. I will have to run some performance tests to verify that even if we get hundreds of login requests at the same time, the authentication server behaves well. This is quite important, especially in the event the server crashes as it is likely to cause a flood of players trying to log back at the same time.

I spent a while fighting against wrong character codes in URLs. As it turns out, the “+” symbol gets replaced by a whitespace if you forget to URL-encode some of your parameters. Sounds pretty obvious to any well-experience web developper, but I am not ( still learning all that stuff ) therefore I’m making basic mistakes all along the way.

I used to do a bit of web development a few years ago in the days of Linux / apache / php / MySQL. Remember our old site ? Yeah, the one that got hacked. That was all php + MySQL. However nowadays we’re using Microsoft technologies ( ASP.NET MVC more specifically ) which is a very different beast. Prior to the Kickstater campaign I had absolutely zero experience with that, so in december I had to ramp up pretty quickly on how it all works. And I must say, contrary to my expectations, it has been a very fun learning experience. It allowed me to improve my general C# experience too.

Speaking of C#, I think I’ve fallen in love with the async… await mechanism. If you’re not a programmer just skip to the next paragraph as I’ll get technical. C# makes it very easy to do asynchronous programming, and once you get the hang of it ( and learn a few tricks along the way, nothing is perfect ) it becomes a real pleasure to work with. I’d even go as far to say that there’s a form of beauty in seing the execution flow of your program. C++ has introduced a similar pattern with future & promises, but we aren’t actively using them just yet in the engine as that will need quite a bit of refactoring. Keith will probably work on that in the upcoming months.

The forums seem to go down fairly frequently nowadays. Once a day, the forums stops working and return HTTP 500 Internal server errors. The main problem is that the accounts database needs access to Discourse before it can register a new account for the website, which means that during the time the forums are down, nobody can register a new account, even on the main site. These problems started to appear a few weeks ago when we upgraded the Discourse version, although it is possible that the problem is unrelated. We’re still investigating the issue and apologize for any inconvenience. For now you can simply try again when the forums are back up.

Today I will continue to work on access rights and display the available servers list in the launcher, so that a player can select to which server he wants to connect to. For the prototype launch we’ll probably just have one, but at least the system will be ready to handle any number of servers in the future in case we need to quickly scale up. I also have more work to do on logging sessions, so that we can have a track of who logs in on which server and for how long.

37 Likes
7

Friday, 5 Feb 2016

These past 2 days haven’t been the most productive ones ever, at least in amounts of code.

Most of the work has gone into doing database updates and slight redesigns. The access rights code that I wrote was checking if a transaction/pledge existed in the database ( of course it was not just that, it also needed to retrieve the pledge level and verify the user identity ) but we realized that we wanted the ability to grant access levels to the game even if a transaction didn’t exist. Like, if you’re one of our developpers. Yup, we had overlooked our own ability to access the game. This required to rewrite the SQL queries a few times.

I also started to run some performance tests to check how the queries scale up with a more complex database. To that end I wrote a script that generates as many random users and pledges as I want. Everything looked good until I reached about ~12K users and queries, at which point SQL manager studio started to reject my queries due to an “out of memory” error. Well, nope. I was nowhere near exhausting my computer’s memory ( checking at the task manager ) so I came to the conclusion that there is a software limit somewhere that prevents me from going higher. I’ll have to investigate to see how to get around that limit. Nowadays, my new best friend is called Google.

I’m having a bunch of IRL issues due to humidity infiltrations in my appartment. Seeing droplets of water fall from my kitchen’s cupboard (which doesn’t contain any liquids) is a bit worrying. I’m suspecting a small break in the wall behind the cupboard or in the ceiling. In any case, the apprt’s owner, building managers and plumber are having a lot of fun distracting me with calls and setting up appointments. I wish they would stop to bully me and leave me alone. I have a game to make, damn it :wink:

33 Likes
8

Saturday, 6 Feb 2016

Plumber came yesterday and was able to determine the origin of the leaks: my top neighboor’s washing machine. I’m glad it didn’t take too long, and most of all, that we’re not going to have to destroy a wall to figure out if a pipe was broken. Going ahead, all the walls of my appartment are going to have to be refreshed/repainted, and since my living room is where I work ( and in terms of area it’s not super big either ), this is going to be problematic. Anyway we’ll cross that bridge later.

I haven’t been able to figure out why SQL Server management studio refuses to execute my stress-test queries. I’m still hitting this weird 12K limit with an “out of memory” error, when I clearly still have memory left. I’m thinking that it tries to execute the queries asynchronously, so each query might spawn a thread and consume a lot of memory. to execute itself. To test that theory, I wrote a C# script that connects to the DB and submits the queries from a file instead, and guess what ? It worked instantly. One bad point for Mgt. Studio.

I’ve tested our web api code ( the one handling connection requests, logins, access rights etc… ) with a variable amount of users and pledges. I was anxious to see the result since we had never tested the code with more than a single user so far. So I started to fill up the DB with 5000 users, 50000 and then 500000. The queries became slower to execute, but not by an order of magnitude, more like a dozen percents each time. Which is really not all that bad. In the end, working with a half-million users/pledges DB is only 50% slower than working with a single user DB. Considering SQL Server is in production all over the world, from bigger companies and much bigger databases, I feel silly for even worrying in the first place, but it’s good at least to have a practical confirmation that it shouldn’t be an issue. And that was for a single test machine ( my desktop ), while in actual production in the Azure cloud we can scale up as much as needed.

While I was in the mood of working on scripts, I wrote one to port our old tasks/bugs database from Mantis to Jixee. It only took a bit more than an hour, and we now have 160 “old” tasks/bugs in our new database, from which 50% were closed already. Unfortunately it wasn’t possible to map all the data perfectly between the old and new formats, so we’re going to have to review and sort out all these old tasks at some point. Nothing immediately urgent though.

I’ve started to investigate the SSPI library and the TSL/SSL protocols to secure our connections between the game client and server. We want to avoid man-in-the-middle attacks which could impersonate a user and mess up with the logins/stats and feed false information to our systems. While I was working on it, I became curious about the RSA algorithm, which is used to generate public and private keys and encrypt all the data. There is some seriously fascinating stuff here. If you ever wondered how a client and a server can establish an encrypted session, completely secure even if somebody is listening to their exchange, I have one answer for you: the magic of mathematics :smile:

32 Likes
9

Sunday, 7 Feb 2016

Yesterday was pretty uneventful. Lots of little fixes to the code to make it easier to use in the future. For instance, some settings are now pulled off the registry instead of being hard-coded. It is possible to force unsecure connections ( disabled by default in production of course, but for testing during development it’s quite handy ). I also added an interface to the game server to handle multiple types of authenticators and introduced a “private” authenticator for the future when people will be hosting their own private servers. The difference being, the “public” authenticator registers itself on our website and checks for hosting access rights while the private does not.

Looking ahead, there are still a few things I need to finish by next week:

  • SSPI + TSL to encrypt authentication between the game client and server.
  • displaying the list of active servers in the launcher
  • handling the maximum concurrent players connections limit
  • some visual redesign and testing of the launcher

The SSPI thingy is currently the most “unknown” one, I have no idea if I can make it work in a day or if it’ll need multiple days for that. The rest are, in comparison, not so big, so they should go smoothly.

After that I will probably move on to resuming work on the actual client/server, while Keith finishes the installer.

24 Likes
10

Tuesday, 9 Feb 2016

The SSPI / Schannel interface is now working and integrated into the engine. It was a nightmare to make it work. All examples found on the net are convoluted and mixed up with sockets code which I didn’t need and I had to figure out how to handle the certificates on the server side. I learnt a lot about certificates, CAs and stuff in the process so it was a great learning exercice, but it took many hours which as you know we’re short on at this time. One area which is still quite blurry for me atm are certificate chains, but I’m not going to spend more time on that so I’ll come back to it later if needed.

The next step is to use that interface in our own sockets code to encrypt the network communications between the client and server. SSPI was mostly research/experimentation ( in amounts of code, it wasn’t all that big; I maybe added 1500 lines of code to the engine for it ). I don’t think it will be too hard, but who knows… hopefully by tomorrow encryption will be finished and running and I can move on to the remaining tasks.

21 Likes
11

Wednesday, 10 Feb 2016

SSPI/TLS Encryption is up and running on our sockets code, all there is left to do is some cleanup and a bit more testing…

… or not. Every time I finish writing a system in our engine, I run a bunch of stress tests to verify that we aren’t leaking any memory, and that we’re not running into surprise performance issues. The good news is that there are no leaks. The bad news is that performance isn’t exactly at the level I was expecting…

To be more precise, it takes around 10 milliseconds to finish the TLS handshake. Now, 10 ms doesn’t sound all that bad but you have to remember that a lot of people could be trying to establish a connection within a short time frame. Like when the server reboots and lots of people try to reconnect to it. The root issue seems to come from the Schannel API calls, which are synchronously blocking, but which I suspect are doing some waiting behind the scenes. CPU usage doesn’t seem to increase during these calls so it’s definitely not doing anything CPU-intensive. I wonder how long those calls could take to complete with a complete certificate chain. Could they be doing some web requests to the certificate authority server ? I definitely need to investigate that. Unfortunately, a quick Google search didn’t give any meaningful answer on this subject.

In any case, for the prototype it’s not a blocker, as we’ll have a few hundred players at most with connections distributed in time. I’m going to wrap up all that code and move on to the rest of the tasks today. By the end of the week I’ll do a review of the list ( which I posted in the Jan 2016 summary ), and I should be ready to move on to updates that are more game specific.

26 Likes
12

Friday, 12 Feb 2016

I’ve been doing some good progress on the server infrastructure systems. Today and tomorrow I’m planning on doing some polish and finalization work on many systems, as well as more tests. Most of the systems are functionnal now. Let’s have a look at the list I posted a week and a half ago… I’ll use icons to show the status: a green checkmark when the task is done or almost done; a hammer when it’s still a WIP but has seen some significant progress; and a red checkmark when the task wasn’t started or is still far from completion:

  • :white_check_mark: server authentication (players will be able to host their own dedicated servers in the future, it has to be secure)
  • :white_check_mark: client authentication
  • :white_check_mark: access rights verification (who has access to the prototype/game)
  • :white_check_mark: security/encryption of requests and tokens
  • :hammer: automatic bug and :negative_squared_cross_mark:crash reporting
  • :white_check_mark: sessions logging (both for client and server)
  • :white_check_mark: kickstarter database importing
  • :hammer: indiegogo database polling and importing (because people can keep pledging)
  • :hammer: rewards claiming / matching databases (lots of people used a different email to pledge)
  • :hammer: versionning and patching ( full or incremental, nobody wants to download a 4 GB patch for every single minor update )
  • :hammer: stats, error logs and backups
  • :white_check_mark: available servers list displayed in the launcher
  • :white_check_mark: connection limitations / :negative_squared_cross_mark: logging queue
  • :hammer: launcher, account management
  • :hammer: performance and ability to scaling up

Let’s now have a quick review of the WIP tasks:

  • automatic bug or crash reports: the launcher has a tab where people can report an issue or a bug, and it is automatically submitted to our infrastructure. This works, but we need to add some security to it, to ensure people don’t abuse that functionality. Automatic crash reports hasn’t been done yet.
  • IGG database polling and importing: Keith did some work on that but AFAIK we still have the problem of being notified when a user just pledged, so that he can play asap.
  • rewards claiming: Keith did some work on it, but I’m not sure of its exact status, so I marked it as a WIP, but it’s possible it’s already complete.
  • versionning and patching: this is the #1 task that needs the most work, and is currently the less advanced. We’re still at least a week away from having it working.
  • stats, error logs etc…: we’re already logging various important events ( who logs in and at what time and from what IP ) to the Azure storage DB, we can do more but it’s now low priority.
  • logging queue: low priority, I’ll come back on this in a second…
  • launcher, account management: the launcher is visually a mess, it is functionnal but looks quite amateurish. Not sure how much time we can allocate to improving its look before we launch. I guess Dev-access backers wouldn’t really care. Account management is basic: it simply shows the list of pledges you’ve done so far, as well as some account details ( your name, your backer level etc… ). It’s read-only at the moment.
  • performance: will eternally be a WIP :wink:

Yesterday I worked on displaying the active servers list in the launcher. I spent more time on stupid WPF issues ( DataGrid doesn’t allow double-clicks, and tricks found on the net to solve it aren’t compatible with the metro theme we’re currently using, grrrr ) than on the web code. Anyways, the servers list is now functional. It displays for each server its name, description, player count + capacity and owner name. One thing it’s lacking is ping, but we can’t ping the server directly at the moment ( due to using our own custom network protocol ) so there’s no easy solution to displaying the server latency. We’ll probably address that much later.

A server has a maximum players capacity and the launcher will refuse to connect to that server if that capacity is reached. This is already working, however I’d like at some point to add a queue where you simply wait your turn before you connect to the server. I set the default capacity to 50 players, however this is only as a start. Once we verify that the server isn’t lagging / crashing, we’ll slowly increase this threshold if that’s needed. Considering that we have something like 300-400 dev-access backers and that only a fraction of these will be concurrently playing at a given time, it is possible that we won’t even need to increase it. I guess we’ll see.

36 Likes
13

Sunday, 14 Feb 2016

I’ve been doing the first “finalization” pass through all our infrastructure systems and started to implement some “details” that were missing, but that were still quite important.

For instance, logging disconnections and de-authentications. Like when a server is shutting down, all user sessions must be closed down. Clients are getting notified with a dialog box, instead of the server just stopping to respond.

I added more information to the user logs, which will be very useful for us in the future. We collect hardware stats such as the cpu type, amounts of ram, operating system and video card type. I’m also planning on adding framerate stats, such as min / average / max framerate per minute of play. This allows us to track some bugs, such as resource leaks ( in the current version of the prototype, some resources aren’t properly unallocated which leads to a framerate drop over time ) and verify that the client’s behavior is as expected. I’ll eventually add a “do not track” option too for those that do not wish to send this data.

I added a private server registration button to the launcher’s server browser. We aren’t enabling private servers just yet ( we don’t want to spread our user base thin that soon ) but at least the functionality will be ready, and it will be useful for us developers when we want to connect to an internal server and not to the public’s one.

31 Likes
14

Tuesday, 16 Feb 2016

I pretty much finished to wrap up my web infrastructure tasks. A lot of it was detail work, but still important. Such as logging in more data about user sessions ( resolution, quality level, game state when disconnecting, polycount, scene graph entities etc… ), reporting the min client version requirements versus the server’s ( and adding a check in the launcher ), passing the user’s account name as his call sign the first time he launches the game ( it defaulted to “unknown” before ), and most important… better error reporting / logging.

Keith has been doing some good progress on the installer / patcher, but as he already explained in the last weekly update, we’re not sure we’ll be able to finish it in time for the end of february. So we’re considering launching the game as a .zip package that dev-access backers would have to install manually. The problem with this approach is that it’s a gamble. If the game is relatively stable at launch, why not ? However if there are frequent crashes or major problems, we’re gonna ask people to download and re-install 2+ GBs every few hours. So the question becomes: what will be the most frustrating for the user ? Another delay if we can’t launch in time, or having to download and manually reinstall patches at a high rate for a week or two ( assuming there are major bugs we need to address asap ), until we can finish the automatic patching system ? The question remains open…

At some point this week, or early next week, we’re going to do a first deployment installation and test the new infrastructure. This means DB schemas changes, KS/IGG data imported ( the code is already ready to do so ) and being able to test the game launcher with the actual data, and not just test/simulation data. Those are pretty major changes on the back-end, however the front-end will stay the same. Ideally, you wouldn’t even notice any change other than a period of downtime.

26 Likes
15

Wednesday, 17 Feb 2016

Yay, it is Christmas ! Vulkan is out ! Woo hooo… better late than never :wink:

I haven’t spent too much time on it yet, only an hour or two browsing the API docs, samples, and simply getting a “feel” for what this new API looks like. Here are my initial thoughts ( which may very well be wrong, you know, those are just first impressions ).

  • The concepts look pretty much identical to DX12’s. The total lack of surprise is a surprise in itself :smile:
  • No, seriously. At this point I’m wondering why the trouble of coming up with a different API. They’re so similar. The only difference is that Vulkan is not MS-proprietary, but other than that, if you simply renamed a bunch of functions from DX12 you’d end up with an API that looks pretty close…
  • It feels like DX12 and Vulkan are little brothers to each other… I won’t go as far as to say twins, because there are still some differences, but conceptually… yeah…

Now the positives/negatives that I could spot:

  • :heavy_plus_sign: :heavy_minus_sign: The API seems to have been designed with future expansions in mind. Lots of parameters are currently left unused for future use. That’s both a plus and minus, a plus because it should theorically make future expansions easier; a minus because it makes it look more complex than DX12.
  • :heavy_minus_sign: :heavy_minus_sign: The API is awfully verbose. By that I mean that they chose to pass function call arguments as structs, and sometimes structs-into-structs. This means that every VK call requires a ton of lines, a lot more than DX12. Sometimes it gets to ridiculous levels, I’ve seen an example with a single function call requiring as much as 60 lines of setup, just to pass parameters. Look at that one for instance: 2800 lines for a… cube ? Wtf ?
  • :heavy_plus_sign: At first glance, synchronization through events or fences seem easier. I’m not sure I’ve understood the difference with mutexes just yet.
  • :heavy_plus_sign: Resources creation / allocation also seems easier than in DX12 at first glance.
  • :heavy_plus_sign: Vulkan introduced the concept of a Render Pass, which I think wasn’t present in DX12 unless I missed it ? Not sure how useful it will be, but it might potentially be interesting.
  • :heavy_minus_sign: :heavy_minus_sign: Extensions crap, please, nooooo. Release day and already a dozen extensions. You know that OGL got annoying because of its extension hell, right ? With hundreds of often vendor-specific extensions and multiple ways of doing the same thing. It was a jungle. Do not repeat the same mistake. The fact that we need an extension to create a swap chain ( one of the most basic things in creating a rendering surface ), day #1, already makes me angry. I get it, you did that because it’s OS dependent, but don’t tell me there was no way to make that part of the core, I won’t believe you.

So in the end, are we going to switch to Vulkan or DX12 ? The fact that Vulkan is cross platform is a big advantage. The fact that it will be available on Windows 7 ( whereas DX12 is Win 10 only ) is another big one. So even if the flaws mentionned above end up being very real, I think the good outweights the bad. That doesn’t mean we’re jumping in adding Vulkan support to the engine any time soon though. First of all, we currently lack the resources to do so. This will be a major technical challenge. The entire renderer will have to be remade, which is man-months of work. This might have significant implications on to the rest of the engine too, maybe the scene graph. So the first step is simply to wait for the API to become more mature, to gain more experience with it ( we’ll probably run some experiments with it first ), and once we gain confidence, assuming ressources permit, we’ll switch to Vulkan. That definitely won’t happen until at least next year, if ever.

25 Likes
16

Friday, 19 Feb 2016

A couple days ago, we did a bunch of editor improvements / fixes. Some things were getting really annoying, for instance opening the model viewer was taking minutes, even for a simple object. After investigation we discovered that it was caused by the wireframe material…

  • There isn’t just one, but actually two wireframe materials. They’re not compiled on-demand ( like when you press the wireframe button ), but the first time you open a model…

  • Those wireframe materials were compiled for all shader models ( SM3, 4, 4.1, 5 ) even though only one is in use at a time.

  • Those materials were compiled with max shader optimization level, when the user probably doesn’t care about performance in the first place.

I fixed the 2 last points, which reduced the initialization time from a few minutes down to around 10 seconds. That’s still a lot just to open a cube model, I think we’re definitely going to have to implement a caching or precompilation mechanism for some special materials.

In other news, I’ve started to investigate decal materials. Before I get into details, I must make it clear we’re not talking about decals as in “decal billboards” such as damage effects here. We’re talking about decal geometry, such as what Star Citizen uses to add all their details to their models. There is plenty of information on the technique on the net, so google for it if you’re interested in the topic.

This is a major art pipeline change that we’ve wanted to use for a while now. It does have a lot of benefits compared to the old “uv-map, skin and make unique textures for every single 3D model” pipeline:

  • Higher quality: decal geometry can re-use texture chunks, so all details are much higher quality than what you could do in a unique texture. Imagine a screw in a cockpit for instance. With a unique texture, there’s no way you’re gonna have any decent resolution for the screw details. With decal geometry, you simply add a quad which samples another texture with the details at a much higher resolution.

  • Lower VRAM: this is actually the #1 motivation for using this technique. The prototype currently uses close to 3.5 GB of VRAM, and it’s not like it features dozens of different ships. We really need to decrease our VRAM usage, and decals help a lot.

  • Faster work: a side effect is that you no longer need to model a high-poly model, uv map it and compute the difference to generate a normal map. That saves a lot of modelling time. You still need normal maps for the details though.

It also has some downsides:

  • Higher polycount: decal geometry is geometry, even if simplistic, it definitely increases the total model’s polycount.

  • How to do unique texture details, such as rust, scratches, etc… When those details are isolated, you can do them as decals, it’s easy. But when those details are shape / geometry dependant ( such as rust/weathering near plating edges ) it becomes problematic. We think that’s probably the reason with SC’s ships look all shiny & clean. Our artists are currently investigating what are the limits of this technique in this area.

  • Level-of-detail and Z-fighting. I don’t think Z-fighting will be too much of a problem ( SC seems to have lots of it when far away, but we do Z-buffer tricks due to the nature of our planetary engine, which makes those problems disappear ). LOD however, well, you can’t simplify the decal geometry since it’s already quite simplistic to start with ( if a screw is a quad, you can’t really simplify it… ). The other problem is that if an asset has a lower LOD level, if you use the decals from a higher LOD level, the two don’t match anymore. We planned to do automatic LOD, but I’m starting to think that won’t be an option due to this issue. Artists might have to manually create every single LOD and their matching decals.

On the engine side, I’ve started to implement the missing features to support these geometric decals & materials. Exposing things such as a ZOffset/Zscale to the user ( to avoid z-fighting ), a render order index ( to ensure that the material is rendered after its base material ), writing masks ( some decals might only want to write normals, others only diffuse, others both… ), alpha blending or masking etc…

Alpha blending ended up being a major issue with our deferred renderer, and incompatible with our current GBuffers setup. I’m going to have to change it so that we can do decals blending. I’ll probably work on that today the entire day.

25 Likes
17

Monday, 22 Feb 2016

This has been a pretty hectic week-end.

We have conducted our first deployment test. As you can see from your ability to read this and to still go over our forums and website, it hasn’t been a total disaster, ahah. In fact, you could say it’s been pretty successful. The Kickstarter / IndieGogo databases have been merged to our I-Novae “live” DB, so people with Dev access should now be able to see the Dev-restricted forums. Everybody should have their forum badges ( we haven’t had time to replace the graphics just yet, so it’s using generic icons for now ). And the first version of our web api is up and running, which will be the foundation for running game servers.

Speaking of which, I’ve uploaded a first build of the server to our Azure virtual machine, and it runs but… game clients can’t connect to it. A new issue with server certificates… dang it. I’ll be investigating that in the coming days. But overall, everything is more or less working as expected.

The next step ?

We are tentatively going to make the dev-access game prototype release this next week-end. At this moment, we can’t announce the exact date or time yet, but it’s probably going to be saturday 27 or sunday 28. That’s assuming we can fix all our last issues and that we don’t discover new tricky ones.

To verify if you are elligible for playing, use this page to check for your game access:

https://inovaestudios.com/Battlescape/Access

The procedure is pretty simple, you simply have to associate your Kickstarter/IGG e-mail to your I-Novae account. Either your main account’s e-mail addresses already match, and you’re good; either they don’t match, and you need to add a secondary e-mail address ( and verify it ). Make sure you don’t introduce typos, because those can’t easily be fixed.

If you think you have entered all these information correctly but your access still doesn’t look right, shoot us an e-mail ( or send me a private message on these forums ). Make sure to copy/paste the content of the box from the access page, and include your user name and e-mail address(es). You can also leave a comment in this thread:

23 Likes
18

Tuseday, 23 Feb 2016

We are pretty satisfied with how things have been going, deployment-wise. Which is not to say that there are no issues… but to be honnest we were kindda expecting more issues. However we haven’t announced the access page “officially” yet ( it’s only referenced from this forum ), but at some point in the coming days, once we’ll be more comfortable with our ability to release the prototype this week-end, we’ll send a newsletter and make an official announcement.

Here’s the list of currently known issues:

  • website refreshing issues, mostly under Firefox. When you sign up for a new account or log in, firefox doesn’t handle the redirection correctly, leaving you thinking nothing happens. Just hit F5, or use another browser. We’ll address these issues once we have more time… which means, probably not too soon :frowning:

  • desynchronization issues: the main website and the forums ( Discourse ) use 2 different databases, which we try to keep synchronized. Sometimes it happens that you’re seen as logged in in one but not the other. You might have to re-login regularly on the forums. We’re still not sure how this happens. Clearing your cookies might help. Testing another browser too.

  • changing your main e-mail address on the forums: a lot of people went in the forums to change their e-mail address for their account to match their Kickstarter’s. Unfortunately, due to the 2 databases being distinct, the main account’s still using the old one. Until we fix this, you can simply use the “add e-mail” feature from the Access page to add your new e-mail. Try to avoid changing your e-mail from Discourse; Ideally we’d simply disable the feature, except that changing the Discourse sources is not easy, since it’d get overwritten by any update we do to the forums…

  • developper forum access and badges: when you add a secondary e-mail to your account, the access rights are updated immediately, however the dev access and the badges are only refreshed every 24h to avoid stressing the database. So you won’t immediately get them. Just be patient.

  • secondary e-mails: the add a secondary e-mail functionnality is still experimental, but we’d like to remind you that these e-mails still have to be confirmed before you get granted access. Make sure to check out your spam folder !

Yesterday I was able to connect the client to the game server. These damn certificates are driving me crazy. Anyway, this now works. I was able to test the public server registration and how it appears and gets refreshed in real-time in the launcher’s server browser. I added some better error handling and messages. I tested the client on a brand new machine, and started to run into trouble in terms of dependencies.

The goal is still to provide a .zip that contains the launcher and the game client. However, there are many redistributable packages that need to be installed for the executables to run. Those currently are:

  • OpenAL redist
  • .NET framework 4.6.1
  • VC 2013 x64 redist
  • VC 2015 x64 redist
  • Universal CRT redist

These will be included in the .zip for easier installation, but you’ll still have to install them manually.

Today, a lot more little fixes and a polish pass on the launcher. Time to redesign some screens a little bit !

28 Likes
19

Friday, 26 Feb 2016

I won’t lie: it’s getting tough. Release definitely won’t happen saturday, but there’s still hope for sunday.

On the engineering side we’re almost good to go. On the content side however there’s still a bit of work to do. We need to integrate some of the new assets / placeholders, generate the configuration files for them, finalize the new ship ( placing lights, sockets for thrusters/guns etc… ), tweak the physics parameters, update the planets textures and a ton of other things like that.

Once everything is done, we still need to do a final cooking for all the assets ( to optimize them ), generate the build ( which involves running a bunch of scripts ) and upload it. The entire process, to generate a single build, takes 4-5 hours.

So yeah, I’m quite busy and I don’t think I’ll have much time for a new progress report until the release.

41 Likes
20

Monday, 29 Feb 2016

Warning: long post incoming.

So, here we are. The long anticipated “developer-access” launch has happened !

We were crunching hard all during these past days, some of us not going to sleep before 4 to 7 am. I personally have been crunching for most of February, but even so, this was a “soft crunch”. These past 3 days however… oh boy… last time we worked that hard was the days before the Kickstarter launch. Needless to say, we’re all pretty tired and we’re gonna slow down a bit for the coming days.

So, how did the launch go ? In fact it went pretty smoothly. Which is not to say that it was perfect, it absolutely wasn’t, and many people had issues with their accounts access rights or the installer. But there wasn’t any major issue ( other than some well known bugs ) in the game itself. No major stability issue, no major framerate issues ( a lot of users seemed to be pretty happy with the performance, and was above their expectations. A few even ran it on laptops at decent framerates ).

But overall, we are most than satisfied with the launch, and we think it has gone as well as we could realistically expect considering our low resources and budget.

Personally, on a scale from 1 to 10 ( 1 being a catastrophic failure and 10 the perfect launch ) I’d say it was a solid 7 or a 8.

The game was launched yesterday at around 8 pm GMT.

The server hasn’t crashed once during that time, and we had a peak of 35 players a few hours after the launch. It was really fun to see players getting used to the controls of the game, fly around, explore planets and shoot at each other. Then press ESCAPE by accident. My evil plan has succeeded, it seems… :smiling_imp:

And so much for the “cease fire” gentlemen’s agreement. Within minutes, everybody was firing at each other and chasing others around the asteroids. I even got killed a few times :rage:

While the server itself was stable, it seems like there was a few network hiccups, with sometimes players failing to update for many seconds. After a while it stabilized again. The server isn’t exactly too powerful and it has limited bandwidth, but I don’t think we were close to the limits. We might have to investigate later with a better server and more stable bandwidth.

So far feedback from our testers on the prototype has been overwhelmingly positive. Of course, the game isn’t perfect and features a lot of placeholders and systems that are barely functional ( I’m looking at you, warp systems :wink: ). There seems to be a bug with flight auto-assist disabled that prevents you from side strafing; the annoying cockpit flickering bug once you die is still there; but overall it’s working pretty well for a first release.

I’d like to share two quotes ( I won’t name the authors, they’ll recognize themselves ) that made me burst into laughter when I read them.

“Man, I must’ve been following this off and on for 9 years. It’s kinda surreal to be playing this prototype”

and

“whaouu this new cargo ship is awesome !!”

The first one is self-explicit. The second one made me laugh because the “cargo ship” is an untextured placeholder, and I definitely wouldn’t quality that as “awesome”.

A lot of the new content on the asteroid base didn’t even exist a few days ago. Two or our artists, Jan and Dan, worked on it until the last minute ( adding the hangars where you spawn for example ) to give the asteroid base some sense of scale from these industrial insallations. It’ll definitely look much better once textured and lit.

Kristian has made new texture packs for planets ( I love some of the new Mars-like surfaces ) and the new asteroids. They are more colorful and varied than the ones from the Kickstarter, although I’m sure nobody remembers :smile:

The UI also got a revamp, but it is functionaly identical to the KS one. It’s still a placeholder though. In a few months we’ll do a serious pass on it, both in terms of functionality and visual look.

The bomber ( the new flyable ship ) currently behaves more like a fighter. This is a conscious choice, since I didn’t want player’s first experience to be with a sluggish ship that has no missiles or torpedoes yet. Once we have an actual fighter implemented into the game, there will be massive tweaks to all controls and physical parameters to make the ships feel more like what their roles imply.

Before I close this post I should list 2 VODs of the event from yesterday:

and

I’d also like to mention Caesar’s 4K screenshots which are absolutely stunning:

31 Likes
21

Wednesday, 2 March 2016

Yesterday the plumber came again to repaint my appartment ( yup, you read that right, the plumber is also a painter. Yeah, I was surprised ). Therefore this has been a forced vacation day.

… or almost. I still had a bit of time to investigate some issues, work on the facebook authentication problems, do a couple of bug fixes, and help some people who seemed to have access issues. I can’t say it has been the most productive day though.

Since I was forced to early retirement for a day, I investigated some hardware replacements that I had been delaying for a while. In particular, my main screen monitor is borderline garbage now ( it’s 8 years old ) with washed-out colors. I’m planning on ordering a new QHD ( 2560 x 1440 ) monitor. I won’t go higher in resolution for price reasons and lack of a video card that could handle UHD anyways…

Short-term road-map ( March )

We are planning to release a first patch to the game client today. This will be an optional patch, and will mostly address some of the most urgent / pending issues, like those people being unable to connect through Facebook. There will be a bunch of minor tweaks too.

Looking ahead, this month we are going to focus on stabilizing the game, sorting out access rights / launch problems or other technical issues, adding new placeholders and doing some tweaks.

  • Keith will mostly be busy on finishing the launcher’s automatic patching system.

  • He also has to do some website updates, and resume work on our custom pledge system.

  • We want to get ride of IndieGoGo as soon as possible. The way we’re forced to poll at regular intervals to get the list of backers is extremely annoying and taxing server resources. There are also update delays, which means backers have to wait a bit for their data to be refreshed before they can access the game. Quite annoying for everybody.

  • Our artists are going to work on concept art for, well, the entire game.

  • They’ll also continue working on the existing assets, experimenting with the new artist pipeline ( based on decals ), the cockpit, better structures for the asteroid base, some new lights placements…

  • And, most importantly, producing a new set of placeholder assets ( those will be untextured too ) which we can use to iterate game on fairly quickly.

  • As for me, I’m going to keep working on client issues and gameplay.

  • For example my top priority will be to figure out the weird cockpit flickering bug. I’ve already tried a few hours during the KS campaign, but due to being so busy I couldn’t keep working on it. Now the dev-access release is done, I’ll have enough time to investigate it properly.

  • I’ll also address some of the easiest / most frequently seen remarks or bugs. Drifting failing while flight auto assist is off ? Sounds like it’ll be an easy bug to fix. There’s probably going to be a lot like that.

  • I’m also going to start some of the more heavy work on the client/server. A lot of data is client-sided at the moment, which is no good. I did it that way before the KS because it was faster to implement, but of course this cannot stay like that.

  • I’ll also work on adding the new placeholder content to the game.

  • When I have a bit of free time I’m going to do some experimental improvements on the look of planets. They’re not yet at our target level in terms of visual quality.

  • One thing I will NOT do just yet is to address the elephant in the room: the networking hit combat issues. Those aren’t easy to solve, in fact going forward it’s probably going to be our #1 challenge. We’ll wait until the game has stabilized a bit until we revisit it.

Mid-term road-map ( April - summer )

  • We’ll address the network issues at this stage.

  • We also need to do add better multithreading to the engine, do a bunch of optimizations, implement streaming and more engine work.

  • We’ll start experimenting with Vulkan, although I can’t give a timeline on that just yet.

  • We’ll also start doing experiments for our in-game GUI solution.

  • We also have to update or replace a bunch of old third-party libraries: physics, sound ( OpenAL is really obsolete at this stage, we’re planning on using Wwise ).

  • That’s pretty much when we’ll add new ships, new weapons and iterate on the real gameplay: attacking/defending objectives like factories or NPC ships.

  • We’ll add more planetary environments at this stage too.

Long-term road-map ( Q4 2016 and later )

  • Content in the game will still be pretty thin at this stage, but we’ll work diligently on increasing it. There will still be some placeholder content remaining.

  • Alpha will be released around that time.

  • Balancing the game “seriously” will also start around that time.

  • Small ships gameplay should be faily well established at this point, but capital ships gameplay will require a lot more work.

  • Note that the longer ahead we are looking at, the less certain we are about development priorities. A lot of things can change depending on how things are going. So take these roadmaps with a grain of salt.

40 Likes
22

Saturday, 5 March 2016

Patch 0.1.2.0 is out, woooh hoo. Adresses some issues with Facebook authentication in the launcher, detail textures for the Glimmerfall asteroid ( we’ll soon extend these detail textures to the roids in the rings ), tweaks to the HUD and a bunch of bug fixes.

Our artists have started to work on the new concept art for all the game’s content, which will be followed up by a bunch of placeholders. I’m not sure yet how the ships placeholders will be integrated into the game since we have no GUI to select a startup ship. Maybe a ship will be selected randomly when spawning, or maybe we could add a preference setting in the config file. We’ll figure that out later.

A few users are still having issues to launch the game. It seems related to fonts. One of them has a problem with the launcher’s fonts ( which uses Windows standard fonts, so not sure why it’d be a problem ) and the other has a problem with a game’s custom font ( the function we use to load / install that font is quite simplistic so I’m not sure what could go wrong ). Still investigating.

I’d like to remind everybody that warp mechanics are barren. I’ve only worked on them for a couple hours in the past months, so they’re in need of some serious love. The sudden acceleration / slow down is one thing I want to address soon. Overall I’d like the warp to work like an Alcubierre drive, special effects coming at a later stage, but functionally it shouldn’t affect the ship’s current velocity. The current implementation of the warp simply changes the acceleration by a factor which is depending on gravity ( so the closer you are to a planet, the lower the warp factor ). That gravity dependency is something I’d like to keep, however the fact that the “real” velocity changes during warp is something I’d like to remove. Oh, and for long-term travel, we’ll probably use a different mechanism which provides FTL. We want players to “jump” to the other side of the solar system in like 30s, not half an hour…

Some of you might have noticed “Glimmerfall Ops” in the middle of the rings, far away where the Glimmferall asteroid actually is. It’s not technically a bug, simply a debug test I forgot to remove from the server entities config. My bad. Will remove it next time the server crashes. Which can take a while. As I am speaking, the server’s uptime is close to 5 days. Before that, it did reboot… because Windows forced an update and had to reboot. Prior to that it hadn’t stopped since the launch last sunday. Therefore, it’s official: the server hasn’t crashed once. Of course, we haven’t changed much since the KS campaign, but going forward once we resume severe work on the server, it’s probably going to crash a lot more frequently.

A lot of people have reported bugs or issues though our launcher’s report tool. It seems to work pretty well. On the list of issues, some are false positives, like players who did not immediately understand how things work. That’s fine, keep reporting everything, we can sort out everything on our end :slightly_smiling:

One “bug” that I see people report often is in fact not a bug: the lights flickering on the ship in 3rd-person mode. We have a basic damage system; when a light gets hit by a blaster shot, it gets damaged and flickers randomly forever. But I admit it looks like a bug. However what is a real bug is the cockpit visibility flickering, which I’m going to work on diligently in the coming days ( hoping, not weeks… )… but it looks like a tough nut to crack.

There are a few bugs which I haven’t been able to replicate so if anybody has hints, feel free to make a post:

  • Players showing up as “Unknown” via Google login: each time I tested it used my account’s name ( INovaeFlavien ) so I’m not sure how to replicate it.

  • A few crashes: please send your log file, not your crash file. I think we’re going to get rid of crash files, cause they contain less relevant information than log files, and most players actually send those instead of the log files :frowning:

  • It is useless to send us a crash report without the log file. As a reminder: the log file is located in C:\Users\YourName\Documents\Infinity Battlescape\Logs . A report just saying “I launched the game and it crashed” is unfortunately not very useful to us.

  • NPC ships lights shining through building walls: technically not a bug, we just haven’t enabled shadowing for NPC ship lights. We might do it later. I agree it’s sometimes confusing when you’ve spawned in a hangar and see weird lights shining from the walls :sunny:

32 Likes
23

Friday, 11 March 2016

Unfortunately these last few days I’ve had a bunch of IRL issues related to my Internet connection and digital TV. I won’t go into details, but at this point, I’ve wasted more than a dozen hours with my provider’s tech support, replacing equipment and having appointments with technicians to inspect my installation. The good news is that everything has been upgraded and that my problems are apparently solved ( although the problem kind of solved magically itself in the end so I’m not confident it won’t happen again ). It really sucks and frustrates me that I have to spend so much time on these issues, but oh well… for now everything seems to be working again.

On the code side I’ve started to do some refactoring of the client/server. This is only a first step and there’s still a lot more to do in the future, but it’s still an important first step. I was also investigating the ability for Visual Studio to debug a child process and came up with a cool plug-in which does it. Just install the plug-in and when you call win32 api’s CreateProcess it’ll appear in the debugger, call stack and threads list. Why isn’t this a standard in VS yet ? It seems to work perfectly and I see no immediate reason it shouldn’t be available in standard…

Next step now is to do a bunch of additions to our material editor. We’re missing some nodes which could be handy to our artists. Some of these nodes are already implemented on the engine side and were “forgotten” ( yeah that happens ) when we created the material editor ( like absolute value, 1 - x, swizzle ), and should take only an hour or two to add. Since Jan worked on the detail textures for the Gimmerfall asteroid we also realized some Photoshop-style filters would become handy, like an overlay node. It’s possible to recreate the overlay formula via other nodes, but obviously it requires the artist to assemble dozens of nodes to do so, so these new nodes will save effort and reduce the material node graph complexity by a lot. All these improvements should be done by this week-end.

After that, it’ll be time to start the serious investigations about this cockpit / thrusters flickering bug. I have a few ideas on what the cause could be, so I’ll start with that, and hopefully it won’t take too much time before I narrow it down.

24 Likes