Last week my team and I spent some time in Bellvue working with Valve's Linux team on their port of Left 4 Dead 2.

It was the most successful work trip I've ever had. The guys at Valve were amazing to work with. They are sharp, driven, and have an aggressive plan. Looking at how far they've come and the short amount of time in which they've done it, I have every confidence that they're going to kick their plan right in the butt. It's going to be a good time to be a Linux gamer.

We had three main goals going in:

  1. Help them tune their game for our driver / hardware.
  2. Find out where our performance is lacking.
  3. Find out what OpenGL features they need / want.

I think we scored on every point. We helped them find some performance bugs in their vertex buffer management (which also affected other GPUs / drivers) and some places where the accidentally triggered shader recompiles. This gave some healthy performance improvements.

We also found some areas where our driver really, really needs to improve. They have a couple shaders that devolve into register spilling nightmares. There are also a few places where we eat way, way too much CPU. A lot of these problems mirror issues that we've seen with other game engines (e.g., Unigine).

These have been a lot easier to diagnose on L4D2 because we have access to their source code. Being able to take a profile that shows times in the driver and in the application makes a world of difference. Being able to tweak little things in the app (what happens if I do this...) is also helpful for diagnosing performance problems. Eric has already started landing patches for L4D2 performance, and there will be many more over the coming weeks.

The funny thing is Valve guys say the same thing about drivers. There were a couple times where we felt like they were trying to convince us that open source drivers are a good idea. We had to remind them that they were preaching to the choir. :) Their problem with closed drivers (on all platforms) is that it's such a blackbox that they have to play guess-and-check games. There's no way for them to know how changing a particular setting will affect the performance. If performance gets worse, they have no way to know why. If they can see where time is going in the driver, they can make much more educated guesses.

We also got some really good feedback about features. The biggest feature they want is better output from GL_ARB_debug_output. They really want to know when they do things that fall off performance paths, trigger shader recompiles, etc. We hacked out some initial versions of this, and it was a big help. Some patches in that area should hit the mailing list soon.

They're also interested in what they call "smart vsync." Swaps that are scheduled soon enough will have vsync, and the application will be vsync limited. Swaps that are scheduled too late happen immediately. In their words, "Tearing is bad, but dropping to 30fps is worse." On GLX, we can expose this with GLX_EXT_swap_control_tear.

I fully agree: The more information we as developers can get about what is going on in the drivers, the easier it is to make our code fast and bug-free. Reading that sentence again, this comes off as somewhat self-evident.

What appears to not be entirely self-evident, is the fact that driver-devs often spend great effort tweaking drivers for specific games. PC driver-devs oftentimes mention the large effort they have to put into creating workarounds for bugs in games. Gamedevs bizarrely have less information available on the PC-platform compared to consoles (ps3/x360). This coupled with a lower priority for the platform, and thus less time to develop the tools themselves, results in games running on exactly the drivers available at release-date... if that.

I thoroughly believe a great way of remedying this situation, is to provide information through the opengl-debug-context. Using the dbg-context as a verbose-mode, warning about hw-specific performance-issues, doing additional internal error-checking etc. and reporting it all to the game-developer. Using a legacy-part of OpenGL that is emulated (slowly)? Warn the user. I think this could help further the development of the OpenGL-API too.

An interesting effect of the existence of the dbg-context is that it opens up for new changes to the API; it is no longer a complete no-no that you can crash the driver, as long as the dbg-context is able to catch and warn about the error. Examples of this are the Nvidia bindless vertex-attributes and texture- extensions, exposing great performance-improvements while catching driver-crashing errors in the debug-context. Gamedevs also working on consoles are used to this level of responsibility, as long as the information is there.

It was a great joy to see that the new Intel4000HD drivers support the debug-context, and I look very much forward to using it to improve future games.

Best regards,

Comment by Mikkel Gjoel Fri 20 Jul 2012 02:56:39 AM PDT
The article is dated 2012-07-19 and talks about "previous week". The performance patch linked to seems to be initially dated 2012-07-07. How do these dates fit together?
Comment by jmayer Fri 20 Jul 2012 03:21:51 AM PDT
Hopefully we will see more pro-linux workgroups in the gaming industrie: having Valve as leader is a first step, though. Actually they started doing what the community said for years: bring both company teams together.
Comment by Lelala Fri 20 Jul 2012 10:14:55 AM PDT
GIT keeps the date when a patch is first committed. We had been looking at L4D2 code for a couple weeks before we visited them, and Eric (and Ken) had some patches in progress before we got there. There are a couple earlier patches that say things like "a closed-source app we're looking at" in the commit messages. Those are L4D2 also. We just wanted to make sure they were okay with us mentioning their product by name before accidentally stepping on any toes.
Comment by IanRomanick Fri 20 Jul 2012 10:32:00 AM PDT

I am curious to hear about how you profile something as complex as kernel drivers and something as messy as game code.

Some in house Intel tools?

Comment by Amadeus Fri 20 Jul 2012 03:44:58 PM PDT
The interesting parts are all in user space. We used sysprof. For figuring out where big chunks of time go, it's a really good tool.
Comment by IanRomanick Fri 20 Jul 2012 05:33:12 PM PDT

Thanks for the feedback! My team was at your talk at the OpenGL BoF at SIGGRAPH last year. We took a lot of notes. :) You should come to the BoF again this year. It's going to be... big.

Any chance of Brink coming to Linux? :D

Comment by IanRomanick Fri 20 Jul 2012 05:42:24 PM PDT

Quote: "In their words, "Tearing is bad, but dropping to 30fps is worse." "

In my opinion tearing is much worse. I realize this is a matter of taste and you can't really discuss about it. So please make Smart-VSync optional and VSync still enforceable.

Comment by Basto Sat 21 Jul 2012 02:47:12 AM PDT
Are most of these fixes in the i965 driver, or are there a significant number in core Mesa too?
Comment by Bryan Cain Sat 21 Jul 2012 01:50:56 PM PDT

Very exciting development!

Carmack have for years pushed GPU vendors for direct memory access, so he can update a texture without having to go though OpenGL, and just change a pointer directly.

Have you talked with him about what features he would like to see?

Comment by Sandra Mon 23 Jul 2012 04:06:32 AM PDT
vsync settings can currently be over-ridden by the user via the driconf option vblank_mode. I don't expect that would change. People often want to turn vsync off for benchmarking, etc. Keep in mind that the use case for the smart vsync is when there are a couple frames that occasionally take too long. The trade-off is an occasional tear (maybe... games tend to put menus, etc at the top to mask this) vs an occasional stutter. In a twitch action game, I'll take the occasional tear every time.
Comment by IanRomanick Mon 23 Jul 2012 10:54:53 AM PDT
Most of the initial low-hanging fruit is primarily in the i965 driver. I suspect that there will be changes in core Mesa as well. Many of the cases where too much CPU is used are either drivers doing too much (or redundant) validation or core Mesa routines not getting out of the way early enough. The former requires just driver changes, and the latter requires Mesa infrastructure changes.
Comment by IanRomanick Mon 23 Jul 2012 10:57:32 AM PDT

It's really hand in hand work that is going to improve the situation, for both developers, vendors and end-users. That's a first step in the right direction, but we should definitely head toward more openess.

Keep up the good work!

Comment by Chris Thu 02 Aug 2012 10:55:24 PM PDT
You say you liked to be in touch with a game team and have their source to poke at. Then why don't you just help open-source software in a similar manner? Second Life for example is a very heavy and difficult OpenGL consumer and is majorly limited by driver performance, and while commercial, the client source code is freely available. I maintain my own, highly divergent fork of the client.
Comment by Siana Gearz Sat 04 Aug 2012 04:09:17 AM PDT
Whom do i e-mail with information about Smart V-Sync which i'm not sure is good idea to be posted in public?
Comment by Siana Gearz Sat 04 Aug 2012 04:22:52 AM PDT

Go Intel! Go Valve! Looking forward to L4D2 (and others!) on Linux.

Comment by JT Wed 08 Aug 2012 09:32:37 AM PDT
I hope they(and others in our industry ) realize the power of OSS from this experience. Developers shouldn't have to guess what drivers or libraries are doing under the hood.
Comment by Anonymous Tue 14 Aug 2012 08:53:59 AM PDT
Hey! With articles like this it really makes me want to reconsider ATI for their cards. For the longest time their drivers have been what has kept me with NVIDIA, but ATI is really ramping up their game. +1 (linux is the best)
Comment by Benjamin Temple Thu 16 Aug 2012 08:21:54 AM PDT