Testing Graphics Hardware Acceleration

Why does changing a car’s oil in take an hour in California? In Texas, this is a 15 minute process, in and out. I’ve never been a fan of wasting time, so while waiting on my car this morning, I started researching one of the most exciting test opportunities we have in the upcoming Gecko 1.9.2 platform: Graphics Hardware Acceleration.

Most computers that people use today have some amount of GPU acceleration under the hood. So, this should result in improved rendering performance, which we will need as the web becomes more graphically interesting.

Our current test suites do not have any notion of hardware accelerated versus software rendering. In fact, on Linux we run our test suites in an X virtual frame buffer, which never touches any graphics hardware. And the graphics hardware on our boxes is pretty bottom of the barrel. I’d bet that the chip rendering this text for you is more advanced.

So…how on earth are we going to test this? I’m mostly concerned about functional testing, but I recognize that whatever solution we come up with here might also be needed to aid the graphics team with their unit tests. My thoughts and research are just beginning on this effort, so this isn’t a complete plan, this is a preview to my plan and a sincere request for help.

First Steps

We need to determine what the top video cards/drivers are for each of our top three platforms: Windows, Linux, and Mac. We can then grab a small handful of machines that have those configurations. I’m thinking maybe just 3-6 machines here, nothing extravagant. I think we would still want to target middle-of-the-road hardware, not top of the line gaming systems. We want to test what the average web user will be running.

At the same time, we need to figure out what graphics benchmarks exist and whether or not these can be leveraged or re-implemented into a graphics benchmark for the web. Since I’m not interested in how we perform writing instructions into the XPCOM components that make up the underbelly of the graphics support, I’m thinking we should create these benchmarks/tests in JavaScript and canvas. We ought to use this new graphics support the way that real developers will be using it once it is released.

Further Thoughts

What we might use here would be a reftest-like framework that would be augmented to compare the test that is hardware accelerated with a reference image that is software rendered. I imagine there are pitfalls this way–with hardware support, the test image may be more detailed than the reference, and we would need a mechanism to detect this condition to flag it as a “pass” rather than a “failure”. I definitely need to think about this some more.

Vision

There are two basic parts here. First, we need a simple test infrastructure that can test our rendering in both accelerated and software modes. We need a set of representative machines to run these tests on and we need those machines automated into our normal build and test reporting structure.

However, our tiny sample of machines will not give us the coverage we really need. So, the other reason this test infrastructure must be simple is that I want to invite all interested users to run it for us. It must be something that the users can download and run on their nightly builds. Once the test finishes, the users should be able to send us the results data from that run with a click of a button. The anonymized results will help us understand how well our code is handling device/driver/OS combinations.

I wanted to start a conversation here, and throw a line out into the wide world of the net to see who might be interested in helping us figure it out. Thoughts and ideas are most welcome. I’ll keep y’all up to date as my plans progress.