The Core i9 Gaming Benchmarks Intel Commissioned Against AMD Are a Flat Lie

Everyone in the enthusiast community knows that manufacturer-provided benchmarks must be taken with a grain of salt. One could write a book on the various ways that companies tend to shade the truth to paint their own products in a positive light. Some of these practices are defensible, at least to a certain degree -- a company that chooses to put its best foot forward by selecting tests in which it performs well may have a perfectly defensible argument if the tests it chooses are well-known industry standards and represent workloads the component is expected to run.

Other times, however, the changes companies make when comparing their hardware to other systems aren't defensible. And sometimes, they cross the line from "favors our own products" into "blatantly misrepresents the performance of the competition."

Intel -- or to be precise, a company Intel hired to create a whitepaper on Core i9 gaming performance -- has crossed that line. According to Forbes(Opens in a new window), Intel contracted with Principled Technologies to distribute a whitepaper(Opens in a new window) containing various claims about gaming performance between Intel's upcoming Core i9-9900K and Core i7-8700K and the AMD Threadripper 2990WX, 2950X, and Ryzen 7 2700X(Opens in a new window). With AMD having surged into competitive positioning in the past 18 months and Intel taking heat from its 10nm delays, Chipzilla has every reason to push a narrative that puts it in the driving seat of gaming. But Intel is using this whitepaper to claim that it's up to 50 percent faster than AMD in gaming based on Ashes of the Singularity in particular, and that's where the problems start. The Intel results are somewhat higher than we'd expect, but the AMD CPUs -- particularly the Ryzen 7 2700X -- are crippled.

There are several problems with the AMD benchmarks as run by Principled Technologies. PT was careful to document its own configuration steps on both systems, which is why we know what, precisely, the company did wrong.

First, the Ryzen systems were tested without XMP enabled. XMP is the high-end memory timing standard that enthusiast kits use to hit maximum performance and Ryzen gaming performance is often tied directly to its RAM clock and sub-timings. Using substandard timing could lower Ryzen's performance by 5-15 percent.

Second, all of the benchmarks in question were run using a GTX 1080 Ti and a resolution of just 1080p. If you wanted to create a report tailor-made to Ryzen's weaknesses, that's the resolution you'd use. Unfair? Not necessarily -- it's the most common resolution after all. But there's a reason we include 1440p and 4K results in our resolutions comparisons for gaming, and Intel/Principled didn't do so.

Third, Principled Technologies notes that it enabled "Game Mode" in AMD's Ryzen Master utility. The implication is that it did this on both systems. This can have serious side effects on how well an AMD system benchmarks. On Threadripper, engaging Game Mode cuts the CPU core count in half and enables NUMA to allow the remaining CPU cores to schedule workloads on the cores closest to the memory controllers. On Ryzen 7, clicking Game Mode just cuts the core count in half. That's why AMD's user guide for Ryzen 7 specifically states that Game Mode is reserved principally for Threadripper and that Ryzen customers shouldn't use it:

If Principled had consulted AMD's documentation, it would've seen that it shouldn't be using this test mode for Ryzen 7 in any case. If it didn't consult AMD's documentation, it had no business using Ryzen Master to adjust Ryzen 7 CPU settings. But the 50 percent performance gain that Intel claims for itself is exactly the kind of result we'd expect if the 2700X had been crippled by having its CPU neutered.

If you need additional evidence of how crazy these scores are, consider our own 2700X review, which we also test with a GTX 1080 Ti(Opens in a new window) in the CPU focused benchmark (the same one PT used).

Their Core i7-8700K is actually a touch slower than ours, but our Ryzen 7 2700X(Opens in a new window) is a massive 1.36x faster. While our results use different detail settings, TechSpot(Opens in a new window) actually checked the exact results with AotS benchmarks of their own. In the graphs below, red bars indicate Principled Technologies results.

Their Assassin Creed Origin tests are similarly broken:

Because they're effectively benchmarking the Ryzen 7 2700X as a quad-core CPU with lousy memory timings, it's no particular surprise that the Ryzen 7 ends up getting its ass kicked. This goes beyond simply adjusting a few game settings in a way that favors your hardware but subtly disadvantages the competition. The Ryzen 7 2700X has been configured to run with half its cores disabled in a non-optimized memory configuration with sub-optimal timings while the Intel system was configured with an ideal memory subsystem and all of its cores and threads enabled.

Misrepresenting product performance by 3-5 percent is a tilt. Misrepresenting it by 1.2x (AotS) and almost 1.25x (as in ACO) is a lie. And that means these results are lies. They may be lies of ignorance or error rather than the result of a deliberate malicious intervention, but given Intel's history, enthusiasts are unlikely to extend much benefit of the doubt. Even a casual readthrough of the document ought to have caught these mistakes -- if, in fact, they were mistakes. And even in the most charitable reading, Principled had no business using an application like Ryzen Master if they weren't going to read the documentation AMD provides to tell you how to use the damn thing. Anybody can have a test run go poorly or mistype a number, but TechSpot found evidence of manipulation in every single benchmark they checked. Either the 8700K was strangely faster than it ought to have been, the 2700X was significantly slower, or both.

What makes the entire affair that much more perplexing is that we'd expect Intel to win this comparison anyway. There was no need to resort to crippling the 2700X to pull ahead. The company could've done that simply by using 1080p and choosing tests where Ryzen doesn't compete as well. The sharp-eyed would call foul, but people are used to taking vendor tests as preliminary indications at best. Instead, Principled Technologies has called into question its own expertise and raised serious questions about what, exactly, Intel was attempting to accomplish with this whitepaper.

When asked for comment by Forbes, Intel responded:

"We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world’s best gaming processor. Principled Technologies conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third party testing in the coming weeks."

Guys, I don't know what you think "in spec" looks like, but running the 2700X with half its cores disabled doesn't fit the bill.