Fall 2013 Supercomputer Rankings Released; Achieving Max Efficiency With Oil
by Ryan Smith on November 22, 2013 3:00 PM ESTTurning our eyes back towards SC13 and supercomputers one more time this week, we have the biannual supercomputer rankings, which are released every spring and every fall. These rankings are composed of the Top500 and Green500, which rank supercomputers based on total performance and power efficiency respectively. These lists are admittedly mostly for bragging rights, but they do give us an interesting look at what the top computers in the world currently look like.
Top500 Top 6 Supercomputers | ||||||
Supercomputer | Architecture | Performance (Rmax, TFLOPS) | Power Consumption (kW) | Efficiency (MFLOPS/W) | ||
Tianhe-2 | Xeon + Xeon Phi | 33862.7 | 17808 | 1901.5 | ||
Titan | Opteron + Tesla | 17590.0 | 8209 | 2142.7 | ||
Sequoia | BlueGene/Q | 17173.2 | 7890 | 2176.5 | ||
K Computer | SPARC64 | 10510.0 | 12660 | 830.1 | ||
Mira | BlueGene/Q | 8586.6 | 3945 | 2176.5 | ||
Piz Daint | Xeon + Tesla | 6271.0 | 1753 | 3185.9 |
Starting with the Top500, the top of the list remains unchanged from what we saw in June. Further supercomputers have come online in the last 6 months, but none large enough in scale to compete with the top supercomputers. This means that the Xeon + Xeon Phi powered Tianhe-2 remains as the top supercomputer in the world, offering a staggering 33.8 PFLOPS of performance, almost twice that of the next fastest computer. This also continues Intel’s rule of the top of the list, with the Xeon powered supercomputer having now held the top for two iterations of the list.
Moving down the list, we have the Opteron + Tesla based Titan, the Bluegene/Q based Sequoia and Mira, and the SPARC64 based K Computer. The first new supercomputer on the list comes in at #6, and that’s the Swiss National Computing Center’s Piz Daint. Piz Daint was first announced back in March and is now fully up and running, based on a Cray XC30 which in turn is a combined Xeon + Tesla system. Piz Daint weighs in at 6.2 PFLOPS, thereby making it the most powerful supercomputer in Europe. This also marks the second Tesla K20 system to make the top 10, tying Intel’s pair of Xeon Phi systems but still well behind the 4 Bluegene/Q systems that occupy the top of the list.
Also released this week was the Green500 list, which is a derivation of the Top500 list sorted by power efficiency instead of total performance. Since the Green500 list isn’t decided on raw performance it often sees a much higher rate of turnover than the Top500 list, as smaller scale computers can enter the top of the list if they’re efficient enough.
Green500 Top 5 Supercomputers | ||||||
Supercomputer | Architecture | Performance (Rmax, TFLOPS) | Power Consumption (kW) | Efficiency (MFLOPS/W) | ||
TSUBAME-KFC | Xeon + Tesla | 150.4 | 27.7 | 4503.1 | ||
Wilkes | Xeon + Tesla | 239.9 | 52.6 | 3631.8 | ||
HA-PACS TCA | Xeon + Tesla | 277.1 | 78.7 | 3517.8 | ||
Piz Daint | Xeon + Tesla | 6271.0 | 1753.6 | 3185.9 | ||
Romeo | Xeon + Tesla | 254.9 | 81.4 | 3130.9 |
This November has seen exactly the above happen, with a number of smaller installations coming online and pushing older systems further down the list. For June you needed an efficiency level of 2,299 MFLOPS/W to make the top 10; for November you need 2358 MFLOPS/W. Which doesn’t sound like a huge difference, but it was enough to push systems 4 through 10 off of the top 10 list, meaning most of the list is composed of new supercomputers.
Interestingly, every new system on the list is some kind Xeon + Tesla K20 configuration, which in turn has led to the entire top 10 now being composed of those systems. In fact you have to go down past #11 – the Xeon + FirePro S10000 based SANAM computer – to find a system that isn’t GPU powered. The Green500 list is based on total power consumption – system plus cooling – so it isn’t just a measurement of system power efficiency, but except in the cases of Eurora, TSUBAME-KFC, and other computers that are using exotic cooling, everyone else is using standard air conditioning and as such system power efficiency is the deciding factor. To that end it’s clear that from an efficiency perspective GPUs have the market cornered, edging out both traditional CPU setups and even Xeon Phi setups (as GPU-like as they are).
Looking at some of the individual systems, the new champion of the Green500 is an interesting and admittedly small scale system out of Japan called TSUBAME-KFC. At 27.78 kW in power consumption and 150 TFLOPS, TSUBAME-KFC isn’t very powerful, but at 4,503 MFLOPS/W it’s by far the most efficient system on the list. The secret – and to our knowledge this is the only ranked supercomputer like this – is that TSUBAME-KFC is the only oil immersion supercomputer on the list. As we’ve seen in the past with systems such as Eurora, supercomputer builders are playing with increasingly exotic forms of cooling in order to keep power consumption down and to further increase efficiency, as the largest scale installations are bounded more by power/cooling needs than they are equipment costs.
TSUBAME-KFC (Kepler Fluid Cooling, we’re told) takes this idea to the extreme by using non-conductive oil rather than air as the heat exchange medium. Oil doesn't have quite the immense heat capacity of water, but it allows for full immersion of the system nodes rather than requiring the use of tubing and water blocks. Coupled with system level optimizations undertaken by the system builders, such as turning down the GPU clockspeeds for efficiency rather than absolute performance, the builders were able to get the system up to 4,503 MFLOPS/W.
At 25% more efficient than the next closest supercomputer, TSUBAME-KFC is probably a safe bet for holding the top spot for another iteration. Any significant move on the list will require equally exotic cooling, or more likely a newer installation using newer processors, which is admittedly just a matter of time. Whether a traditional air cooled system can beat that mark remains to be seen, but we certainly expect all the usual parties – including the new IBM/NVIDIA partnership – to be potential contenders as their new processors and new manufacturing processes come online.
29 Comments
View All Comments
marc1000 - Sunday, November 24, 2013 - link
haha, oil-like cool!bobbozzo - Friday, November 22, 2013 - link
Hi, how is the oil cooled?e.g. via air-cooled heatsinks/exchangers, or freon-cooled, or ?
Thanks!
errorr - Friday, November 22, 2013 - link
Answered this elsewhere but the data sheet said heat exchanger to building water (hot or cold) hookup. So yay tap water.SuckRaven - Friday, November 22, 2013 - link
My computer needs an oil change. =)mfenn - Friday, November 22, 2013 - link
KFC obviously stands for Kentucky Fried Chicken. Kepler Fluid Cooling is just a respectable-sounding bacronym. ;)Belegost - Friday, November 22, 2013 - link
I was hoping the KFC here meant the hot oil could be used for frying poultry. I am greatly disappointed.xdrol - Friday, November 22, 2013 - link
You should check your facts.. Mineral oil has about one third heat capacity than water, just you can submerge chips into it (unlike water).errorr - Friday, November 22, 2013 - link
Um, just a correction that I would only make on a site like this but oil has nowhere near the heat capacity of water. Besides pure hydrogen or helium only anhydrous ammonia exceeds the heat capacity of water.I believe they use "white" mineral oil (baby oil, liquid parrafin) with additives of Teflon like chemicals to boost heat capacity a bit. Still the heat capacity is still probably less than 1/3 of water.
Of course there is much more mass so the heat capacity of the whole system is better than if you could somehow use distilled water in a sealed vessel.
My understanding is also that the system is cooled by using tap water and a heat exchanger to answer someone's question.
Tchamber - Friday, November 22, 2013 - link
It may have lower heat capacity than water, but what it is good at is absorbing heat...I would guess that it absorbs heat at a higher rate, much like aluminum absorbs heat faster than steal. With enough volume, it has better cooling efficiency than any closed-loop H2O system.meacupla - Saturday, November 23, 2013 - link
I doubt that, it is an oil after all and has nowhere near the same amount of bonds as water can achieve.the advantage this system has is sheer reservoir volume, its massive heat exchange surface area and because it is devoid of air, which performs worse than oil.