Skip to main content

Event Details

  • Tuesday, May 23, 2017
  • 10:00 - 10:30

Sustained Petascale Production-Mode DNS of Secondary Flows with a Very High-Order IMEX Solver on Three of the Most Powerful Supercomputers in the World

Corners in rectangular ducts induce flows in the cross-stream plane, so-called secondary flows that enhance the cross-sectional mixing. These secondary flows are turbulence driven. It is an open question as to whether Reynolds numbers greater than 30,000 strengthen or dampen the secondary flows, and how flow properties are affected. We demonstrate a tool capable of analyzing these questions by performing highly efficient extreme scale direct numerical simulations of secondary flows using very high-order spectral element methods. Single-node efficiency is achieved by auto-generated assembly implementations of small matrix multiplies and key vector-vector operations, aggressive loop merging, and selective single precision evaluations. Scalability is maintained by bounded nearest neighbor communication and global coarse solvers based on algebraic multigrid. Using 294,912 cores (9216 nodes) on Trinity XC40, we demonstrate greater than 50% of peak memory bandwidth sustained over the entire solver (500 TB/s in aggregate), sustained petascale performance, and a 30% reduction in power consumption compared to native Fortran on Shaheen XC40. Similar performance is achieved on the emerging Xeon Phi architecture. On 3072 nodes of Cori we achieved 378 TFLOP/s with an aggregated bandwidth of 310 TB/s, corresponding to 2.11x faster time-to-solution compared to the same number of Haswell nodes.