The end of Dennard scaling, resulting in the use of low-power processor technologies in HPC, is leading to the deployment of large-scale parallelism on a single compute node to obtain high performance. By 2024, the cost of flops on HPC-capable systems, in terms of dollars and power, is expected to decrease by a factor of 100, relative to what it was in 2014. However, the performance of memory systems for these processors will not keep pace, due to power constraints. The ratio of the peak aggregate flop rate to the bandwidth between DRAM and the floating point units, as well as ratio of flop rate to overall size of DRAM, will be significantly less than what we have been used to for the last 20 years. Traditional numerical methods for partial differential equations have low arithmetic intensity (AI). Such methods can only obtain a small fraction of peak performance on such systems: performance is limited by rate at which data can get to the floating point units. In this talk, we will give several examples of how these new architectures lead to new tradeoffs in the design of numerical algorithms for PDE, trading flops for bytes in a way that reduces the overall data traffic and memory footprint required to obtain a given level of accuracy in a calculation. We will focus our attention on structured-grid and particle methods; in those cases, the most obvious example of this approach is the use of methods that are higher-order accurate than the ones customarily used (mostly second order) in multi-physics calculations. High-order methods can, in principle, obtain a given level of accuracy for fewer degrees of freedom, and the arithmetic intensity increases with the order of accuracy of the method. The design of new discretization methods is central to the success of this endeavor. However, there are other issues that need to be addressed in order for this approach to be successful. The design of high-order accurate methods that will retain their apparent advantages for realistic applications is nontrivial, and significant effort is required to realize the theoretically predicted performance by careful implementation.

We report recent progress on wall-resolved and wall-modeled large-eddy simulation (LES) of wall-bounded turbulent flows, including model development and applications. The sub-grid-scale (SGS) model employed is the stretched spiral vortex SGS model, originally proposed by Misra & Pullin (Phys. Fluids 1997). The present wall-model is an extension of the original one proposed by Pullin & co-workers for LES of high Reynolds number flows in a channel, turbulent boundary layer, and a pipe. Our extension to the wall model includes three modifications: relaxation of the log-law assumption, extension into two-dimensional wall model and reformulation in generalized curvilinear coordinates. Five flow configurations are examined in detail. The first case is that of a zero pressure gradient turbulent boundary layer in which our objective is to address whether the mean velocity profile obeys a log-law or a power-law. The second case is that of a separated and re-attached turbulent boundary layer; where the two-dimensional wall model is shown to reasonably predict the wall stress in the separated flow. The third case is flow over period hill, a benchmark case for LES modeling. The fourth and perhaps more challenging case is that of a flow over a circular cylinder with emphasis on reproduction of the drag crisis at Re~O(105) and quantification of skin friction on the surface. The final case is that of flows over airfoils, which is a prerequisite step for applying the present curvilinear wall model on cases with even more complex geometry. In all these, our emphasis is on validation and quantitative comparisons with experimental data when available. Acknowledgement: The work was supported by the KAUST office of Competitive Research Funds under Award No. URF/1/1394-01. The Shaheen-Cray XC40 at KAUST was utilized for all the simulations. Collaborators: Dale Pullin, Wan Cheng, Wei Zhang and Wei Gao

The talk will present simulations based on a Lattice-Boltzmann Method, which allows efficient handling of large-scale and complex geometries. Since the turbulent flow needs to be spatially and temporally resolved, the simulations are usually carried out on a large number of computational cores for adequate turnaround times. Methodology and HPC requirements will be explained and examples range from aerospace to oil & gas applications.

CFD is a key technology enabler for obtaining improved performance for land-based and airborne turbomachinery engines at GE. We will present the state-of-the-art ways GE utilizes CFD in the design process for turbomachinery blade rows and discuss the complementary strategies related to high performance computing. We will discuss the interplay between RANS and LES in the engineering process. We will close with a discussion of future challenges and highlight ways academic research can impact industrial CFD.

Saudi Aramco R&DC scientists are working on enhancing the performance of multiphase gravity separator vessels (GSV), utilizing CFD to better design the internals of these separator vessels, and enhance their separation efficiency. The multiphase CFD modeling of such an application is computationally demanding, as it needs to address the many involved phenomena, in addition to the relatively large domain sizes. Typical Saudi Aramco vessels are 40 m in length and 4 m in diameter. Typical complex features of the multiphase separation process are the poly-dispersed nature of the oil emulsion, complex emulsion rheology, transient nature of the process, and the need to resolve turbulence structures that affect droplets coalescence and breakage, etc. These competing phenomena put a high demand on computational resources. Utilization of enough parallel processes of an HPC system, such as KAUST’s supercomputer Shaheen II, should expedite the turnover rate of such simulations, which will enable Saudi Aramco engineers to explore more design options, and forecast performance of the multiphase separators under foreseen conditions. In collaboration with the KAUST Supercomputing Lab, Saudi Aramco R&DC scientists are investigating the acceleration of the turnover rate of these particular simulations when using a parallel multiphase CFD code, such as ANSYS Fluent, to predict the complex flow inside Saudi Aramco GSVs, and running the simulations on a massively parallel HPC resource such as Shaheen II. The results of the scalability study performed on Shaheen II will be presented in this talk. The results show excellent scalability until 16k cores. The results of arguably the largest run of FLUENT ever, on any supercomputer, 200k cores, will be presented as well, conditional on the results being available on the date of the presentation.

Advances in massively parallel computing hardware enabled high fidelity simulations of turbulent combustion at unprecedented physical scales with chemical complexities. The presentation will start with an overview of current status in the KARFS (KAUST Adaptive Reacting Flows Solver) project, a new direct numerical simulation (DNS) code for reacting flows for hybrid CPU+X architecture. The new code has been applied to a number of physical studies, such as turbulent premixed flame at high Reynolds/Karlovitz numbers, bluff-body flame stabilization, and auto-ignition of premixed reactants in the presence of temperature fluctuations. Key challenges and scientific findings from these studies will be discussed.

In many applications the simulated fluid incorporates a rather small amount of particles and each single particle needs to be resolved. In biological systems the high viscosity of the surrounding cytosol plays a dominant role since the hydrodynamic interaction between the fluid and the particle serves as dedicated communication pathway within the overall system. So called direct numerical simulation is necessary. We derived a fully coupled and stable method for an appropriate formulation of the interacting forces. Many methods introduce external forces to describe the interaction between fluid and particles. Such an approach typically leads to a saddle point problem. We utilize a finite volume discretization of the Navier-Stokes equations. Its theoretical treatment as a so-called Petrov-Galerkin method enables to develop a fully coupled model, naturally incorporating the particles into the equations. By rigorous extension of the solution space and also the test space of the bilinear operator the internal forces can be formulated in a rather natural way on the basis of the underlying model equations. Our approach leads to an implicit representation of the according forces. The main benefit is the avoidance of a saddle point formulation. Consequently, the solving procedure is conducted in one step and the coupling of the interaction forces can be preserved. Furthermore, in contrast to other approaches, no stabilizing terms need to be introduced to achieve stability: By exploiting the comparability to a finite element method our method can even be proved to be stable. In addition, the immersed boundary is represented as sharp interface in the discrete equations. Second order convergence in space was reached. Finally, the resulting discrete scheme is consistent with the original, unrestricted equations without particles.