Rocstar Simulation Suite
Over the past decade, a team of faculty, staff, and student researchers at the Center for Simulation of Advanced Rockets (CSAR) have been developing Rocstar, a comprehensive, integrated, highly parallel software suite for complex three-dimensional multiphysics simulations. Its individual component codes are based on fundamental research and development in turbulence modeling, multiphase flow, constitutive modeling, combustion chemistry, computational mechanics, coupling methodology, etc. Now relatively mature, Rocstar provides by far the most advanced simulation capability available today for analyzing performance of existing solid rocket motors and for virtual prototyping of new designs. Through collaborations with CSAR, the solid propulsion programs of DOD, NASA, and the U.S. rocket industry increasingly rely upon Rocstar. Recent examples include analyses for NASA of a proposed five-segment version of the Space Shuttle booster for use in the lunar exploration program, simulations for the U.S. Air Force to characterize the distribution of aluminum particle impingement on rocket nozzles, and multiscale modeling of solid propellants for ATK.
The initial target for Rocstar has been the simulation of solid propellant rocket engines. Rocstar has capabilities well beyond solid propellant motors however — it has a flexible, general-purpose framework and can effectively simulate a wide variety of multicomponent systems involving fluid dynamics, struc-tural dynamics, combustion, and their interactions. Rocstar features multiple, state-of-the-art solvers for various types of physical components, but what sets it apart is its sophisticated integration framework that facilitates the solution of tightly coupled problems using independently developed physics modules with minimal changes. Rocstar also provides an array of services including parallel I/O, performance profiling, accurate and conservative data transfer, automatic mesh modification and load rebalancing, surface propagation, and overall parallel orchestration. Rocstar has been applied to a number of fluid-structure interaction and multiphase problems in addition to rocketry, including simulations of noise from helicop-ter blades, fuel injectors, volcanoes, and the effects of wind on tall buildings. In addition, Rocstar has ex-hibited excellent scalability on multiple thousands of processors and is expected to perform efficiently on petascale systems.
Rocstar Features
Rocstar is a fully integrated, general-purpose solver for the numerical simulation of fully-coupled, time-dependent fluid, structure, and combustion interaction problems. It consists of a suite of physics applica-tions coupled together by means of a powerful integration framework. All components of Rocstar are designed to run efficiently on massively parallel computers, enabling the use of detailed, science based physical models in complex 3-D geometries.
The Rocstar physics applications solvers are general-purpose and discipline-specific code modules that interact by calling a comprehensive set of library routines to register and share their data and to take advantage of the advanced meshing, data transfer, surface propagation, I/O, and other capabilities included in the integration framework. A key design goal has been to minimize the number of modifications required to integrate a new solver with Rocstar.
The fluid and structural dynamics applications integrated in Rocstar solve partial differential equations on moving, body-fitted computational meshes. While this choice can provide maximum solution accuracy near domain boundaries, maintaining acceptable mesh quality despite significant geometrical changes has required the development of robust and efficient mesh modification capabilities. For physical problems for which boundary fitted meshes are impractical, solvers with fixed computational meshes may be used, taking full advantage of the advanced data transfer and surface propagation schemes in Rocstar. Geometry descriptions and computational meshes are generated off-line using commercial packages plus preprocessors specific to each solver. The preprocessors are invoked by a comprehensive Rocstar dataset preparation tool that automates problem set-up.
Computational Environment
Rocstar runs efficiently on platforms ranging from single-processor workstations to the world's fastest massively parallel supercomputers. Our staff has access to local computing facilities that include an in-house 1536-processor Macintosh G5 cluster, and on the University of Illinois campus at NCSA are three large Linux clusters, each with 2048 to 4800 processors, and a 384-processor IBM AIX system. Scalability of Rocstar on these platforms enables reasonably short wall-clock times to complete simulations on computational meshes with tens of millions of elements for grid-converged solutions.
Fig.1: IllinoisRocstar, LLC builds on strong success of existing U of I code base. Rocstar results from fully coupled, 3-D fluid/structure/combustion simulations of NASA Space Shuttle solid rocket booster shortly after ignition (a-c) and Air Force BATES motor (d). (a) Head end section shows local deformation of propellant due to pressure in core gas (top) and resultant surface temperatures (bottom). (b) Gas temperatures (top; red is hotter) and aluminum particle size and location (bottom; red is larger) in core flow. (c) Solid propellant deformation and temperature isosurfaces in fluid core. (d) Number of impacting aluminum droplets on rocket motor nozzle (top; red is higher) and mean diameter of droplets (bottom; red is larger).
Rocstar Component Modules
Fluid Dynamics.
Rocstar includes two complementary compressible flow solvers, Rocflu and Rocflo, which share a substantial portion of their code base. The fluid equations are formulated on moving meshes to handle geometrical changes without the loss of accuracy typically incurred transferring the so-lution to new meshes. Three classes of turbulence models are available, including Large Eddy Simulation (LES), Reynolds Averaged Navier-Stokes (RANS), and hybrid models (ether LES with a near-wall model or Detached Eddy Simulation). Any combination of these turbulence models can be used in different re-gions of a single simulation, and various subgrid scale models and wall-layer models can be selected de-pending upon the flows being considered.
Rocflu operates on unstructured tetrahedral or mixed element-type meshes. It employs a novel high-order WENO-like approach, as well as the HLLC scheme to handle strong transient flows including shocks. Time integration is by explicit multistage Runge-Kutta or by a Newton-Krylov-based implicit scheme that utilizes the PETSc library. Rocflo uses either a centered or an upwind scheme with Roe flux splitting on multi-block structured meshes. In addition to explicit time stepping, Rocflo can use a dual time stepping algorithm to take time steps longer than the Courant limit. Rocflo and Rocflu are integrated with physics modules for simulating turbulent multiphase fluid flows. Non-ideal gases, chemical species, droplets, smoke, and radiation (flux-limited diffusion approximation) can all be included with full, two-way coupling. Burning aluminum droplets are treated by tracking Lagrangian super-particles, each repre-senting many droplets, while smoke particles are evolved using our novel Equilibrium Eulerian method.
Solid Mechanics.
Two finite-element structural dynamics solvers are available in Rocstar — Rocfrac and Rocsolid. Both are finite-deformation solvers that use an ALE formulation to account for the conversion of solid propellant into the gas phase, solve three-dimensional heat conduction, and include a variety of element types and constitutive models.
Rocfrac optionally can employ Cohesive Volumetric Finite Elements placed between ordinary finite elements to simulate crack propagation. An explicit time stepping scheme follows any rapidly propagat-ing cracks. Rocfrac features several other specialized element types, as well as a variety of material mod-els including micromechanics-based material models that account for nonlinear interface debonding.
Rocsolid is a variationally-based finite-strain structural solver that employs an implicit Newmark time integrator and several high-performance sparse iterative solvers, including multigrid and stabilized bi-conjugate gradients (BiCGSTAB). Matrix-vector multiplies are implemented using an element-by-element framework, and a matrix-free approach is used to decrease storage and CPU time. Rocsolid in-cludes enhanced assumed-strain solid elements for modeling metallic components, and mixed-mode ele-ments for nearly incompressible material response. Rocsolid optionally utilizes several micromechanics based models to account for the evolution of the microstructure and include the effects of void formation and growth, damage (dewetting), and strain hardening. An objective integration algorithm is used for rate formulations in order to eliminate the generation of spurious stresses in rigid body motions. Rocsolid op-tionally includes cohesive elements to describe the particle-matrix decohesion in the quasi-static regime, and also employs the Generalized Finite Element Method (GFEM) to capture discontinuities, such as moving fronts or cracks, without cumbersome computational geometry.
Combustion.
Rocburn determines the rate of propellant deflagration as computed by one of three com-bustion submodules. These three physical models are pressure-dependent and are one-dimensional (nor-mal to the surface) in formulation. Thus, for a given model, the local burn rate is obtained independently at each cell face on the burning surface. The simplest model assumes steady state burning for which the regression speed is proportional to the local gas pressure raised to the power n. Two dynamic burn-rate models may also be selected, both of which solve a time-dependent heat conduction equation for the tem-perature profile near the propellant surface in order to capture unsteady events. The first model is based on the Zeldovich-Novozhilov approach, and ignores ignition events. The second model has two parts, one that models ignition transients and the other that models deflagration. Prior to ignition, the propellant sur-face is heated by igniter gases that are calculated by the fluid solvers of Rocstar. The propellant locally begins to burn when it reaches a critical temperature. Flame spread is then captured and modeled as an ignition wave. After ignition, Rocburn uses a heat-flux look-up table computed off-line by Rocfire, a de-tailed three-dimensional heterogeneous aluminized propellant combustion simulation code. This enables accurate reproduction of dynamic burning behavior at every cell face in a large rocket at minimal addi-tional computational cost.
Rocstar Architecture
Code Architecture.
The Rocstar code suite is integrated by Roccom, an interface that facilitates the ex-change of data and functions among modules, even those written in different programming languages. By making calls to Roccom, physics modules communicate with each other transparently and access a variety of useful services, including mesh modification, data transfer, I/O, performance profiling, and surface propagation (Figure 2).
Coupling and Time Stepping
A modular approach has been adopted in Rocstar for time stepping, wherein execution alternates between separate domain-specific physics solvers. An orchestration module, Rocman, controls the execution of the physics applications to perform coupled time steps. Current options include the Simple Staggered Scheme with optional Predictor-Corrector iterations, and the Improved Staggered Scheme of Farhat. Additional coupling schemes are under development that promise to provide greater accuracy, stability, and efficiency. Rocman also enforces interface jump conditions (derived from conservation of mass, momentum, and energy) specific to the particular coupled problem.
Interface Data Transfer
The Rocface module enables physics applications for abutting domains to ex-change quantities across the interface between them. By construction, the data transfer scheme exactly conserves mass, momentum, and energy at the interface, even when their respective meshes do not match. Conservation is achieved by using a common refinement of the two meshes, each subdivision of which lies entirely within a cell face in both surface meshes. Interpolation errors are minimized in the least squares sense, leading to a scheme that is several orders of magnitude more accurate than previous conservative methods. Although minimizing interpolation errors involves solving linear systems, this procedure consumes only a small fraction of the run time for a typical Rocstar simulation.
Interface Propagation
The surface propagation module, Rocprop, computes the motion of a surface given a local regression velocity, which for rockets is the burn rate. Rocprop includes a new, robust, accurate, and efficient surface propagation scheme called the Face-Offsetting Method (FOM), which is based on a generalized Huygens principle. Unlike Eulerian methods such as level-set methods that require a volume mesh, FOM operates directly on a Lagrangian surface mesh. Unlike traditional Lagrangian methods, FOM propagates faces and then reconstructs vertices through an eigenvalue analysis performed locally at each vertex to resolve the normal and tangential motion of the interface simultaneously. The method rig-orously maintains the smoothness and integrity of the surface as it evolves, and produces accurate physi-cal solutions even in the presence of singularities and large curvature.
Mesh Modification
Mesh modification schemes in Rocstar are employed at two levels. Mesh smoothing without changing the number of mesh vertices for unstructured meshes is accomplished through subrou-tine calls to the Mesquite package developed at Sandia National Laboratories. If smoothing cannot main-tain acceptable mesh quality, then either local mesh repair or global remeshing is performed using the Simmetrix commercial meshing package. After remeshing, the new mesh is repartitioned and the solution is accurately transferred to the new mesh.
Rocstar Verification and Validation, and Uncertainty Quantification
Verification.
Many test cases have been run to verify Rocstar and its component modules.
Super-seismic shock The super-seismic shock problem is a coupled fluid-structure interaction problem in which a shock wave moves through a fluid along the fluid-solid boundary at a speed that is greater than the speed of sound in the solid, and is thus termed “super-seismic”. The interaction of the shock in the fluid with the solid boundary causes a deformation in the solid that has an analytically predictable angle. Rocstar simulation results match analytic results for three different shock intensities to within one percent of the analytical results.
Supersonic free vortex The supersonic free vortex is a test case that models isentropic flow of a com-pressible fluid between two concentric cylindrical walls, with the velocity magnitude varying inversely with the radius. The analytical solution to this problem is available, and both fluid-alone (Rocflu) and coupled cases (Rocflu-Rocfrac) have been run to test grid-convergence using five different grid spacings. The results of both Rocflu standalone and coupled Rocstar runs show 2nd order convergence, as expected.
Manufactured solutions The method of manufactured solutions is a means for verifying computer pro-grams by defining a problem with a known source term and result, and specifically coding the source term into the program to generate the result. The ALE scheme in Rocsolid has been verified using a manufactured solution involving a bar or an elliptical cavity with a constant load applied to one surface. The analytical solution is known for the surface displacement and internal stresses in this problem in the absence of burning. On the surface of the burning bar or cavity, the stress is imposed that was computed from the analytical solution at the same location within the non-burning bar. The computed displacement (due to deformation) of the burning surface is then compared with the corresponding displacement in the non-burning bar. A second manufactured solution problem has been used to verify a physical model before its integration into Rocstar. A one-dimensional coupled fluid-solid heat transfer model has been developed and run. A manufactured solution has been constructed using combinations of Dirichlet and Neumann boundary conditions on either the solid or fluids side of the heat transfer interface. Forms of the problem with both moving and non-moving boundaries have been constructed and run in order to verify the basic 1-D model. The 1-D solution matches the analytical solution within fractions of a percent.
Mass Conservation Since Rocstar is used to model coupled fluid-solid problems with burning boundaries, it is important to ensure mass transferred from the solid to the fluid is conservative. Simple 1-D burning-bar problems have been constructed with sealed fluid domains to enable verification of mass conservation in Rocstar. Two rectangular domains share one face that is burning. As mass is injected from the solid into the fluid, the total mass of each of the solid and fluid systems are output at each time step. It has been verified that Rocstar conserves total mass to machine precision.
Validation.
Rocstar has been validated using test stand and flight data available for several full-scale rockets. Individual instrumented tests are available for the large RSRM motor, multiple configurations with limited data are available for the Air Force BAllistic Test and Evaluation System (BATES) motor, and design data with limited testing results are available for a small attitude control motor (ACM) from Aerojet. Navy laboratory-scale rockets with limited data are also available.
RSRM The NASA Reusable Solid Rocket Motor (RSRM) is large, with complex geometry, and data are available from many firings. Block-structured hexahedral grids as well as several unstructured tetrahedral grids have been generated for use with Rocflo and Rocflu, respectively, as well as a tetrahedral solids grid to model the propellant with Rocfrac. Figure 1(a-c) depicts some of the most recent results obtained for RSRM simulations with Rocstar. Figure 3 shows the axial pressure profile from two instrumented firings along with Rocstar results.
BATES The Rocpart module of Rocstar enables simulation of multiphase flows. Rocpart has been vali-dated using an Air Force laboratory rocket motor test series known as the BAllistic Test and Evaluation System (BATES), in which aluminum particles embedded in the propellant are injected into the flow as the propellant burns. As these particles move through the flow, they burn and change diameter, as well as produce aluminum oxide smoke. The effect of different aluminum loadings on the efficiency of aluminum burning in a 15-lb BATES motor has been performed, as well as simulations with a 70 lb version of the motor. These studies were designed for comparison with Air Force experimental results. Figure 1(d) shows the Al particle impingement on the BATES nozzle cone from one of these simulations. Validation is performed by comparing ISP between the simulation and experiment, and we have found that ISP matches within a few percent.
Titan IV In November 1991 a Titan IV rocket motor exploded during its test-stand firing due to an aero-elastic interaction that caused the propellant to slump inward, choking off the flow and causing the head end pressure to increase catastrophically. Subsequent design modifications eliminated this problem, but CSAR researchers have used the original design as a test of the ability of Rocstar to simulate the dynamic fluid-structure interaction that caused the failure. Several different fluid and solid meshes have been developed, and various material models have been investigated. Rocstar is able to simulate the aeroelastic effects of the slumping propellant and resulting change in pressure.
LabScale/Staraft Motors Experimental data are available for two Navy laboratory-scale rockets we have used in validation studies with Rocstar. One such rocket (“Motor 13”) has a simple cylindrical-bore. It has been extensively modeled with Rocstar and with all combinations of its fluid and solid modules. Simulations using the dynamic burning model in the Rocburn module of Rocstar have been able to simulate the dynamic burning peak and the ignition transient during the initial pressure transient to within a few percent of the (single) experimental result available. “Motor 6” is a more complex aft-star-grain configuration that has been used for studies of the accuracy of the advanced mesh motion and time-zooming models in Rocstar.
Flexible Panel in Shock Tube An experimental shock-tube study where a Mach 1.21 shock in air is intro-duced at one end of a shock tube has been modeled using Rocstar. The shock travels down the tube until it encounters a thin, flat, steel panel projecting vertically into the flow. The panel is fixed at the bottom end, and free at the top and sides. Measurements and observations of the shock profiles and plate move-ment are provided. Agreement between the Rocstar simulation and experimental measurements is excel-lent. The qualitative behavior of the transmitted and reflected shocks in the simulation is identical to the shadowgraph of the experiment.