SIERRA -- A Computational Framework for …

1 1 SIERRA -- A Computational Framework forEngineering mechanics R. Stewart Carter Edwards Sciences CenterSandia National LaboratoriesAlbuquerque, NM2 Outline SIERRA concepts and overview Basic Framework services Advanced Framework services3 DOE / ASCI Developed for Department of Energy (DOE) Accelerated Strategic Computing Initiative (ASCI) Challenges Coupled multiphysics: solid +fluid+thermal+chem. Large unstructured meshes: 100,000,000s elements Massively parallel computing: 1,000s of processors Advanced algorithms: mesh erosion, h-adaptivity,multilevel, dynamic load balancing4 SIERRA ConceptApplications share a single Framework whichprovides common capabilities Simplify utilization of ASCI supercomputers Consolidate common capabilities Eliminate redundant development and maintenance Encourage architecturally similar applicationsApplication developers work in a commonsoftware development environment Uniform access to ASCI resources Utilize a common source code repository Coordinate development efforts Consolidate the set of software development toolstools Share software development processes5 SIERRA Contributors Framework Noel Belcourt Kevin Copps Carter Edwards

2 Jonathan Rath Greg Sjaardema Jim Stewart Alan Williams Tools Kathy Aragon Dorothy Brethauer Christi Forsythe Mark Hamilton Erik Illescas+ many Application Developers!6 Current stand-alone codesVIPARP arachute performance code, vortexmethod coupled with transient dynamicsPRONTOT ransient dynamicsLagrangian solid mechanicsJASQ uasi-static solid mechanicsCOYOTET hermal mechanics with chemistryGOMAI ncompressible fluid mechanics with free surfaceSALINASS tructural dynamicsSACCARAC ompressible fluid mechanicsDAKOTAD esign optimizationVIPARP arachute performance code, vortexmethod coupled with transient dynamicsPRONTOT ransient dynamicsLagrangian solid mechanicsJASQ uasi-static solid mechanicsCOYOTET hermal mechanics with chemistryGOMAI ncompressible fluid mechanics with free surfaceSALINASS tructural dynamicsSACCARAC ompressible fluid mechanicsDAKOTAD esign optimizationMigration of SandiaApplication CodesSIERRAC urrent and future SIERRA -based codes FUEGO/ SYRINXPREMO(SACCARA)SALINASKRINOARIA (GOMA)CALORE(COYOTE)ADAGIO(JAS)PRESTO(PR ONTO)

3 ANDANTEVIPARDAKOTA7 Outline SIERRA concepts and overview Basic Framework services Advanced Framework services8 Basic Framework ServicesUser Input ParsingBulk Mesh Data I/OMechanics MgmtField MgmtMesh MgmtParallel CommunicationsSIERRAK ernelMaster Element I/FLinear Solver I/F9 Mesh Management Unstructured mesh Arbitrary mesh object connections Mix element topologies (hex, tet, quad, ..) Fully distributed mesh data structure Dynamic creation/deletion of mesh objects Can define mesh subsets Define by part, material type, boundary, constraint, .. Define unions and intersections of subsets10 Field Management Application defined fields ( variables) Text name Type (int, real, vector, full tensor, symmetric tensor.)

4 Aggregate types ( collection of material variables) Optionally associated with a master element(interpolation field, integration field) Fields are defined on mesh objects Associated with a mesh subset ( field , mesh-object subset ) allocated value ( field , mesh-object subset ) NO value11 Mesh/Field Managementdata100data200data300data430da ta410data420 Collection of mesh objectsdata442data441data444data44330040 0 Finite Element Mesh410430440420441442443444100200300 Contiguous workset arraydata444data441data430data410data200 data300data100data443data420data44212 Mesh/Field ManagementWorkset arrays are: Delivered to the mechanicsalgorithm in Fortran array order Sized to minimize cache missesdata444data441data430data410data20 0data300data100data443data420data44213 Finite Element Services SIERRA provides a master element interface Elements are implemented and shared byapplication developers Current (partial) list of master elements (eachwith fully integrated, uniform gradient, and controlsurface/volume versions) 8-node hexahedron 4-node tetrahedron 4-node quadrilateral (in 2D and 3D, including shell) 3-node triangle (in 3D, including shell)

5 6-node wedge Others, and more on the Management A mechanics module consists of algorithmsand supporting data Associate mesh subsets with mechanics modules Declare fields to support mechanics modules Uses zero-to-many master elements mechanics modules are supplied byapplication Examples: solvers, BC s, external forces, etc Nest mechanics modules inside other modules15 mechanics Module HierarchyDomainProcedure (time step control)Region A(single step of physics A)MechanicsMesh and FieldsRegion B(single step of physics B)MechanicsMesh and FieldsTransfer16 Parallel Communication Services Simple scalar reductions ( global dotproduct) Global assembly Update/sum field values on subdomain boundary Inter-processor operations Communicate fields between processors (MPI) Multiple domain decompositions Redistribute mesh between decompositions SIERRA uses the Zoltan (SNL)

6 Partitioning library17 Finite Element Interface (FEI) Prometheus Trilinos Spooles PETSc Others Linear Solver InterfaceElement, Boundary, and Constraint ContributionsSolution ValuesApplication MechanicsLinear Solver Interface18 User Input ParsingApplication sMechanicsParameterValuesUserUserInput FileCommandSpecifications HTML PagesGenerateDocumentationCommandSpecifi cations XML DatabaseQuerySpecificationsSIERRAP arserParseCommandsCommandRegistration19 Scaling Tests: 3 ApplicationsAdagio Scaling for 2K Elements/Processor Relative to 32 Processors on of ProcessorsScaling of Nonlinear IterationASCI-RedASCI-Blue-MtnASCI-Blue- PacPresto Scaling for 2K Elem ents/Processor Relative to 32 Processors on of ProcessorsScalingASCI-RedASCI-Blue-MtnAS CI-Blue-PacCalore Scaling for 10K Elements/Processor Relative to 32 Processors on of ProcessorsScalingASCI-RedASCI-Blue-MtnAS CI-Blue-Pac Presto: explicit dynamics Adagio: quasi-statics Calore.

7 Thermal conductionand enclosure radiation20 Outline SIERRA concepts and overview Basic Framework services Advanced Framework services21 Basic Framework ServicesUser Input ParsingBulk Mesh Data I/OMechanics MgmtField MgmtMesh MgmtParallel CommunicationsSIERRAK ernelMaster Element I/FLinear Solver I/F22 Advanced Framework ServicesMechanics MgmtField MgmtMesh MgmtParallel CommunicationsH-AdaptivityField TransfersElement DeathLoad BalancingSIERRAK ernelUser Input ParsingBulk Mesh Data I/OMaster Element I/FLinear Solver I/F23 Key Concept:Multiple Domain DecompositionsOriginal Finite Element Mesh Primary Decomposition - graph based,best suited for element computations Secondary Decomposition geometry based,best suited for search algorithms.

8 mechanics algorithm chooses an efficient domain decomposition mechanics algorithms are independent of a specific decompositionFour Processor Example24 Dynamic Load Balancing:Surface Example(b)Primary Decomposition Topology BasedAll of surface is on P0P0P1(c)Secondary Decomposition Geometry BasedDistributed surface on P0 & P1P0P1(a)Original Undecomposed ModelSurfaceLoad BalancingDecompositionPrimary Surface nodes faces edgesNOT balanced forsurface algorithmPrimary Surface nodes faces edgesNOT balanced forsurface algorithmSecondary Surface nodes faces edgesBalanced forsurface algorithmSecondary Surface nodes faces edgesBalanced forsurface algorithmTwo copies of the same surface (nodes, edges, faces) using exactly the same amount of memory.

9 However, they are distributed over theprocessors differently for each Technology:Parallel Communication Specification CommSpec Relation: Source-Mesh Destination-Mesh { (MeshObject,SrcProc) (MeshObject,DestProc) } Symmetric: Relation = converse(Relation) Shared mesh objects on inter-processor boundaries Nonsymmetric Arbitrary on and off processor mesh object connectivity Inter-mesh transfers, load balancing, periodic boundaryconditions, contact algorithms, ..26 Determine common geometricdecomposition for rendezvousParallel Transfer Algorithm Transfer nodal field from source to destination Mesh decompositions not geometrically aligned must geometrically rendezvous the meshesDestinationMeshSourceMeshMesh Data(CommSpec B)Field & Mesh Data(CommSpec A)SourceRendezvousMeshDestinationRendezv ousMeshOn-processor search(CommSpec C)On-processor interpolationInterp.

10 Field Data(CommSpec BT)Search Results(CommSpec AToC)Interpolated Results(CommSpec BToCToA)27 Coupled Calore/Fuego(Thermal/Flow) Pipe Flow ProblemCalore Mesh both pipe and fluid Transfer fluid temperatureto FuegoFuego Different mesh for fluid Transfer fluid velocity toCalore28H-Adaptivity: Overview Goal: Selectively refine elements toachieve solution accuracy more efficiently Application SIERRA EitherSolve PhysicsYesProceed to Next TimestepEstimate SolutionError DistributionStopping CriterionSatisfied?NoMark Elements(Adaptive Strategy)ResolveMarkers (2:1)RestrictVariablesGlobal Mesh UpdateProlongVariables29H-AdaptivityGlob al Mesh UpdateGlobal Mesh UpdateRestrictVariablesProlongVariablesU nrefine MeshRebalance LoadRefine MeshRebalancing is done when the mesh is smallest !