Panzer - A Finite Element Assembly Engine within the ...

1 , , onalLaboratoriesMarch3,2017 Sandia National Laboratories is a multi-mission laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed MartinCorporation, for the Department of Energy s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2017-1977 CWhatisPanzer?C++LibraryGeneralfiniteele mentassemblyengineformul -physicssimula onSupports1-,2-,&3-Dunstructuredmeshcalc ula onsSuppliesquan esneededforadvancedsolu onandanalysisalgorithmsresidualsJacobian sparametersensi vi esstochas physicsapplica onsarelight-weightfrontendsMassivelypara llelforcomplexphysicsLeveragestemplate-b asedgenericprogramming[5]toassemblequan esofinterestIncorporates35 TrilinospackagesWhatisPanzernot?Applica ,FEniCS,MFEM,MOOSE,SundancePanzer sHistoryLessonslearnedfromSandia sPDEphysicscodesCharon1,MPSalsa, on libraryofpackagesCapabili esexplored/developed Phalanx,PanzerPanzer sHistoryDrekar: advancedalgorithmdemonstra onApplica ons(Drekar,Charon2,EMPIRE,etc.)

2 DrivePanzer srequirements,designgoalsCoupledmul -physicsLargescalesimula on(>100kcores)Finiteelementfocussed(curr ently)Embeddedanalysis(AD,sensi vi es)TechnologysharinganddeploymentPanzerp rovidesapplica onswithflexibleinfrastructure,coretechno logiesPanzerEnablesApplica onsTurbulentCFDM agnetohydrodynamicsSemiconductordevicesS uppor ngtechnologiesAlgebraicmul gridBlockprecondi oningUncertaintyquan fica onIMEXPDE constrainedop miza onCompa blediscre za onsElement&PhysicsBlocksElementBlock0 ElementBlock1 SolidPhysicsBlockFluidPhysicsBlockUsersd ividethedomainintoelementblocksEacheleme ntblockmapstoasinglephysicsblockPhysicsb lockscontainalistofequa onsetsEqua onSetsSolidFluidEqua onsetsdefinetheformofthePDED etailsarefilledinusingclosuremodelsNavie r-Stokes u t+u u ( u) + p=f u= 0 Energy T t+u T ( T) = 0 DataMappingU li esFiniteelementdiscre za onshavechangedHistoricallyusednodalequal -orderfiniteelementsNewcodeembracesmixed discre za onsAlsousinghigh-ordercompa blediscre za onsHgrad(nodal),Hcurl(edge),Hdiv(face)Re quiresextradatamanagement(orienta ons)HgradHcurlHdivDataMappingU li esThreeprimarypieces.

3 FieldPa ernDescribesbasislayout&con nuityoffieldsConnManagerMeshtopologyfrom fieldpa ern(meshabstrac on)DOFM anagerManagesandcomputesunknownfieldnumb ersPanzer=mesh-agnos cpanzer_stk=concreteimplementa onofConnManagerFieldPa ernLinear pressure ,temperatureQuadra cveloci esux,uy,p,T u t+u u ( u) + p=f u= 0 T t+u T ( T) = 0 FieldPa ernLinear pressure ,temperatureQuadra cveloci esux,uy,p,Tp,T u t+u u ( u) + p=f u= 0 T t+u T ( T) = 0 FieldPa ernLinear pressure ,temperatureQuadra cveloci esux,uy,p,Tp,Tux,uy u t+u u ( u) + p=f u= 0 T t+u T ( T) = 0 FieldPa ernLinear pressure ,temperatureQuadra cveloci esux,uy,p,Tp,Tux,uyux,uy,p,T = u t+u u ( u) + p=f u= 0 T t+u T ( T) = 0 ConnManagerLinear pressure,temperatureQuadra cveloci esSolid(T)Fluid(ux,uy,p,T)Numbering=mesh topology01234567891011121314151617181920 2122232425262728293031323334353637383940 41424344 DOFM anager[1]Linear pressure,temperatureQuadra cveloci esSolid(T)Fluid(ux,uy,p,T)Unknownfieldnu mbering012,3,4,5678,9,10,11121314,15,16, 1718,19,20,2122,23,24,2526,27,28,2930,31 ,32,3334,35,36,3738,39,40,4142,4344,4546 ,4748,4950,5152,5354,5556,5758,5960,6162 ,6364,6566,6768,6970,7172,73~x= DAG-BasedAssembly(Phalanx)[2,3,4]Decompo secomplexmodelintographofsimplekernelsRa piddevelopment,separa onofconcerns,extensibilityAutomateddepen dencytrackingTopologicalsorttoorderevalu a ons iu uu uiui u iupi p iptransientMfluxMRCfluxE fsourceMRMRMRC iT T TiTi T iTtransientE RERERM= ( u t+u u ( u) + p f) vd ,RC= ( u) vd ,RE= ( T t+u T ( T)) vd DAG-BasedAssembly(Phalanx)

4 [2,3,4]NodescanbeswappedoutSepara onofdataandkernelsopera ngonthedataMul -physicscomplexityhandledautoma callyEasytoaddequa ons,changemodels,testinisola on iu uu uiui u iupi p iptransientMfluxMRCfluxE fsourceMRMRMRC iT T TiTi T iTtransientE RERERM= ( u t+u u ( u) + p f) vd ,RC= ( u) vd ,RE= ( T t+u T ( T)) vd EvaluatorsDeclarefieldstoevaluate(ortoco ntributeto)DeclaredependentfieldsFunc ontoperformevalua onTemplatedonevalua ontypeSpecializa onsforsca ers&gathersUsercodereusedforresidual,Jac obian,Hessian,etc. iu uu uiui u iupi p iptransientMfluxMRCfluxE fsourceMRMRMRC iT T TiTi T iTtransientE RERERM= ( u t+u u ( u) + p f) vd ,RC= ( u) vd ,RE= ( T t+u T ( T)) vd u(x,y) +k2u(x,y) =f(x,y),(x,y) = (0,1) (0,1)u(x,y) = 0,(x,y) u(x,y) +k2u(x,y) =sin(2 x)sin(2 y),(x,y) u(x,y) = 0,(x,y) u(x,y) + (1 8 2)u(x,y) =sin(2 x)sin(2 y),(x,y) u(x,y) = 0,(x,y) u(x,y) + (1 8 2)u(x,y) =sin(2 x)sin(2 y),(x,y) u(x,y) = 0,(x,y) WeakForm u vd + (1 8 2) uvd = sin(2 x)sin(2 y)vd AnExampleProblemR= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRCreateanEqu a onSet// <typenameEvalT>classMyEquationSet:publicpanzer::Equatio nSet_DefaultImpl <EvalT>{public:MyEquationSet(.)}

5 ;voidbuildAndRegisterEquationSetEvaluato rs(..)const;private:std :: stringdofName_;} // endof classMyEquationSetAddtheDegreeofFreedoma nditsGradient// <typenameEvalT>MyEquationSet<EvalT>::MyEquationSet(..){..dofName_ ="U";std :: stringbasisType("HGrad");intbasisOrder(1 ), integrationOrder(2);this->addDOF(dofName _, basisType,basisOrder, integrationOrder);this->addDOFGrad(dofNa me_);..this->setupDOFs();} // endof ConstructorR= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRAddtheLapla cianTerm// stillin <typenameEvalT>voidMyEquationSet::buildAndRegisterEquat ionSetEvaluators(PHX ::FieldManager< Panzer ::Traits>& fm,..)const{Teuchos::RCP< Panzer ::IntegrationRule>ir =this->getIntRuleForDOF(dofName_);Teucho s::RCP< Panzer ::BasisIRLayout>basis =this->getBasisIRLayoutForDOF(dofName_); ..R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRAddtheLapla cianTermstd :: stringlaplacianName("RESIDUAL_"+ dofName_ +"_LAPLACIAN");Teuchos::ParameterList p; ("ResidualName", laplacianName); ("FluxName", "GRAD_" + dofName_); ("IR ", ir); ("Basis", basis); ("Multiplier", );Teuchos::RCP<PHX ::Evaluator< Panzer ::Traits>>op =Teuchos::rcp(newpanzer::Integrator_Grad BasisDotVector <EvalT, Panzer ::Traits>(p));this-> templateregisterEvaluator<EvalT>(fm, op).

6 R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRAddtheHelmh oltzTerm// stillinbuildAndRegisterEquationSetEvalua tors()std :: stringhelmholtzName("RESIDUAL_"+ dofName_ +"_HELMHOLTZ");Teuchos::ParameterList p; ("ResidualName", helmholtzName); ("ValueName", dofName_); ("IR ", ir); ("Basis", basis); ("Multiplier",( - * M_PI * M_PI));Teuchos::RCP<PHX ::Evaluator< Panzer ::Traits>>op =Teuchos::rcp(newpanzer::Integrator_Basi sTimesScalar <EvalT, Panzer ::Traits>(p));this-> templateregisterEvaluator<EvalT>(fm, op);..R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRAddtheSourc eTerm// stillinbuildAndRegisterEquationSetEvalua tors()std :: stringsourceName("RESIDUAL_" +dofName_ +"_SOURCE");Teuchos::ParameterList p; ("ResidualName", sourceName); ("ValueName", dofName_ +"_SOURCE"); ("IR ", ir); ("Basis", basis); ("Multiplier", );Teuchos::RCP<PHX ::Evaluator< Panzer ::Traits>>op =Teuchos::rcp(newpanzer::Integrator_Basi sTimesScalar <EvalT, Panzer ::Traits>(p));this-> templateregisterEvaluator<EvalT>(fm, op).

7 R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRAddtheResid ual// stillinbuildAndRegisterEquationSetEvalua tors()std :: vector<std :: string>residualOperatorNames{laplacianName,helm holtzName, sourceName};this->buildAndRegisterResidu alSummationEvaluator(fm,dofName_, residualOperatorNames);} // endofbuildAndRegisterEquationSetEvaluato rs()R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRCreatetheSo urceFunc on// <typenameEvalT,typenameTraits>classMySourceTerm:publicPHX ::EvaluatorWithBaseImpl <Traits>,publicPHX ::EvaluatorDerived<EvalT,Traits>{public:MySourceTerm(..);voidpostRegistr ationSetup(..);voidevaluateFields(..);pr ivate:PHX ::MDField<EvalT::ScalarT, Panzer ::Cell, Panzer ::Point>result;intirDegree_, irIndex_;} // endof classMySourceTermR= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRCreatetheSo urceFunc on// (typenameTraits::EvalDataworkset){consta uto& coords = [irIndex]->ip_coordinates;Kokkos::parall el_for( ,[=] (constpanzer::index_t c){for(intp(0);p < (1); ++p){constdouble& x(coords(c, p, 0)),y(coords(c, p, 1));result(c, p) = sin(2 * M_PI * x)* sin(2 * M_PI * y);} // endloopovertheIPs});// endloopoverthecells} // endof evaluateFields()R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRCreatetheClosureModelFactory// <typenameEvalT>classMyClosureModelFactory:publicpanzer: :ClosureModelFactory<EvalT>{public:typedefstd :: vector<Teuchos::RCP<PHX ::Evaluator< Panzer ::Traits>>>EvalVec;typedefTeuchos::RCP<EvalVec>EvalVecRCP.

8 EvalVecRCP buildClosureModels(..)const;} // endof classMyClosureModelFactoryR= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u iuHelmholtzLaplacianfSourceRRCreatetheCl osureModelFactory// <typenameEvalT>EvalVecRCP MyClosureModelFactory <EvalT>::buildClosureModels(..,constTeuchos::RC P< Panzer ::IntegrationRule >&ir, ..)const{EvalVecRCP evaluators =Teuchos::rcp(newEvalVec);..Teuchos::RCP<PHX ::Evaluator< Panzer ::Traits>>e =Teuchos::rcp(newMySourceTerm<EvalT, Panzer ::Traits>("U_SOURCE", *ir));evaluators->push_back(e);..returne valuators;} // endof buildClosureModels()R= u vd + (1 8 2) uvd sin(2 x)sin(2 y)vd = 0 iuuui u CreateanEqua Addtheresidual2. Createthesourcefunc on3. CreatetheClosureModelFactoryConcludingre marksApplica ondevelopersfocusoncomplexi esinphysicsmodels,boundarycondi ons, veeaseAdvancedanalysis=freeUsePanzer useTrilinosHowIuseTrilinosEvery-dayuse: Panzer ,Teuchos,Thyra,Phalanx,Epetra/Tpet raEveryonceinawhile: NOX,LOCA,Piro,TekoReferences[1] ,MarkHoemmen, ,andBenSeefeldt.

9 ParallelUnknownNumberingfortheFinite-Ele mentMethod .In:(2017).Submi edtoACMTOMS.[2] , , Graph-BasedSo wareDesignforManagingComplexityandEnabli ngConcurrencyinMul physicsPDESo ware . ( ),1:1 1:21. :0098-3500. : [3] , , Automa ngEmbeddedAnalysisCapabili esandManagingSo wareComplexityinMul physicsSimula on,PartI:Template-basedGenericProgrammin g . ( ), 219. :1058-9244. : [4] , , , , ,andMa Automa ngEmbeddedAnalysisCapabili esandManagingSo wareComplexityinMul physicsSimula on,PartII:Applica ontoPar alDifferen alEqua ons . (July2012), 345. :1058-9244. : [5] EfficientExpressionTemplatesforOperatorO verloading-basedAutoma cDifferen a on .In: (2012).