Oracle Real Application Clusters (RAC) on Oracle VM

1 <Insert Picture Here> Oracle real Application Clusters (RAC) on Oracle VM Nitin Vengurlekar, Markus Michalewicz, Daniel Dibbets RAC is the cornerstone of Oracle s Grid infrastructure RAC is the enabler of Grid Higher Availability RAC systems can be configured to have no single point of failure Scalability RAC allows database systems to scale-out rather than having to scale-up Affordability Can be built using low-cost servers and low-cost storage Manageability RAC with EM, allows users to manage a cluster of database servers as if they were single systems. Oracle RAC Overview Improved server consolidation RAC databases with underutilized CPU resources or peaky CPU utilization can be consolidated with other similar workloads Rapid provisioning of RAC servers and resources RAC nodes can be quickly deployed including fully patched and configured system images Sub-capacity licensing Oracle VM can be configured using hard partitioning Hard partitioning allows you to license only CPUs used by the partition instead of all CPUs on a physical server Virtual cluster Create RAC environment on a single physical server Perfect environment for test.

2 Q/A, or demo clustersBenefits of RAC on Oracle VM (OVM)Servicepublic networkNode1 Operating SystemOracle Clusterwareinstance 1 ASMVIP1 ListenerNode 2 Operating SystemOracle Clusterwareinstance 2 ASMVIP2 ListenerServiceNode nOperating SystemOracle Clusterwareinstance nASMVIPnListenerService/../Redo / Archive logs all instancesshared storageDatabase / Control filesOCR and Voting DisksManaged by ASMRAW / Block DevicesOracle RAC Architecture Configuration Oracle Database 32bit Guest OS : Oracle Enterprise Linux or higher Requires Oracle VM Requires para-virtualized mode guests Static mode support only Live Migration, dynamic resource changes (CPU / Memory) currently not supported Timeline for Certification RAC-OVM Template coming soon 64bit CYQ1 11g CYQ2 Dynamic resources changes CYQ2 Live Migration possibly CYQ2 Oracle VM RAC DatabaseSupport Why RAC on OVM Certification is needed Oracle Clusterware / RAC is very time and scheduling sensitive Follow best practices and guidelines to avoid surprises Validation/Certification - Areas of focus.

3 Clock drift Scheduling Timeout sensitivity Dynamic changes OS profile and statistics CPU, Memory affinity ..Server Virtualization Certification Size as you would a bare-metal server Same traditional questions: How much memory do I need? How many CPUs do I need? How much I/O and network bandwidth should I dedicate?Sizing and Configuration Guest and Dom0 Minimum of 2 Oracle VM servers Ensure enough memory to accommodate DomUcomponents (OS, Oracle Clusterware, and RAC databases) Do not overload Dom0 with unnecessary packages, products and workload Configure Time synchronization Configure NTPD to run in Dom0 and DomU Set guests to manage their own system clocks by setting / file.

4 RAC on OVM Server Configuration CPU configuration Assign a minimum of 1 core to Dom0. Certain circumstances may require to assign 1> core to Dom0. Follow guidelines in Oracle VM Server User s Guide Over-committing CPUs or cores is not supported Don t create more vcpu s then real CPU s in one DomU Dynamic VCPU and memory changes are currently not supportedRAC on OVM Server Configuration cont dD o m a i n - 0G u e s t D o m a i n 1D a t a b a s e 1D a t a b a s e 1I n s t a n c e 1A S M I n s t a n c e+ A S M 2A S M I n s t a n c e+ A S M 1D a t a b a s e 1I n s t a n c e 2O r a c l e C l u s t e r W a r eO r a c l e E n t e r p r i s e L i n u xO r a c l e C l u s t e r W a r eO r a c

5 L e E n t e r p r i s e L i n u xG u e s t D o m a i n 2 RAC on OVM Non-ProductionDeployment Domain-0 Guest Domain 1 Database 1 Instance 1 Database 2 Instance 1 Database 1 Instance 2 Database 2 Instance 2 ASM Instance+ASM1 ASM Instance+ASM1 ASM Instance+ASM2 ASM Instance+ASM2 Oracle ClusterWareOracle Enterprise LinuxOracle ClusterWareOracle Enterprise LinuxOracle ClusterWareOracle Enterprise LinuxOracle ClusterWareOracle Enterprise LinuxGuest Domain 2 Domain-0 Guest Domain 1 Guest Domain 2 Database 1 Database 2 RAC on OVM Production Deployment Two or more HBAs need for redundancy HBAs should be configured as multipath-ed devices in Dom0 Only physical disks based VBD are supported File or block based storage configuration okay for single instance Configure device persistency in Dom0 Generally provided by multipathing driver If not available, use udev in DomU Optionally configure physical block devices (in DomU) as ASMLIB disksRAC on OVM Disk Configuration# xen config file example for RAC Guest Domainname = vmrac1"memory = 8192"disk = [ 'phy:/dev/mapper/mpath3p1,xvda,w!]

6 ','phy:/dev/mapper/mpath4p1,xvdb,w!','ph y:/dev/mapper/mpath5p1,xvdc,w!',]vif = ['mac=00:16:3E:00:00:08, bridge=xenbr0','mac=00:16:3E:10:A5:96, bridge=xenbr1',]vfb = ["type=vnc,vncunused=1"]uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728"bo otloader="/usr/bin/pygrub"vcpus=8on_rebo ot = 'restart'on_crash = 'restart'RAC on OVM Disk Configuration D o m a i n - 0O r a c l e V MG u e s t 1G u e s t 2p o r t - 0p o r t - 1S A N S t o r a g eP r o c e s s o r 0P r o c e s s o r 1S A N S w i t c h 1S A N S w i t c h 2M u l t i p a t h i n g D r i v e rV B D D r i v e rV B D D r i v e rL U N - 1/ d e v / s d a/ d e v / x v d a/ d e v / x v d aS A N H o s t B u s A d a p t e rA S M I n s t a n c e+ A S M 1A S M I n s t a n c e+ A S M

7 2 RAC on OVM ASMDisk Configuration Domain-0 Oracle VMGuest 1 Guest 2port-0port-1 SAN StorageProcessor 0 Processor 1 SAN Switch 1 SAN Switch 2 Multipathing DriverVBD DriverVBD DriverLUN-1/dev/sda/dev/xvda/dev/xvdaSAN Host Bus AdapterOCFS filesystemRAC on OVM OCFSDisk Configuration Use dedicated network devices For each host a minimum of two 1Gb Ethernet NICs is recommended For production RAC environments provide network traffic separation for public and private Extra Ethernet NICs required if network based storage connectivity (iSCSI or NFS) is used These networks can be shared with multiple RAC VM guests, but network bandwidth and capacity needs to reviewed For complete network redundancy, 4-6 NICs recommended Bond ethNinterfaces in Dom0 for HA and/or bandwidthRAC on OVM Network Configuration# xen config file example for RAC Guest Domainname = vmrac1"memory = 8192"disk = [ 'phy:/dev/mapper/mpath3p1,xvda,w!]

8 ','phy:/dev/mapper/mpath4p1,xvdb,w!','ph y:/dev/mapper/mpath5p1,xvdc,w!',]vif = ['mac=00:16:3E:00:00:08, bridge=xenbr0','mac=00:16:3E:10:A5:96, bridge=xenbr1',]vfb = ["type=vnc,vncunused=1"]uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728"bo otloader="/usr/bin/pygrub"vcpus=8on_rebo ot = 'restart'on_crash = 'restart'RAC on OVM Network Configuratione t h 0e t h 1e t h 2e t h 3x e n b r 0x e n b r 1e t h 0e t h 1e t h 0e t h 1b o n d 0b o n d 1D o m a in - 0P u b l ic n e t w o r kP u b lic n e t w o r kP r iv a t e n e t w o r kP r iv a t e n e t w o r kO r a c le V MG u e s t 1G u e s t 2P u b lic N e t w o r kS w it c h 1P u b lic N e t w o r kS w it c h 2P r iv a t e N e t w o r kS w it c h 1P r iv a te N e t w o r kS w it c h 2 RAC on OVM

9 Network Configurationeth0eth1eth2eth3xenbr0xenbr 1eth0eth1eth0eth1bond0bond1 Domain-0 Public networkPublic networkPrivate networkPrivate networkOracle VMGuest 1 Guest 2 Public NetworkSwitch 1 Public NetworkSwitch 2 Private NetworkSwitch 1 Private NetworkSwitch 2eth4eth5xenbr2bond2 Storage NetworkSwitch 1 Storage NetworkSwitch 2eth2eth2 Storage networkStorage networkNetApp :/volume01/u01/oradata/u01/oradataRAC on OVM NFS based Virtual cluster Disk Configuration Make sure device persistancy is correctly setup with udev in Dom0 Make sure permissions are set persistently and correct using udev in DomU Network Configuration Make sure bridges are configured correctly in Dom0 Use brctl showto verify Ensure firewall in DomU does not affect interconnect traffic The default firewall in the Oracle supplied Linux templates blocks RAC traffic.

10 Either make it a trusted device:iptables A input i ethN j ACCEPT, or disable firewall Typical Installation Issues[root@strab03 ~]# xm listName ID Mem VCPUs State Time(s)Domain-0 0 751 4 r----- 110240 2 -b---- 210240 2 -b---- [root@strab03 ~]# brctl showbridge name bridge id STP enabled no no Example for having both RAC guests on one server. Make sure the same bridges, using the same physical networks are used in both RAC Network Setup Each RAC node must be deployed on a separate physical server for production environments Do not over-commit CPUs Configure disk multi-pathing and NIC bonding i