这是参考ORACLEFC+ASM+UDEV+FAILGROUP，自已搭建的基于ORACLE12.2.0.1.0，用两台主机加VM的5台虚拟机实现该个实验搭建。
Redhatextentrac一.环境规划主机IP，主机名hostnamepublic_ippriv-ipvipsidscan-ipexrac1192.168.0.70192.168.4.30192.168.0.71extora1192.168.0.74exrac2192.168.0.75192.168.4.31192.168.0.73extora2192.168.0.74ASM磁盘组DATA（10G)FRA(5G)REDO(2G)OCR(8G)GIMR(40G)T_OCR(8G)FGADATA_AFRA_AREDO_AOCR_AGIMR_AT_OCRFGBDATA_BFRA_BREDO_BOCR_BGIMR_BFGQ/OCR/VOTE_DISK数据文件归档文件日志文件OCR文件资料管理库临时OCR盘存储规划存储A（内网IP192.168.4.40,外网问IP192.168.0.40）
卷容量备注OCR_A8GFRA_A5GREDO_A2GGIMR_A40GDATA_A10GT_OCR8G存储B（内网IP192.168.4.41,外网问IP192.168.0.42）卷容量备注OCR_B8GFRA_B5GREDO_B2GIMR_B40GDATA_B10G
NFS卷容量备注/ocr/vote_disk8g二、存储部置1.存储ABLOCKDEVICEMANAGERMENT
VOLUMEGROUPMANAGEMENT
T_OCR
OCR_A
GIMR-A
ManageServices
把ISCSCITARGET点击modifyboot的ENABLE把CURRENTSTATUS置为RUNNING也就是点击STARTTARGETCONFIGURATION
这里存储A为iqn.2020-08.com.openfiler:tsn.extend1LUNSMAPPING
2.存储BBLOCKDeviceManagement
gimr-b
data-b
fra-b
ocr-b
NETWORKACCESSCONFIGURATION
这里的网段为192.168.0.0，192.168.4.0
这里存储B为iqn.2020-08.com.openfiler:tsn.extend2
三、在两台服务器连接二台存储1.服务器A连接存储A
vi/etc/iscsi/iscsid.confnode.session.auth.authmethod=CHAP
node.session.auth.username=rootnode.session.auth.password=password123保存退出后重启一下ISCISID服务systemctlrestartiscsid.service
iscsiadm-mdiscovery-tsendtargets-p192.168.4.40
iscsiadm-mnode-Tiqn.2020-08.com.openfiler:tsn.extent1-p192.168.4.40:3260-l然后fdisk-l
2.服务器A连接存储B
iscsiadm-mdiscovery-tsendtargets-p192.168.4.41
iscsiadm-mnode-Tiqn.2020-08.com.openfiler:tsn.extent2-p192.168.4.41：3260-l然后fdisk-l
3.服务器B连接存储A
iscsiadm-mnode-Tiqn.2020-08.com.openfiler:tsn.extend1-p192.168.4.40:3260-l4.服务器B连接存储B
iscsiadm-mnode-Tiqn.2020-8.com.openfiler:tsn.extend2-p192.168.4.41:3260-l然后fdisk-l四修改/etc/hostsvi/etc/hosts
192.168.0.70exrac1192.168.0.75exrac2
192.168.0.71exrac1-vip192.168.0.73exrac2-vip
192.168.4.40exrac1-priv192.168.4.41exrac2-priv
192.168.0.79exrac-cluster-scan五、内核参数文件vi/etc/sysctl.conffs.aio-max-nr=1048576fs.file-max=6815744
kernel.shmmax=7730941132
kernel.shmall=1887436kernel.shmmni=4096kernel.sem=25032000100128net.ipv4.ip_local_port_range=900065500net.ipv4.tcp_rmem=4096873804194304net.ipv4.tcp_wmem=4096163844194304net.core.rmem_default=262144net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=1048576
然后sysctl-p六、修改资源限制文件vi/etc/security/limits.confgridsoftnproc2047gridhardnproc16384gridsoftnofile1024gridhardnofile65536gridsoftstack10240gridhardstack32768oraclesoftnproc2047oraclehardnproc16384oraclesoftnofile1024oraclehardnofile65536oraclesoftstack10240oraclehardstack32768七、关闭防火墙systemctlstopfirewalld.servicesystemctldisablefirewalld.service八、关闭SELINUXvi/etc/selinux/configselinux=disabled九配置MTU参数（心跳网络将mtu调大至9000）：vi/etc/sysconfig/network-scripts/ifcfg-ens39MTU=9000十配置YUM源mkdir/mnt/systemmount-olooprhel-server-7.3-x86_64-dvd.iso/mnt/systemcd/etc/yum.repos.d/virhel.repo[rhel7]name=rhel7baseurl=file:///mnt/systemenabled=1gpgcheck=0
修改共享内存分区大小将如下参数添加到/etc/fstab，具体大小数值根据实际情况调整，因为这个数值和物理内存以及MEMORY_TARGET有关。echoshm/dev/shmtmpfssize=12g00>>/etc/fstab修改后，只需重新对shm进行挂载即可：mount-oremount/dev/shm
十一通过YUM命令安装RPMyum-yinstallbcbinutilscompat-libcap1compat-libstdc++-33compat-libstdc++-33.i686e2fsprogse2fsprogs-libsglibcglibc.i686glibc-develglibc-devel.i686libaiolibaio.i686libaio-devellibaio-devel.i686libX11libX11.i686libXaulibXau.i686libXilibXi.i686libXtstlibXtst.i686libgcclibgcc.i686libstdc++libstdc++.i686libstdc++-devellibstdc++-devel.i686libxcblibxcb.i686makesmartmontoolssysstatunziplrzszopenssh-clientslibXrender-devel.i686libXrender-devel.x86_64gccgcc-c++kshelfutils-libelf-devel.x86_64十二。创建用户groupadd-g500oinstallgroupadd-g501dbagroupadd-g502opergroupadd-g503asmadmingroupadd-g504asmdbagroupadd-g505asmoperuseradd-u501-goinstall-Gasmadmin,asmdba,asmoper,oper,dbagriduseradd-u502-goinstall-Gdba,asmdba,asmadminoraclepasswdgridpasswdoracle
十四、禁用NTPsystemctlstopntpd.servicesystemctldisablentpd.servicemv/etc/ntp.conf/etc/ntp.conf.original
关闭LINUX透明大页修改/etc/default/grub文件,如下是修改之后的效果：[root@rac1default]#cat/etc/default/grubGRUB_TIMEOUT=5GRUB_DISTRIBUTOR="$(sed's,release.*$,,g'/etc/system-release)"GRUB_DEFAULT=savedGRUB_DISABLE_SUBMENU=trueGRUB_TERMINAL_OUTPUT="console"GRUB_CMDLINE_LINUX="rhgbquiettransparent_hugepage=never"---->>>transparent_hugepage=never是新加的GRUB_DISABLE_RECOVERY="true"[root@rac1default]#grub2-mkconfig-o/boot/grub2/grub.cfg---->>执行该命令关闭transparent_hugepage的配置
iftest-f/sys/kernel/mm/redhattransparenthugepage/enabled;thenechonever>/sys/kernel/mm/redhattransparenthugepage/enabledfi修改NSSWITCH.CONFvi/etc/nsswitch.conf修改行hosts:filesdns为hosts:filesdnsnis配置NOZEROCONFvi/etc/sysconfig/network增加以下内容NOZEROCONF=yes检查cat/sys/block/sda/queue/schedulernoop[deadline]cfq十五停止avahi-daemon服务
十八多路径配置
rpm-qa|grepdevice-mapperyuminstalldevice-mapper-y
modprobedm-multipathmodprobedm-round-robinlsmod|grepmultipath
systemctlenablemultipathd.servicesystemctlstartmultipathd.service
mpathconf--enable--find_multipathsy--with_moduley--outfile/etc/multipath.confmultipath-v0加载
foriincdefghijklmnop;doecho"KERNEL"sd*",BUS"scsi",PROGRAM"/lib/udev/scsi_id--whitelisted--replace-whitespace--device=/dev/$name",RESULT"/lib/udev/scsi_id--whitelisted--replace-whitespace--device=/dev/sd$i",NAME="asm-disk$i",OWNER="grid",GROUP="asmadmin",MODE="0660""done
vi/etc/multipath/bindingsmpatha14f504e46494c45526a635065336d2d654f76762d41517564mpathb14f504e46494c455269734d7856772d466d79472d6e47394bmpathc14f504e46494c4552483046696e432d6e5350772d47527768mpathd14f504e46494c45526856784562792d304763722d586a4776mpathe14f504e46494c45527066467151652d6377705a2d744b6670mpathf14f504e46494c45524750494269302d6c4735332d7a744547mpathg14f504e46494c45526431566d78322d633965502d767a517ampathh14f504e46494c4552337256367a702d52436f692d784a4177mpathi14f504e46494c45525137537a67352d6b6633472d4343477a
vi/etc/multipath/wwids/14f504e46494c45526a635065336d2d654f76762d41517564//14f504e46494c455269734d7856772d466d79472d6e47394b//14f504e46494c4552483046696e432d6e5350772d47527768//14f504e46494c45526856784562792d304763722d586a4776//14f504e46494c45527066467151652d6377705a2d744b6670//14f504e46494c45524750494269302d6c4735332d7a744547//14f504e46494c45526431566d78322d633965502d767a517a//14f504e46494c4552337256367a702d52436f692d784a4177//14f504e46494c45525137537a67352d6b6633472d4343477a
配置多路径文件blacklist{devnode"^sda"}multipaths{multipath{wwid14f504e46494c45527066467151652d6377705a2d744b6670aliasFRA_Auid0mode0600}multipath{wwid14f504e46494c45526856784562792d304763722d586a4776aliasDATA_Auid0mode0600}multipath{wwid14f504e46494c4552483046696e432d6e5350772d47527768aliasGIMR_Auid0mode0600}multipath{wwid14f504e46494c455269734d7856772d466d79472d6e47394baliasOCR_Auid0mode0600}multipath{wwid14f504e46494c45526a635065336d2d654f76762d41517564aliasT_OCRuid0mode0600}multipath{wwid14f504e46494c45525137537a67352d6b6633472d4343477aaliasOCR_Buid0mode0600}multipath{wwid14f504e46494c4552337256367a702d52436f692d784a4177aliasFRA_Buid0mode0600}multipath{wwid14f504e46494c45526431566d78322d633965502d767a517aaliasDATA_Buid0mode0600}multipath{wwid14f504e46494c45524750494269302d6c4735332d7a744547aliasGIRM_Buid0mode0600}}
/sbin/udevadmtrigger--type=devices--action=change/sbin/udevadmcontrol--reload
二十创建ASM磁盘安装ASMLIB（PRM包i在/opt目录）：rpm-ivhkmod-oracleasm-2.0.8-17.el7.x86_64.rpmrpm-ivhoracleasm-support-2.1.8-3.el7.x86_64.rpmrpm-ivhoracleasmlib-2.0.12-1.el7.x86_64.rpm配置asmlib/etc/init.d/oracleasmconfigure-iConfiguringtheOracleASMlibrarydriver.
Thiswillconfiguretheon-bootpropertiesoftheOracleASMlibrarydriver.Thefollowingquestionswilldeterminewhetherthedriverisloadedonbootandwhatpermissionsitwillhave.Thecurrentvalueswillbeshowninbrackets('[]').Hittingwithouttypingananswerwillkeepthatcurrentvalue.Ctrl-Cwillabort.
Defaultusertoownthedriverinterface[]:gridDefaultgrouptoownthedriverinterface[]:asmadminScanforOracleASMdisksonboot(y/n)[y]:yWritingOracleASMlibrarydriverconfiguration:doneInitializingtheOracleASMLibdriver:[OK]ScanningthesystemforOracleASMLibdisks:[OK]创建ASMDISK:[root@exrac2opt]#oracleasmdeletediskDATA_AClearingdiskheader:doneDroppingdisk:done[root@exrac2opt]#oracleasmcreatediskDATA_A/dev/mapper/DATA_AWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskDATA_B/dev/mapper/DATA_BWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskFRA_A/dev/mapper/FRA_AWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskFRA_B/dev/mapper/FRA_BWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskOCR_A/dev/mapper/OCR_AWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskOCR_B/dev/mapper/OCR_B\Writingdiskheader:doneInstantiatingdisk:done
[root@exrac2opt]#oracleasmcreatediskGIMR_A/dev/mapper/GIMR_AWritingdiskheader:doneInstantiatingdisk:done[root@exrac2opt]#oracleasmcreatediskGIMR_B/dev/mapper/GIMR_BWritingdiskheader:doneInstantiatingdisk:done
[root@exrac2opt]#oracleasmcreatediskT_OCR/dev/mapper/T_OCRWritingdiskheader:done
[root@exrac2opt]#oracleasmlistdisksDATA_ADATA_BFRA_AFRA_BGIMR_AGIMR_BOCR_AOCR_BREDO_AREDO_BT_OCR
[root@exrac1opt]#oracleasmscandisksReloadingdiskpartitions:doneCleaninganystaleASMdisks...ScanningsystemforASMdisks...Instantiatingdisk"DATA_A"Instantiatingdisk"REDO_A"Instantiatingdisk"GIMR_A"Instantiatingdisk"T_OCR"Instantiatingdisk"FRA_A"Instantiatingdisk"OCR_A"Instantiatingdisk"DATA_B"Instantiatingdisk"REDO_B"Instantiatingdisk"GIMR_B"Instantiatingdisk"FRA_B"Instantiatingdisk"OCR_B"[root@exrac1opt]#oracleasmlistdisksDATA_ADATA_BFRA_AFRA_BGIMR_AGIMR_BOCR_AOCR_BREDO_AREDO_BT_OCR
二十一、挂载NFS共享盘
vi/etc/exports/ocr192.168.0.0/24(rw,sync,root_squash)
Exportlistfor192.168.0.43:
mkdir/ocrmount-tnfs4192.168.0.43:/ocr/ocrvi/etc/fstab192.168.0.43:/ocr/ocrnfs4rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=60000
检查配置环境/u01/app/grid/runcluvfy.shstage-precrsinst-nexrac1,exrac2-fixup-verbose对GRID进行安装
[grid@exrac1grid]$exportDISPLAY=192.168.0.123:0.0[grid@exrac1grid]$./gridSetup.sh-applyOneOffs/opt/29301687/29301676/
设置暂时OCR
先点击SpecifyFailureGroups，配置FGA,FGB两个故障组
这里密码为gridpass
在root用户下依次运行ORAINSTROOT.sh和root.sh
ora....SM.lsnrora....er.typeONLINEONLINEexrac1ora.GIMR.dgora....up.typeONLINEONLINEexrac1ora....ER.lsnrora....er.typeONLINEONLINEexrac1ora....AF.lsnrora....er.typeOFFLINEOFFLINEora....N1.lsnrora....er.typeONLINEONLINEexrac1ora.MGMTLSNRora....nr.typeONLINEONLINEexrac1ora.OCR.dgora....up.typeONLINEONLINEexrac1ora.asmora.asm.typeONLINEONLINEexrac1ora.chadora.chad.typeONLINEONLINEexrac1ora.cvuora.cvu.typeONLINEONLINEexrac1ora....C1.lsnrapplicationONLINEONLINEexrac1ora.exrac1.onsapplicationONLINEONLINEexrac1ora.exrac1.vipora....t1.typeONLINEONLINEexrac1ora....C2.lsnrapplicationONLINEONLINEexrac2ora.exrac2.onsapplicationONLINEONLINEexrac2ora.exrac2.vipora....t1.typeONLINEONLINEexrac2ora.mgmtdbora....db.typeONLINEONLINEexrac1ora....networkora....rk.typeONLINEONLINEexrac1ora.onsora.ons.typeONLINEONLINEexrac1ora.proxy_advmora....vm.typeOFFLINEOFFLINEora.qosmserverora....er.typeONLINEONLINEexrac1ora.scan1.vipora....ip.typeONLINEONLINEexrac1配置ASM磁盘组
grid@exrac1~]$sqlplus/assysasmSQL>creatediskgroupDATANORMALREDUNDANCY2FAILGROUPFG_ADISK'/dev/mapper/DATA_A'3FAILGROUPFG_BDISK'/dev/mapper/DATA_B'4ATTRIBUTE'au_size'='1M',5'compatible.asm'='12.2',6'compatible.rdbms'='12.2';
SQL>creatediskgroupFRANORMALREDUNDANCY2FAILGROUPFG_ADISK'/dev/mapper/FRA_A'3FAILGROUPFG_BDISK'/dev/mapper/FRA_B'4ATTRIBUTE'au_size'='1M',5'compatible.asm'='12.2',6'compatible.rdbms'='12.2';SQL>CREATEDISKGROUPOCR2NORMALREDUNDANCY2FAILGROUPFG_ADISK'/dev/mapper/OCR_A'3FAILGROUPFG_BDISK'/dev/mapper/OCR_B'4ATTRIBUTE'au_size'='1M',5'compatible.asm'='12.2',6'compatible.rdbms'='12.2';安装ORACLE软件[oracle@exrac1database]$exportDISPLAY=192.168.0.121:0.0[oracle@exrac1database]$./runInstallerStartingOracleUniversalInstaller...
执行ROOT.SH
[root@exrac1~]#/u01/app/oracle/product/db_home/root.shPerformingrootuseroperation.
Thefollowingenvironmentvariablesaresetas:ORACLE_OWNER=oracleORACLE_HOME=/u01/app/oracle/product/db_home
Enterthefullpathnameofthelocalbindirectory:[/usr/local/bin]:Thecontentsof"dbhome"havenotchanged.Noneedtooverwrite.Thecontentsof"oraenv"havenotchanged.Noneedtooverwrite.Thecontentsof"coraenv"havenotchanged.Noneedtooverwrite.
Entrieswillbeaddedtothe/etc/oratabfileasneededbyDatabaseConfigurationAssistantwhenadatabaseiscreatedFinishedrunninggenericpartofrootscript.Nowproduct-specificrootactionswillbeperformed.[root@exrac2~]#/u01/app/oracle/product/db_home/root.shPerformingrootuseroperation.
Entrieswillbeaddedtothe/etc/oratabfileasneededbyDatabaseConfigurationAssistantwhenadatabaseiscreatedFinishedrunninggenericpartofrootscript.Nowproduct-specificrootactionswillbeperformed.