数据库RAC添加新节点的操作步骤
安装pdsrac的新节点
1.在两个节点分别安装vnc
wget /dist/vnc-4.0-x86_linux.tar.gz
tar -xzvf vnc-4.0-x86_linux.tar.gz
./vncinstall /usr/local/bin
mkdir -p /usr/local/vnc/classes
cp java/* /usr/local/vnc/classes
vncserver
2.添加网卡
参数说明：
DEVICE=物理设备名
IPADDR=IP地址
NETMASK=掩码值
NETWORK=网络地址
BROADCAST=广播地址
GATEWAY=网关地址
ONBOOT=[yes|no]（引导时是否激活设备）
USERCTL=[yes|no]（非root用户是否可以控制该设备）
BOOTPROTO=[none|static|bootp|dhcp]（引导时不使用协议|静态分配|BOOTP协议|DHCP协议）
HWADDR = 你的MAC地址
在/etc/sysconfig/network-scripts/下创建虚拟网卡bond0的配置文件ifcfg-bond0
cd /etc/sysconfig/network-scripts/
vi ifcfg-bond1
DEVICE=bond1
BOOTPROTO=none
BORADCAST=192.168.220.255
IPADDR=192.168.220.12
NETMASK=255.255.255.0
NETWORK=192.168.220.0
ONBOOT=yes
TYPE=Ethernet
分别修改原来网卡配置文件的信息，删除其中的IP 地址、子网掩码等信息
vi ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=yes
vi ifcfg-eth1
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=yes
vi ifcfg-eth2
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet DEVICE=eth2
HWADDR=00:24:E8:56:4E:88
ONBOOT=yes
BOOTPROTO=none
MASTER=bond1
SLAVE=yes
USERCTL=yes
HOTPLUG=no
vi ifcfg-eth3
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet DEVICE=eth3
HWADDR=00:24:E8:56:4E:8A
ONBOOT=yes
MASTER=bond1
USERCTL=yes
SLAVE=yes
BOOTPROTO=none
HOTPLUG=no
3.配置驱动模块参数：/etc/modprobe.conf
alias bond0 bonding
options bond0 miimon=100 mode=1
alias eth0 bnx2x
alias eth1 bnx2x
alias bond1 bonding
options bond1 miimon=100 mode=1
alias eth2 bnx2x
alias eth3 bnx2x
aliasscsi_hostadaptercciss
aliasusb-controller uhci-hcd
alias usb-controller1 ehci-hcd
alias scsi_hostadapter1 qla2xxx
optionshangcheck-timer hangcheck_tick=30 hangcheck_margin=180
说明：
（1）、miimon是链路监测的时间间隔单位是毫秒，miimon=100的意思就是，每100毫秒检测网卡和交换机之间是否连通，如不通则使用另外的链路。
（2)、mode=0 表示负载均衡方式，两块网卡都工作，需要交换机作支持
mode=1 表示冗余方式，网卡只有一个工作，一个出问题启用另外的
mode=6 表示负载均衡方式，两块网卡都工作，不需要交换机作支持
设置开机绑定
[root@BJ-FT-1F-120-12 etc]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/usr/sbin/ntpdate &
/sbin/modprobehangcheck_timer
/usr/local/sbin/oraraw.sh
service network restart
ifenslave bond eth2 eth3
modprobehangcheck-timer hangcheck_tick=30 hangcheck_margin=180
重启网络
service network restart
cat /proc/net/bonding/bond0
cat /proc/net/bonding/bond1
4.创建用户
groupadd -g 200 dba
groupadd -g 201 oinstall
useradd -u 300 -g oinstall -G dba -d /home/oracle -s /bin/bash -m oracle
vi /home/oracle/.bash_profile
export ORACLE_BASE=/apps/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0
export CRS_HOME=$ORACLE_BASE/crs
export
PATH=$CRS_HOME/bin:$ORACLE_HOME/jdk/bin:/usr/kerberos/bin:/usr/local/bin:/bin:$PATH:/us r/bin:/usr/X11R6/bin:/home/oracle/bin:$ORACLE_HOME/bin
export ORACLE_OWNER=oracle
export LD_LIBRARY_PATH=:$ORACLE_HOME/Slib
export NLS_LANG=American_america.zhs16gbk
export ORACLE_TERM=xterm
export AGENT_HOME=$ORACLE_BASE/agent10g
aliasmsh='cd $ORACLE_BASE/admin/msh'
export ORACLE_SID=pdsdb1
5.修改hosts
vi /etc/hosts
192.168.120.11 BJ-FT-1F-120-11
192.168.120.12 BJ-FT-1F-120-12
192.168.120.13 BJ-FT-1F-120-13
192.168.120.14 BJ-FT-1F-120-14
192.168.120.21 ftpds_vip1
192.168.120.22 ftpds_vip2
192.168.120.23 ftpds_vip3
192.168.120.24 ftpds_vip4
192.168.220.21 ftpds_priv1
192.168.220.22 ftpds_priv2
192.168.220.23 ftpds_priv3
192.168.220.24 ftpds_priv4
6.配置信任关系
[oracle@BJ-FT-1F-120-12 ~]$
mkdir ~/.ssh
chmod 755 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
[root@BJ-FT-1F-120-11 ~]#
cp /home/oracle/.ssh/authorized_keys rac1.authorized_keys
scp rac1.authorized_keys shexiang@192.168.106.10:/192.168.120.12/root/home/oracle
[root@BJ-FT-1F-120-12 ~]#
chownoracle:dba rac1.authorized_keys
[oracle@BJ-FT-1F-120-12 ~]$
mv rac1.authorized_keys .ssh/
cd .ssh/
mv rac1.authorized_keys authorized_keys
cat id_rsa.pub >>authorized_keys
cat id_dsa.pub >>authorized_keys
cpauthorized_keys rac2.authorized_keys
[root@BJ-FT-1F-120-12 .ssh]#
scp /home/oracle/.ssh/rac2.authorized_keys shexiang@192.168.106.10:/192.168.120.14/root/home/oracle
[oracle@BJ-FT-1F-120-14 ~]#
chownoracle:dba /home/oracle/rac2.authorized_keys
[oracle@BJ-FT-1F-120-14 ~]$
mkdir ~/.ssh
chmod 755 ~/.ssh
/usr/bin/ssh-keygen -t rsa
/usr/bin/ssh-keygen -t dsa
mv rac2.authorized_keys .ssh/
cd .ssh/
mv rac2.authorized_keys authorized_keys
cat id_rsa.pub >>authorized_keys
cat id_dsa.pub >>authorized_keys
cpauthorized_keys rac4.authorized_keys
[oracle@BJ-FT-1F-120-14 ~]#
scp /home/oracle/.ssh/rac2.authorized_keys shexiang@192.168.106.10:/192.168.120.11/root/home/oracle
scp /home/oracle/.ssh/rac2.authorized_keys shexiang@192.168.106.10:/192.168.120.12/root/home/oracle
scp /home/oracle/.ssh/rac2.authorized_keys shexiang@192.168.106.10:/192.168.120.13/root/home/oracle
[root@BJ-FT-1F-120-11 ~]#
chownoracle:dba rac4.authorized_keys
mv rac2.authorized_keys .ssh/
cd .ssh/
cpauthorized_keysauthorized_keys.old
mv rac2.authorized_keys authorized_keys
在每一个节点执行
ssh BJ-FT-1F-120-11 date
ssh BJ-FT-1F-120-12 date
ssh BJ-FT-1F-120-13 date
ssh BJ-FT-1F-120-14 date
ssh ftpds_priv1 date
ssh ftpds_priv2 date
ssh ftpds_priv3 date
ssh ftpds_priv4 date
7.修改核心参数
vi /etc/sysctl.conf
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
kernel.core_uses_pid = 1
kernel.panic = 1
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 9000 65000
# For 11g, recommended value for file-max is 6815744
# For 10g, uncomment 'fs.file-max 327679', comment other entries for this parameter and re-run sysctl -p
# fs.file-max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 1048576
# For 11g, recommended value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this parameter and re-run sysctl -p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this parameter and re-run sysctl -p
# net.core.wmem_max:262144
fs.aio-max-nr = 3145728
# For 11g, recommended value for ip_local_port_range is 9000 65500
# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for this parameter and re-run sysctl -p
# net.ipv4.ip_local_port_range:1024 65000
sysctl -p 立刻生效
vi /etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
vi /etc/pam.d/login
session required pam_limits.so
vi /etc/rc.d/rc.local
/sbin/modprobehangcheck_timer
设置hangcheck内核模块参数：
vi /etc/modprobe.conf
optionshangcheck-timer hangcheck_tick=30 hangcheck_margin=180 设置hangcheck内核模块开机加载
vi /etc/rc.d/rc.local
/sbin/modprobehangcheck-timer
要立即加载模块，执行：
modprobe -v hangcheck-timer
检查加载情况：
[root@zhh1 ~]# lsmod | grephangcheck_timer
hangcheck_timer 5593 0
8.安装ASM包，配置ASM，并搜索ASM
--在rac3上安装ASMLibs, tools, support 三个rpm文件
# rpm -ivh *.rpm --nodeps --force
-- 运行/etc/init.d/oracleasm configure
回答oracle , dba, y, y 就可以了
vi /usr/local/sbin/oraraw.sh
#####
# Oracle Cluster Registry (OCR) devices
#####
chownroot:oinstall /dev/mapper/ocr
chmod 660 /dev/mapper/*
raw /dev/raw/raw1 /dev/mapper/ocr
sleep 2
chownroot:oinstall /dev/raw/raw1
chmod 660 /dev/raw/raw1
#####
# Oracle Cluster Voting disks
#####
chownoracle:oinstall /dev/mapper/vtdsk
raw /dev/raw/raw2 /dev/mapper/vtdsk
sleep 2
chownoracle:oinstall /dev/raw/raw2
chownoracle:oinstall /dev/mapper/arch
chownoracle:oinstall /dev/mapper/data*
chownoracle:oinstall /dev/mapper/redo*
9.安装CRS
cd /apps/software/clusterware/cluvfy
./cluvfy stage -post hwos -n BJ-FT-1F-120-11,BJ-FT-1F-120-13
./cluvfy stage -post hwos -n BJ-FT-1F-120-11,BJ-FT-1F-120-12,BJ-FT-1F-120-13,BJ-FT-1F-120-14
./runcluvfy.sh stage -pre crsinst -n BJ-FT-1F-120-11,BJ-FT-1F-120-12,BJ-FT-1F-120-13,BJ-FT-1F-120-14
./runInstaller -updateNodeList "CLUSTER_NODES={BJ-FT-1F-120-11,BJ-FT-1F-120-13}" LOCAL_NODE="BJ-FT-1F-120-11" "ORACLE_HOME=$ORACLE_HOME" -local
. 为新节点添加CRS Home
启动vnc http://192.168.120.13:5801/
xhost+
su - oracle
export LANG=en
在【BJ-FT-1F-120-13】上运行CRS Home下的addNode.sh，启动OUI添加节点。
原计划在BJ-FT-1F-120-11上执行，但是上面报错，报错内容为
SEVERE: OUI-25023: The local node is not selected for installing this product. Include the local node in the cluster list or perform t
he installation on the nodes on which the install is to be performed.
未找到解决办法，在节点13上执行通过。
cd /apps/oracle/crs/oui/bin
./addNode.sh
按照提示填写节点3公网私网虚拟名称
执行安装
分别执行3个脚本
在节点BJ-FT-1F-120-12运行orainstRoot.sh
/apps/oracle/oraInventory/orainstRoot.sh
在节点BJ-FT-1F-120-13运行rootaddnode.sh
/apps/oracle/crs/install/rootaddnode.sh
最后会报错
/apps/oracle/crs/bin/srvctl add nodeapps -n bj-ft-1f-120-12 -A BJ-FT-1F-120-12-vip/255.255.255.0/bond0 -o /apps/oracle/crs
PRKO-2109 : 地址字符串无效: BJ-FT-1F-120-12-vip/255.255.255.0/bond0
手动执行
/apps/oracle/crs/bin/srvctl add nodeapps -n bj-ft-1f-120-12 -A ftpds_vip2/255.255.255.0/bond0 -o /apps/oracle/crs
最后在节点BJ-FT-1F-120-12运行/apps/oracle/crs/root.sh
[root@BJ-FT-1F-120-12 raw]# /apps/oracle/crs/root.sh
WARNING: directory '/apps/oracle' is not owned by root
No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS = /dev/raw/raw1
OCR backup directory '/apps/oracle/crs/cdata/crs' does not exist. Creating now Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/apps/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node<nodenumber>: <nodename><private interconnect name><hostname> node 1: bj-ft-1f-120-11 ftpds_priv1 bj-ft-1f-120-11
node 2: bj-ft-1f-120-13 ftpds_priv3 bj-ft-1f-120-13
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
bj-ft-1f-120-11
bj-ft-1f-120-13
bj-ft-1f-120-12
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
检查su - oracle
crs_stat -t
Name Type Target State Host ------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINEbj-f...0-11 ora....11.lsnr application ONLINE ONLINEbj-f...0-11 ora....-11.gsd application ONLINE ONLINEbj-f...0-11 ora....-11.ons application ONLINE ONLINEbj-f...0-11 ora....-11.vip application ONLINE ONLINEbj-f...0-11 ora....-12.gsd application ONLINE ONLINEbj-f...0-12 ora....-12.ons application ONLINE ONLINEbj-f...0-12 ora....-12.vip application ONLINE ONLINEbj-f...0-12 ora....SM2.asm application ONLINE ONLINEbj-f...0-13 ora....13.lsnr application ONLINE ONLINEbj-f...0-13 ora....-13.gsd application ONLINE ONLINEbj-f...0-13 ora....-13.ons application ONLINE ONLINEbj-f...0-13 ora....-13.vip application ONLINE ONLINEbj-f...0-13
ora.pdsdb.db application ONLINE ONLINEbj-f...0-11 ora...._taf.cs application ONLINE ONLINEbj-f...0-13 ora....db1.srv application ONLINE ONLINEbj-f...0-11 ora....db2.srv application ONLINE ONLINEbj-f...0-13 ora....b1.inst application ONLINE ONLINEbj-f...0-11 ora....b2.inst application ONLINE ONLINEbj-f...0-13
10.添加ONS远程端口
[oracle@BJ-FT-1F-120-12 ~]$ ocrdump -stdout | grep ONS_HOSTS [DATABASE.ONS_HOSTS]
[DATABASE.ONS_HOSTS.bj-ft-1f-120-11]
[DATABASE.ONS_HOSTS.bj-ft-1f-120-11.PORT]
[DATABASE.ONS_HOSTS.bj-ft-1f-120-13]
[DATABASE.ONS_HOSTS.bj-ft-1f-120-13.PORT]
ocrdump -stdout -keyname 'DATABASE.ONS_HOSTS.bj-ft-1f-120-13.PORT' cd $CRS_HOME/bin
./racgonsadd_config bj-ft-1f-120-12:6200
ocrdump -stdout | grep ONS_HOSTS
11.为新节点添加RDBMS Home
[oracle@rac101 bin]$ cd $ORACLE_HOME/oui/bin
[oracle@rac101 bin]$ ./addNode.sh
[root@BJ-FT-1F-120-12 oracle]# /apps/oracle/product/10.2.0/root.sh
12.为新节点添加listener netca
只选择“rac103”
13.为新节点添加ASM 运行DBCA
crs_stat -t | grepasm
ora....SM1.asm application ONLINE ONLINEbj-f...0-11 ora....SM3.asm application ONLINE ONLINEbj-f...0-12 ora....SM2.asm application ONLINE ONLINEbj-f...0-13 ora....SM4.asm application ONLINE ONLINEbj-f...0-14 14.为新节点添加instence
这步稍慢，主要时间花在重新配置dbconsole。
耐心等待，结束之后就大功告成了。
最后，不要忘记重新注册一下本地监听，否则会出现问题的。
alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.120.22)(PORT=1521))' scope = both sid = 'pdsdb3';
alter system set local_listener = '(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.120.24)(PORT=1521))' scope = both sid = 'pdsdb4';
启动em检测，需要在各个节点启动
emctl startdbconsle。
