RHCS+oracle安装配置实施
一、Information
二、system base config：（every node）
Selinux off、firewall off、NetworkManager off、Hostname、IP、GATEWAY、hosts、yum等
A、seliunx off
vi /etc/selinux/config
SELINUX=disabled
B firewall off
chkconfig iptables off
service iptables stop
C、yum
/etc/yum.repos.d/cdrom.repo（使用目录下默认文件copy一份名为cdrom文件）[base]
name=cdrom
baseurl=file:///mnt/cdrom
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[HighAvailability]
name=HighAvailability
baseurl=file:///mnt/cdrom/HighAvailability
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[LoadBalancer]
name=LoadBalancer
baseurl=file:///mnt/cdrom/LoadBalancer
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[ResilientStorage]
name=ResilientStorage
baseurl=file:///mnt/cdrom/ResilientStorage
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
D、NetworkManager off （server noed除外）
chkconfig NetworkManager off
service NetworkManager stop
E、software install
yum -y installfence-* ricci cman rgmanager
cluster-glue resource-agentspacemakerlvm2-cluste （gfs2-utils 如需要可以安装）yum -y install luci（只在管理节点安装）
F、start service
chkconfig ricci on
chkconfig rgmanager on
chkconfig cman on
service ricci start
service rgmanager start
service cman start
只在管理节点设置启动
chkconfig luci on
service luci start
修改ricci用户的密码
Passwdricci,改为root用户密码即可(也可以使用其他密码)
G、安装IPMI相关的程序包
freeipmi-0.7.16-3.el6.x86_64.rpm
freeipmi-bmc-watchdog-0.7.16-3.el6.x86_64.
freeipmi-ipmidetectd-0.7.16-3.el6.x86_64.rpm
ipmitool-1.8.11-13.el6.x86_64.rpm
OpenIPMI-2.0.16-12.el6.x86_64.rpm
OpenIPMI-libs-2.0.16-12.el6.x86_64.rpm
建议升级freeipmi到下面这个版本
freeipmi-ipmidetectd-1.2.1-6.el6_5.x86_64
freeipmi-bmc-watchdog-1.2.1-6.el6_5.x86_64
freeipmi-1.2.1-6.el6_5.x86_64
H、安装多路径软件
yum -y install device-mapper*
chkconfig multipathd on
service multipathd start
使用多路径后硬盘设备会存放在/dev/mapper/下
使用 mpathconf --enable生成multipath.conf文件
使用multipath -F 删除现有路径
使用 multipath -v2 格式化路径
使用 multipath -ll 查看多路径
为了确保两台服务器存储硬盘ID与硬盘别名始终保持一致需要修改/etc/multipath.conf（两台服务器同步此文件，别名和ID对应关系可以使用multipath –ll查看到）
在multipaths 参数下添加
multipath {
wwid 360050768028191c2a00000000000004e
alias mpathb
}
multipath {
wwid 360050768028191c2a000000000000048
alias mpathc
}
multipath {
wwid 360050768028191c2a000000000000045
alias mpathd
}
multipath {
wwid 360050768028191c2a000000000000047
alias mpathe
}
multipath {
wwid 360050768028191c2a00000000000004c
alias mpathf
}
multipath {
wwid 360050768028191c2a000000000000046
alias mpathg
}
multipath {
wwid 360050768028191c2a000000000000049
alias mpathh
}
multipath {
wwid 360050768028191c2a00000000000004a
alias mpathi
}
multipath {
wwid 360050768028191c2a00000000000004b
alias mpathj
}
multipath {
wwid 360050768028191c2a00000000000004d
alias mpathk
}
multipath {
wwid 360050768028191c2a00000000000004f
alias mpathl
}
}
I、hosts
52.208.110.171 cgvch-podb1
52.208.110.172 cgvch-podb2
52.208.110.161 cgvch-admin
52.208.110.170 cgvchpodbvip
52.208.205.151 IMM2-podb1
52.208.205.152 IMM2-podb2
J、看一下node1能否管理node2电源模块
[root@host1~]# ipmitool-Ilan-H52.208.205.213-UUSERID-PPASSW0RDpowerstatus Chassis Power is on
K、创建QDISK盘
mkqdisk -c /dev/mapper/mpathl -l qdisk
mkqdisk -L 查看仲裁盘
chkconfig quota_nld on
service quota_nld start
三、RHCS cluster config
1、登录lucihttps://IP:8084
使用系统用户名root登录
2、添加节点名和成员(在添加节点是必须有yum及系统盘否则会提示无法安装包)
添加节点时使用的是ricci设置的password，Ricci Hostname 是使用在hosts表中定义的主机名（勾选enable shared storage support，如果勾选Reboot Nodes Before Joining Cluster 添加节点会重启服务器）。
3、createsfance devices
点击add，选择IPMI lan，输入Name、fence device IP、Login：USERID、password:PASSW0RD、Authentication Type：Password（各厂家的FENCE设备不一样根据情况配置）。
创建完成之后跟节点关联
先添加fence method，方法名随便，之后关联起来创建的fence实例。一一对应即可
4、creates Failover Domains
点击add，输入Failover Domains name、勾选Prioritized、No Failback，在Member勾选添加在此Failover Domains 的Node，并设置Priority。
Restricetd：限定运行服务节点的机器范围，
no fackback：不回切，当故障机器恢复可用以后，服务不回切。
Priority：优先级的值越低，优先级别越高，0不可用。
5、Add Resources
我们添加一个服务IP地址资源、一个oracle数据库文件系统资源、一个oracle数据库启动、关闭、监控脚本资源（数据库脚本请查看Note1）
6、Creates Service Groups
输入Service Name，勾选Automatically Start This Service,选择Failover Domain，选择
Recovery Policy：Relocate
run exclusive：服务只会在“没有运行其他服务的节点”上运行，可以不选。Add Resource ：将之前我们创建的Resource添加到这个Service Groups
7、Configure QDISK
Linux RHCS基本配置完成
四、RHCS常用命令
1、节点切换
clusvcadm -r oradb(服务名)
Trying to relocate service:oradb....Success
service:oradb is now running on cgvch_podb2
2、查看群集状态
[root@cgvch-podb1 ~]# clustat
Cluster Status for PODB_CLUSTER @ Sun Jun 28 14:20:48 2015
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
cgvch-podb1 1 Online, Local, rgmanager
cgvch-podb2 2 Online, rgmanager
Service Name Owner (Last) State ------- ---- ----- ------ -----
service:oradb cgvch-podb2 started
3、查看HA日志
tail -f /var/log/messages
tail-f/var/log/cluster/rgmanager.log
4、查看RHCS配置内容
cat /etc/cluster/cluster.conf
5、查看服务IP
ip addr list
6、查看oracle进程
ps -ef|grep ora
Note1、oracle数据库启动、关闭、监控脚本
主脚本：oracledb.sh（配置到resources中的脚本）从脚本：startdb.sh stopdb.sh statusdb.sh
1、oracledb.sh
#!/bin/sh
#
# Cluster service script to start, stop, and check status of oracle
#
cd /home/oracle
case $1 in
start)
su - oracle -c ./startdb.sh
exit 0
;;
stop)
su - oracle -c ./stopdb.sh
exit 0
;;
status|monitor)
./statusdb.sh
exit $?
;;
#restart)
# su - oracle -c ./stopdb.sh
# su - oracle -c ./startdb.sh
#;;
esac
echo $? >>/home/oracle/oracle.log
2、startdb.sh
#!/bin/sh
set -x
export ORACLE_SID=CGVOLN
export ORACLE_BASE=/oradb/app/oracle
export ORACLE_HOME=/oradb/app/oracle/product/11.2.0.4
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:$LD_LIBRARY_PATH
export PATH=$PATH:${ORACLE_HOME}/bin
$ORACLE_HOME/bin/sqlplus /nolog >>/home/oracle/startdb.log<< EOF
conn /as sysdba
WHENEVER SQLERROR EXIT SQL.SQLCODE
startup;
exit;
EOF
if [ $? -eq 0 ];then
echo "startup normal">>/home/oracle/startdb.log
else
exit 1 >>/home/oracle/startdb.log
fi
lsnrctl start LISTENER_CGVOLN
#emctl start dbconsole
echo $? >>/home/oracle/startdb.log
3、stopdb.sh
#!/bin/sh
set -x
ORACLE_RELEASE=11.2.0
export ORACLE_SID=CGVOLN
export ORACLE_BASE=/oradb/app/oracle
export ORACLE_HOME=/oradb/app/oracle/product/11.2.0.4
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:$LD_LIBRARY_PATH
export PATH=$PATH:${ORACLE_HOME}/bin
lsnrctl stop LISTENER_CGVOLN
#emctl stop dbconsole
$ORACLE_HOME/bin/sqlplus /nolog>>/home/oracle/stopdb.log<< EOF
conn /as sysdba
WHENEVER SQLERROR EXIT SQL.SQLCODE
shutdown immediate;
quit;
EOF
4、statusdb.sh
#!/bin/sh
ORACLE_RELEASE=11.2.0
export ORACLE_SID=CGVOLN
export ORACLE_BASE=/oradb/app/oracle
export ORACLE_HOME=/oradb/app/oracle/product/11.2.0.4
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:$LD_LIBRARY_PATH
export PATH=$PATH:${ORACLE_HOME}/bin
ps -ef | grep ora_pmon_CGVOLN | grep -v grep >> /home/oracle/statusdb.log if [ $? -eq 0 ];then
echo $?
echo "startup normal">>/home/oracle/statusdb.log
else
echo "it seems its not normal" >>/home/oracle/statusdb.log
exit 1
fi
lsof -i:1521 >>/home/oracle/statusdb.log
if [ $? -eq 0 ];then
echo $?
echo "startup normal">>/home/oracle/statusdb.log
else
echo "it seems its not normal" >>/home/oracle/statusdb.log
exit 1
fi
标红处需要注意根据实际情况修改，在监控状态脚本中的ora_pmon_CGVOLN 需要根据实际情况进行修改