丢失的ASM磁盘组

  • 格式:doc
  • 大小:95.00 KB
  • 文档页数:10

近日遇到一个问题,ASM的磁盘组无法挂载,之前是正常的,由于一些其他的操作,数据库启动失败,当问题排除时候,发现在数据库整体启动时,挂载磁盘组的环节出现问题。

环境介绍#########################################硬件:Vmware ESX虚拟机OS: Red hat linux 5Oracle version: 11.2.0.2ASM disk是通过asmlib挂载的这个磁盘组只有一个虚拟出的硬盘,是/dev/sdb1.#########################################下面是我整个分析的过程1. 首先通过ASM alert.log,发现如下错误,磁盘挂载失败,无法找到磁盘组1.SQL> alter diskgroup DATA mount2.NOTE: cache registered group DATA number=1incarn=0xc28a1e2d3.NOTE: cache began mount (first) of group DATA number=1incarn=0xc28a1e2d4.Tue Dec 11 18:06:55 20125.ERROR: no PST quorum in group: required 2, found 0 <<<<<<<<<<<6.NOTE: cache dismounting (clean) group 1/0xC28A1E2D (DATA)7.NOTE: dbwr not being msg'd to dismount8.NOTE: lgwr not being msg'd to dismount9.NOTE: cache dismounted group 1/0xC28A1E2D (DATA)10.N OTE: cache ending mount (fail) of group DATA number=1incarn=0xc28a1e2d11.N OTE: cache deleting context for group DATA 1/0xc28a1e2d12.G MON dismounting group 1 at 8 for pid 17, osid 3216313.E RROR: diskgroup DATA was not mounted14.O RA-15032: not all alterations performed15.O RA-15017: diskgroup "DATA" cannot be mounted16.O RA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"17.E RROR: alter diskgroup DATA mount2. 首先检查ASM pfile 文件,未发现异常1.asm_diskgroups='DATA'2.instance_type='asm'rge_pool_size=12M4.remote_login_passwordfile='EXCLUSIVE'3. 尝试通过以下命令检查磁盘是否物理存在,是如何对应物理设备的,发现查询不到ASM 磁盘1.[grid@lgto_test ~]$ kfod disks=all2.3.----Non output----4.5.6.7.[grid@lgto_test peer]$ cd /dev/oracleasm/disks/8.[grid@lgto_test disks]$ ls9.10.----Non output----11.12.13.[grid@lgto_test disks]$ /etc/init.d/oracleasm listdisks14.15.----Non output----4. 但是直接检查物理设备,/dev/sdb1是存在的,说明OS已经识别该硬盘设备,只是ASMlib无法正常识别:1.查询对应的物理硬盘2.[oracle@OEL ~]$ /etc/init.d/oracleasm querydisk -d disk1 Disk "DISK1" is a valid ASMdisk on device [8,17]3.[oracle@OEL ~]$ ls -l /dev/ |grep 8|grep 17 brw-r----- 1 root disk 8, 17 Oct 1614:01 sdb14.5.[root@lgto_test ~]# ls -lst /dev/sd*6.0 brw-r----- 1 root disk 8, 0 Dec 11 19:29 /dev/sda7.0 brw-r----- 1 root disk 8, 2 Dec 11 19:29 /dev/sda28.0 brw-r----- 1 root disk 8, 16 Dec 11 19:29 /dev/sdb9.0 brw-r----- 1 root disk 8, 17 Dec 11 19:29 /dev/sdb1 <<<<<<<This is the missed diskgroup10.0 brw-r----- 1 root disk 8, 1 Dec 11 11:29 /dev/sda15. 起先是考虑是否是磁盘头损害,导致无法ASMlib识别该磁盘,dump磁盘头发现没有问题.1.#od -c /dev/sdb12.……3.0000040 O R C L D I S K D A T A D G 0 14.……5.7760040 O R C L D I S K D A T A D G 0 1这里补充下,如果磁盘头信息丢失,将会显示如下1.0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0如果显示这个结果,需要通过以下方式renamedisk,具体可以参考文档Oracleasm Listdisks Cannot See Disks (Doc ID 392527.1)e the "oracleasm renamedisk" utility to add an asmlib label to the disk:2./etc/init.d/oracleasm renamedisk /dev/<device><asmlib_label>3.If it fails, use the "-f" switch:4./etc/init.d/oracleasm renamedisk -f /dev/<device><asmlib_label>6. 重启ASMLib ,检查是否是ASMLib 问题1.[root@lgto_test ~]# /etc/init.d/oracleasm restart2.Dropping Oracle ASMLib disks:3.[ OK ]4.Shutting down the Oracle ASMLib driver: [FAILED]检查文件系统oracleasm文件系统已经成功挂载1.[root@lgto_test ~]# df -ha2.Filesystem Size Used Avail Use% Mounted on3.……4.oracleasmfs 0 0 0 - /dev/oracleasm7. 检查/dev/sdb1状态,查看是否已经marked为ASM disk,显示已经标记成功1.[root@lgto_test ~]# oracleasm querydisk /dev/sdb12.Device "/dev/sdb1" is marked an ASM disk with the label "DATADG01"3.4.[root@lgto_test ~]# /sbin/service oracleasm scandisks5.Scanning the system for Oracle ASMLib disks:6.[ OK ]7.8.[root@lgto_test ~]# /etc/init.d/oracleasm listdisks9.----Non output---8. 检查rpm package也没有问题1.[grid@lgto_test ~]$ rpm -qa|grep oracleasm2.oracleasmlib-2.0.4-1.el53.oracleasm-support-2.1.7-1.el54.oracleasm-2.6.18-308.el5-2.0.5-1.el59. 收集Kfed logs,没有检查到异常新信息。

1.[root@lgto_test ~]# /oracle/ora11g/product/app/grid/bin/kfed read /dev/sdb12.3.kfbh.endian: 1 ; 0x000: 0x014.5.kfbh.hard: 130 ; 0x001: 0x826.7.kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD8.9.kfbh.datfmt: 1 ; 0x003: 0x0110.11.k fbh.block.blk: 0 ; 0x004: T=0NUMB=0x012.13.k fbh.block.obj: 2147483648 ; 0x008: TYPE=0x8NUMB=0x014.15.k fbh.check: 3351358462 ; 0x00c: 0xc7c1abfe16.17.k fbh.fcn.base: 0 ; 0x010: 0x0000000018.19.k fbh.fcn.wrap: 0 ; 0x014: 0x0000000020.21.k fbh.spare1: 0 ; 0x018: 0x0000000022.23.k fbh.spare2: 0 ; 0x01c: 0x0000000024.25.k fdhdb.driver.provstr: ORCLDISKDATADG01 ; 0x000: length=1626.27.k fdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x4154414428.29.k fdhdb.driver.reserved[1]: 825247556 ; 0x00c: 0x3130474430.31.k fdhdb.driver.reserved[2]: 0 ; 0x010: 0x0000000032.33.k fdhdb.driver.reserved[3]: 0 ; 0x014: 0x0000000034.35.k fdhdb.driver.reserved[4]: 0 ; 0x018: 0x0000000036.37.k fdhdb.driver.reserved[5]: 0 ; 0x01c: 0x0000000038.39.k pat: 186646528 ; 0x020: 0x0b20000040.41.k fdhdb.dsknum: 0 ; 0x024: 0x000042.43.k fdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL44.45.k fdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER46.47.k fdhdb.dskname: DATADG01 ; 0x028: length=848.49.k fdhdb.grpname: DATA ; 0x048: length=450.51.k fdhdb.fgname: DATADG01 ; 0x068: length=852.53.k fdhdb.capname: ; 0x088: length=054.55.k fdhdb.crestmp.hi: 32977140 ; 0x0a8: HOUR=0x14DAYS=0x7MNTH=0xc YEAR=0x7dc56.57.k fdhdb.crestmp.lo: 1642529792 ; 0x0ac: USEC=0x0MSEC=0x1c1SECS=0x1e MINS=0x1858.59.k fdhdb.mntstmp.hi: 32977140 ; 0x0b0: HOUR=0x14DAYS=0x7MNTH=0xc YEAR=0x7dc60.61.k fdhdb.mntstmp.lo: 1664549888 ; 0x0b4: USEC=0x0MSEC=0x1c1SECS=0x33MINS=0x1862.63.k fdhdb.secsize: 512 ; 0x0b8: 0x020064.65.k fdhdb.blksize: 4096 ; 0x0ba: 0x100066.67.k fdhdb.ausize: 1048576 ; 0x0bc: 0x0010000068.69.k fdhdb.mfact: 113792 ; 0x0c0: 0x0001bc8070.71.k fdhdb.dsksize: 204797 ; 0x0c4: 0x00031ffd72.73.k fdhdb.pmcnt: 3 ; 0x0c8: 0x0000000374.75.k fdhdb.fstlocn: 1 ; 0x0cc: 0x0000000176.77.k fdhdb.altlocn: 2 ; 0x0d0: 0x0000000278.79.k fdhdb.f1b1locn: 2 ; 0x0d4: 0x0000000280.81.k fdhdb.redomirrors[0]: 0 ; 0x0d8: 0x000082.83.k fdhdb.redomirrors[1]: 0 ; 0x0da: 0x000084.85.k fdhdb.redomirrors[2]: 0 ; 0x0dc: 0x000086.87.k fdhdb.redomirrors[3]: 0 ; 0x0de: 0x000088.89.k fdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a10000090.91.k fdhdb.grpstmp.hi: 32977140 ; 0x0e4: HOUR=0x14DAYS=0x7MNTH=0xc YEAR=0x7dc92.93.k fdhdb.grpstmp.lo: 1642390528 ; 0x0e8: USEC=0x0MSEC=0x139SECS=0x1e MINS=0x1894.95.k fdhdb.vfstart: 0 ; 0x0ec: 0x0000000096.97.k fdhdb.vfend: 0 ; 0x0f0: 0x0000000098.99.k fdhdb.spfile: 58 ; 0x0f4: 0x0000003a100.101.kfdhdb.spfflg: 1 ; 0x0f8: 0x00000001102.103.kfdhdb.ub4spare[0]: 0 ; 0x0fc: 0x00000000104.105.kfdhdb.ub4spare[1]: 0 ; 0x100: 0x00000000106.107.kfdhdb.ub4spare[2]: 0 ; 0x104: 0x00000000108.109.kfdhdb.ub4spare[3]: 0 ; 0x108: 0x00000000110.111.kfdhdb.ub4spare[4]: 0 ; 0x10c: 0x00000000112.113.kfdhdb.ub4spare[5]: 0 ; 0x110: 0x00000000114.115.kfdhdb.ub4spare[6]: 0 ; 0x114: 0x00000000116.117.kfdhdb.ub4spare[7]: 0 ; 0x118: 0x00000000118.119.kfdhdb.ub4spare[8]: 0 ; 0x11c: 0x00000000120.122.123.kfdhdb.ub4spare[10]: 0 ; 0x124: 0x00000000 124.125.kfdhdb.ub4spare[11]: 0 ; 0x128: 0x00000000 126.127.kfdhdb.ub4spare[12]: 0 ; 0x12c: 0x00000000 128.129.kfdhdb.ub4spare[13]: 0 ; 0x130: 0x00000000 130.131.kfdhdb.ub4spare[14]: 0 ; 0x134: 0x00000000 132.133.kfdhdb.ub4spare[15]: 0 ; 0x138: 0x00000000 134.135.kfdhdb.ub4spare[16]: 0 ; 0x13c: 0x00000000 136.137.kfdhdb.ub4spare[17]: 0 ; 0x140: 0x00000000 138.139.kfdhdb.ub4spare[18]: 0 ; 0x144: 0x00000000 140.141.kfdhdb.ub4spare[19]: 0 ; 0x148: 0x00000000 142.143.kfdhdb.ub4spare[20]: 0 ; 0x14c: 0x00000000 144.145.kfdhdb.ub4spare[21]: 0 ; 0x150: 0x00000000 146.147.kfdhdb.ub4spare[22]: 0 ; 0x154: 0x00000000 148.149.kfdhdb.ub4spare[23]: 0 ; 0x158: 0x00000000 150.151.kfdhdb.ub4spare[24]: 0 ; 0x15c: 0x00000000 152.153.kfdhdb.ub4spare[25]: 0 ; 0x160: 0x00000000 154.155.kfdhdb.ub4spare[26]: 0 ; 0x164: 0x00000000 156.157.kfdhdb.ub4spare[27]: 0 ; 0x168: 0x00000000 158.159.kfdhdb.ub4spare[28]: 0 ; 0x16c: 0x00000000 160.161.kfdhdb.ub4spare[29]: 0 ; 0x170: 0x00000000 162.163.kfdhdb.ub4spare[30]: 0 ; 0x174: 0x00000000 164.166.167.kfdhdb.ub4spare[32]: 0 ; 0x17c: 0x00000000 168.169.kfdhdb.ub4spare[33]: 0 ; 0x180: 0x00000000 170.171.kfdhdb.ub4spare[34]: 0 ; 0x184: 0x00000000 172.173.kfdhdb.ub4spare[35]: 0 ; 0x188: 0x00000000 174.175.kfdhdb.ub4spare[36]: 0 ; 0x18c: 0x00000000 176.177.kfdhdb.ub4spare[37]: 0 ; 0x190: 0x00000000 178.179.kfdhdb.ub4spare[38]: 0 ; 0x194: 0x00000000 180.181.kfdhdb.ub4spare[39]: 0 ; 0x198: 0x00000000 182.183.kfdhdb.ub4spare[40]: 0 ; 0x19c: 0x00000000 184.185.kfdhdb.ub4spare[41]: 0 ; 0x1a0: 0x00000000 186.187.kfdhdb.ub4spare[42]: 0 ; 0x1a4: 0x00000000 188.189.kfdhdb.ub4spare[43]: 0 ; 0x1a8: 0x00000000 190.191.kfdhdb.ub4spare[44]: 0 ; 0x1ac: 0x00000000 192.193.kfdhdb.ub4spare[45]: 0 ; 0x1b0: 0x00000000 194.195.kfdhdb.ub4spare[46]: 0 ; 0x1b4: 0x00000000 196.197.kfdhdb.ub4spare[47]: 0 ; 0x1b8: 0x00000000 198.199.kfdhdb.ub4spare[48]: 0 ; 0x1bc: 0x00000000 200.201.kfdhdb.ub4spare[49]: 0 ; 0x1c0: 0x00000000 202.203.kfdhdb.ub4spare[50]: 0 ; 0x1c4: 0x00000000 204.205.kfdhdb.ub4spare[51]: 0 ; 0x1c8: 0x00000000 206.207.kfdhdb.ub4spare[52]: 0 ; 0x1cc: 0x00000000 208.210.211.kfdhdb.acdb.aba.seq: 0 ; 0x1d4: 0x00000000212.213.kfdhdb.acdb.aba.blk: 0 ; 0x1d8: 0x00000000214.215.kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000216.217.kfdhdb.acdb.ub2spare: 0 ; 0x1de: 0x000010. 阶段总结,通过以上的分析,得出以下总结1. ASMLib 正常2. RPM包正常3. 磁盘头没有损坏和丢失信息4. 该硬件已经被系统正常识别当前问题就是为什么ASMLib 不能正常扫描并识别到该硬盘11. 最后在检查文件/etc/sysconfig/oracleasm时,发现问题,我们需要扫描到的磁盘是/dev/sdb1,可是在这个配置文件中却排除扫描sdb*的磁盘,和我们希望的是相悖的,将ORACLEASM_SCANEXCLUDE="" 设为空,并重启ASMLib,最后问题解决。