11gR2 on Linux: Prepare the shared storage/Prereq

Main Post: Installing 11gR2 RAC on Linux

After installing the basic installation of linux there are certain requirments which need to be fulfilled for successful installation of RAC 11g.

Install Required Packages
# From Enterprise Linux DVD (if you have different CDs you can copy the Server folder RPMs on some location eg; /rpm4oracle/Server and then install them)

cd /media/cdrom/Server

binutils-2.17.50.0.6-14.el5.i386.rpm
compat-libstdc++-33-3.2.3-63.i386.rpm
elfutils-libelf-devel-0.137-3.el5.i386.rpm
elfutils-libelf-devel-static-0.137-3.el5.i386.rpm
gcc-4.1.2-48.el5.i386.rpm
gcc-c++-4.1.2-48.el5.i386.rpm
glibc-devel-2.5-49.i386.rpm
glibc-headers-2.5-49.i386.rpm
kernel-headers-2.6.18-194.el5.i386.rpm
libacl-devel-2.2.39-6.el5.i386.rpm
libaio-devel-0.3.106-3.2.i386.rpm
libgomp-4.4.0-6.el5.i386.rpm
libstdc++-devel-4.1.2-48.el5.i386.rpm
sysstat-7.0.2-3.el5.i386.rpm
unixODBC-2.2.11-7.1.i386.rpm
unixODBC-devel-2.2.11-7.1.i386.rpm
kernel-debug-devel-2.6.18-164.0.0.0.1.el5
kernel-debug-2.6.18-164.0.0.0.1.el5
oracleasm-2.6.18-164.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5
oracleasm-2.6.18-164.el5debug-2.0.5-1.el5

You can check whether the required package is installed or not.
[root@rac1 ~]# rpm -qa | grep kernel
If required package is not installed then you will install like below from the media.
[root@rac1 ~]# rpm -Uvh unixODBC-*

Determine your current kernel.
[oracle@rac1 ~]$ uname -rm
2.6.18-164.el5 i686
[oracle@rac1 ~]$

Check shared memory file system is big enough for Automatic Memory Manager to work.
[root@rac1 ~]# df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       44G   11G   32G 25% /
/dev/sda1              99M   19M   76M 20% /boot
tmpfs                 1.5G 874M 613M 59% /dev/shm

if it is not enoug then you can change like below.
[root@rac1 ~]# umount tmpfs
[root@rac1 ~]# mount -t tmpfs shmfs -o size=1500m /dev/shm
[root@rac1 ~]# vi /etc/fstab
tmpfs /dev/shm tmpfs size=1500m 0 0

Modify /etc/hosts
Modify the hosts file as per your network. The SCAN should not be in the hosts file but for the VM we will use in hosts file.

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
#::1       localhost6.localdomain6 localhost6
########Public ##############
132.35.21.177   rac1.localdomain rac1
132.35.21.178   rac2.localdomain rac2
########Private ##############
10.10.10.1   rac1-priv.localdomain rac1-priv
10.10.10.2   rac2-priv.localdomain rac2-priv
########Virtual ##############
132.35.21.187   rac1-vip.localdomain rac1-vip
132.35.21.188   rac2-vip.localdomain rac2-vip
########SCAN ##############
132.35.21.198   racscan.localdomain racscan

Modify kernel parameters
[root@rac1 ~]# vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 1054504960
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586

In order to take effect these parameters , run the following
[root@rac1 ~]# /sbin/sysctl -p
Modify /etc/security/limits.conf
Add the following lines to the "/etc/security/limits.conf" file.

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Modify /etc/pam.d/login
Add the following lines to the "/etc/pam.d/login" file, if it does not already exist.

session required pam_limits.so

Deconfigure NTP.
[root@rac1 ~]# service ntpd stop
Shutting down ntpd: [FAILED]
[root@rac1 ~]# chkconfig ntpd off
[root@rac1 ~]# mv /etc/ntp.conf /etc/ntp.conf.org
[root@rac1 ~]# rm /var/run/ntpd.pid
rm: cannot lstat `/var/run/ntpd.pid': No such file or directory

Add User and Group
[root@rac1 ~]# groupadd -g 500 oinstall
[root@rac1 ~]# groupadd -g 501 dba
[root@rac1 ~]# groupadd -g 502 oper
[root@rac1 ~]# groupadd -g 503 asmadmin
[root@rac1 ~]# useradd -u 1000 -g oinstall -G dba,oper,asmadmin oracle
[root@rac1 ~]# passwd oracle
Changing password for user oracle.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@rac1 ~]#

Make required directories
[root@rac1 ~]# mkdir -p /u01/app/11.2.0/grid
[root@rac1 ~]# mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
[root@rac1 ~]# chown -R oracle:oinstall /u01
[root@rac1 ~]# chmod -R 775 /u01

Modify .bash_profile
Make the necessary modification for user specific environment for "oracle" user.
[root@rac1 ~]# vi /home/oracle/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

# User specific environment and startup programs

#PATH=$PATH:$HOME/bin

#export PATH

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac1.localdomain; export ORACLE_HOSTNAME
ORACLE_UNQNAME=rac; export ORACLE_UNQNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
ORACLE_SID=rac1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
Make environement files (optional)Create a file called "/home/oracle/grid_env" with the following contents.
[root@rac1 ~]# vi /home/oracle/grid_env
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

Create a file called "/home/oracle/db_env" with the following contents.
[root@rac1 ~]# vi /home/oracle/db_env
ORACLE_SID=rac1; export ORACLE_SID
ORACLE_HOME=$DB_HOME; export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

You will be able to switch between environments as follows.

$ grid_env
$ echo $ORACLE_HOME
/u01/app/11.2.0/grid

Switch environment

$ db_env
$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/dbhome_1
$
Shutdown Node1
[root@rac1 ~]# shutdown –h now

Create shared DISKs
Make a shared folder where shared disks will be created to be used by ASM eg;
D:\Virtual Machines\SharedASM
Add the desired number of harddisk (virtual) using the VMware Infrastructure web access panel. I added 5 disks. After adding the disks , make their partitions.

[root@rac1 ~]# cd /dev/
[root@rac1 dev]# ls sd*
sda sda1 sda2 sdb sdc sdd sde sdf
[root@racha1 dev]# fdisk sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):
Using default value 1305

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Note: Do the same for other disks.
Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command.

# cd /dev
# ls sd*
sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1 sdf sdf1
#

Configure ASMLib
[root@rac1 ~]# oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]:

Writing Oracle ASM library driver configuration: done

Load the kernel module

[root@rac1 ~]# /usr/sbin/oracleasm init

Loading module "oracleasm": oracleasm

Mounting ASMlib driver filesystem: /dev/oracleasm

If you have any problems, run the following command to make sure you have the correct version of the driver.

[root@rac1 ~]# /usr/sbin/oracleasm update-driver

Mark/create shared disks

[root@rac1 ~]# /usr/sbin/oracleasm createdisk ASMDSK1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]# /usr/sbin/oracleasm createdisk ASMDSK2 /dev/sdc1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]# /usr/sbin/oracleasm createdisk ASMDSK3 /dev/sdd1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]# /usr/sbin/oracleasm createdisk ASMDSK4 /dev/sde1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]# /usr/sbin/oracleasm createdisk ASMDSK5 /dev/sdf1

Writing disk header: done

Instantiating disk: done

Now run the "scandisks" command to refresh the ASM disk configuration.

[root@rac1 ~]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac1 ~]#
We can see the disk are now visible to ASM using the "listdisks" command.

[root@rac1 ~]# /usr/sbin/oracleasm listdisks
ASMDSK1
ASMDSK2
ASMDSK3
ASMDSK4
ASMDSK5
[root@rac1 ~]#

Some related screen shots below: