Home
 | 
Articles
 | 
Installation
 | 
Oracle Grid Infrastructure and Database 12c RAC (2 Nodes)
 | 
Grid Infrastructure and Database 12c Installation (2 Node RAC)
 on Oracle Linux 7.3
Overview
Recent Article
All Archives
Topics
Comments 
Last modified: January 2017
»»
This article describes the installation of Oracle Grid Infrastructure and Cluster Database 12c Release 1 (2 nodes) on Oracle Linux 7.3 using VirtualBox.
In order to save time, I will build the first guest machine (node 1 which I would call it as ol-alpha.shannura.com) and install the prerequisite packages, create users, groups and installation paths etc. and then clone it to create another node (ol-beta.shannura.com). In a real world, we will otherwise be building an identical (hardware and software) two or more physical machines to participate in the cluster.
Virtual Machine Setup (VirtualBox)
As a first thing, we must setup our own Host-only networks as shown shown here. I am not using the default one which comes with 192.168.56.x IP range. Host-only network will avoid any IP address conflict with any other existing hosts available in the network where the actual PC (host machine) is connected to. Please remember, we will be using the public interface and virtual interface and SCAN name using this address range (172.168.190.x).
VirtualBox Manager -> Preferences -> Network -> Host-only Networks
I have installed Oracle Linux 7.3 (actually installed 7.2 version and did a full update using yum ('yum update'). So, I'm not going through those steps here as they are simple and straightforward. Here is the configuration I used. My host machine is having 16G of physical memory so I can allocate 4G to each machine.
OS: Oracle Linux 7.3
Memory: 4G
Storage: 40G
Network Interface(s):	
Adapter 1 (attached to Host-only Adapter) - this will be our public interface
Adapter 2 (attached to Internal Network)  - this will be our private interface
I am going to set the hostname as 'ol-alpha.shannura.com' for my first node:
$
[root@localhost ~]# hostnamectl set-hostname ol-alpha.shannura.com
Operating System (Oracle Linux 7) Recommendation for 12c: Oracle Linux 7 with the Unbreakable Enterprise kernel: 3.8.13-33.el7uek.x86_64 or later. Mine is okay to proceed.
$
[root@ol-alpha ~]# uname -r
3.8.13-118.16.2.el7uek.x86_64
Stop and disable the firewall:
$
[root@localhost ~]# systemctl stop firewalld.service
[root@localhost ~]# systemctl disable firewalld.service
Stop and disable Avahi daemon (however, don't remove it)
$
[root@localhost ~]# systemctl stop avahi-daemon.socket avahi-daemon.service
[root@localhost ~]# systemctl disable avahi-daemon.socket avahi-daemon.service
Installing the preinstall packages
Oracle PreInstall RPM for Oracle Linux is one the biggest benefits of adopting Oracle Linux. You can either use Oracle public yum repository (if your Linux machine is not connected to Internet) or manually download it from "http://public-yum.oracle.com/repo/OracleLinux/OL7/addons/x86_64/".
Since my VM Guest machine has access to internet, I'm going to use the Oracle public yum repository. Before I can do that, I need to modify the below section inside the file '/etc/yum.repos.d/public-yum-ol7.repo' file as below.
Before:
[ol7_addons]
name=Oracle Linux $releasever Add ons ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL7/addons/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0
After:
[ol7_addons]
name=Oracle Linux $releasever Add ons ($basearch)
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL7/addons/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
Following packages or later must be installed. However, the preinstall package takes care of everything.
binutils-2.23.52.0.1-12.el7.x86_64 
compat-libcap1-1.10-3.el7.x86_64
compat-libstdc++-33-3.2.3-71.el7.i686 
compat-libstdc++-33-3.2.3-71.el7.x86_64
gcc-4.8.2-3.el7.x86_64 
gcc-c++-4.8.2-3.el7.x86_64 
glibc-2.17-36.el7.i686 
glibc-2.17-36.el7.x86_64 
glibc-devel-2.17-36.el7.i686 
glibc-devel-2.17-36.el7.x86_64 
ksh
libaio-0.3.109-9.el7.i686 
libaio-0.3.109-9.el7.x86_64 
libaio-devel-0.3.109-9.el7.i686 
libaio-devel-0.3.109-9.el7.x86_64 
libgcc-4.8.2-3.el7.i686 
libgcc-4.8.2-3.el7.x86_64 
libstdc++-4.8.2-3.el7.i686 
libstdc++-4.8.2-3.el7.x86_64 
libstdc++-devel-4.8.2-3.el7.i686 
libstdc++-devel-4.8.2-3.el7.x86_64 
libXi-1.7.2-1.el7.i686 
libXi-1.7.2-1.el7.x86_64 
libXtst-1.2.2-1.el7.i686 
libXtst-1.2.2-1.el7.x86_64 
make-3.82-19.el7.x86_64 
sysstat-10.1.5-1.el7.x86_64
$
[root@ol-alpha /]# yum -y install oracle-rdbms-server-12cR1-preinstall
We can always double check the packages whether or not installed as shown below, for example:
$
[root@ol-alpha /]# rpm -qa glibc-devel
glibc-devel-2.17-105.0.1.el7.x86_64

[root@ol-alpha /]# rpm -qa gcc
gcc-4.8.5-4.el7.x86_64

[root@ol-alpha /]# rpm -qa binutils
binutils-2.23.52.0.1-55.el7.x86_64
In addition, it is worth installing these packages (includes both 32-bit and 64-bit versions)
$<>  
[root@ol-alpha /]# yum install binutils -y
[root@ol-alpha /]# yum install compat-libcap1 -y
[root@ol-alpha /]# yum install compat-libstdc++-33 -y
[root@ol-alpha /]# yum install compat-libstdc++-33.i686 -y
[root@ol-alpha /]# yum install gcc -y
[root@ol-alpha /]# yum install gcc-c++ -y
[root@ol-alpha /]# yum install glibc -y
[root@ol-alpha /]# yum install glibc.i686 -y
[root@ol-alpha /]# yum install glibc-devel -y
[root@ol-alpha /]# yum install glibc-devel.i686 -y
[root@ol-alpha /]# yum install ksh -y
[root@ol-alpha /]# yum install libgcc -y
[root@ol-alpha /]# yum install libgcc.i686 -y
[root@ol-alpha /]# yum install libstdc++ -y
[root@ol-alpha /]# yum install libstdc++.i686 -y
[root@ol-alpha /]# yum install libstdc++-devel -y
[root@ol-alpha /]# yum install libstdc++-devel.i686 -y
[root@ol-alpha /]# yum install libaio -y
[root@ol-alpha /]# yum install libaio.i686 -y
[root@ol-alpha /]# yum install libaio-devel -y
[root@ol-alpha /]# yum install libaio-devel.i686 -y
[root@ol-alpha /]# yum install libXext -y
[root@ol-alpha /]# yum install libXext.i686 -y
[root@ol-alpha /]# yum install libXtst -y
[root@ol-alpha /]# yum install libXtst.i686 -y
[root@ol-alpha /]# yum install libX11 -y
[root@ol-alpha /]# yum install libX11.i686 -y
[root@ol-alpha /]# yum install libXau -y
[root@ol-alpha /]# yum install libXau.i686 -y
[root@ol-alpha /]# yum install libxcb -y
[root@ol-alpha /]# yum install libxcb.i686 -y
[root@ol-alpha /]# yum install libXi -y
[root@ol-alpha /]# yum install libXi.i686 -y
[root@ol-alpha /]# yum install make -y
[root@ol-alpha /]# yum install sysstat -y
[root@ol-alpha /]# yum install unixODBC -y
[root@ol-alpha /]# yum install unixODBC-devel -y
[root@ol-alpha /]# yum install zlib-devel -y
[root@ol-alpha /]# yum install zlib-devel.i686 -y
'/etc/sysctl.conf' is automatically is optimized by the preinstall package:
$<>  
[root@ol-alpha ~]# cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

# oracle-rdbms-server-12cR1-preinstall setting for fs.file-max is 6815744
fs.file-max = 6815744

# oracle-rdbms-server-12cR1-preinstall setting for kernel.sem is '250 32000 100 128'
kernel.sem = 250 32000 100 128

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmmni is 4096
kernel.shmmni = 4096

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmall is 1073741824 on x86_64
kernel.shmall = 1073741824

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmmax is 4398046511104 on x86_64
kernel.shmmax = 4398046511104

# oracle-rdbms-server-12cR1-preinstall setting for kernel.panic_on_oops is 1 per Orabug 19212317
kernel.panic_on_oops = 1

# oracle-rdbms-server-12cR1-preinstall setting for net.core.rmem_default is 262144
net.core.rmem_default = 262144

# oracle-rdbms-server-12cR1-preinstall setting for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304

# oracle-rdbms-server-12cR1-preinstall setting for net.core.wmem_default is 262144
net.core.wmem_default = 262144

# oracle-rdbms-server-12cR1-preinstall setting for net.core.wmem_max is 1048576
net.core.wmem_max = 1048576

# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.conf.all.rp_filter is 2
net.ipv4.conf.all.rp_filter = 2

# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.conf.default.rp_filter is 2
net.ipv4.conf.default.rp_filter = 2

# oracle-rdbms-server-12cR1-preinstall setting for fs.aio-max-nr is 1048576
fs.aio-max-nr = 1048576

# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
Create OS users & groups
Create the 'grid' user and other OS groups. Groups first.
$
[root@ol-alpha ~]# groupadd -g 54323 oper
[root@ol-alpha ~]# groupadd -g 54327 asmdba
[root@ol-alpha ~]# groupadd -g 54328 asmoper
[root@ol-alpha ~]# groupadd -g 54329 asmadmin
$
[root@ol-alpha ~]# useradd -g oinstall -G dba,asmdba,asmoper,asmadmin grid
[root@ol-alpha ~]# usermod -a -G oper,asmdba oracle
[root@ol-alpha ~]# passwd grid
[root@ol-alpha ~]# passwd oracle
Note: user 'oracle' and groups 'oinstall','dba' are automatically created by the preinstall packages. So, you can skip that step.
$<>  
[oracle@ol-alpha ~]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54327(asmdba)

[grid@ol-alpha ~]$ id
uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54327(asmdba),54328(asmoper),54329(asmadmin)
Ensure resource limits are set as below (/etc/security/limits.conf)
oracle   soft   nofile   1024
oracle   hard   nofile   65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock  134217728
oracle   soft   memlock  134217728
grid     soft   nofile   1024
grid     hard   nofile   65536
grid     soft   nproc    16384
grid     hard   nproc    16384
grid     soft   stack    10240
grid     hard   stack    32768
grid     hard   memlock  134217728
grid     soft   memlock  134217728
Ensure enough shared memory is available...in my case, I have set it to 4G because my VM guest machine doesn't have enough shared memory as the base memory is very low.
$
[root@ol-alpha app]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.0G  152K  2.0G   1% /dev/shm

[root@ol-alpha app]# mount -o remount,size=4G /dev/shm
[root@ol-alpha app]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           4.0G   96K  4.0G   1% /dev/shm
Add this entry into /etc/fstab to make this change persistent after reboot. Otherwise, after every reboot, you will need to manually set it. If the memory isn't sufficient some of the Oracle grid components won't succeed. For example, your ASM wouldn't startup on its own throwing MEMORY_TARGET error.
none      /dev/shm        tmpfs   defaults,size=4G        0 0
Create the directory structures and set ownership and permissions
$
[root@ol-alpha /]# mkdir -p /u01/app/12.1.0/grid
[root@ol-alpha /]# mkdir -p /u01/app/grid
[root@ol-alpha /]# mkdir -p /u01/app/oracle
[root@ol-alpha /]# chown -R grid:oinstall /u01
[root@ol-alpha /]# chown oracle:oinstall /u01/app/oracle
[root@ol-alpha /]# chmod -R 775 /u01/
Setting up the .bash_profile for both 'grid' and 'oracle' user.
$<>  
[oracle@ol-alpha ~]$ cat .bash_profile
#  ---------------------------------------------------
# .bash_profile
#  ---------------------------------------------------
# OS User: oracle
# Application: Oracle Database Software Owner
# Version: Oracle 12c Release 1
#  ---------------------------------------------------
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi  
#  ---------------------------------------------------
# ORACLE_SID
#  ---------------------------------------------------
# Specifies the Oracle system identifier (SID)
ORACLE_SID=oradb; export ORACLE_SID
#  ---------------------------------------------------
# ORACLE_UNQNAME
#  ---------------------------------------------------
# Set ORACLE_UNQNAME equal to the database unique name.
#  ---------------------------------------------------
ORACLE_UNQNAME=oradb; export ORACLE_UNQNAME
#  ---------------------------------------------------   
# JAVA_HOME
#  ---------------------------------------------------
JAVA_HOME=/usr/bin/java; export JAVA_HOME
#  ---------------------------------------------------
# ORACLE_BASE
#  ---------------------------------------------------
# Specifies the base of the Oracle directory structure
# for Optimal Flexible Architecture (OFA) compliant
# database software installations.
#  ---------------------------------------------------
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
#  ---------------------------------------------------
# ORACLE_HOME
#  ---------------------------------------------------
# Directory containing the Oracle Database software.
#  ---------------------------------------------------
ORACLE_HOME=$ORACLE_BASE/product/12.1.0/db_1; 
export ORACLE_HOME
#  ---------------------------------------------------
# ORACLE_TERM
#  ---------------------------------------------------
# Defines a terminal definition. If not set, it
# defaults to the value of your TERM environment
# variable. Used by all character mode products.
#  ---------------------------------------------------
ORACLE_TERM=xterm; export ORACLE_TERM
#  ---------------------------------------------------
# NLS_DATE_FORMAT
#  ---------------------------------------------------
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"
export NLS_DATE_FORMAT
#  ---------------------------------------------------
# TNS_ADMIN
#  ---------------------------------------------------
# Specifies the directory containing the Oracle Net
# Services configuration files like listener.ora,
# tnsnames.ora, and sqlnet.ora.
#  ---------------------------------------------------
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
#  ---------------------------------------------------
# PATH
#  ---------------------------------------------------
# Used by the shell to locate executable programs;
# must include the $ORACLE_HOME/bin directory.
#  ---------------------------------------------------
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/local/bin
export PATH
#  ---------------------------------------------------
# LD_LIBRARY_PATH
#  ---------------------------------------------------
# Specifies the list of directories that the shared
# library loader searches to locate shared object
# libraries at runtime.
#  ---------------------------------------------------
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib 
export LD_LIBRARY_PATH
#  ---------------------------------------------------
# CLASSPATH
#  ---------------------------------------------------
# Specifies the directory or list of directories that
# contain compiled Java classes.
#  ---------------------------------------------------
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
#  ---------------------------------------------------
# THREADS_FLAG
#  ---------------------------------------------------
# All the tools in the JDK use green threads as a
# default. To specify that native threads should be
# used, set the THREADS_FLAG environment variable to
# "native". You can revert to the use of green
# threads by setting THREADS_FLAG to the value
# "green".
#  ---------------------------------------------------
THREADS_FLAG=native; export THREADS_FLAG
#  ---------------------------------------------------
# TEMP, TMP, and TMPDIR
#  ---------------------------------------------------
export TEMP=/tmp
export TMPDIR=/tmp
#  ---------------------------------------------------
# UMASK
#  ---------------------------------------------------
# Set the default file mode creation mask
# (umask) to 022 to ensure that the user performing
# the Oracle software installation creates files
# with 644 permissions.
#  ---------------------------------------------------
umask 022
$<>  
[grid@ol-alpha ~]$ cat .bash_profile
# .bash_profile
# OS User: grid
# Application: Oracle Grid Infrastructure
# Version: Oracle 12c Release 1
#  ---------------------------------------------------
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi  
#  ---------------------------------------------------
# ORACLE_SID
#  ---------------------------------------------------
# Specifies the Oracle system identifier (SID)
# for the Automatic Storage Management (ASM)instance
# running on this node.
#  ---------------------------------------------------
ORACLE_SID=+ASM; export ORACLE_SID
#  ---------------------------------------------------
# JAVA_HOME
#  ---------------------------------------------------
# Specifies the directory of the Java SDK and Runtime
# Environment.
#  ---------------------------------------------------
JAVA_HOME=/usr/bin/java; export JAVA_HOME
#  ---------------------------------------------------
# ORACLE_BASE
#  ---------------------------------------------------
# Specifies the base of the Oracle directory structure
# for Optimal Flexible Architecture (OFA) compliant
# installations. The Oracle base directory for the
# grid installation owner is the location where
# diagnostic and administrative logs, and other logs
# associated with Oracle ASM and Oracle Clusterware
# are stored.
# ---------------------------------------------------
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
#  ---------------------------------------------------
# ORACLE_HOME
#  ---------------------------------------------------
# Specifies the directory containing the Oracle
# Grid Infrastructure software. For grid
# infrastructure for a cluster installations, the Grid
# home must not be placed under one of the Oracle base
# directories, or under Oracle home directories of
# Oracle Database installation owners, or in the home
# directory of an installation owner. During
# installation, ownership of the path to the Grid
# home is changed to root. This change causes
# permission errors for other installations.
#  ---------------------------------------------------
ORACLE_HOME=/u01/app/12.1.0/grid; export ORACLE_HOME
#  ---------------------------------------------------
# ORACLE_TERM
#  ---------------------------------------------------
ORACLE_TERM=xterm; export ORACLE_TERM
#  ---------------------------------------------------
# TNS_ADMIN
#  ---------------------------------------------------
# Specifies the directory containing the Oracle Net
# Services configuration files like listener.ora,
# tnsnames.ora, and sqlnet.ora.
#  ---------------------------------------------------
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
#  ---------------------------------------------------
# PATH
#  ---------------------------------------------------
PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/local/bin
export PATH
#  ---------------------------------------------------
# TEMP, TMP, and TMPDIR
export TEMP=/tmp
export TMPDIR=/tmp
#  ---------------------------------------------------
# UMASK
#  ---------------------------------------------------
umask 022
Setting up ASM
I've already downloaded the grid infrastructure software(s) and kept it under /u02/stage/ location.
-rwxrwxrwx. 1 grid oinstall  646972897 May  7 19:54 linuxamd64_12102_grid_2of2.zip
-rwxrwxrwx. 1 grid oinstall 1747043545 May  7 20:24 linuxamd64_12102_grid_1of2.zip
ASMlib packages:
~<>  
kmod-oracleasm (kmod-oracleasm-2.0.8-17.0.1.el7.x86_64)     -- this can be installed straight off public reposity
oracleasm-support (oracleasm-support-2.1.8-3.1.el7.x86_64)  -- this can also be installed straight off public respository
oracleasmlib-2.0.12-1.el7.x86_64.rpm                        -- download this from http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol7-2352094.html 
$<>  
[root@ol-alpha /]# yum install kmod-oracleasm
[root@ol-alpha /]# yum install oracleasm-support
[root@ol-alpha tmp]# wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.12-1.el7.x86_64.rpm
[root@ol-alpha tmp]# rpm -Uvh oracleasmlib-2.0.12-1.el7.x86_64.rpm 
Configure ASM:
At this point ASMLib and its support packages are now installed, however not configured yet. Before we start to configure ASMLib, ensure following os user and os groups are created. In our case, it's already done.
$
[root@ol-alpha soft]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
This can be verified using:
$
[root@ol-alpha soft]# /usr/sbin/oracleasm configure
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
Initialize the asmlib with the oracleasm init command to load oracleasm module and mount the oracleasm filesystem.
$
[root@ol-alpha soft]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size 
Mounting ASMlib driver filesystem: /dev/oracleasm
When ASMLIB is configured, a special file system is created and mounted as /dev/oracleasm, you should notice the following line from the df command.
$
[root@ol-alpha soft]# df -ha | grep oracle
oracleasmfs        0     0     0    - /dev/oracleasm
By now, if we list the asm disks under /dev/oracleasm/disks/, we won't see anything being listed because we haven't created any ASM disks yet.
Clone the virtual machine
I will quickly shutdown my VM Guest machine and make a clone of this virtual machine to create an exact copy. So, we will have an identifical copy of two guest machines. I will set the hostname for the cloned machine as 'ol-beta.shannura.com'. This is actually our second node.
A quick overview of both machines:
Hostname(s):	Node1: ol-alpha.shannura.com
		Public Interface: 172.168.190.101/255.255.0.0
		Private Interface: 192.168.190.101/255.255.255.0
				
		Nod2: ol-beta.shannura.com
		Public Interface: 172.168.190.102/255.255.0.0
		Private Interface: 192.168.190.102/255.255.255.0
Note: Make notice of the different subnet masks for the public and private interfaces
DNS and SCAN name
Below machine is our third virtual machine which is the DNS server to resolve the participating nodes' hostnames as well as SCAN name(s). Follow this separate post on how to setup a DNS server.
DNS Server:	Hostname: rhel54.shannura.com
		Public Interface: 172.168.190.200
		Gateway: 172.168.190.100
				
		SCAN NAME: shannuracluster-scan.shannura.com
		IP Address:	172.168.190.211
				172.168.190.212
				172.168.190.213
My DNS server looks perfect folloiwng a quick nslookup check. The DNS server returns the correct IP address for the hosts I have set it up
$<>  
[root@rhel54 ~]# nslookup ol-alpha.shannura.com		(should return 172.168.190.101)
[root@rhel54 ~]# nslookup ol-beta.shannura.com		(should return 172.168.190.102)
[root@rhel54 ~]# nslookup ol-alpha-vip.shannura.com	(should return 172.168.190.201)
[root@rhel54 ~]# nslookup ol-beta-vip.shannura.com	(should return 172.168.190.202)
[root@rhel54 ~]# nslookup shannuracluster-scan		(should return 172.168.190.211, 172.168.190.212, 172.168.190.213)
Ensure /etc/resolv.conf in both machines as shown here:
$
[root@ol-alpha ~]# cat /etc/resolv.conf
search shannura.com
nameserver 172.168.190.200
options attempts: 3
options timeout: 3

[root@ol-beta ~]# cat /etc/resolv.conf
search shannura.com
nameserver 172.168.190.200
options attempts: 3
options timeout: 3
Ensure /etc/hosts in both machines have entries as below:
$
127.0.0.1         localhost.localdomain localhost

172.168.190.101   ol-alpha.shannura.com ol-alpha
172.168.190.102   ol-beta.shannura.com ol-beta

172.168.190.201   ol-alpha-vip.shannura.com ol-alpha-vip
172.168.190.202   ol-beta-vip.shannura.com ol-beta-vip

172.168.190.211   shannuracluster-scan.shannura.com shannuracluster-scan
172.168.190.212   shannuracluster-scan.shannura.com shannuracluster-scan
172.168.190.213   shannuracluster-scan.shannura.com shannuracluster-scan

192.168.190.101   ol-alpha-priv.shannura.com ol-alpha-priv
192.168.190.102   ol-beta-priv.shannura.com ol-beta-priv
/etc/ssh/sshd_config GSSAPIAuthentication yes -> no #UseDNS yes -> UseDNS no [root@ol-alpha ~]# cat /etc/ssh/sshd_config | egrep -i 'GSSAPIAuthentication|UseDNS' GSSAPIAuthentication no UseDNS no [root@ol-alpha ~]# systemctl restart sshd.service
SSH delay?
Unpin

vi /etc/ssh/sshd_config
Identify these lines and make changes as below:

GSSAPIAuthentication yes --> GSSAPIAuthentication no
#UseDNS yes --> UseDNS no

[root@ol-alpha ~]# cat /etc/ssh/sshd_config | egrep -i 'GSSAPIAuthentication|UseDNS'
GSSAPIAuthentication no
UseDNS no

[root@ol-alpha ~]# systemctl restart sshd.service
Virtual Shared Disks
I will shutdown ol-alpha.shannura.com & ol-beta.shannura.com momentarily to create virtual shared disks and the same would be converted as ASM disks to create DISKGROUPS where OCR, Voting Disk, ASM SPFILE and our actual database would reside.
I plan to create 6 disks in total. Remember the very first diskgroup (which I would create as OCRDG) must have a minimum of 4g. Because in the same diskgroup in addition to OCR and Voting Disk, _mgmtdb would also be created. I'm considering 6g.

vdisk01	[3g]
vdisk02	[3g] --> i will use this 6g for OCR (e.g. OCRDG)

vdisk03	[4g]
vdisk04	[4g] --> i will use this 8g for flash recovery area (e.g. FRADG)

vdisk05	[6g]
vdisk06	[6g] --> i will use this 12g for DATA (e.g. DATADG)
Note: This is not a standard but can be created with as many disks as possible to meet your requirement or you have enough disk space in your host machine and create ASM diskgroups of your choice.
Select ol-alpha machine and click on Settings and navigate to Storage and follow the steps as shown in the screenshots.
You have will to repeat the above steps to create the remaining 5 disks. With all the disks created, it should look as below:
Then, navigate to Virtual Media Manager and select vdisk01.vdi and then click on 'Modify', choose 'Shareable' and press OK. Repeat the same for the remaining 5 disks.
Then select ol-beta machine and go to Settings and navigate to Storage and follow the below steps. Here you will choose the existing disk that we created earlier. Again repeat the same step for the remaining 5 disks.
Now the disks should be visible on both machines but cannot be used until we partition them. So, let's power on both machines once again.
I'm logged on to my first node (ol-alpha). Let's view the disk devices using 'fdisk -l /dev/sd[b-g]' command.
$<>  
[root@ol-alpha ~]# fdisk -l /dev/sd[b-g]

Disk /dev/sdb: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdc: 3221 MB, 3221225472 bytes, 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sde: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdf: 6442 MB, 6442450944 bytes, 12582912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdg: 6442 MB, 6442450944 bytes, 12582912 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Partition them one by one.
$<>  
[root@ol-alpha ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x3766ba87.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-6291455, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-6291455, default 6291455):
Using default value 6291455
Partition 1 of type Linux and of size 3 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
Repeat the same for the remaining 5 disks. At the end, it would look like this:
$<>  
[root@ol-alpha ~]# fdisk -l /dev/sd[b-g]1

Disk /dev/sdb1: 3220 MB, 3220176896 bytes, 6289408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdc1: 3220 MB, 3220176896 bytes, 6289408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd1: 4293 MB, 4293918720 bytes, 8386560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sde1: 4293 MB, 4293918720 bytes, 8386560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdf1: 6441 MB, 6441402368 bytes, 12580864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdg1: 6441 MB, 6441402368 bytes, 12580864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Now we are ready to create the ASM disks...
$
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDB1 /dev/sdb1
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDC1 /dev/sdc1
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDD1 /dev/sdd1
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDE1 /dev/sde1
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDF1 /dev/sdf1
[root@ol-alpha disks]# oracleasm createdisk ASMDISK_SDG1 /dev/sdg1
$
[root@ol-alpha disks]# cd /dev/oracleasm/disks
[root@ol-alpha disks]# ls -lrt
total 0
brw-rw----. 1 grid asmadmin 8, 17 Jan 16 21:52 ASMDISK_SDB1
brw-rw----. 1 grid asmadmin 8, 33 Jan 16 21:52 ASMDISK_SDC1
brw-rw----. 1 grid asmadmin 8, 49 Jan 16 21:53 ASMDISK_SDD1
brw-rw----. 1 grid asmadmin 8, 65 Jan 16 21:53 ASMDISK_SDE1
brw-rw----. 1 grid asmadmin 8, 81 Jan 16 21:53 ASMDISK_SDF1
brw-rw----. 1 grid asmadmin 8, 97 Jan 16 21:53 ASMDISK_SDG1
Let's check if the disks are visible in the other machine. If not, use 'oracleasm scandisks' as below:
$
[root@ol-beta ~]# cd /dev/oracleasm/disks
[root@ol-beta disks]# ls -lrt
total 0
$
[root@ol-beta disks]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMDISK_SDB1"
Instantiating disk "ASMDISK_SDC1"
Instantiating disk "ASMDISK_SDD1"
Instantiating disk "ASMDISK_SDE1"
Instantiating disk "ASMDISK_SDF1"
Instantiating disk "ASMDISK_SDG1"
$
[root@ol-beta disks]# ls -lrt
total 0
brw-rw----. 1 grid asmadmin 8, 17 Jan 16 21:55 ASMDISK_SDB1
brw-rw----. 1 grid asmadmin 8, 33 Jan 16 21:55 ASMDISK_SDC1
brw-rw----. 1 grid asmadmin 8, 49 Jan 16 21:55 ASMDISK_SDD1
brw-rw----. 1 grid asmadmin 8, 65 Jan 16 21:55 ASMDISK_SDE1
brw-rw----. 1 grid asmadmin 8, 81 Jan 16 21:55 ASMDISK_SDF1
brw-rw----. 1 grid asmadmin 8, 97 Jan 16 21:55 ASMDISK_SDG1
Now we are almost set ready to start the Grid Infrastructure installation... Click the next page to continue.
You might want to read this:
Grid Infrastructure and Database 11g R2 Installation (2 Node RAC) on RHEL 5.4
1
2
3
4