Configure Direct NFS Client (DNFS) on Linux

Step by Step - Configure Direct NFS Client (DNFS) on Linux for Oracle11g:

1 Shutdown the database
                sqlplus "/as sysdba"
                shutdown immediate
                exit

2   Enable the Direct NFS Client
                cd $ORACLE_HOME/rdbms/lib
                make -f ins_rdbms.mk dnfs_on

Verify:

  ls -l $ORACLE_HOME/lib/lib*od*11.so

-rw-r--r--. 1 oracle oinstall 60431 Sep 17  2011 $OH/lib/libnfsodm11.so  << dnfs

-rw-r--r--. 1 oracle oinstall 60431 Oct 20 19:45 $OH/lib/libodm11.so     # Oracle used

-rw-r--r--. 1 oracle oinstall 12259 Sep 17  2011 $OH/lib/libodmd11.so    << no dnfs

   ( Disable the Direct NFS Client

       cd $ORACLE_HOME/rdbms/lib

       make –f ins_rdbms.mk dnfs_off   )

Oracle processes use /etc/mtab to access NFS settings

3 Configure ORANFSTAB

Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux) to determine the mount point settings for NFS storage devices.

This file is required only for configuring the Direct NFS for load balancing and specfic to single database.You can still enable the Direct NFS without configuring oranfstab file.DNFS will take mount point settings for NFS from /etc/mtab on Linux

In RAC,the oranfstab must be configured on all nodes and keep /etc/oranfstab file synchronized on all nodes.

(When the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database. In this case, all nodes running an Oracle RAC database use the same ORACLE_HOME/dbs/oranfstab file.

When the oranfstab file is placed in /etc, then it is globally available to all Oracle databases, and can contain mount points used by all Oracle databases running on nodes in the cluster, including single-instance databases. However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file /etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the /etc/fstab file.

                cd /etc

                grant root/root access to oranfstab

                update oranfstab file with /oradata and /orabkup mount point

Example file provided below for MYDBDEV:

$ORACLE_HOME/dbs/oranfstab  or  cat /etc/oranfstab

server: stgasm <=== NFS server Host name

path: 10.177.52.158 <--- First path to NFS server ie NFS server NIC

local: 10.177.52.151 <--- First client-side NIC

path: 10.177.52.159 <--- Second path to NFS server ie NFS server NIC (For load balance purpose)

local: 10.177.52.151 <--- Second client-side NIC (For load balance purpose)

export: /oraclenfs mount: /oradata1

or

                server: (Storage server Name)

                path:  <IP 1.1.1.1>

                #path:

                #path:

                #path:

                export: /vol/<XXXXXXX>: /<MP>

               

                server: (Storage server Name)

                path:  <IP 1.1.1.1>

                #path:

                #path:

                #path:

                export: /vol/<XXXXXXX>: /<MP>

               

                dontroute

4              Verify mtab

                cd /etc

                Verify mount point entries on mtab for which you want to setup the DNFS.

               

                Example: MYDBDEV

                <Storage host name >:/vol/mydbdev_MP /MP nfs rw,nolock,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=10.1.1.1  0 0

                <Storage host name >:/vol/mydbdev_MP /MP nfs rw,nolock,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,addr=10.1.1.1  0 0

5 . What is oradism? When is it used by Direct NFS?

Direct NFS client uses oradism executable to perform functions that require root privileges. oradism is configured by default on a single instance and for RAC if not using shared $ORACLE_HOME.

It is used for the following special purposes:

Direct NFS client needs root privileges to start communication with the NFS filer. It uses oradism to obtain the root file handle for the exported volume and the NFS server port and NFS mount port. Once the root handle and port information is obtained, all future communication is issued by Oracle user processes using normal privileges.

Oradism executable need always to be owned by root otherwise instance will not come up and returns the error in alert log:

Direct NFS: please check that oradism is setuid

Verify oradism

                cd $ORACLE_HOME/bin

                verify the permissions on oradism binary. It should have root oradba grants

                If the binary doesn't have the right grants: Follow the steps below

                # chown root:oradba oradism / chmod 4755 oradism /cd $ORACLE_HOME/bin/oradism

                If ‘oradism’ is not configured correctly, this error in seen the alert log:

                “Direct NFS: please check that oradism is setuid”

6. Restart the database server

                sudo reboot –p

7. Verify the status of the database

                If it's not started. Syntax: sqlplus "/as sysdba" , startup

How can the DNFS usage be verified?

1. Verify the status of DNFS

Vi my_db_dnfs_layout.sql

spool db_dnfs_layout.html

                SET MARKUP HTML ON

                set echo on

              set pagesize 200

                alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS';

                select 'THIS DB REPORT WAS GENERATED AT: ==)> ' , sysdate " "  from dual;

                select 'HOSTNAME ASSOCIATED WITH THIS DB INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';

                select * from v$dnfs_servers;

                select * from v$dnfs_files;

                select * from v$dnfs_channels;

                select * from v$dnfs_stats;

                select * from v$datafile;

                select * from dba_data_files;

                select * from DBA_TEMP_FILES;

                select * from v$logfile;

                select * from  v$log;

                select * from v$version;

                show parameter

                spool off

                exit

               

  sqlplus "/as sysdba"

@ my_db_dnfs_layout.sql

2.  Just after the initialization parameters are listed in the alert log, you will see the following entry

Direct NFS client is enabled by checking the below message

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version2.0

3.  Along with the message from the alert log, this query on v$dnfs_servers ensures that dNFS is truly being used (returns !=0 value):

                select count(*) from v$dnfs_servers

( sometimes it isn’t true ;)

4.  $ ps -ef | grep dbw

oracle   23126     1  0 06:53 ?        00:00:00 ora_dbw0_MYINST1

oracle   23128     1  0 06:53 ?        00:00:00 ora_dbw1_MYINST1

oracle   23246 22377  0 07:04 pts/0    00:00:00 grep dbw

$  lsof –p < dbw pid >

/usr/sbin/lsof -p 23126

# DNFS OFF

lsof -p 725 | grep data01.dbf

oracle  725 oracle  262u   REG ... /nfsimp/data01.dbf (192.168.51.21:/u01)

# DNFS ON

lsof -p 6540 | grep 192.168.51.21

oracle  6540 oracle   32u  IPv4 ... TCP dbhost:26171->nfsserver:nfs (ESTABLISHED)

5. alert.log

Direct NFS: channel id [0] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP

Direct NFS: channel id [1] path [IPnfs] to filer [KUKARACHA] via local [IPdb] is UP

About DNFS things to remember for knowledge purpose::

1         DNFS oranfstab

Parameter	Usage
Server	Unique identifier for the NFS server
Local	Network paths (up to 4) on the database host
Path	Network paths (up to 4) on the NFS server
Export	The exported volume on the NFS server
Mount	The local mount point for the exported volume
mnt_timeout	Time in seconds to wait for the first mount
dontroute	Prevents OS routing of outgoing messages
management	Network path for NFS server management interface

2   Direct NFS searches for mount entries in the following order

$ORACLE_HOME/dbs/oranfstab # an instance specific

/etc/oranfstab

/etc/mtab

3 You must restart an instance after adjusting oranfstab file

Disadvantages of DNFS ::

·         It is relatively difficult to troubleshoot

·         You will not find many DNFS messages in alert.log or trace files

·         Sometimes Oracle SILINTLY switches from dNFS to kNFS

·         This sections gives you useful troubleshooting hints

VVI Note::

Don’t mix old and new methods

●        Old method

cd $ORACLE_HOME/lib

mv libodm11.so libodm11.so_bak

ln -s libnfsodm11.so libodm11.so

●        New/current method

make -f ins_rdbms.mk dnfs_on

●        If you enable dNFS using one method disable it the same way.