Deinstall and cleanup Oracle 11g RAC instance

Deinstall and Clean-up  11g RAC installaton:


1) Shutdown all the DB and cluster services on all the RAC node
From root:
crsctl stop cluster -all
and
crsctl stop crs  on both node



2) From Oracle User:

Run the below:

soatsdb1.com[GSOATS1]$ cd /opt/oracle/grid/11.2.0/deinstall

soatsdb1.com[GSOATS1]$ ls
bootstrap.pl  deinstall  deinstall.pl  deinstall.xml  jlib  readme.txt  response  sshUserSetup.sh

soatsdb1.com[GSOATS1]$ ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /opt/oracle/oraInventory/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################### CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /opt/oracle/grid/11.2.0
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /opt/oracle/product
Checking for existence of central inventory location /opt/oracle/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /opt/oracle/grid/11.2.0
The following nodes are part of this cluster: soatsdb1,soatsdb2
Checking for sufficient temp space availability on node(s) : 'soatsdb1,soatsdb2'

## [END] Install check configuration ##

Traces log file: /opt/oracle/oraInventory/logs//crsdc.log
Enter an address or the name of the virtual IP used on node "soatsdb1"[null]
 >
soatsdb1-v.com
The following information can be collected by running "/sbin/ifconfig -a" on node "soatsdb1"
Enter the IP netmask of Virtual IP "10.30.207.150" on node "soatsdb1"[255.255.255.0]
 >

Enter the network interface name on which the virtual IP address "10.30.207.150" is active
 >

Enter an address or the name of the virtual IP used on node "soatsdb2"[10.30.207.150]
 >

The following information can be collected by running "/sbin/ifconfig -a" on node "soatsdb2"
Enter the IP netmask of Virtual IP "soatsdb2-vip" on node "soatsdb2"[255.255.255.0]
 >

Enter the network interface name on which the virtual IP address "soatsdb2-vip" is active
 >

Enter an address or the name of the virtual IP[]
 >


Network Configuration check config START

Network de-configuration trace file location: /opt/oracle/oraInventory/logs/netdc_check2012-02-02_11-59-38-PM.log

Network Configuration check config END

Asm Check Configuration START

ASM de-configuration trace file location: /opt/oracle/oraInventory/logs/asmcadc_check2012-02-02_11-59-40-PM.log

ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: n
ASM was not detected in the Oracle Home

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /opt/oracle/grid/11.2.0
The cluster node(s) on which the Oracle home deinstallation will be performed are:soatsdb1,soatsdb2
Oracle Home selected for deinstall is: /opt/oracle/grid/11.2.0
Inventory Location where the Oracle home registered is: /opt/oracle/oraInventory
ASM was not detected in the Oracle Home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/opt/oracle/oraInventory/logs/deinstall_deconfig2012-02-02_11-56-20-PM.out'
Any error messages from this session will be written to: '/opt/oracle/oraInventory/logs/deinstall_deconfig2012-02-02_11-56-20-PM.err'

######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /opt/oracle/oraInventory/logs/asmcadc_clean2012-02-02_11-59-55-PM.log
ASM Clean Configuration END

Network Configuration clean config START

Network de-configuration trace file location: /opt/oracle/oraInventory/logs/netdc_clean2012-02-02_11-59-55-PM.log

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END


---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.

Run the following command as the root user or the administrator on node "soatsdb2".

/tmp/deinstall2012-02-02_11-56-11PM/perl/bin/perl -I/tmp/deinstall2012-02-02_11-56-11PM/perl/lib -I/tmp/deinstall2012-02-02_11-56-11PM/crs/install /tmp/deinstall2012-02-02_11-56-11PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp"

Run the following command as the root user or the administrator on node "soatsdb1".

/tmp/deinstall2012-02-02_11-56-11PM/perl/bin/perl -I/tmp/deinstall2012-02-02_11-56-11PM/perl/lib -I/tmp/deinstall2012-02-02_11-56-11PM/crs/install /tmp/deinstall2012-02-02_11-56-11PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode

Press Enter after you finish running the above commands

<----------------------------------------


---------------------------------------->

The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.

Press Enter after you finish running the above commands

<----------------------------------------

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/opt/oracle/grid/11.2.0' from the central inventory on the local node : Done

Delete directory '/opt/oracle/grid/11.2.0' on the local node : Done

The Oracle Base directory '/opt/oracle/product' will not be removed on local node. The directory is in use by Oracle Home '/opt/oracle/product/11.2.0'.

Detach Oracle home '/opt/oracle/grid/11.2.0' from the central inventory on the remote nodes 'soatsdb2' : Done

Delete directory '/opt/oracle/grid/11.2.0' on the remote nodes 'soatsdb2' : Done

The Oracle Base directory '/opt/oracle/product' will not be removed on node 'soatsdb2'. The directory is in use by Oracle Home '/opt/oracle/product/11.2.0'.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2012-02-02_11-56-11PM' on node 'soatsdb1'
Clean install operation removing temporary directory '/tmp/deinstall2012-02-02_11-56-11PM' on node 'soatsdb2'

## [END] Oracle install clean ##


######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
Oracle Clusterware is stopped and successfully de-configured on node "soatsdb2"
Oracle Clusterware is stopped and successfully de-configured on node "soatsdb1"
Oracle Clusterware is stopped and de-configured successfully.
Successfully detached Oracle home '/opt/oracle/grid/11.2.0' from the central inventory on the local node.
Successfully deleted directory '/opt/oracle/grid/11.2.0' on the local node.
Successfully detached Oracle home '/opt/oracle/grid/11.2.0' from the central inventory on the remote nodes 'soatsdb2'.
Successfully deleted directory '/opt/oracle/grid/11.2.0' on the remote nodes 'soatsdb2'.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

soatsdb1.com[GSOATS1]$


3) run the given command on

Node 2:

[root@soatsdb2 ~]# /tmp/deinstall2012-02-02_11-56-11PM/perl/bin/perl -I/tmp/deinstall2012-02-02_11-56-11PM/perl/lib -I/tmp/deinstall2012-02-02_11-56-11PM/crs/install /tmp/deinstall2012-02-02_11-56-11PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp"
Using configuration parameter file: /tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'soatsdb2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'soatsdb2'
CRS-2673: Attempting to stop 'ora.crf' on 'soatsdb2'
CRS-2677: Stop of 'ora.mdnsd' on 'soatsdb2' succeeded
CRS-2677: Stop of 'ora.crf' on 'soatsdb2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'soatsdb2'
CRS-2677: Stop of 'ora.gipcd' on 'soatsdb2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'soatsdb2'
CRS-2677: Stop of 'ora.gpnpd' on 'soatsdb2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'soatsdb2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@soatsdb2 ~]#


4) Run the given command on

Node 1:


[root@soatsdb1 deinstall]# /tmp/deinstall2012-02-02_11-56-11PM/perl/bin/perl -I/tmp/deinstall2012-02-02_11-56-11PM/perl/lib -I/tmp/deinstall2012-02-02_11-56-11PM/crs/install /tmp/deinstall2012-02-02_11-56-11PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode
Using configuration parameter file: /tmp/deinstall2012-02-02_11-56-11PM/response/deinstall_Ora11g_gridinfrahome1.rsp

CRS-5702: Resource 'ora.cssd' is already running on 'soatsdb1'
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'soatsdb1'
CRS-2673: Attempting to stop 'ora.crsd' on 'soatsdb1'
CRS-2677: Stop of 'ora.crsd' on 'soatsdb1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'soatsdb1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'soatsdb1'
CRS-2677: Stop of 'ora.mdnsd' on 'soatsdb1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'soatsdb1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'soatsdb1'
CRS-2677: Stop of 'ora.cssd' on 'soatsdb1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'soatsdb1'
CRS-2677: Stop of 'ora.gipcd' on 'soatsdb1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'soatsdb1'
CRS-2677: Stop of 'ora.gpnpd' on 'soatsdb1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'soatsdb1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node
[root@soatsdb1 deinstall]#


5) run below command on all the node.

[root@soatsdb1 stage]# rm /etc/oracle/*
rm: cannot lstat `/etc/oracle/*': No such file or directory
[root@soatsdb1 stage]#
[root@soatsdb1 stage]# rm -f /etc/init.d/init.cssd
[root@soatsdb1 stage]# rm -f /etc/init.d/init.crs
[root@soatsdb1 stage]# rm -f /etc/init.d/init.crsd
[root@soatsdb1 stage]# rm -f /etc/init.d/init.evmd
[root@soatsdb1 stage]# rm -f /etc/rc2.d/K96init.crs
[root@soatsdb1 stage]# rm -f /etc/rc2.d/S96init.crs
[root@soatsdb1 stage]# rm -f /etc/rc3.d/K96init.crs
[root@soatsdb1 stage]# rm -f /etc/rc3.d/S96init.crs
[root@soatsdb1 stage]# rm -f /etc/rc5.d/K96init.crs
[root@soatsdb1 stage]# rm -f /etc/rc5.d/S96init.crs
[root@soatsdb1 stage]# rm -Rf /etc/oracle/scls_scr
[root@soatsdb1 stage]# rm -f /etc/inittab.crs
[root@soatsdb1 stage]# ps -ef | grep crs
root      7139 34131  0 10:54 pts/5    00:00:00 grep crs
[root@soatsdb1 stage]# ps -ef | grep evm
root      7143 34131  0 10:54 pts/5    00:00:00 grep evm
[root@soatsdb1 stage]# ps -ef | grep css
root      7146 34131  0 10:54 pts/5    00:00:00 grep css
[root@soatsdb1 stage]# rm -f /var/tmp/.oracle
[root@soatsdb1 stage]# rm -f /tmp/.oracle
[root@soatsdb1 stage]#


6) Any files should not be there if deinstall has completed without any failure.

7) This will also delete all the contents which is on grid home.

8) We only need to remove ORALCE_HOME with rm command.

Data guard related issues and its fixes

If error is   Warning: ORA-16789: standby redo logs not configured

DGMGRL> DGMGRL> show configuration

Configuration - abcprd

  Protection Mode: MaxPerformance

  Databases:

    fc_abcprd - Primary database

      Warning: ORA-16789: standby redo logs not configured

    dr_abcprd - Physical standby database

      Error: ORA-16525: the Data Guard broker is not yet available

Fast-Start Failover: DISABLED

Configuration Status:

ERROR

DGMGRL>

Add Standby redo log group on to  Primary db as well as DR site:

SQL> ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo04.log' size 50M;

SQL>  ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo05.log' size 50M;

SQL> ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo06.log' size 50M;

And on  DR Site: 

SQL> alter database recover managed standby database cancel;

SQL>  ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo04.log' size 50M;

SQL> ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo05.log' size 50M;

SQL> ALTER DATABASE ADD STANDBY LOGFILE '/oradata/ABCPRD/stdby_redo06.log' size 50M;

After the above steps completion start DR in managed recovery:

SQL> alter database recover managed standby database disconnect from session;

On primary now run the dgmgrl :

DGMGRL>  show configuration;

Configuration - abcprd

  Protection Mode: MaxPerformance

  Databases:

    fc_abcprd - Primary database

    dr_abcprd - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS

If error is below after above steps then on Primary site

Warning: ORA-16826: apply service state is inconsistent with the DelayMins property

DGMGRL> remove database DR_ABCPRD;

Removed database "dr_abcprd" from the configuration

DGMGRL> add database DR_ABCPRD as connect identifier is  DR_ABCPRD maintained as physical;

Database "dr_abcprd" added

Then Run :

Show configuration

It should not show any error message after that.  But if you are still facing any issue then please leave a message to me, will try to help you. 

Regards,

Amaresh

Script to list missing and INVALID Objects in the database

REM Script to list missing and INVALID Objects in the database
REM
REM      MISSING.SQL                                                  
REM
REM      This script recompiles all objects that have become invalidated    
REM
REM      It should be run as SYS or SYSTEM
REM

set pagesize 0
set linesize 120
set feedback off
set trimspool on
set termout on

spool missing.txt

select A.Owner Oown,
       A.Object_Name Oname,
       A.Object_Type Otype,
       'Miss Pkg Body' Prob
  from DBA_OBJECTS A
 where A.Object_Type = 'PACKAGE'
   and A.Owner not in ('SYS','SYSTEM')
   and not exists
        (select 'x'
           from DBA_OBJECTS B
          where B.Object_Name = A.Object_Name
            and B.Owner = A.Owner
            and B.Object_Type = 'PACKAGE BODY')
union
select Owner Oown,
       Object_Name Oname,
       Object_Type Otype,
       'Invalid Obj' Prob
  from DBA_OBJECTS
 where Owner not in ('SYS','SYSTEM')
   and Status != 'VALID'
 order by 1,4,3,2
/
spool off

ERROR :: ORA-04030: out of process memory when trying to allocate 16328 bytes

Issue: ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pl/sql vc2)

Following ORA-04030 error is encountered every time when the PGA allocation reaches 15GB:


The incident trace shows 15G used by pl/sql:
=======================================
TOP 10 MEMORY USES FOR THIS PROCESS
---------------------------------------
100%   15 GB, 1008569 chunks: "pl/sql vc2                "  PL/SQL
        koh-kghu call   ds=fffffc7ffc6f51f8  dsprt=c715710
0%   15 MB, 15763 chunks: "free memory               "
        pga heap        ds=c715710  dsprt=0


This is due to bug 14119856 when real free allocator is used even though pga_aggregate_target is set more than 16GB.
Use below query to check if real free allocator is used:

SQL> col name format a30
col cur_val format a20
select i.ksppinm name , v.ksppstvl cur_val, v.ksppstdf default_val,v.ksppstvf
from x$ksppi i, x$ksppcv v where i.indx = v.indx and i.ksppinm in
('_realfree_heap_pagesize_hint', '_use_realfree_heap');SQL> SQL>   2    3

NAME                           CUR_VAL              DEFAULT_V   KSPPSTVF
------------------------------ -------------------- --------- ----------
_realfree_heap_pagesize_hint   65536                TRUE               0
_use_realfree_heap             TRUE                 TRUE               0


Technique 1:


 Step 1:
• Restart the database and  server in order to fix the issue
Or
Change the upper limit at either the OS or at the database level:


• Change the page count at the OS level:

by root user,
$ more /proc/sys/vm/max_map_count
$ sysctl -w vm.max_map_count=200000 (for example)

OR at database level,
• Adjust the realfree heap pagesize within the database by setting the following parameters in the init/spfile and restart the database.

_use_realfree_heap=TRUE
_realfree_heap_pagesize_hint = 262144
- OR -
Use Workaround:

 Set "_use_realfree_heap=false" and restart database instance.

Or

 Apply patch <="" a="">14119856> if available for your platform and Oracle version or request for a one-off patch.

 Reference  MI note :: Doc ID 1506315.1 and Thanks for the giving time and reading the post .

ORA-00119: invalid specification for system parameter LOCAL_LISTENER

ERROR  ::   ORA-00119: invalid specification for system parameter LOCAL_LISTENER

Issue: ORA-00119: invalid specification for system parameter LOCAL_LISTENER

ORA-00132: syntax error or unresolved network name 'LISTENER_DATABASE NAME'

Couse: 1) You have not made the entry of local listener in TNSNAMES.ora file and parameter is added in spfile or pfile . In order to avoid the above error we have two options as per my understanding (if you have plz comment and share your solution.)

Technique 1:

 Step 1: Make an entry in the tnsnames.ora as mentioned below

LISTENER_<NAME/DATABASE NAME> =

  (ADDRESS = (PROTOCOL = TCP)(HOST = my.host.com)(PORT = 1521))

Where LISTENER_<NAME/DATABASE NAME>  is the parameter value for LOCAL_LISTENER which is mentioned in pfile/spfile

PORT is the listener port you are using.

Also there is no need to modify the pfile/spfile.

Technique 2 :

Step 2:

a)      If you are using a spfile for starting up your database, create a pfile from it and remove the LOCAL_LISTNER parameter and Create spfile from the pfile .

b)      Strat the database .

        sqlplus / as sysdba

              startup ;

c)        start the listener  lsnrctl start

ORA-00845: MEMORY_TARGET not supported on this system and ORA-00093: _shared_pool_reserved_min_alloc must be between 4000 and 0

Database not coming up due to below errors (ORA-00845: MEMORY_TARGET not supported on this system and ORA-00093: _shared_pool_reserved_min_alloc must be between 4000 and 0)

Scene 1)

##########################################
SQL> startup nomount;

ORA-00845: MEMORY_TARGET not supported on this system

SQL> shut abort

ORACLE instance shut down.

SQL> exit

Disconnected

Scene 2)
#########################################

SQL> startup nomount;

ORA-01078: failure in processing system parameters

ORA-00093: _shared_pool_reserved_min_alloc must be between 4000 and 0

SQL> shut abort;

ORACLE instance shut down.

SQL> exit

Disconnected

Scene 3)
#########################################

SQL> startup nomount

ORA-01078: failure in processing system parameters

ORA-00838: Specified value of MEMORY_TARGET is too small, needs to be at least 3072M

SQL>

SQL> exit

Disconnected

#########################################

These are all due to memory Memory related parameter which is not specified as it should be. 

Solution: 

Scene 1) This error always come if /dev/shm FS size is less then memory target parameter.   

To overcome from this situation either increase the /dev/shm to greater than memory target or keep less value of memory target from /dev/shm. 

Other way is to use SGA_MAX_SIZE instead of MEMORY_TARGET. 

Scene 2) This error only came if SGA_MAX_SIZE is set but aggregate value of  individual memory parameter (db_cache_size, shared_pool_size, java_pool_size, stream_pool_size etc..) value is more than SGA_MAX_SIZE

To overcome from this situation either increase the SGA_MAX_SIZE or decrease aggregate value of  individual memory parameter.

Scene 3) in case by mistaken SGA_MAX_SIZE and MEMORY_TARGET both has been define, and SGA_MAX_SIZE is greater than MEMORY_TARGET. 

To overcome from this situation remove or keep size 0  either SGA_MAX_SIZE or MEMORY_TARGET parameter value

[oracle@abcdbhost01 dbs]$ sqlplus '"as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Wed Sep 10 20:48:05 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>  startup nomount

ORACLE instance started.

Total System Global Area 3340451840 bytes

Fixed Size                  2232960 bytes

Variable Size             771755392 bytes

Database Buffers         2550136832 bytes

Redo Buffers               16326656 bytes

SQL>

Database not starting due to ORA 00205 error in identifying control file

Database not starting up due to ORA-00205 error in identifying control file

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2085360 bytes
Variable Size            1040190992 bytes
Database Buffers         1090519040 bytes
Redo Buffers               14688256 bytes
ORA-00205: error in identifying control file, check alert log for more info

If you are getting above error during starting up database,

shutdown the database and check database datafiles and controlfiles file system mount option. 

sometime it is not mounted with nolock option  in mostly linux system. 

run mount command to check current mount option:

mount

If output is like below for FS which contains your datafiles 

xxxxxxxxxxxxxxxxxx:/vol/xxxxxxxxx01/oradata on /oradata type nfs (rw,hard,wsize=32768,rsize=32768,addr=10.20.16.52)

Check FSTAB file

cat /etc/fstab

take backup of fstab  file

sudo cp /etc/fstab /etc/fstab_bkp

change the same entry in fstab

 sudo vi /etc/fstab

edit the /etc/fstab file like below: 

xxxxxxxxxxxxxxxx:/vol/xxxxxxxxxxxxxxxxxoradata /oradata        nfs      rw,nolock,hard,wsize=32768,rsize=32768,actimeo=120,timeo=600,tcp   1 2

Aftet this,  unmount the /oradata FS

sudo umount /oradata

Remount the same:

sudo mount  xxxxxxxxxxxxxxxxxxx:/vol/xxxxxxxxxxxxxx/oradata /oradata

If it doesn't work then 

use mount the same Fs with below comand: 

mount the FS with proper mount option for nfs

sudo mount -t nfs -o rw,nolock,hard,wsize=32768,rsize=32768,actimeo=120,timeo=600,tcp fcgsnprdsng03b:/vol/gemdbval01/oradata /oradata

check again for mount option

 mount

if it looks good this time then 

start the database