Celerra Health Check with CLI Commands Here are the first commands I¡¯ll type when I suspect there is a problem with the Ce lerra, or if I want to do a simple health check. 1. <watch> /nas/sbin/getreason. This will quickly give you the current status o f each data mover. 5=up, 0=down/rebooting. Typing watch before the command will run the command with continuous updates so you can monitor a datamover if you a re purposely rebooting it. 10 ¨C slot_0 primary control station 5 ¨C slot_2 ed 5 ¨C slot_3 ed 2. nas_server -list. This lists all of the datamovers and their current state. It¡¯s a good way to quickly tell which datamovers are active and which are standby. 1=nas, 2=unused, 3=unused, 4=standby, 5=unused, 6=rdf id 1 2
type acl slot groupID state name 1 0 2 0 4 0 3 0
server_2 server_3
3. server_sysstat. This will give you a quick overview of memory and U utiliz ation. server_2 : threads runnable threads blocked threads I/J/Z memory free(kB) u idle_% 4. nas_checkup.
= = = = =
6 4001 1 2382807 70 This runs a system health check.
Check Version: 5.6.51.3 Check Command: /nas/bin/nas_checkup Check Log : /nas/log/checkup-run.110608-143203.log ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª-Checks¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ªControl Station: Checking if file system usage is under limit¡¡¡¡.. Control Station: Checking if NAS Storage API is installed correctly¡¡.. ¡ 5. server_log server_2. This shows the current alert log. Alert logs are also stored in /nas/log/webui. 6. vi /nas/jserver/logs/system_log.
This is the java system log.
7. vi /var/log/messages. This displays system messages. -------------------
EMC NAS / VNX Health Checkup using command line August 7, 2012 Leave a comment using nas and the system¡¯s health, type:
$ /nas/bin/nas_checkup The checkup command reports back on the state of the Control Station, Data Mover s, and storage system. Note: This health check ensures that there are no major errors in the system tha t would prevent the system from being turned on during the power up process. [nas@VNXCS01 ~]$ /nas/bin/nas_checkup Check Version: 7.0.51.3 Check Command: /nas/bin/nas_checkup Check Log : /nas/log/checkup-run.120807-113919.log ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª-Checks¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ªControl Station: Checking statistics groups database¡¡¡¡¡¡¡.. Control Station: Checking if file system usage is under limit¡¡¡¡.. Control Station: Checking if NAS Storage API is installed correctly¡¡.. Control Station: Checking if NAS Storage APIs match¡¡¡¡¡¡¡¡ Control Station: Checking if NBS clients are started¡¡¡¡¡¡¡.. Control Station: Checking if NBS configuration exists¡¡¡¡¡¡¡. Control Station: Checking if NBS devices are accessible¡¡¡¡¡¡.. Control Station: Checking if NBS service is started¡¡¡¡¡¡¡¡ Control Station: Checking if PXE service is stopped¡¡¡¡¡¡¡¡ Control Station: Checking if standby is up¡¡¡¡¡¡¡¡¡¡¡ Control Station: Checking integrity of NASDB¡¡¡¡¡¡¡¡¡¡. Control Station: Checking if primary is active¡¡¡¡¡¡¡¡¡.. Control Station: Checking all callhome files delivered¡¡¡¡¡¡¡ Warn Control Station: Checking resolv conf¡¡¡¡¡¡¡¡¡¡¡¡.. Control Station: Checking if NAS partitions are mounted¡¡¡¡¡¡.. Control Station: Checking ipmi connection¡¡¡¡¡¡¡¡¡¡¡. Control Station: Checking nas site eventlog configuration¡¡¡¡¡¡ Control Station: Checking nas sys mcd configuration¡¡¡¡¡¡¡¡ Control Station: Checking nas sys eventlog configuration¡¡¡¡¡¡. Control Station: Checking logical volume status¡¡¡¡¡¡¡¡¡. Control Station: Checking valid nasdb backup files¡¡¡¡¡¡¡¡. Control Station: Checking root disk reserved region¡¡¡¡¡¡¡¡ Control Station: Checking if RDF configuration is valid¡¡¡¡¡¡.. N/A Control Station: Checking if fstab contains duplicate entries¡¡¡¡.. Control Station: Checking if sufficient swap memory available¡¡¡¡.. Control Station: Checking for IP and subnet configuration¡¡¡¡¡¡ Control Station: Checking auto transfer status¡¡¡¡¡¡¡¡¡.. Warn Control Station: Checking for invalid entries in etc hosts¡¡¡¡¡.. Control Station: Checking the hard drive in the control station¡¡¡¡ Control Station: Checking if Symapi data is present¡¡¡¡¡¡¡¡ Control Station: Checking if Symapi is synced with Storage System¡¡¡. Blades : Checking boot files¡¡¡¡¡¡¡¡¡¡¡¡¡ Blades : Checking if primary is active¡¡¡¡¡¡¡¡¡.. Blades : Checking if root filesystem is too large¡¡¡¡¡¡ Blades : Checking if root filesystem has enough free space¡¡¡ Blades : Checking network connectivity¡¡¡¡¡¡¡¡¡.. Blades : Checking status¡¡¡¡¡¡¡¡¡¡¡¡¡¡. Blades : Checking dart release compatibility¡¡¡¡¡¡¡.. Blades : Checking dart version compatibility¡¡¡¡¡¡¡.. Blades : Checking server name¡¡¡¡¡¡¡¡¡¡¡¡.. Blades : Checking unique id¡¡¡¡¡¡¡¡¡¡¡¡¡. Blades : Checking CIFS file server configuration¡¡¡¡¡¡. Blades : Checking domain controller connectivity and configuration. Blades : Checking DNS connectivity and configuration¡¡¡¡¡ Blades : Checking connectivity to WINS servers¡¡¡¡¡¡¡ Blades : Checking I18N mode and unicode translation tables¡¡¡ Blades : Checking connectivity to NTP servers¡¡¡¡¡¡¡. Warn Blades : Checking connectivity to NIS servers¡¡¡¡¡¡¡.
Blades : Checking virus checker server configuration¡¡¡¡¡ Blades : Checking if workpart is OK¡¡¡¡¡¡¡¡¡¡.. Blades : Checking if free full dump is available¡¡¡¡¡¡. Blades : Checking if each primary Blade has standby¡¡¡¡¡. Blades : Checking if Blade parameters use EMC default values¡¡. Blades : Checking VDM root filesystem space usage¡¡¡¡¡¡ N/A Blades : Checking if file system usage is under limit¡¡¡¡.. Blades : Checking slic signature¡¡¡¡¡¡¡¡¡¡¡.. Storage System : Checking disk emulation type¡¡¡¡¡¡¡¡¡¡ Storage System : Checking disk high availability access¡¡¡¡¡¡.. Storage System : Checking disks read cache enabled¡¡¡¡¡¡¡¡. Storage System : Checking disks and storage processors write cache enabled. Storage System : Checking if FLARE is committed¡¡¡¡¡¡¡¡¡. Storage System : Checking if FLARE is ed¡¡¡¡¡¡¡¡¡. Storage System : Checking array model¡¡¡¡¡¡¡¡¡¡¡¡.. Storage System : Checking if microcode is ed¡¡¡¡¡¡¡¡ N/A Storage System : Checking no disks or storage processors are failed over¡ Storage System : Checking that no disks or storage processors are faulted.. Storage System : Checking that no hot spares are in use¡¡¡¡¡¡.. Storage System : Checking that no hot spares are rebuilding¡¡¡¡¡. Storage System : Checking minimum control lun size¡¡¡¡¡¡¡¡. Storage System : Checking maximum control lun size¡¡¡¡¡¡¡¡. N/A Storage System : Checking maximum lun address limit¡¡¡¡¡¡¡¡ Storage System : Checking system lun configuration¡¡¡¡¡¡¡¡. Storage System : Checking if storage processors are read cache enabled¡.. Warn Storage System : Checking if auto assign are disabled for all luns¡¡¡ N/A Storage System : Checking if auto tres are disabled for all luns¡¡. N/A Storage System : Checking storage processor connectivity¡¡¡¡¡¡. Storage System : Checking control lun ownership¡¡¡¡¡¡¡¡¡. N/A Storage System : Checking if Fibre Channel zone checker is set up¡¡¡. N/A Storage System : Checking if Fibre Channel zoning is OK¡¡¡¡¡¡.. N/A Storage System : Checking if proxy arp is setup¡¡¡¡¡¡¡¡¡. Storage System : Checking if Product Serial Number is Correct¡¡¡¡.. Storage System : Checking SPA SPB communication¡¡¡¡¡¡¡¡¡. Storage System : Checking if secure communications is enabled¡¡¡¡.. Storage System : Checking if backend has mixed disk types¡¡¡¡¡¡ Storage System : Checking for file and block enabler¡¡¡¡¡¡¡.. Storage System : Checking if nas storage command generates discrepancies¡ Storage System : Checking if Repset and CG configuration are consistent¡. Storage System : Checking block operating environment¡¡¡¡¡¡¡. Storage System : Checking thin pool usage¡¡¡¡¡¡¡¡¡¡¡. N/A Storage System : Checking for domain and federations health on VNX¡¡¡ ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¨C One or more warnings have occurred. It is recommended that you follow the instructions provided to correct the problem then try again. ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¨CInformation¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ªControl Station: Check if standby is up Information HC_CS_27389984778: The standby Control Station is currently powered on. It will be powered off during upgrade, and then later restarted and upgraded. ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¨C ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ªWarnings¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª Control Station: Check all callhome files delivered Warning HC_CS_18800050328: There are 36 undelivered Call Home incidents and 3 scheduled Call Home files left in the /nas/log/ConnectHome directory(es)
Action : Check the /nas/log/connectemc/ConnectEMC log to ensure the connection is established correctly. To test your Callhome configuration, you can run /nas/sbin/nas_connecthome -test { -email_1 | -email_2 | -ftp_1 | -ftp_2 | -modem_1 | -modem_2 } command. View the RSC*.xml files under the /nas/log/ConnectHome directory(es) and inspect the CDATA content to find out and possibly resolve the problem. To remove the call home incidents and files, run the command ¡°/nas/sbin/nas_connecthome service clear¡±. Otherwise escalate this issue through your organization. Control Station: Check auto transfer status Warning HC_CS_18800050417: The automatic transfer feature is disabled. Action : EMC recommends the automatic transfer feature to be enabled via command: /nas/tools/automaticcollection -enable or from Unisphere: 1. Select VNX > [VNX_name] > System. Click the link for ¡°Manage Log Collection for File¡± Under Service Tasks. 2. Select Enable Automatic Transfer. 3. Click Apply. By default, materials will be transferred to ftp.emc.com, but you can modify the location in the /nas/site/automaticcollection.cfg file. For more information, search the Knowledgebase on Powerlink as follows: 1. to http://powerlink.emc.com and go to > Knowledgebase Search> Solutions Search. 2. Use ID emc221733 to search. Blades : Check connectivity to NTP servers Warning HC_DM_18800115743: * server_2: Only one NTP server is configured. It is recommended to define at least two different NTP servers for a high availability. If the clock of the Data Mover is not correct, potential errors during Kerberos authentication may happen (timeskew). Action : Use the server_date command to define another NTP server on the Data Mover. Read the man pages for details and examples. Storage System : Check if storage processors are read cache enabled Warning HC_BE_18799984735: SPA Read Cache State on VNX FCN0xxxxxxxx5 is not enabled Action : Please EMC Customer Service for assistance. Include this log with your request. Storage System : Check if storage processors are read cache enabled Warning HC_BE_18799984735: SPB Read Cache State on VNX FCNxxxxxxxxx5 is not enabled Action : Please EMC Customer Service for assistance. Include this log with your request. ¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¨C [nas@VNXCS01 ~]$