Cloudibee

linux - storage - virtualization

Just saw this article in linux.com, where NetFlix architect presents his Linux Performance Observability tools diagram. Perhaps the most inclusive synopsis I’ve ever come across.

 

 

http://www.brendangregg.com/blog/2014-08-23/linux-perf-tools-linuxcon-na-2014.html

I tried to install some xml related rpms on my Fedora, so checked yum list to see what xml rpms are installed. But the system gave me the below error.

[[email protected] home]# yum list | grep -i xml
 rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
 error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
 error: cannot open Packages index using db3 -  (-30974)
 error: cannot open Packages database in /var/lib/rpm
 CRITICAL:yum.main:
 Error: rpmdb open failed
[[email protected] home]#

I inititally thought it was some issue with yum and its cache, so tried yum clean. That did not work!.

[[email protected] home]# yum clean
 rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
 error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
 error: cannot open Packages index using db3 -  (-30974)
 error: cannot open Packages database in /var/lib/rpm
 CRITICAL:yum.main:
 Error: rpmdb open failed
[[email protected] home]# rpm -qa
 rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
 error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
 error: cannot open Packages index using db3 -  (-30974)
 error: cannot open Packages database in /var/lib/rpm
 rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
 error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
 error: cannot open Packages database in /var/lib/rpm

Taking a closer look, it sounded like rpm db is corrupted.

[[email protected] home]# ls /var/lib/rpm
Basenames  Conflictname  __db.001  __db.002  __db.003  __db.004  Dirnames  Filedigests  Group  Installtid  Name  Obsoletename  Packages  Providename  Provideversion  Pubkeys  Requirename  Requireversion  Sha1header  Sigmd5  Triggername
[[email protected] home]#
[[email protected] home]# rpm --rebuilddb
rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
[[email protected] home]#

Deleted the __db* files and did a “rpm –rebuilddb”. It worked perfectly and I was able to do rpm and yum commands.

[[email protected] home]#
[[email protected] home]# rm -f /var/lib/rpm/__*
[[email protected] home]# rpm --rebuilddb
[[email protected] home]# rpm -qa | wc -l
629
[[email protected] home]#

This post contains the list of commands that will be most used and will come handy when managing or monitoring or troubleshooting a Netapp filer in 7-mode.

  • sysconfig -a : shows hardware configuration with more verbose information
  • sysconfig -d : shows information of the disk attached to the filer
  • version : shows the netapp Ontap OS version.
  • uptime : shows the filer uptime
  • dns info : this shows the dns resolvers, the no of hits and misses and other info
  • nis info : this shows the nis domain name, yp servers etc.
  • rdfile : Like “cat” in Linux, used to read contents of text files/
  • wrfile : Creates/Overwrites a file. Similar to “cat > filename” in Linux
  • aggr status : Shows the aggregate status
  • aggr status -r : Shows the raid configuration, reconstruction information of the disks in filer
  • aggr show_space : Shows the disk usage of the aggreate, WAFL reserve, overheads etc.
  • vol status : Shows the volume information
  • vol status -s : Displays the spare disks on the filer
  • vol status -f : Displays the failed disks on the filer
  • vol status -r : Shows the raid configuration, reconstruction information of the disks
  • df -h : Displays volume disk usage
  • df -i : Shows the inode counts of all the volumes
  • df -Ah : Shows “df” information of the aggregate
  • license : Displays/add/removes license on a netapp filer
  • maxfiles : Displays and adds more inodes to a volume
  • aggr create : Creates aggregate
  • vol create : Creates volume in an aggregate
  • vol offline : Offlines a volume
  • vol online : Onlines a volume
  • vol destroy : Destroys and removes an volume
  • vol size [+|-] : Resize a volume in netapp filer
  • vol options : Displays/Changes volume options in a netapp filer
  • qtree create : Creates qtree
  • qtree status : Displays the status of qtrees
  • quota on : Enables quota on a netapp filer
  • quota off : Disables quota
  • quota resize : Resizes quota
  • quota report : Reports the quota and usage
  • snap list : Displays all snapshots on a volume
  • snap create : Create snapshot
  • snap sched : Schedule snapshot creation
  • snap reserve : Display/set snapshot reserve space in volume
  • /etc/exports : File that manages the NFS exports
  • rdfile /etc/exports : Read the NFS exports file
  • wrfile /etc/exports : Write to NFS exports file
  • exportfs -a : Exports all the filesystems listed in /etc/exports
  • cifs setup : Setup cifs
  • cifs shares : Create/displays cifs shares
  • cifs access : Changes access of cifs shares
  • lun create : Creates iscsi or fcp luns on a netapp filer
  • lun map : Maps lun to an igroup
  • lun show : Show all the luns on a filer
  • igroup create : Creates netapp igroup
  • lun stats : Show lun I/O statistics
  • disk show : Shows all the disk on the filer
  • disk zero spares : Zeros the spare disks
  • disk_fw_update : Upgrades the disk firmware on all disks
  • options : Display/Set options on netapp filer
  • options nfs : Display/Set NFS options
  • options timed : Display/Set NTP options on netapp.
  • options autosupport : Display/Set autosupport options
  • options cifs : Display/Set cifs options
  • options tcp : Display/Set TCP options
  • options net : Display/Set network options
  • ndmpcopy : Initiates ndmpcopy
  • ndmpd status : Displays status of ndmpd
  • ndmpd killall : Terminates all the ndmpd processes.
  • ifconfig : Displays/Sets IP address on a network/vif interface
  • vif create : Creates a VIF (bonding/trunking/teaming)
  • vif status : Displays status of a vif
  • netstat : Displays network statistics
  • sysstat -us 1 : begins a 1 second sample of the filer’s current utilization (crtl – c to end)
  • nfsstat : Shows nfs statistics
  • nfsstat -l : Displays nfs stats per client
  • nfs_hist : Displays nfs historgram
  • statit : beings/ends a performance workload sampling [-b starts / -e ends]
  • stats : Displays stats for every counter on netapp. Read stats man page for more info
  • ifstat : Displays Network interface stats
  • qtree stats : displays I/O stats of qtree
  • environment : display environment status on shelves and chassis of the filer
  • storage show <disk|shelf|adapter> : Shows storage component details
  • snapmirror intialize : Initialize a snapmirror relation
  • snapmirror update : Manually Update snapmirror relation
  • snapmirror resync : Resyns a broken snapmirror
  • snapmirror quiesce : Quiesces a snapmirror bond
  • snapmirror break : Breakes a snapmirror relation
  • snapmirror abort : Abort a running snapmirror
  • snapmirror status : Shows snapmirror status
  • lock status -h : Displays locks held by filer
  • sm_mon : Manage the locks
  • storage download shelf : Installs the shelf firmware
  • software get : Download the Netapp OS software
  • software install : Installs OS
  • download : Updates the installed OS
  • cf status : Displays cluster status
  • cf takeover : Takes over the cluster partner
  • cf giveback : Gives back control to the cluster partner
  • reboot : Reboots a filer
If you are not aware of the complete details of these commands and need more information on these commands, refer the Netapp Data Ontap administration manual from now site.

More netapp blog posts at : http://linux.cloudibee.com/tag/netapp/

OCFS2 is a POSIX-compliant shared-disk cluster file system for Linux capable of providing both high performance and high availability.  Cluster-aware applications can make use of parallel I/O for higher performance. OCFS2 is mostly used to host Oracle Real application clusters (RAC) database on Linux clusters. 
The below steps shows how to create ocfs2 filesystem on top a multipath’d SAN lun and mount it on Linux clusters.
  1. Identify the nodes that will be part of your cluster.
  2. Export/Zone the LUNs on the SAN end and check whether they are accessible on all the hosts of the cluster. (fdisk -l or multipath -ll)
  3. If you need multipathing, configure multipath and the multipathing policy based on your requirement. For Linux multipath setup, refer Redhat’s multipath guide.
  4. Create OCFS2 configuration file (/etc/ocfs2/cluster.conf) on all the cluster nodes.
  5. The example presents you a sample cluster.conf for a 3 node pool. If you have heartbeat IP configured on these cluster nodes, use the heartbeat IP for ocfs2 cluster communication and specify the hostname (without FQDN). Copy the same file to all the hosts in the cluster.

    [[email protected] ~]# cat /etc/ocfs2/cluster.conf
    node:
            ip_port = 7777
            ip_address = 203.21.2.101
            number = 0
            name = oracle-cluster-1
            cluster = ocfs2

    node:
            ip_port = 7777
            ip_address = 203.21.2.102
            number = 1
            name = oracle-cluster-2
            cluster = ocfs2

    node:
            ip_port = 7777
            ip_address = 203.21.2.103
            number = 2
            name = oracle-cluster-3
            cluster = ocfs2

    cluster:
            node_count = 3
            name = ocfs2

    [[email protected] ~]#

  6. On each node check the status of OCFS2 cluster service and stop “o2cb” if the service is already running.

    # service o2cb status
    # service o2cb stop

  7. On each node, load the OCFS2 module.

    # service o2cb load

  8. Make the OCFS2 service online on all the nodes.

    # service o2cb online
  9. Now your OCFS2 cluster is ready.
  10. Format the SAN lun device from any one of the cluster node.

    # mkfs.ocfs2 -b 4k -C 32k -L oraclerac /dev/mapper/mpath0

    -b : Block size (values are 512, 1K, 2K and 4K bytes per block)
    -C : Cluster size (values are 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K and 1M)
    -L : Label


    Note : Replace /dev/mapper/mpath0 with your device name.
  11. Update /etc/fstab on all the nodes in the cluster with the mount point.

    Like : /dev/mapper/mpath0 /u01 ocfs2 _netdev 0 0

  12. Mount the /u01 volume using mount command

    # mount /u01

  13. Enable ocfs and o2b service at runlevel 3.

    # chkconfig –level 345 o2cb on ; chkconfig –level 345 ocfs2 on

  14. The /u01 repository setup on a SAN Lun is done.
  15. You can now configure Oracle RAC database on this filesystem.