Rsync command syntax

Below given is the recommended Rsync syntax which can be used to sync 2 file-systems. However appropriate options can be chosen based on the requirement:
 Source machine# nohup rsync -atlrzvuop --progress /<source folder>/  <Destination IP/Hostname>:/<Dest folder>/  &
-a  :  Archive mode
-t  :   Preserve modification time
-l  :   Preserve symbolic links
-r :   Copy recursively
-v :   Verbose
-z  :  Zip the files before transfer
-u :  Skips files which exist on the destination and have a modified time newer than the source file.
-p :  Preserve permissions
-o :  Preserve owner
--bwlimit=<kpbs> : To control the I/O bandwidth if you are syncing across WAN (e.g. --bwlimit= 30000  - to sync at 30 Mbps)
--delete  :   Delete all the extra files on Destination which aren’t present in the Source.
--exclude=<pattern>  :   Exclude files from syncing whose filenames matching the PATTERN  (e.g. --exclude=.zip )

Automated Rsync script:
http://smart-scripts-for-sysadmins.blogspot.com/2012/03/sample-rsync-script.html

Finding Start time of a Linux Process

In Linux, finding Starting time of a process using 'top' output will always tend to confuse us. Below is a sample 'top' output of a bunch of 'cpio' process which are running on a Server. The TIME+ column shows some time but it is quite confusing to interpret.

Following are the command-set that I use to find actual Start-time of a Process and how long it runs:

[root@linuxhost ~]# ps -eo pid,lstart,cmd | grep -v grep | grep 14115

14115 Sat Jun  2 01:15:38 2012 cpio -pdmu /NA_cdb01_ud4001

[root@iss-365-tdb02 ~]# date

Sat Jun  2 02:01:41 PDT 2012

[root@linuxhost ~]# ps -eo pid,etime,cmd | grep -v grep | grep 14115

14115       46:23 cpio -pdmu /NA_cdb01_ud4001

[root@iss-365-tdb02 ~]#

The above output shows the Process with PID 14115 started at 01:15:38 AM PDT and it is running for more than 45 minutes.

Script to generate file-system usage report

This script is developed specifically for checking the File-system usage of SAN Mounts on Linux or any UNIX flavors. 
However this can be used to check any Unix filesystem such as /var, /opt, /root etc. This script takes Access, Modified and Change Time as parameters on the specified Mount points and followed by an E-mail address. Upon execution, it will send you the resultant output of the script to the E-mail address that you specified in sorted form with the total size of matching files. In addition a copy of the script output will be stored on the path where you executed the script in the following format: Hostname-Month-Day-Year.

URL to download: http://dl.dropbox.com/u/50666315/scripts/SAN_Mount_Check.pl

Supported platforms: Any Unix platform with Perl version 5.x or above installed.

How to use:

The syntax for executing this script is as follows:

# SAN_Mount_Check.pl [-t]  [-a <value> -m <value> -c <value>]  <Mount points  separated by space>

“-t” for Total disk space

“-a” for access time

“-m” for modification time

“-c” for change time

Upon executing the above command will prompt you to enter the E-mail address to which the file-system usage report needs to be sent. Please note the switch “-t” is optional whereas the other switches “ -a, -m, -c “ are must (atleast one or more). You can give multiple filesystem pathnames separated by space but I suggest giving just one in-order to avoid confusion in sorted output; where the files will be sorted in accordance to its size.

Few Sample Syntax:

# perl SAN_Mount_Check.pl -t -a 1000 /san_mount1      <-- List Files which are accessed 1000 days before.

# perl SAN_Mount_Check.pl -t -c 1000 /Netapp_filer_mount2  <-- Files which are changed 1000 days before.

# perl SAN_Mount_Check.pl -t -m 300  /var                <-- Files modified 300 days before in /var

# perl SAN_Mount_check.pl -t -a 500 -m 750 /u01      <-- Files which are accessed 500 days before and modified 750 days before on /u01.

# perl SAN_Mount_Check.pl -t -a 365 /san_mnt1 /san_mnt2     <-- Files which are accessed 365 days before on 2 SAN mounts

How it works:

This script uses &wanted subroutine which comes along with standard Perl Module “File::Find” for traversing through all the files in a desired filesystem. For each file it finds, it checks whether it matches with the parameters supplied and if it is true, it will output the file information along with file size information.  This is a non-intrusive script, as this is equivalent to executing a ‘find’ command. 

Enabling ACL in /etc/fstab

I wish to set FACL for a file-system "/ua1003" with Read-Write permission for my user-account 'ashok'. When I try to do it, I noticed the file-system isn't mounted with ACL feature enabled. I have added the 'acl' option in /etc/fstab for that mount point and did a remount. After that, I was able to set the FACL. Below shown is the sequence which I did to make the ACL work:

[root@hostxyz ua1003]# setfacl -m u:ashok:rwx /ua1003                          
setfacl: /ua1003: Operation not supported                                                  <-- ACL not enabled
[ashok@hostxyz /]$ grep /ua1003 /etc/fstab
LABEL=/ua1003           /ua1003                   ext3    defaults        0  0        
[ashok@hostxyz /]$ vi /etc/fstab
[root@hostxyz ~]# grep /ua1003 /etc/fstab
LABEL=/ua1003           /ua1003                   ext3    defaults,acl        0  0   <-- Enabled ACL feature in /etc/fstab
[root@hostxyz ~]# mount -o remount /ua1003                                           <-- Remounting /ua1003
[root@hostxyz ~]# setfacl -m u:ashok:rwx /ua1003                                    <-- Setting FACL to the parent folder
[root@hostxyz ~]# getfacl /ua1003
getfacl: Removing leading '/' from absolute path names
# file: ua1003
# owner: applprod
# group: dba
user::rwx
user:ashok:rwx
group::r-x
mask::rwx
other::r-x
[root@hostxyz ~]# setfacl -R -m u:ashok:rwx /ua1003                               <--Setting FACL recursively
[root@hostxyz ~]# getfacl /ua1003
getfacl: Removing leading '/' from absolute path names
# file: ua1003
# owner: applprod
# group: dba
user::rwx
user:ashok:rwx
group::r-x
mask::rwx
other::r-x
[root@hostxyz ~/#

PS: To check if ACL is enabled on a given file-system, execute 'tune2fs -l <devicename>' command and look out for "Default mount options".

How to encrypt and decrypt a file in Linux ?

There could be few ways to Encrypt and Decrypt a file in Linux. The one which I use is, gpg (GnuPG).
Below shown are the steps to encrypt and decrypt a file called "confidential.txt".

[root@hostxyz ashok]# echo 'newpassXYZ' > confidentail.txt
[root@hostxyz ashok]# cat confidentail.txt
newpassXYZ
[root@hostxyz ashok]# gpg -c confidential.txt
Enter passphrase:  <secret word>
Repeat passphrase: <secret word>
[root@hostxyz ashok]# ls -l confidentail.*
-rw-r--r-- 1 root root 11 May 22 22:01 confidential.txt
-rw-r--r-- 1 root root 66 May 22 22:01 confidential.txt.gpg
[root@hostxyz ashok]# cat confidential.txt.gpg
ê2Ãà pisÃu?î^ó5Ã\<dÃ
â[root@hostxyz ashok]# mv confidential.txt.gpg /tmp
[root@hostxyz ashok]# cd /tmp
[root@hostxyz tmp]# gpg -d confidential.txt.gpg
gpg: CAST5 encrypted data
Enter passphrase:  <secret word>
[root@hostxyz tmp]# cat confidential.txt
newpassXYZ
[root@hostxyz tmp]#

The config file for 'gpg' is /root/.gnupg/gpg.conf

Finding number of occurrence of a string in a file.

Let's say you have a file (filename: testfile) as shown below and you want to find number of occurrence of a string 'snapshot' in this file.  Please note the specified string could occur more than once in a line. 

"A disk "Snapshot" is a copy of the virtual machine disk file (VMDK) at a certain point in time. It preserves the disk file system,

system memory of your VM by enabling you to revert to the snapshot in case something goes wrong. Snapshot can 

upgrading or patching applications and servers. This article will go over everything you need to know about using snapshot,

including what they are, how they work and advanced techniques. A virtual machine provides several operations for managing

snapshots and snapshot chains. These operations let you create snapshots, revert to any snapshot in the chain, and remove snapshots. You can create extensive snapshot trees."

Command is:  grep -io snapshot testfile | wc -l

Re-executing a command from History after substitution

At times we might be executing a long command on Shell prompt and later we might have to re-run that same command-set with one small change. It will be real pain to type the same command-set again just for a one parameter change. Linux has a Bash built-in command called 'fc' (stands for find command) using which we can make this task simple.

Illustration:
Let's say you have executed the following command (bit long) some time ago:
"bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s300 -r iss-365-rhel5664-tmpl.xyz.com -c196"
Now you wish to execute the same command with one value changed in it. Instead of -s300, you want to re-run the command with the new value -s200.
hostxyz # fc -l               <-- Will list 16 most recent commands
427      ls
428      ls -l
429      vim NMON_startup_script.sh-May-16-12.log
430      sh /root/itc/hc/dyn/NMON_startup_script.sh
431      cat NMON_startup_script.sh-May-16-12.log
432      uptime
433      perl /opt/sysinfo.pl
434      ls -l /var/lock/subsys
435      bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s300 -r iss-365-rhel5664-tmpl.xyz.com -c196
436      cd ~ashok
437      ls
438      ps -ef | grep nmon
439      kill -15 3384
440      date
441      fc -l
442      ps -ef | grep nmon
hostxyz # fc -s s300=s200 bash      
bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s200 -r iss-365-rhel5664-tmpl.xyz.com -c150
hostxyz #
With the `fc -s [pat=rep ...] [command]' format, the command starting with 'bash' is re-executed after the substitution OLD=NEW is performed.
Note:  Typing just 'fc' will open the last command in Text Editor. If you want to open range of commands that you executed before in Text Editor, you can use the following syntax:  # fc [Start no]  [End no].

FUNC - Fedora Unified Network Controller

FUNC is an open source automation tool developed by using Python programming language. We can use this tool for automating system admin tasks such as status check, configuration tweak, file transfer, rebooting the systems etc in multiple Linux-based systems. It uses the typical Client-Server model, where the server is called by the term "overlord" and the clients which are binded to the server are called as "minions".

Check out this link for more details with examples:
http://advanced-sysadmin-stuff.blogspot.in/2011/03/func-fedora-unified-network-controller.html

I have developed an Installation bundle which consists of Script for installing and configuring the FUNC client with all the required packages. URL to download the software bundle : http://dl.dropbox.com/u/50666315/blog/func_pack.tar

Just download it, extract and install the FUNC client by executing the 'install_func_client.sh' script. http://dl.dropbox.com/u/50666315/scripts/install_func_client.sh

PS: If you are looking for a definitive solution on permanent basis to address all your challenges in remote administration of Linux-based servers, I would recommend to go for "Puppet" (http://puppetlabs.com) . With Puppet, you can achieve lot more things than FUNC. URL to download a book on Puppet: http://db.tt/LtZOcfZk

How to force a user to reset his password upon next login ?

Syntax     :  chage -d 0 [user-name]
Example   :  chage -d 0 ashok

How to use 'ls' command to list only directories?

# ls -ld */
(or)
# ls -lp | grep ^dr
(or)  
# ls -pl | grep /$


To display the size of each folder listed above:

# ls -ld */ | awk '{print $9}' | xargs du -sh

How to know File creation time in Linux ?

In Linux or any Unix flavors, file-creation time isn't maintained. It will get change when the file-content changes. I tested and confirmed this 1.5 years before and shared my test results in this post.
http://ashok-linux-tips.blogspot.com/2010/11/finding-file-creation-time-in-linux.html

Now the good news is, with 'EXT4' file-system it is possible to get the File-creation time.

Here's the test results performed on a EXT4 file-system:

root@Linux-Mach # df -h
Filesystem            Size    Used   Avail   Use% Mounted on
/dev/sda2             3.8G  2.2G    1.4G   62%   /
/dev/sda3             4.6G  139M  4.3G    4%     /home
/dev/sda1              46M   11M   33M   25%   /boot
tmpfs                   333M     0      333M  0%    /dev/shm
root@Linux-Mach # cd /opt
root@Linux-Mach # touch sample_file
root@Linux-Mach # ls -l sample_file
-rw-r--r-- 1 root root 0 2012-04-24 18:35 sample_file                    # File-creation time is 18:35

root@Linux-Mach # date
Tue Apr 24 18:40:12 CDT 2012
root@Linux-Mach # cat > sample_file           
This is test file to check creation time of this file.
Ctrl+D
root@Linux-Mach # ls -l sample_file  
-rw-r--r-- 1 root root 29 2012-04-24 18:40 sample_file
root@Linux-Mach # ls -i sample_file  
135528 sample_file                                                                    # inode number  
root@Linux-Mach # debugfs -R 'stat <135528>' /dev/sda2          
Inode 14552801   Type regular    Mode  0644   Flags 0x80000
Generation 340511001    Version 0x0000000000000001
User  1000   Group  1000   Size 29 
File ACL 0    Directory ACL 0 
Links 1   Blockcount 8 
Fragment  Address 0    Number 0    Size 0
 ctime 0x3f1da5b513cbff4 -- Tue Apr 24 18:40:51 2012
 atime 0x3f1da5ec8725434 -- Tue Apr 24 18:40:54 2012
 mtime 0x3f1da5b513cbff4 -- Tue Apr 24 18:40:51 2012
crtime 0x3f1cacc966104fc -- Tue Apr 24 18:35:28 2012             # we could see file creation time    
Size of extra inode fields 28
EXTENTS
(0) 44665199
debugfs 1.41.11 (14-Mar-2010)

root@Linux-Mach #

Please note the entry "crtime"  which displays the file-creation time. In my view, this information would be of great help in certain situations.

Listing and Extracting files from a RPM

As far I know, there are 3 ways available to list the files inside a RPM and they are as follows:
 To list only files:
# rpm -qlp  packagename.rpm

To list files with permission with ownership:
# rpm2cpio packagename.rpm | cpio -tv

 To list files with detailed description about the RPM:
# less packagename.rpm

Next to extract a file from a RPM, we can again use the 'rpm2cpio' command along with 'cpio' with different switches as shown as follows: 

 [root@hostxyx tmp]# rpm2cpio packagename.rpm | cpio -idvm
./etc/blkid
./etc/mke2fs.conf
./sbin/badblocks
./sbin/blkid
./sbin/debugfs
./sbin/dumpe2fs
./sbin/e2image
./sbin/fsck
./sbin/logsave
./sbin/resize2fs
./usr/bin/chattr

.

.

<Output truncated>

The above command will create directories such as ./etc, ./sbin, ./usr under the folder where RPM is kept. Please note this wouldn't install the RPM on the system, it will only extract the files. Once you grab the file that you wish to, just delete all the directories that are created. Never execute this command being in /, as it will mess up the entire file-system.

How to log commands executed by all the users in Linux?

By adding the following entry in /etc/bashrc, we can log the commands executed by all the users on a Linux machine. 
This would be certainly helpful for tracking commands on Critical servers.

PROMPT_COMMAND='history -a >(logger -t "$USER[$PWD] $SSH_CONNECTION")'

After you add the above entry at the end of /etc/bashrc file, execute the command 'source /etc/bashrc' or logout and login back to your session. Now the commands executed by all the users will be logged in /var/log/messages.

Note: If you wish to log the commands on to a different file, please check the solution given in the comments section.

Sample test result:

Apr 18 13:35:21 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: uptime
Apr 18 13:35:24 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: cd /opt
Apr 18 13:35:26 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: ls -lR
Apr 18 13:35:35 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: iostat -x 2
Apr 18 13:35:39 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: cd /root
Apr 18 13:35:39 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: ls -l
Apr 18 13:35:51 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: cd /home
Apr 18 13:35:52 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: ls
Apr 18 13:35:56 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: httpd -t

Apr 18 13:51:24 Linux-Mach test1[/home/test1] 192.168.6.8 9106 172.16.0.252 22: ls -l
Apr 18 13:53:20 Linux-Mach test1[/var/lock/subsys] 192.168.6.8 9106 172.16.0.252 22: cd /var/lock/subsys
Apr 18 13:53:30 Linux-Mach test1[/var/lock/subsys] 192.168.6.8 9106 10.160.0.252 22: ls -ltr

Updated one:

# echo $PROMPT_COMMAND

RETRN_VAL=$?;logger -p local3.debug "$(whoami)  $remoteip  [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//") [$RETRN_VAL]"

On RHEL 8.x

[root@rhel8 ]# cat /etc/profile.d/prompt.sh 

remoteip=$(who am i | awk '{print $5}' | sed "s/[()]//g")

export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local3.debug "$(whoami)  $remoteip  [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//") [$RETRN_VAL]"'

[root@rhel8 ]# 

Retrieving the file-system from a lost Linux partition

When you lost a Linux partition on Storage device, there is a 80% chance that you can retrieve the file-system in it just by Re-creating the lost Partition. I have tried couple of times in the recent past and it worked successfully.

Here's the sequence of steps which I performed to retrieve the file-system "/us2001" on a lost partition from a SAN disk:

[root@server-tap04 mapper]# mount -t ext3 /dev/mapper/us2001np1 /us2001

mount: special device /dev/mapper/us2001np1 does not exist

[root@server-tap04 mapper]# fdisk -l /dev/mapper/us2001n

 Disk /dev/mapper/us2001n: 10.7 GB, 10737418240 bytes

64 heads, 32 sectors/track, 10240 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/mapper/us2001n doesn't contain a valid partition table

[root@server-tap04 mapper]#

[root@server-tap04 mapper]# multipath -l | grep -A5 us2001n

us2001n (350002ac009290783)

[size=10 GB][features="1 queue_if_no_path"][hwhandler="0"]

\_ round-robin 0 [active]

\_ 1:0:0:1 sdc 8:32 [active]

\_ 2:0:0:1 sdb 8:16 [active]

[root@server-tap04 mapper]# fdisk -l /dev/sdc

 Disk /dev/sdc: 10.7 GB, 10737418240 bytes

 64 heads, 32 sectors/track, 10240 cylinders

 Units = cylinders of 2048 * 512 = 1048576 bytes

 Disk /dev/sdc doesn't contain a valid partition table

[root@server-tap04 mapper]# fdisk /dev/sdc

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-10240, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-10240, default 10240):

Using default value 10240

Command (m for help): w

The partition table has been altered!

 Calling ioctl() to re-read partition table.

Syncing disks.

[root@server-tap04 mapper]# kpartx -a /dev/mapper/us2001n

[root@ server-tap04 mapper]# kpartx -l /dev/mapper/us2001n

us2001np1 : 0 20971488 /dev/mapper/us2001n 32

[root@rica-chi-tap04 mapper]# mount -t ext3 /dev/mapper/us2001np1 /us2001

[root@rica-chi-tap04 mapper]# df -h /us2001
Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/us2001np1

                      9.9G  5.8G  4.1G  59% /us2001

[root@rica-chi-tap04 mapper]# cd /us2001                <-- Partition recreated and mounted  

[root@rica-chi-tap04 us2001]# ls -l                           <-- All the files are retrieved

total 28

-rw-r--r--  1 root     root     0 Jan 18  2011 2

-rw-r--r--  1 root     root     0 Jan 18  2011 abc

drwxr-xr-x  5 appltest dba   4096 Aug 10  2010 BI_Disco10g

drwx------  2 root     root 16384 Aug  6  2010 lost+found

drwxr-xr-x  4 appltest dba   4096 Mar 11  2011 product

[root@server-tap04 us2001]# touch xyz
[root@server-tap04 us2001]#