Environment variables in Linux

On a Linux system, we can view the Environment variables that are set and exported using 3 built-in commands: 

'set', 'env' & 'export -p'.

Ever wondered how can we see the other 'unset' environment variables belongs to Bash shell or some command or tool ?

The technique that I follow is to look in 2 places: One in the Binaries of the program which the application uses and other in the MAN page.

To view all the printable strings in a Bash shell:  # strings /bin/bash | grep -P '[A-Z]+'

In this output, there will be a lot of additional stuff and you will have to look for environment variables carefully

To print all the ENV variables related to 'history' command:

# man bash | sed 's/[[:cntrl:]].//g' | egrep -x ' +HIST[[:alpha:]]*'

       HISTCMD

       HISTCONTROL

       HISTFILE

       HISTFILESIZE

       HISTIGNORE

       HISTSIZE

       HISTTIMEFORMAT

A URL that contains all the Bash variables explaining each:

http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_02.html

Improving Read performance of Disks using 'blockdev'

The Read performance of a Disk can be improved by increasing a parameter called "Read+Ahead" using 'blockdev' command. By default the Linux OS will read 128 KB of data in advance so that it is already in Memory cache before the program needs it. This value can be increased so as to get better Read Performance.

Steps to check and increase the Read-Ahead value:

To check the current 'blockdev' status of all block device:   # blockdev --report
[root@linuxserver ~]# blockdev --report
RO    RA   SSZ   BSZ   StartSec     Size    Device
rw 16384   512  4096          0   14680064  /dev/sda
rw 16384   512  1024         63     208782  /dev/sda1
rw 16384   512   512     208845   10265535  /dev/sda2
rw 16384   512  4096          0 2147483648  /dev/sdb
rw 16384   512  4096          0   35651584  /dev/sdc
rw 16384   512  2048         63   35648172  /dev/sdc1
rw 16384   512  4096          0  104857600  /dev/sdd

To check the 'Read-Ahead' value of an individual disk (let's take sda)
# blockdev --getra /dev/sda
(or)
# cat /sys/block/sda/queue/read_ahead_kb

To change the 'Read-Ahead' value to 8 MB (16384 times of 512 bytes blocks).
# blockdev --setra 16384 /dev/sda

To make it permanent upon system reboot, just add this command entry in /etc/rc.local.

Tweaking 'ps' command to display full user-name

By default, the UID column in ‘ps’ output has a character length of 8. If the user name exceeds 8 characters, it will convert the User-name to its corresponding UID and displays it. This might cause inconvenience when we want to check the process list run by a user-account with more than 8 characters. 'ps' command has an option to display full user-name.

Normal 'ps' command:

[longusername@hostxyz ~]$ id longusername
uid=55062(longusername) gid=55062(longusername) groups=55062(longusername)
[longusername@ora-prod-inf-d2 ~]$ ps -ef | grep tail
55062    11708 11470  0 02:11 pts/2    00:00:00 tailf a
55062    11739 11470  0 02:11 pts/2    00:00:00 tailf b
55062    11754 11470  0 02:11 pts/2    00:00:00 tailf c
55062    20637 11470  0 02:33 pts/2    00:00:00 grep tail

A tweaked ‘ps’ output with User column set as 20 characters 

[longusername@hostxyz ~]$ ps -o user:20,pid,ppid,c,stime,tty,cmd | grep tail
longusername         11708 11470  0 02:11 pts/2    tailf a
longusername         11739 11470  0 02:11 pts/2    tailf b
longusername         11754 11470  0 02:11 pts/2    tailf c
longusername         20641 11470  0 02:33 pts/2    grep tail
[longusername@hostxyz ~]$

In this you can see the full user-name getting displayed.

Additional command :

List out the processes based on CPU and Memory consumption

# ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

Try 'compgen' - an interesting command

'compgen' is a handy Bash built-in command to list all the Commands, Aliases and Bash Built-ins functions available for a logged-in user.
To list all the commands     :  compgen -c
To list all the Aliases          :   compgen -a
To list all the Bash Built-ins :  compgen -b

How to check when a patch was last installed or upgraded ?

Syntax: rpm -qa --last | grep <package name>

Example:  Suppose you want to know when the MySql packages was last installed or upgraded.

The syntax should be:  
[root@Hostxyz ~]# rpm -qa --last | grep -i mysql
MySQL-client-5.6.5_m8-1.rhel5                 Fri 13 Jul 2012 02:35:01 AM PDT
MySQL-server-5.6.5_m8-1.rhel5                Fri 13 Jul 2012 02:34:44 AM PDT
php54-mysql-5.4.0-1.el5                           Fri 13 Jul 2012 01:17:21 AM PDT
[root@Hostxyz ~]#

To list the entire package list, just execute # rpm -qa --last

Complex Arithmetic using Linux Binary calculator

Lets say you want to execute the below arithmetic expression using Binary calculator (bc):
((10*10)+100)/4. The expected value for this arithmetic expression is 50.

The syntax using 'bc' would be :
# echo "10*10;.+100;./4" | bc
100
200
50

#

Please note the semicolon(;) and period (.) used in-between.

Other version using 'last' keyword:

# echo "10*10;last+100;last/4" | bc

100

200

50

#

The period (.) or 'last' keyword denotes result of previous execution.

Rsync command syntax

Below given is the recommended Rsync syntax which can be used to sync 2 file-systems. However appropriate options can be chosen based on the requirement:
 Source machine# nohup rsync -atlrzvuop --progress /<source folder>/  <Destination IP/Hostname>:/<Dest folder>/  &
-a  :  Archive mode
-t  :   Preserve modification time
-l  :   Preserve symbolic links
-r :   Copy recursively
-v :   Verbose
-z  :  Zip the files before transfer
-u :  Skips files which exist on the destination and have a modified time newer than the source file.
-p :  Preserve permissions
-o :  Preserve owner
--bwlimit=<kpbs> : To control the I/O bandwidth if you are syncing across WAN (e.g. --bwlimit= 30000  - to sync at 30 Mbps)
--delete  :   Delete all the extra files on Destination which aren’t present in the Source.
--exclude=<pattern>  :   Exclude files from syncing whose filenames matching the PATTERN  (e.g. --exclude=.zip )

Automated Rsync script:
http://smart-scripts-for-sysadmins.blogspot.com/2012/03/sample-rsync-script.html

Finding Start time of a Linux Process

In Linux, finding Starting time of a process using 'top' output will always tend to confuse us. Below is a sample 'top' output of a bunch of 'cpio' process which are running on a Server. The TIME+ column shows some time but it is quite confusing to interpret.

Following are the command-set that I use to find actual Start-time of a Process and how long it runs:

[root@linuxhost ~]# ps -eo pid,lstart,cmd | grep -v grep | grep 14115

14115 Sat Jun  2 01:15:38 2012 cpio -pdmu /NA_cdb01_ud4001

[root@iss-365-tdb02 ~]# date

Sat Jun  2 02:01:41 PDT 2012

[root@linuxhost ~]# ps -eo pid,etime,cmd | grep -v grep | grep 14115

14115       46:23 cpio -pdmu /NA_cdb01_ud4001

[root@iss-365-tdb02 ~]#

The above output shows the Process with PID 14115 started at 01:15:38 AM PDT and it is running for more than 45 minutes.

Script to generate file-system usage report

This script is developed specifically for checking the File-system usage of SAN Mounts on Linux or any UNIX flavors. 
However this can be used to check any Unix filesystem such as /var, /opt, /root etc. This script takes Access, Modified and Change Time as parameters on the specified Mount points and followed by an E-mail address. Upon execution, it will send you the resultant output of the script to the E-mail address that you specified in sorted form with the total size of matching files. In addition a copy of the script output will be stored on the path where you executed the script in the following format: Hostname-Month-Day-Year.

URL to download: http://dl.dropbox.com/u/50666315/scripts/SAN_Mount_Check.pl

Supported platforms: Any Unix platform with Perl version 5.x or above installed.

How to use:

The syntax for executing this script is as follows:

# SAN_Mount_Check.pl [-t]  [-a <value> -m <value> -c <value>]  <Mount points  separated by space>

“-t” for Total disk space

“-a” for access time

“-m” for modification time

“-c” for change time

Upon executing the above command will prompt you to enter the E-mail address to which the file-system usage report needs to be sent. Please note the switch “-t” is optional whereas the other switches “ -a, -m, -c “ are must (atleast one or more). You can give multiple filesystem pathnames separated by space but I suggest giving just one in-order to avoid confusion in sorted output; where the files will be sorted in accordance to its size.

Few Sample Syntax:

# perl SAN_Mount_Check.pl -t -a 1000 /san_mount1      <-- List Files which are accessed 1000 days before.

# perl SAN_Mount_Check.pl -t -c 1000 /Netapp_filer_mount2  <-- Files which are changed 1000 days before.

# perl SAN_Mount_Check.pl -t -m 300  /var                <-- Files modified 300 days before in /var

# perl SAN_Mount_check.pl -t -a 500 -m 750 /u01      <-- Files which are accessed 500 days before and modified 750 days before on /u01.

# perl SAN_Mount_Check.pl -t -a 365 /san_mnt1 /san_mnt2     <-- Files which are accessed 365 days before on 2 SAN mounts

How it works:

This script uses &wanted subroutine which comes along with standard Perl Module “File::Find” for traversing through all the files in a desired filesystem. For each file it finds, it checks whether it matches with the parameters supplied and if it is true, it will output the file information along with file size information.  This is a non-intrusive script, as this is equivalent to executing a ‘find’ command. 

Enabling ACL in /etc/fstab

I wish to set FACL for a file-system "/ua1003" with Read-Write permission for my user-account 'ashok'. When I try to do it, I noticed the file-system isn't mounted with ACL feature enabled. I have added the 'acl' option in /etc/fstab for that mount point and did a remount. After that, I was able to set the FACL. Below shown is the sequence which I did to make the ACL work:

[root@hostxyz ua1003]# setfacl -m u:ashok:rwx /ua1003                          
setfacl: /ua1003: Operation not supported                                                  <-- ACL not enabled
[ashok@hostxyz /]$ grep /ua1003 /etc/fstab
LABEL=/ua1003           /ua1003                   ext3    defaults        0  0        
[ashok@hostxyz /]$ vi /etc/fstab
[root@hostxyz ~]# grep /ua1003 /etc/fstab
LABEL=/ua1003           /ua1003                   ext3    defaults,acl        0  0   <-- Enabled ACL feature in /etc/fstab
[root@hostxyz ~]# mount -o remount /ua1003                                           <-- Remounting /ua1003
[root@hostxyz ~]# setfacl -m u:ashok:rwx /ua1003                                    <-- Setting FACL to the parent folder
[root@hostxyz ~]# getfacl /ua1003
getfacl: Removing leading '/' from absolute path names
# file: ua1003
# owner: applprod
# group: dba
user::rwx
user:ashok:rwx
group::r-x
mask::rwx
other::r-x
[root@hostxyz ~]# setfacl -R -m u:ashok:rwx /ua1003                               <--Setting FACL recursively
[root@hostxyz ~]# getfacl /ua1003
getfacl: Removing leading '/' from absolute path names
# file: ua1003
# owner: applprod
# group: dba
user::rwx
user:ashok:rwx
group::r-x
mask::rwx
other::r-x
[root@hostxyz ~/#

PS: To check if ACL is enabled on a given file-system, execute 'tune2fs -l <devicename>' command and look out for "Default mount options".

How to encrypt and decrypt a file in Linux ?

There could be few ways to Encrypt and Decrypt a file in Linux. The one which I use is, gpg (GnuPG).
Below shown are the steps to encrypt and decrypt a file called "confidential.txt".

[root@hostxyz ashok]# echo 'newpassXYZ' > confidentail.txt
[root@hostxyz ashok]# cat confidentail.txt
newpassXYZ
[root@hostxyz ashok]# gpg -c confidential.txt
Enter passphrase:  <secret word>
Repeat passphrase: <secret word>
[root@hostxyz ashok]# ls -l confidentail.*
-rw-r--r-- 1 root root 11 May 22 22:01 confidential.txt
-rw-r--r-- 1 root root 66 May 22 22:01 confidential.txt.gpg
[root@hostxyz ashok]# cat confidential.txt.gpg
ê2Ãà pisÃu?î^ó5Ã\<dÃ
â[root@hostxyz ashok]# mv confidential.txt.gpg /tmp
[root@hostxyz ashok]# cd /tmp
[root@hostxyz tmp]# gpg -d confidential.txt.gpg
gpg: CAST5 encrypted data
Enter passphrase:  <secret word>
[root@hostxyz tmp]# cat confidential.txt
newpassXYZ
[root@hostxyz tmp]#

The config file for 'gpg' is /root/.gnupg/gpg.conf

Finding number of occurrence of a string in a file.

Let's say you have a file (filename: testfile) as shown below and you want to find number of occurrence of a string 'snapshot' in this file.  Please note the specified string could occur more than once in a line. 

"A disk "Snapshot" is a copy of the virtual machine disk file (VMDK) at a certain point in time. It preserves the disk file system,

system memory of your VM by enabling you to revert to the snapshot in case something goes wrong. Snapshot can 

upgrading or patching applications and servers. This article will go over everything you need to know about using snapshot,

including what they are, how they work and advanced techniques. A virtual machine provides several operations for managing

snapshots and snapshot chains. These operations let you create snapshots, revert to any snapshot in the chain, and remove snapshots. You can create extensive snapshot trees."

Command is:  grep -io snapshot testfile | wc -l

Re-executing a command from History after substitution

At times we might be executing a long command on Shell prompt and later we might have to re-run that same command-set with one small change. It will be real pain to type the same command-set again just for a one parameter change. Linux has a Bash built-in command called 'fc' (stands for find command) using which we can make this task simple.

Illustration:
Let's say you have executed the following command (bit long) some time ago:
"bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s300 -r iss-365-rhel5664-tmpl.xyz.com -c196"
Now you wish to execute the same command with one value changed in it. Instead of -s300, you want to re-run the command with the new value -s200.
hostxyz # fc -l               <-- Will list 16 most recent commands
427      ls
428      ls -l
429      vim NMON_startup_script.sh-May-16-12.log
430      sh /root/itc/hc/dyn/NMON_startup_script.sh
431      cat NMON_startup_script.sh-May-16-12.log
432      uptime
433      perl /opt/sysinfo.pl
434      ls -l /var/lock/subsys
435      bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s300 -r iss-365-rhel5664-tmpl.xyz.com -c196
436      cd ~ashok
437      ls
438      ps -ef | grep nmon
439      kill -15 3384
440      date
441      fc -l
442      ps -ef | grep nmon
hostxyz # fc -s s300=s200 bash      
bash /root/itc/hc/dyn/nmon -f -t -m /var/log/nmon -s200 -r iss-365-rhel5664-tmpl.xyz.com -c150
hostxyz #
With the `fc -s [pat=rep ...] [command]' format, the command starting with 'bash' is re-executed after the substitution OLD=NEW is performed.
Note:  Typing just 'fc' will open the last command in Text Editor. If you want to open range of commands that you executed before in Text Editor, you can use the following syntax:  # fc [Start no]  [End no].

FUNC - Fedora Unified Network Controller

FUNC is an open source automation tool developed by using Python programming language. We can use this tool for automating system admin tasks such as status check, configuration tweak, file transfer, rebooting the systems etc in multiple Linux-based systems. It uses the typical Client-Server model, where the server is called by the term "overlord" and the clients which are binded to the server are called as "minions".

Check out this link for more details with examples:
http://advanced-sysadmin-stuff.blogspot.in/2011/03/func-fedora-unified-network-controller.html

I have developed an Installation bundle which consists of Script for installing and configuring the FUNC client with all the required packages. URL to download the software bundle : http://dl.dropbox.com/u/50666315/blog/func_pack.tar

Just download it, extract and install the FUNC client by executing the 'install_func_client.sh' script. http://dl.dropbox.com/u/50666315/scripts/install_func_client.sh

PS: If you are looking for a definitive solution on permanent basis to address all your challenges in remote administration of Linux-based servers, I would recommend to go for "Puppet" (http://puppetlabs.com) . With Puppet, you can achieve lot more things than FUNC. URL to download a book on Puppet: http://db.tt/LtZOcfZk