How to log commands executed by all the users in Linux?

By adding the following entry in /etc/bashrc, we can log the commands executed by all the users on a Linux machine. 
This would be certainly helpful for tracking commands on Critical servers.

PROMPT_COMMAND='history -a >(logger -t "$USER[$PWD] $SSH_CONNECTION")'

After you add the above entry at the end of /etc/bashrc file, execute the command 'source /etc/bashrc' or logout and login back to your session. Now the commands executed by all the users will be logged in /var/log/messages.

Note: If you wish to log the commands on to a different file, please check the solution given in the comments section.

Sample test result:

Apr 18 13:35:21 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: uptime
Apr 18 13:35:24 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: cd /opt
Apr 18 13:35:26 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: ls -lR
Apr 18 13:35:35 Linux-Mach root[/opt] 192.168.4.5 51650 172.16.0.252 22: iostat -x 2
Apr 18 13:35:39 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: cd /root
Apr 18 13:35:39 Linux-Mach root[/root] 192.168.4.5 51650 172.16.0.252 22: ls -l
Apr 18 13:35:51 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: cd /home
Apr 18 13:35:52 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: ls
Apr 18 13:35:56 Linux-Mach root[/home] 192.168.4.5 51650 172.16.0.252 22: httpd -t

Apr 18 13:51:24 Linux-Mach test1[/home/test1] 192.168.6.8 9106 172.16.0.252 22: ls -l
Apr 18 13:53:20 Linux-Mach test1[/var/lock/subsys] 192.168.6.8 9106 172.16.0.252 22: cd /var/lock/subsys
Apr 18 13:53:30 Linux-Mach test1[/var/lock/subsys] 192.168.6.8 9106 10.160.0.252 22: ls -ltr

Updated one:

# echo $PROMPT_COMMAND

RETRN_VAL=$?;logger -p local3.debug "$(whoami)  $remoteip  [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//") [$RETRN_VAL]"

On RHEL 8.x

[root@rhel8 ]# cat /etc/profile.d/prompt.sh 

remoteip=$(who am i | awk '{print $5}' | sed "s/[()]//g")

export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local3.debug "$(whoami)  $remoteip  [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//") [$RETRN_VAL]"'

[root@rhel8 ]# 

Retrieving the file-system from a lost Linux partition

When you lost a Linux partition on Storage device, there is a 80% chance that you can retrieve the file-system in it just by Re-creating the lost Partition. I have tried couple of times in the recent past and it worked successfully.

Here's the sequence of steps which I performed to retrieve the file-system "/us2001" on a lost partition from a SAN disk:

[root@server-tap04 mapper]# mount -t ext3 /dev/mapper/us2001np1 /us2001

mount: special device /dev/mapper/us2001np1 does not exist

[root@server-tap04 mapper]# fdisk -l /dev/mapper/us2001n

 Disk /dev/mapper/us2001n: 10.7 GB, 10737418240 bytes

64 heads, 32 sectors/track, 10240 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/mapper/us2001n doesn't contain a valid partition table

[root@server-tap04 mapper]#

[root@server-tap04 mapper]# multipath -l | grep -A5 us2001n

us2001n (350002ac009290783)

[size=10 GB][features="1 queue_if_no_path"][hwhandler="0"]

\_ round-robin 0 [active]

\_ 1:0:0:1 sdc 8:32 [active]

\_ 2:0:0:1 sdb 8:16 [active]

[root@server-tap04 mapper]# fdisk -l /dev/sdc

 Disk /dev/sdc: 10.7 GB, 10737418240 bytes

 64 heads, 32 sectors/track, 10240 cylinders

 Units = cylinders of 2048 * 512 = 1048576 bytes

 Disk /dev/sdc doesn't contain a valid partition table

[root@server-tap04 mapper]# fdisk /dev/sdc

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-10240, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-10240, default 10240):

Using default value 10240

Command (m for help): w

The partition table has been altered!

 Calling ioctl() to re-read partition table.

Syncing disks.

[root@server-tap04 mapper]# kpartx -a /dev/mapper/us2001n

[root@ server-tap04 mapper]# kpartx -l /dev/mapper/us2001n

us2001np1 : 0 20971488 /dev/mapper/us2001n 32

[root@rica-chi-tap04 mapper]# mount -t ext3 /dev/mapper/us2001np1 /us2001

[root@rica-chi-tap04 mapper]# df -h /us2001
Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/us2001np1

                      9.9G  5.8G  4.1G  59% /us2001

[root@rica-chi-tap04 mapper]# cd /us2001                <-- Partition recreated and mounted  

[root@rica-chi-tap04 us2001]# ls -l                           <-- All the files are retrieved

total 28

-rw-r--r--  1 root     root     0 Jan 18  2011 2

-rw-r--r--  1 root     root     0 Jan 18  2011 abc

drwxr-xr-x  5 appltest dba   4096 Aug 10  2010 BI_Disco10g

drwx------  2 root     root 16384 Aug  6  2010 lost+found

drwxr-xr-x  4 appltest dba   4096 Mar 11  2011 product

[root@server-tap04 us2001]# touch xyz
[root@server-tap04 us2001]# 

Enforcing to set Strong password in Linux

From release 4, RedHat comes with a Pam module called "pam_cracklib" using which we can enforce the user to set Strong password.

Lets say we have to set the Password requirement as follows:
Minimum length of password should be 8
Minimum number of lower case letters should be 1
Minimum number of upper case letters should be 2
Minimum number of digits should be 2
Minimum number of other characters should be 1
To setup these password restrictions, edit the /etc/pam.d/system-auth file and add/change the following pam_cracklib arguments highlighted in bold:

auth        required      /lib/security/$ISA/pam_env.so
auth        sufficient    /lib/security/$ISA/pam_unix.so likeauth nullok
auth        required      /lib/security/$ISA/pam_deny.so
account     required      /lib/security/$ISA/pam_unix.so
account     sufficient    /lib/security/$ISA/pam_succeed_if.so uid < 100 quiet
account     required      /lib/security/$ISA/pam_permit.so
password    requisite     /lib/security/$ISA/pam_cracklib.so retry=3 minlen=8 lcredit=-1 ucredit=-2 dcredit=-2 ocredit=-1
password    sufficient    /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow
password    required      /lib/security/$ISA/pam_deny.so
session     required      /lib/security/$ISA/pam_limits.so
session     required      /lib/security/$ISA/pam_unix.so

Now verify that the new password restrictions work for new passwords for normal user. To test it, simply login as a non-root user and change the password using the 'passwd' command. Note that the above requirements are not enforced if you run the 'passwd' command under root. 

Settings in tabular form:

pam_cracklib.so	minlen=8	Minimum length of password is 8
pam_cracklib.so	lcredit=-1	Minimum number of lower case letters is 1
pam_cracklib.so	ucredit=-2	Minimum number of upper case letters is 2
pam_cracklib.so	dcredit=-2	Minimum number of digits is 2
pam_cracklib.so	ocredit=-1	Minimum number of other characters is 1

Commands to get the PCI device information

# lspci
# lspci -v
# kudzu -p
# dmidecode --type 9
# lshw -short  (3rd party utility, package needs to be installed)

Steps to handle a Read-Only file-system problem

Try these commands at your own risk !

Lets assume the device name of the mounted file-system on "/data1" which got into Read-only mode is "/dev/sda5".

#  mount -o remount,rw /data1   (provided fstab have the entry for mount point)

If this works, well and good. Otherwise you need to go through the pain of trying the following commands:

#  hdparm -r 0 /dev/sda5           <-- To turn-off the Read-only mode.

 # blockdev --setrw /dev/sda5   <-- To set the block (mounted) device to Read-Write mode.

If the above commands didn't the fix the problem, then go with the regular fix:
1. Check the processes that are accessing the mount point.
2. Request the process/application owner to stop the processes that are accessing the mount point.
3. Try to un-mount the filesystem (without any option).
4. Mount the file-system back on its mount point.
5. If Step 4 didn't work,  run a fsck on the un-mounted file-system (without any option).Otherwise continue Step 6.
6. Verify the mounted file-system is in Read-Write mode.

Incase if the step 3 doesn’t work:
i)  Check the processes again that are accessing the mount point and ask the Application owner to confirm that the stopped the services properly and ask them to kill any hung sessions.
ii)    Try to unmount again without any option.
iii)   If it still didn’t work, use lazy option “umount -l”   (if this works, continue from Step 4).
iv)   If lazy option didn’t work, we are in a chaotic  situation.  Try Forced un-mount by using “umount -f”  & continue from Step 4.
v)    After the above step, incase you couldn’t mount the file-system back, the best choice would be to go for a System reboot. (I’m not sure in this situation, forced Mount using ‘mount -f’ will work). 

PS: Please note this procedure will not be applicable for OS file-systems such as /, /var, /opt etc. In that case, a system reboot would be the only solution, if the remount doesn't work. 

Script to generate Complex password

Description:
This script can be used to generate complex password with predefined password length.

Script:
#!/bin/bash
#Filename : randompass.sh

#Sets the length of the password the script will generate
MAXSIZE=8
# Holds valid password characters. I choose alpha-numeric + the shift-number keyboard keys
array1=(
w e r t y u p a s d f h j k z x c v b m Q W E R T Y U P A D
F H J K L Z X C V B N M 2 3 4 7 8 ! @ $ % \# \& \* \= \- \+ \?
)

# Used in conjunction with modulus to keep random numbers in range of the array size
MODNUM=${#array1[*]}

# Keeps track of the number characters in the password we have generated
pwd_len=0

while [ $pwd_len -lt $MAXSIZE ]
do
  index=$(($RANDOM%$MODNUM))
  password="${password}${array1[$index]}"
  ((pwd_len++))
done
echo $password


Sample Execution: 
[root@sysllm01 create_password]# ./random_pass.sh
#V?Z27NN
[root@sysllm01 create_password]# ./random_pass.sh
D34EXL*3
[root@sysllm01 create_password]#

Find syntax to list huge-sized files and calculate total size

Situation

The disk space in one of the file-system (say /data1) is more 90% and you are asked to list out all the files which are more than 100 MB size and find the sum of total space occupied by listed files in Megabytes.

Solution

To list all the files in /data1, which are more than 100 MB size.

# find /data1 -size +102400k -exec du -sh {} \;

To find the sum of total space occupied by listed files.

In Megabytes:

# find /data1 -size +102400k -ls | awk '{sum += $7} END {printf "Total size: %8.4f MB\n", sum/1024/1024}'\;

In Gigabytes:

# find /data1 -size +102400k -ls | awk '{sum += $7} END {printf "Total size: %6.2f GB\n", sum/1024/1024/1024}'\;

Command-line to check the details of Perl modules installed on Linux

To find the path of Perl modules installed:  
# perl -e 'print join "\n", @INC'

To find the path of each individual Perl modules:
# find `perl -e 'print "@INC"'` -name '*.pm' -print

To check if a given Perl module is installed on the system:
Let say you want to check if the Perl module "File::Compare" is installed or not. The syntax would be as follows:
# perl -e 'use File::Compare; print "Installed\n"'
Installed
#

How to reset the permissions of installed RPM packages ?

Situation:  

One of our Developer has changed the permission of all the files under /etc to 777 by inadvertently executing the chmod command with 777 permission. Due to this, the system went into non-usable state.

Resolution: 

The Linux 'rpm' command  comes with an interesting switch known as '--setperms' using which we can restore the file-permissions of either one or more packages. 

Syntax:  

# rpm --setperms [package name]

To reset one specific RPM package:  

# for pkg in $(rpm -qa | grep -w pkgname); do rpm --setperms $pkg; done
Example:  
To reset the 'zip' package.
# rpm -qa | grep zip                           <-- Check to see if the package exists
# for pkg in $(rpm -qa | grep -w zip); do rpm --setperms $pkg; done

To reset the entire packages installed ( applicable in the situation I mentioned above):

# for allpkg in $(rpm -qa); do rpm --setperms $allpkg; done

Note:  To restore the Group permission, we need to use the switch "--setugids".

Linux system info script - latest version

Whenever I work on Performance issue of a Linux server or when I do any Server Audit, I used to wonder how great it would be if I have a script that will give me all the vital information about the system including the Health status and Security settings. I put my thought into action and this Perl script is a result of it.

Upon execution with the syntax "perl sysinfo.pl", it will display all the vital System parameter details, Health Status and the Security Information on ONE screen.

Supported Platform :  Any Physical or VMware Server running on Linux OS with Perl installed.

Packages required : The Linux machine should contain 'sysstat' and 'dmidecode-2.xx' package,

URL to download the script: https://tinyurl.com/yb97snab
Once you download, please run the following command to convert the script into Unix format: # dos2unix sysinfo.pl

SAMPLE OUTPUT

Note: I shall keep adding new features to this script. Hence please use the given download link to get the latest version.

PS: The Logics used in this script are chosen based on appropriate theory on Linux to ensure the accuracy of output.
Any feedback or suggestions are most welcome!! 

Automatically logout inactive SSH sessions

Typically in an Enterprise setup, we would see Users login from various terminals via SSH but never bother to disconnect the established session. This might cause a slight overload on the Network, since these Established sessions have to maintain their connections by sending Alive packets. So I guess it would be appropriate if we make these Users automatically Logout after a certain period of Inactivity (say 1 hour).

Solution 1:

Create a file called "autologout.sh" under /etc/profile.d with execute permission.

# touch /etc/profile.d/autologout.sh

# chmod 711 /etc/profile.d/autologout.sh

Add the following entries to it (Assuming we have to automatically Logout the users after 1 hour, which is 3600 seconds).

TMOUT=3600

readonly TMOUT

export TMOUT

Solution 2:

Enable the following directives in SSH config file (/etc/ssh/sshd_config) and reload the 'sshd' service.

ClientAliveInterval 3600

ClientAliveCountMax 0

# service sshd reload  (or)  # service sshd restart

Few useful Bash Shell shortcuts

Ctrl + l       # Clear the screen
Ctrl + u      # Delete backward from the cursor position
Ctrl + k      # Delete till End of Line
Ctrl + r      # Search the History from backwards
Ctrl + e     # Move the cursor to end of line


Alt + Back-space      # Delete backwards word by word
Alt + t                      # Shuffle words
Alt + b                     # Move backwards
Alt + f                      # Move forwards


Press 2 times Tab key       # Lists all available commands on the server
$ with 2 times Tab key       # Display all system variables
Some string followed by 2 times Tab key     # All available commands starting with that string  


Refer: http://wiki.bash-hackers.org/syntax/shellvars

Analyzing past System performance of a Linux server

Assumption: 

Today's date is 13th Aug, 2011.  You are asked to check the System performance of a Linux server on 7th Aug,2011 between 3 AM to 5 AM.

Solution: 

Run the 'sar' command on the respective 'sa' (System Activity) file created for the date "7th Aug,2011" with specifying the Starting and End time.

Illustration:

Go to /var/log/sa

[root@hostxyz sa]# ls -ltr sa??

-rw-r--r-- 1 root root 481776 Aug 5 23:50 sa05
-rw-r--r-- 1 root root 481776 Aug 6 23:50 sa06
-rw-r--r-- 1 root root 481776 Aug 7 23:50 sa07       # File that belongs to 7th Aug,2011
-rw-r--r-- 1 root root 481776 Aug 8 23:50 sa08
-rw-r--r-- 1 root root 481776 Aug 9 23:50 sa09
-rw-r--r-- 1 root root 481776 Aug 10 23:50 sa10
-rw-r--r-- 1 root root 481776 Aug 11 23:50 sa11
-rw-r--r-- 1 root root 481776 Aug 12 23:50 sa12
-rw-r--r-- 1 root root 287824 Aug 13 14:10 sa13

[root@hostxyz sa]#
[root@hostxyz sa]# sar -u -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01   # To check CPU utilization
Linux 2.6.18-92.el5 (hostxyz) 08/07/2011

03:00:01 AM CPU %user %nice %system %iowait %steal %idle
03:10:01 AM all 24.57 0.00 5.16 6.04 0.00 64.23
03:20:01 AM all 24.57 0.10 5.06 6.28 0.00 63.98
03:30:01 AM all 24.33 0.00 4.88 5.64 0.00 65.14
03:40:01 AM all 15.75 0.00 3.93 10.52 0.00 69.80
03:50:01 AM all 12.70 0.00 3.09 19.04 0.00 65.17
04:00:01 AM all 16.80 0.00 3.90 9.40 0.00 69.90
04:10:01 AM all 9.18 0.02 2.26 14.43 0.00 74.11
04:20:01 AM all 8.84 0.10 2.20 9.65 0.00 79.22
04:30:01 AM all 11.42 0.00 3.24 10.50 0.00 74.84
04:40:01 AM all 11.84 0.00 2.43 20.64 0.00 65.09
04:50:01 AM all 17.80 0.00 3.78 17.00 0.00 61.42
05:00:01 AM all 6.46 0.00 1.53 21.80 0.00 70.22
Average: all 15.35 0.02 3.46 12.58 0.00 68.59

[root@hostxyz sa]#

[root@hostxyz sa]#  sar -r -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Memory status

[Output no shown]
.

[root@hostxyz sa]#  sar -q -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Load average

[Output not shown]

.

[root@hostxyz sa]#  sar -b -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01     # To check I/O status

[Output not shown]

.

[root@hostxyz sa]#  sar -n DEV -f /var/log/sa/sa07 -s 03:00:01 -e 05:00:01    # To check Network status

[Output not shown]

.

[root@hostxyz sa]# 

Notes: In Linux, System activity report is collected for every 10 minutes by a cron job "sysstat" located under /etc/cron.d and at end of the day, a summary report is generated and saved in /var/log/sa/saXX file, which we can use for later analysis. 

[root@hostxyz cron.d]# cat sysstat

# run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib64/sa/sa1 1 1

# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

root@hostxyz cron.d]#

Sending mail using 'mutt' from shell

In my opinion, 'mutt' is the best text-based Email client for sending mails with Attachment from Linux shell. The best part with it is, it supports wide variety of MIME ("Multipurpose Internet Mail Extensions") types, which ensures the integrity of various types of attachments such as Binary, jpeg, mp3 files.

Below shown are few command sets for sending mails using Mutt with following specifications.

To address: rdashokraj@yahoo.com

Subject: Test mail

Attachment filename:  song.mp3

Message body:  "This mail has a mp3 attachment"

With attachment:

# echo "This mail has a mp3 attachment" | mutt -s "Test mail" -a song.mp3 rdashokraj@yahoo.com

# mutt -s "Test mail" -a study.dat rdashok@outlook.com < messagefile 

Note: "messagefile" is the filename that contains the message body "This mail has a mp3 attachment"

Without attachment:

# echo "Mail without attachment" | mutt -s "Test mail" rdashok@outlook.com

Without Message body:

# mutt -s "Test mail" -a study.txt rdashok@outlook.com < /dev/null