Wednesday, October 13, 2010

Finding Load average on Linux

Load Average of a Linux server can be find in various ways and they are shown below.  The 3 values which it displays are the system  load averages for the past 1, 5, and 15 minutes.

 [root@S3 ~/]# uptime
 07:18:07 up 57 days,  8:04,  8 users,  load average: 1.82, 1.24, 0.97

[root@S3 ~/]# cat /proc/loadavg
1.67  1.22  0.97  1/283 24487

 [root@S3 ~/]# w
 07:18:19 up 57 days,  8:04,  8 users,  load average: 1.56, 1.21, 0.97
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
adevaraj pts/0    gtlslhh.tcprod. 01:41    0.00s  0.10s  0.03s sshd: adevaraju [priv]
tkhalef  pts/3    gtwash01.tcprod. Mon05   45:35   0.38s  0.03s sshd: tkhalef [priv]
nimmika  pts/4    220.10.245.82   05:49    1:28m  0.02s  0.02s -bash
[root@S3 ~/]#

[root@S3 ~/]# top
top - 07:19:14 up 57 days,  8:05,  8 users,  load average: 1.57, 1.28, 1.00
Tasks: 284 total,   1 running, 281 sleeping,   0 stopped,   2 zombie
Cpu(s):  2.2%us,  0.7%sy,  0.0%ni, 95.9%id,  1.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16634296k total, 14157704k used,  2476592k free,   438604k buffers
Swap:  5406712k total,       56k used,  5406656k free, 12835252k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
21914 apache    15   0     0    0    0 Z  7.7  0.0   0:02.06 httpd <defunct>
24774 root      15   0  2508 1072  720 R  3.8  0.0   0:00.02 top
20504 apache    15   0 50596  24m 4800 S  1.9  0.1   0:03.89 httpd
    1 root      15   0  2072  616  532 S  0.0  0.0   0:08.33 init
.
.
.
.

Sunday, October 10, 2010

What an useradd command in Linux does?

When you execute an useradd command  (# useradd <username>”), the following happens.

1.       Add the new user entry in the following files: /etc/passwd and /etc/shadow

2.       A group will be created with the same username and it will be updated in the following files: /etc/group and /etc/gshadow.

3.       Home folder for the user will be created (/home/<username>) and the default profile settings will be copied from /etc/skel to it.

# ls -la /etc/skel/
total 56
drwxr-xr-x   3 root root  4096 Aug 16 13:03 .
drwxr-xr-x 111 root root 12288 Oct  6 15:10 ..
-rw-r--r--   1 root root    33 Jan 21  2009 .bash_logout
-rw-r--r--   1 root root   176 Jan 21  2009 .bash_profile
-rw-r--r--   1 root root   124 Jan 21  2009 .bashrc
-rw-r--r--   1 root root   515 May 24  2008 .emacs
drwxr-xr-x   4 root root  4096 Jul 26 18:19 .mozilla
-rw-r--r--   1 root root   658 Sep 21  2009 .zshrc
 #


Please note in the case, if you happen to accidently delete either /etc/passwd or /etc/shadow files, you can restore it from its corresponding backup files (i.e /etc/passwd- ,  /etc/shadow- ) respectively. These files are updated upon the system reboot.  So you can’t expect these files to be having the entries for user accounts which are added after the system reboot.

SUID (Set User ID) Explained


The password information of an user account is saved in /etc/shadow file. When you check the file permission of it, you would see that it has Read permission ONLY for root.  So ever wondered how can a normal user will be able to Write on this file while executing the ‘passwd’ command for changing his password ??

[adevaraju@hostx ~]$ ls -l /etc/shadow
-r-------- 1 root root 1436 Oct  6 14:40 /etc/shadow
[adevaraju@hostx ~]$


There comes SUID in picture……..If you check the file permission for ‘passwd’ command, you can see that it has a SUID (Set User ID) set for it as shown below. Now lemme tell the definition of SUID. “When SUID bit is set for any command then whoever executes that command, will execute it with the privilege of file owner”. 

Here w.r.t ‘passwd’ command, when a normal user executes it, then it will run with “root” ownership. As root user can over-write any local files, he can update the /etc/shadow file, though it doesn’t have Write permission on it. And that’s how a normal user can change his password.

[adevaraju@hostx ~]$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 22984 Jan  6  2007 /usr/bin/passwd
[adevaraju@hostx ~]$


How to set SUID ?

# chmod u+s <command/script name>

(or)

# chmod 4755 <command/script name>


How to search files with SUID set?

# find / -perm -4000  -type f -print

Please note while doing security audit on a server, finding and reviewing the existence of executables with SUID set is an important action item that needs to be taken care; as there are very dangerous.

Refer: http://www.bashguru.com/2010/03/unixlinux-advanced-file-permissions.html

Thursday, October 7, 2010

Resolving Too many files open error in Linux

Situation:
Users couldn't execute *ANY* commands. It gives "Too many open files" error as shown below:

orabi@miaash02-t1$ top
ksh: top: /usr/bin/top: cannot execute [Too many open files in system]
dwilliams@miaash02-t1$ ls -l
ksh: ls: /bin/ls: cannot execute [Too many open files in system]


Reason:
Everything in Linux are files; Linux forks most things including devices, sockets and pipes as files.  There is a kernel parameter called “file-max” which controls the maximum number of files that can be opened in a system. The default value is 65K (approx), can be find using the following command:
“sysctl -a | grep file-max”.
To check the count of number of files open, we can use the following command: “lsof | wc –l”. However this will not give you the exact number, because it is possible for a single file to be opened multiple times for readind and each additional concurrent open will increase the count for file-max value.  And in addition even connections to network ports can eat up the ‘file-max’ value.

Resolution:
As a temporary solution, we can increase the ‘file-max’ value by issuing the following command:
# echo “value” > /proc/sys/fs/file-max and then we need to identify the problem by analyzing either the System logs or by using lsof command itself with appropriate options. 

Finding default block size in Linux

How to find default block size in Linux?
Answer:  
# tune2fs -l /dev/sda1 | grep Block
Block count:                4980736
Block size:                    4096
Blocks per group:     32768
From this example, you can see that the default block size for the filesystem on /dev/sda1 partition is 4096 bytes, or 4k. That's the default block size for ext3 filesystem.    


How to define the block size while creating  ext3 filesystem ?

 Answer:    

 # mkfs.ext3 –b <block size>  <Device>
 
Example:   # mkfs.ext3 -b 2048 /dev/sda1                
This creates a filesystem with 2KB block size on partition /dev/sda1.

Going indepth of Linux process

Scenario: 
At times need arises to dig in deep about a running process. We may be seeing a process occupying more system resource, in that case, we have to find what does that process actually doing and which are all the files it is accessing.

Solution:
There are multiple ways:

1.       Using ‘ps’ with appropriate options  (This will give show you the process tree of all process)

# ps axjf 
# ps eauxf

2.       Using ‘lsof’   (‘-p’ and ‘-c’ shows you the files opened by a particular process and command respectively)

# lsof -p  <PID>                

To know PID of a process name (e.g smbd),
# pidof  [process name]      (i.e # pidof smbd)

# lsof -c  <Command name>


Wanna go still deeper………………………………………………………………….??


3.       Use ‘strace’  ( Will list out all the System calls and Signals made by a Process. It records the System calls “C functions” which are called by a  process  and  the  Signals  which  are received by a process)

#  strace -f -p  <PID>


Use 'pidstat' to find out processes wise memory consumption. To find Memory usage of active process :

# pidstat -l -r | sort -k8nr

How to find PID associated with a Port number

Scenario
At times we may run into situation to find the PID associated with a particular Port number. We may not be able to start/restart a particular service (say Jboss), since the Port number which is needed for the service is occupied by another process. You would have seen the error saying “Address already in use”. 

Solution:
Find the PID associated with the particular port and kill it, if not required, and start the service.

Couple of straight forward commands:

# lsof -i      (or)  lsof -i: <port number>

# netstat -lantp   (or)  netstat -lautp

Examples:

[root@hostx ~]# lsof -i:80                        # Second column shows the PID associated with HTTPD service running on port 80.
COMMAND   PID   USER   FD   TYPE   DEVICE SIZE NODE NAME
httpd   28064   root    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28066 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28067 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28068 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28069 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28070 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28071 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28072 apache    4u  IPv6 27763387       TCP *:http (LISTEN)
httpd   28073 apache    4u  IPv6 27763387       TCP *:http (LISTEN)                                                                                                                         

[root@hostx ~]# lsof -i:123                   # 1582 is the PID for NTP service running on port 123
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
ntpd    1582  ntp   16u  IPv4   3830       UDP *:ntp
ntpd    1582  ntp   17u  IPv6   3831       UDP *:ntp
ntpd    1582  ntp   18u  IPv6   3846       UDP [::1]:ntp
ntpd    1582  ntp   19u  IPv6   3847       UDP [fe80::3c8c:7ff:fe96:2705]:ntp
ntpd    1582  ntp   20u  IPv6   3890       UDP [fe80::1c43:d9ff:fe07:dfc0]:ntp
ntpd    1582  ntp   21u  IPv4   3964       UDP localhost.localdomain:ntp
ntpd    1582  ntp   22u  IPv4   3965       UDP kesav01-t2:ntp

 [root@hostx ~]# netstat -lantp | grep java      # 3rd column shows the Local server Port number and last column shows the PID and process associated with it.
tcp        0      0 0.0.0.0:55980               0.0.0.0:*                   LISTEN      20586/java
tcp        0      0 0.0.0.0:11025               0.0.0.0:*                   LISTEN      20586/java
tcp        0      0 10.10.64.78:8088            0.0.0.0:*                   LISTEN      20607/java
tcp        0      0 10.10.64.78:8089            0.0.0.0:*                   LISTEN      20586/java
tcp        0      0 0.0.0.0:42587               0.0.0.0:*                   LISTEN      20586/java
tcp        0      0 0.0.0.0:55933               0.0.0.0:*                   LISTEN      20607/java
tcp        0      0 10.10.64.78:39646           10.10.25.32:55200           ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:39645           10.10.25.32:55200           ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:50681           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:50683           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:50682           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        2      0 10.10.64.78:60506           10.10.65.13:8088            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:42882           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:42881           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:48708           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:48709           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:48706           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:48707           10.10.64.90:1521            ESTABLISHED 20607/java
tcp        0      0 10.10.64.78:48704           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:48705           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:48703           10.10.64.90:1521            ESTABLISHED 20586/java
tcp        0      0 10.10.64.78:8088            10.10.65.16:51863           ESTABLISHED 20607/java
[root@hostx ~]#

[root@hostx ~]# netstat -lautp | grep java       # Difference with ‘u’ option is, it resolves the Hostname and Port number.
tcp        0      0 *:55980                     *:*                         LISTEN      20586/java
tcp        0      0 *:11025                     *:*                         LISTEN      20586/java
tcp        0      0 kseav01-t2:radan-http      *:*                         LISTEN      20607/java
tcp        0      0 kesav01-t2:8089            *:*                         LISTEN      20586/java
tcp        0      0 *:42587                     *:*                         LISTEN      20586/java
tcp        0      0 *:55933                     *:*                         LISTEN      20607/java
[root@hostx ~]#

Memory leakage in Linux

What is Memory leakage?

Memory Leakage basically refers to a situation where the memory allocated to an application (can be a Database query as well) is not getting freed up. This can be due to bug in the program which utilize the memory resource. In my opinion, when an application closes, it should issue the proper exit statement so that it will free up the memory which it occupied and kill all its child process. 
So this Memory leakage is basically an application issue, and any leaked memory will be freed up after the application is killed or stopped.


How to find the Memory leakage?

We would first need to determine which application/process is consuming more memory.  This can be done through data gathering. We need to periodically check what is running on a server and what is the Memory utilization . With the time when the server runs out of available memory to allocate, the leaking application might crash or it may even crash the system.

Valgrind is a popular Open source tool available for detecting the memory leakage of an application (URL:  http://valgrind.org/ ). A sample Valgrind output attached.


How to fix it?

We need to correct the application from using more memory ( Done by a Programmer or DBA).  I don’t think we can do much on the server side since the job of kernel is release the requested memory by an application. Certain things we can do at Server end are, tuning some kernel parameter like restricting the number of child process forked by a single process, restricting number of process forked by an user etc. If the kernel have the intelligence to distinguish, which request for more memory is valid and which one causes memory leaks, it would be great but am not sure if it’s possible to do such level of tuning J


In Linux, Memory leakage can be found using the below given 'ps' command. This will give you an idea about Memory usage of each process in a sorted manner.
# ps aux --sort pmem

Finding the empty folders, files and hidden files

1.     How to find all the empty folders which are there in a filesystem?

# find /path -depth -empty -type d –print

Example:

[root@hostx ~]# find /etc -depth -empty -type d -print
/etc/skel/.mozilla/plugins
/etc/skel/.mozilla/extensions
/etc/opt
/etc/prelink.conf.d
/etc/logwatch/scripts
/etc/X11/sysconfig
/etc/rwtab.d
/etc/oddjob
/etc/jvm-commmon
/etc/racoon/certs
/etc/sasl2
/etc/subversion
/etc/dbus-1/session.d
/etc/gconf/gconf.xml.mandatory
/etc/dnsmasq.d
/etc/NetworkManager/VPN
/etc/NetworkManager/dispatcher.d
/etc/hal/fdi/preprobe
/etc/hal/fdi/information
.
[Output truncated]
.
[root@hostx ~]#
[root@hostx ~]# cd /etc/sysconfig/pgsql
[root@hostx pgsql]# ls -la
total 16
drwxr-xr-x  2 root root 4096 Jan 12  2008 .
drwxr-xr-x 12 root root 4096 Jul 14 05:23 ..                            ß Nothing exists
[root@hostx pgsql]# cd /etc/pm/hooks
[root@hostx hooks]# ls -la
total 16
drwxr-xr-x 2 root root 4096 Jan 22  2009 .                              ß Nothing exists
drwxr-xr-x 5 root root 4096 Oct 12  2009 ..
[root@hostx hooks]#




How to search for hidden files which are there in a particular folder?


# find /path -type f -name ".*" -print


Example:


[root@hostx ~]# find /etc -type f -name ".*" -print
/etc/skel/.bash_profile
/etc/skel/.kde/Autostart/.directory
/etc/skel/.bash_logout
/etc/skel/.zshrc
/etc/skel/.bashrc
/etc/skel/.emacs
/etc/.pwd.lock
/etc/lvm/cache/.cache
/etc/news/.profile
[root@hostx ~]#

How to find empty files in a given folder ?

# find /path -type f -size 0 -print

Example:

# find /etc -type f -size 0 -print
/etc/wvdial.conf
/etc/cron.deny
/etc/security/opasswd
/etc/security/console.apps/xserver
/etc/motd
/etc/dumpdates
/etc/environment
/etc/.pwd.lock
/etc/lsb-release.d/core-3.0-noarch
/etc/lsb-release.d/core-3.0-amd64

Steps to reset MySQL root password

                         [root@xyzhost conf]# mysql -u root -p         <-- Mysql root password isn't working
Enter password:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
[root@xyzhost conf]#
Check if the Mysql process is running (its running here)
[root@xyzhost init.d]# ps -ef | grep -v grep | grep mysql
root      2104     1  0 Jan05 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql     2154  2104  0 Jan05 ?        00:00:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
[root@xyzhost conf]#

Time to reset the Mysql root password !!

Stop the mysql service
[root@xyzhost init.d]# /etc/init.d/mysqld stop
Stopping MySQL:                                            [  OK  ]
Start the Mysql in Safe-mode
[root@xyzhost init.d]# mysqld_safe --skip-grant-tables &
[1] 7256
[root@xyzhost init.d]# Starting mysqld daemon with databases from /var/lib/mysql
[root@xyzhost init.d]# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.82sp1 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> update user set password=PASSWORD("mysqlnewp@$$") where User='root';
Query OK, 3 rows affected (0.00 sec)
Rows matched: 3  Changed: 3  Warnings: 0
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> quit
Bye
Stop the Mysql service
[root@xyzhost init.d]# /etc/init.d/mysqld stop
STOPPING server from pid file /var/run/mysqld/mysqld.pid
110721 12:58:44  mysqld ended
Stopping MySQL:                                            [  OK  ]
[1]+  Done                    mysqld_safe --skip-grant-tables
Start the Mysql service
[root@xyzhost init.d]# /etc/init.d/mysqld start
Starting MySQL:                                            [  OK  ]
Connect to Mysql DB using new password:
[root@xyzhost init.d]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.82sp1 Source distribution
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql                   |
| customers            |
| test                      |
| clients_db            |
+--------------------+
5 rows in set (0.00 sec)
mysql>

Thats it !!

More about 'touch' command in Linux

touch” resemble like a simple command for creating empty file but it is widely used by system for changing the Time stamp of files and for Service management.

Here’s an example of touch command that change the Time stamp of a file:

[adevaraju@host01-tx xivnew_64]$ ls -l
total 3904
-rw-r--r-- 1 adevaraju domain users   52422 Nov 12  2009 sg3_utils-libs-1.25-4.el5.x86_64.rpm
[adevaraju@host01-tx xivnew_64]$


[adevaraju@host01-tx xivnew_64]$ touch sg3_utils-libs-1.25-4.el5.x86_64.rpm
[adevaraju@host01-tx xivnew_64]$ ls -l
total 3904
-rw-r--r-- 1 adevaraju domain users   52422 Sep 15 07:15 sg3_utils-libs-1.25-4.el5.x86_64.rpm                                       # Time stamp changed


[adevaraju@host01-tx xivnew_64]$ touch -t 1009240615 sg3_utils-libs-1.25-4.el5.x86_64.rpm
[adevaraju@host01-tx xivnew_64]$ ls -l sg3_utils-libs-1.25-4.el5.x86_64.rpm
Fri Nov 26 06:30:34 CST 2010           <- Time stamp advanced to 06:30 11/26/2010
[adevaraju@host01-tx xivnew_64]$


Service management:

When you get into the folder /var/lock/subsys, you could see list of empty files with service names. These files are basically created using touch command, when a service (say ntpd) is started/restarted.  Presence of these files ensures that same service willnot be started again and it gives an error message saying the service is already running. Further these files will let us know how long a particular service is being running.


[adevaraju@host01-tx xivnew_64]$ cd /var/lock/subsys
[adevaraju@host01-tx subsys]$ ls -l
total 0
-rw-r--r-- 1 root root 0 Jul 12 20:25 acpid
-rw-r--r-- 1 root root 0 Jul 12 20:25 atd
-rw-r--r-- 1 root root 0 Jul 12 20:25 auditd
-rw-r--r-- 1 root root 0 Jul 12 20:25 autofs
-rw-r--r-- 1 root root 0 Jul 12 20:25 bluetooth
-rw-r--r-- 1 root root 0 Jul 12 20:24 cpuspeed
-rw-r--r-- 1 root root 0 Jul 12 20:25 crond
-rw-r--r-- 1 root root 0 Jul 12 20:25 cups
-rw-r--r-- 1 root root 0 Jul 12 20:26 firstboot
-rw-r--r-- 1 root root 0 Sep 12 04:02 funcd
-rw-r--r-- 1 root root 0 Jul 12 20:25 gpm
-rw-r--r-- 1 root root 0 Jul 12 20:26 haldaemon
-rw-r--r-- 1 root root 0 Jul 12 20:25 hcid
-rw-r--r-- 1 root root 0 Jul 12 20:25 hidd
-rw-r--r-- 1 root root 0 Jul 12 20:25 hpiod
-rw-r--r-- 1 root root 0 Jul 12 20:25 hplip
-rw-r--r-- 1 root root 0 Jul 12 20:25 hpssd.py
-rw-r--r-- 1 root root 0 Jul 12 20:25 iscsi
-rw-r--r-- 1 root root 0 Jul 12 20:24 iscsid
-rw-r--r-- 1 root root 0 Jul 12 20:24 kudzu
-rw-r--r-- 1 root root 0 Jul 12 20:28 local
-rw-r--r-- 1 root root 0 Jul 12 20:25 messagebus
-rw-r--r-- 1 root root 0 Jul 12 20:24 microcode_ctl
-rw-r--r-- 1 root root 0 Jul 12 20:24 multipathd
-rw-r--r-- 1 root root 0 Jul 12 20:25 netfs
-rw-r--r-- 1 root root 0 Jul 12 20:25 network
-rw-r--r-- 1 root root 0 Jul 12 20:25 nfslock
-rw-r--r-- 1 root root 0 Jul 12 20:32 ntpd
-rw------- 1 root root 0 Jul 12 20:25 pcscd
-rw-r--r-- 1 root root 0 Jul 12 20:32 portmap
-rw-r--r-- 1 root root 0 Aug 13 07:44 postfix
-rw-r--r-- 1 root root 0 Jul 12 20:25 rhnsd
-rw-r--r-- 1 root root 0 Jul 12 20:25 rpcidmapd
-rw-r--r-- 1 root root 0 Jul 12 20:25 sdpd
-rw-r--r-- 1 root root 0 Aug 17 03:08 sendmail
-rw-r--r-- 1 root root 0 Jul 12 20:28 smartd
-rw-r--r-- 1 root root 0 Jul 12 20:32 smb
-rw-r--r-- 1 root root 0 Aug 17 03:08 sm-client
-rw-r--r-- 1 root root 0 Jul 12 20:25 sshd
-rw------- 1 root root 0 Jul 12 20:25 syslog
-rw-r--r-- 1 root root 0 Sep  9 14:48 winbindd   # Indicates winbind service was restarted on Sep 9th.
-rw-r--r-- 1 root root 0 Jul 12 20:25 xfs
-rw-r--r-- 1 root root 0 Jul 12 20:25 xinetd
-rw-r--r-- 1 root root 0 Jul 12 20:25 yum-updatesd
[adevaraju@host01-tx subsys]$