One of popular batch scheduler in Supercomputing area is LSF. As many other forks out there, I used LSF for a while until we switched to other one. bhist is one of LSF command to displays historical information about jobs.
I was assigned to develop a shell script to display job list consumed more than certain wall time clock of certain user. So, I've got 2 input parameters; wall clock time and user id. And output should be the list of jobs used more than given wall time clock and it should contain detail information about job. Finally, the output should be suitable for printing and reporting.
This is what I came up with after spending one or two days. To align column or row for printing layout, it includes lots of intentional tabs and white spaces.
Here is sample output;
Extended bhist
=====================================================================================
DATE JOBID USER JOB_NAME PEND PSUSP RUN SSUSP TOTAL
-------------------------------------------------------------------------------------
Aug 29 13:33:51 16464 xxx test1 10 0 82 0 92
Aug 29 13:46:51 16465 xxx test1 10 0 4402 0 4412
...
...
...
Aug 29 15:20:37 16471 xxx test1 8 0 13639 0 13647
=====================================================================================
Total CPU time: 569:53:52
Source code
#!/bin/sh
#
# NAME : ebhist.new
# Display output of bhis with time & date information of each job of LSF.
#
# AUTHOR : Brian Kim
# Supercomputer Center
#
# DATE : SEPTEMBER 12, 2005
#
function print_title {
echo $1 $2 $3
echo "Extended bhist : " $1 $2 $3
echo
}
function print_usage {
echo "Usage: ebhist WALLTIME ACCOUNT"
echo
echo " Display job information of [ACCOUNT]"
echo " which consumed more than [WALLTIME] second"
echo
echo "Example:"
echo " ebhist 1000 guest"
echo
}
function print_heading {
# echo "\tDATE\tJOBID\tUSER\tJOB_NAME\tPEND\tPSUSP\tRUN\tUSUSP\tSSUSP\tUNKWN\tTOTAL"
# echo "==============================================================================="
echo "====================================================================================="
echo "DATE\t\tJOBID\tUSER\tJOB_NAME\tPEND\tPSUSP\tRUN\tSSUSP\tTOTAL"
echo "-------------------------------------------------------------------------------------"
# echo "-------------------------------------------------------------------------------"
}
function print_footer {
a=$1
hh=`expr $a \/ 3600`
tmp=`expr $a \% 3600`
mm=`expr $tmp \/ 60`
ss=`expr $tmp \% 60`
if [ $ss -lt 10 ]
then
ss="0"$ss
fi
if [ $mm -lt 10 ]
then
mm="0"$mm
fi
if [ $hh -lt 10 ]
then
hh="0"$hh
fi
echo "====================================================================================="
# echo "==============================================================================="
echo "\t\t\t\t\t\tTotal CPU time: "$hh:$mm:$ss
# echo $hh:$mm:$ss
}
function set_parameter {
min_cpu_time=$1
id=$2
}
# Start of program
print_title $0 $1 $2
if [ $# -ne 2 ]
then
print_usage
exit
fi
set_parameter $1 $2
print_heading
MYSUM=0
my_date=""
bhist -a -u $id | \
while read jobid user job_name pend psusp run ususp ssusp unkwn total
do
case $jobid in
"") continue ;;
JOBID) continue ;;
Summary) continue ;;
esac
if [ $run -lt $min_cpu_time ]
then
continue
else
if [ ${#job_name} -gt 7 ]
then
NUM_TAB='\t'
else
NUM_TAB='\t\t'
fi
MYSUM=`expr $MYSUM + $run`
# echo $MYSUM $run
# echo `bhist -l $jobid | grep Submitted | cut -d: -f1-3` "\t$jobid\t$user\t$job_name$NUM_TAB$pend\t$psusp\t$run\t$ususp\t$ssusp\t$unkwn\t$total"
my_date=`bhist -l $jobid | grep Submitted | cut -c5-19`
my_date_length=`echo $my_date | wc -c`
if [ my_date_length -lt 16 ]
then
PADDING=" "
else
PADDING=""
fi
echo $my_date$PADDING" $jobid\t$user\t$job_name$NUM_TAB$pend\t$psusp\t$run\t$ssusp\t$total"
fi
done
print_footer $MYSUM
# End of program
Friday, August 29, 2008
Tuesday, July 22, 2008
What is biggest file on my hard disk?
su + sort
About ten years ago, whenever my hard disk space is running low, I used to run shell script using 'du' command to find out which file is taking biggest space and select victim file/directory to kill and save some room on my hard disk. Basically, the script uses 'du' command for each directory recursively from /root directory, after than, 'sort' command sorts that result descending order. Then the first item on the list is taking biggest space and so on.
KDirStat
Now, my SUSE linux has a gui application called 'KDirStat'. It exactly does same thing and shows the result even graphical way to help you more visually figure out which file you should delete to get some extra room on your hard disk .:)
For my experiments, I have 3 Matlab installed on my hard disk and each of it takes about 2GB. Because of these 3 Matlabs, my hard disk is running low and only 1GB is left. Now, I removed oldest version of Matlab and it resulted in 3G available disk space.
About ten years ago, whenever my hard disk space is running low, I used to run shell script using 'du' command to find out which file is taking biggest space and select victim file/directory to kill and save some room on my hard disk. Basically, the script uses 'du' command for each directory recursively from /root directory, after than, 'sort' command sorts that result descending order. Then the first item on the list is taking biggest space and so on.
KDirStatNow, my SUSE linux has a gui application called 'KDirStat'. It exactly does same thing and shows the result even graphical way to help you more visually figure out which file you should delete to get some extra room on your hard disk .:)
For my experiments, I have 3 Matlab installed on my hard disk and each of it takes about 2GB. Because of these 3 Matlabs, my hard disk is running low and only 1GB is left. Now, I removed oldest version of Matlab and it resulted in 3G available disk space.
Friday, March 7, 2008
Ubuntu/VirtualBox/OpenSuSE10.3/Full screen
Ubuntu 7.10/VirtualBox1.5.2_OSE/SuSE 10.3 x86_64
I have been playing with VirtualBox on OpenSuSE 10.3 and have been successful with Windows guest but, Ubuntu guest has been irritated me few days by resisting to be full screen mode. I have installed 'Guest Addition' and did everything I have seen from internet forum. But, all didn't work.
Finally, I found out the reason why it is. When I was installing 'Guest Addition', I just clicked the icon of installation script' file and I thought it's done. Today, I opened a terminal window and typed installation script name to run it and it turns out that I should be a root to install it. I've totally forgot about this.
So I used 'sudo' command to run it under root privilege and it works fine now.
I have been playing with VirtualBox on OpenSuSE 10.3 and have been successful with Windows guest but, Ubuntu guest has been irritated me few days by resisting to be full screen mode. I have installed 'Guest Addition' and did everything I have seen from internet forum. But, all didn't work.
Finally, I found out the reason why it is. When I was installing 'Guest Addition', I just clicked the icon of installation script' file and I thought it's done. Today, I opened a terminal window and typed installation script name to run it and it turns out that I should be a root to install it. I've totally forgot about this.
So I used 'sudo' command to run it under root privilege and it works fine now.
CentOS5/Remote desktop
CentOS5/Remote desktop
Using remote desktop on CentOS5(RHEL5) looks simple until you face
'unable to connect to host: No route to host (113)'
error message.
This is caused by firewall setting of target machine, which is normally a machine you want to connect to.
Don't panic!
It's very simple. Login to the target machine and click the menu 'System/Administration/Security level and Firewall'
and click 'Other ports' then add 5900 port.
Once you have done so far, try again.
Using remote desktop on CentOS5(RHEL5) looks simple until you face
'unable to connect to host: No route to host (113)'
error message.
This is caused by firewall setting of target machine, which is normally a machine you want to connect to.
Don't panic!
It's very simple. Login to the target machine and click the menu 'System/Administration/Security level and Firewall'
and click 'Other ports' then add 5900 port.
Once you have done so far, try again.
Tuesday, November 20, 2007
RHEL dual monitor
Dual monitor test on RHEL 4
Video card
nVidia 128MB PCIe x16 nVidia Quadro NVS 285, Dual DVI or Dual VGA capable
Monitors
Two monitors have different resolution each other;
1)DELL 2007FPW, 1600x1200
2)DELL 1707FPVt, 1280x1024
Test methods
There are two different mode for utilizing 2 monitors on Linux. Both of them can be configured in /etc/X11/xorg.conf file.
TwinView : Single X session
Dual monitor : Dual X session
Results
TwinView shares one X session and it looks virtually one big screen. However, failed to have a optimal resolution since two monitor has different resolution and ether of them does not work in maximum resolution and it looks like big font or low resolution.
Dual monitor looks like you have two machines since it shows you different X session on each monitor. Mouse pointer moves one to another back and forth. It works fine and each monitor has menu bar separately. Best solution for two monitor with different resolution.
References
http://gentoo-wiki.com/HOWTO_Dual_Monitors
http://www.yolinux.com/TUTORIALS/LinuxAndDualMonitors.html
http://http.download.nvidia.com/XFree86/Linux-x86/1.0-8178/README/appendix-p.html
http://support.dell.com/support/edocs/video/P107426/en/usage.htm#display_modes
http://support.dell.com/support/edocs/monitors/2007WFP/en/index.htm
http://support2.jp.dell.com/docs/monitors/1707FPV/en/index.htm
Video card
nVidia 128MB PCIe x16 nVidia Quadro NVS 285, Dual DVI or Dual VGA capable
Monitors
Two monitors have different resolution each other;
1)DELL 2007FPW, 1600x1200
2)DELL 1707FPVt, 1280x1024
Test methods
There are two different mode for utilizing 2 monitors on Linux. Both of them can be configured in /etc/X11/xorg.conf file.
TwinView : Single X session
# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig: version 1.0 (buildmeister@builder26) Wed Mar 29 15:31:59 PST 2006
# Xorg configuration created by system-config-display
Section "ServerLayout"
Identifier "single head configuration"
Screen 0 "Screen0" 0 0
InputDevice "Mouse0" "CorePointer"
InputDevice "Keyboard0" "CoreKeyboard"
EndSection
Section "Files"
# RgbPath is the location of the RGB database. Note, this is the name of the
# file minus the extension (like ".txt" or ".db"). There is normally
# no need to change the default.
# Multiple FontPath entries are allowed (they are concatenated together)
# By default, Red Hat 6.0 and later now use a font server independent of
# the X server to render fonts.
RgbPath "/usr/X11R6/lib/X11/rgb"
FontPath "unix/:7100"
EndSection
Section "Module"
Load "dbe"
Load "extmod"
Load "fbdevhw"
Load "glx"
Load "record"
Load "freetype"
Load "type1"
Load "vnc"
EndSection
Section "ServerFlags"
Option "AllowMouseOpenFail" "yes"
EndSection
Section "InputDevice"
# Specify which keyboard LEDs can be user-controlled (eg, with xset(1))
# Option "Xleds" "1 2 3"
# To disable the XKEYBOARD extension, uncomment XkbDisable.
# Option "XkbDisable"
# To customise the XKB settings to suit your keyboard, modify the
# lines below (which are the defaults). For example, for a non-U.S.
# keyboard, you will probably want to use:
# Option "XkbModel" "pc102"
# If you have a US Microsoft Natural keyboard, you can use:
# Option "XkbModel" "microsoft"
#
# Then to change the language, change the Layout setting.
# For example, a german layout can be obtained with:
# Option "XkbLayout" "de"
# or:
# Option "XkbLayout" "de"
# Option "XkbVariant" "nodeadkeys"
#
# If you'd like to switch the positions of your capslock and
# control keys, use:
# Option "XkbOptions" "ctrl:swapcaps"
# Or if you just want both to be control, use:
# Option "XkbOptions" "ctrl:nocaps"
#
Identifier "Keyboard0"
Driver "kbd"
Option "XkbModel" "pc105"
Option "XkbLayout" "us"
EndSection
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5"
Option "Emulate3Buttons" "no"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Monitor Vendor"
ModelName "Dell 2000FP (Digital)"
HorizSync 31.5 - 80.0
VertRefresh 56.0 - 76.0
Option "dpms"
EndSection
Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "Videocard vendor"
BoardName "NVIDIA PCI-Express Quadro 4 285 NVS"
# for TwinView
BusID "PCI:7:0:0"
Option "TwinView"
Option "MetaModes" "1280x1024, 1280x1024; 1280x1024,1280x1024; 1280x1024; 1024x768,1024x768; 1024x768; 800x600,800x600; 800x600"
Option "TwinViewOrientation" "RightOf"
Option "SecondMonitorHorizSync" "30-81"
Option "SecondMonitorVertRefresh" "56-76"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "SecurityTypes" "VncAuth"
Option "UserPasswdVerifier" "VncAuth"
Option "PasswordFile" "/root/.vnc/passwd"
SubSection "Display"
Viewport 0 0
Depth 16
Modes "1024x768" "800x600" "640x480"
EndSubSection
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1600x1200" "1680x1050" "1400x1050" "1280x1024" "1280x960" "1152x864" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Dual monitor : Dual X session
# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig: version 1.0 (buildmeister@builder26) Wed Mar 29 15:31:59 PST 2006
# Xorg configuration created by system-config-display
Section "ServerLayout"
# Identifier "single head configuration"
Identifier "single head configuration"
Screen "Screen0" LeftOf "Screen1"
Screen "Screen1"
InputDevice "Mouse0" "CorePointer"
InputDevice "Keyboard0" "CoreKeyboard"
EndSection
Section "Files"
# RgbPath is the location of the RGB database. Note, this is the name of the
# file minus the extension (like ".txt" or ".db"). There is normally
# no need to change the default.
# Multiple FontPath entries are allowed (they are concatenated together)
# By default, Red Hat 6.0 and later now use a font server independent of
# the X server to render fonts.
RgbPath "/usr/X11R6/lib/X11/rgb"
FontPath "unix/:7100"
EndSection
Section "Module"
Load "dbe"
Load "extmod"
Load "fbdevhw"
Load "glx"
Load "record"
Load "freetype"
Load "type1"
Load "vnc"
EndSection
Section "ServerFlags"
Option "AllowMouseOpenFail" "yes"
EndSection
Section "InputDevice"
# Specify which keyboard LEDs can be user-controlled (eg, with xset(1))
# Option "Xleds" "1 2 3"
# To disable the XKEYBOARD extension, uncomment XkbDisable.
# Option "XkbDisable"
# To customise the XKB settings to suit your keyboard, modify the
# lines below (which are the defaults). For example, for a non-U.S.
# keyboard, you will probably want to use:
# Option "XkbModel" "pc102"
# If you have a US Microsoft Natural keyboard, you can use:
# Option "XkbModel" "microsoft"
#
# Then to change the language, change the Layout setting.
# For example, a german layout can be obtained with:
# Option "XkbLayout" "de"
# or:
# Option "XkbLayout" "de"
# Option "XkbVariant" "nodeadkeys"
#
# If you'd like to switch the positions of your capslock and
# control keys, use:
# Option "XkbOptions" "ctrl:swapcaps"
# Or if you just want both to be control, use:
# Option "XkbOptions" "ctrl:nocaps"
#
Identifier "Keyboard0"
Driver "kbd"
Option "XkbModel" "pc105"
Option "XkbLayout" "us"
EndSection
Section "InputDevice"
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "IMPS/2"
Option "Device" "/dev/input/mice"
Option "ZAxisMapping" "4 5"
Option "Emulate3Buttons" "no"
EndSection
Section "Monitor"
Identifier "Monitor0"
VendorName "Monitor Vendor"
ModelName "Dell 2000FP (Digital)"
HorizSync 31.5 - 80.0
VertRefresh 56.0 - 76.0
Option "dpms"
EndSection
Section "Monitor"
Identifier "Monitor1"
VendorName "Monitor Vendor"
ModelName "Dell 1707FP (Digital)"
HorizSync 31.5 - 80.0
VertRefresh 56.0 - 76.0
Option "dpms"
EndSection
Section "Device"
Identifier "Videocard0"
Driver "nvidia"
VendorName "Videocard vendor"
BoardName "NVIDIA PCI-Express Quadro 4 285 NVS"
BusID "PCI:7:0:0"
Screen 0
EndSection
Section "Device"
Identifier "Videocard1"
Driver "nvidia"
VendorName "Videocard vendor"
BoardName "NVIDIA PCI-Express Quadro 4 285 NVS"
BusID "PCI:7:0:0"
Screen 1
EndSection
Section "Screen"
Identifier "Screen0"
Device "Videocard0"
Monitor "Monitor0"
DefaultDepth 24
Option "SecurityTypes" "VncAuth"
Option "UserPasswdVerifier" "VncAuth"
Option "PasswordFile" "/root/.vnc/passwd"
SubSection "Display"
Viewport 0 0
Depth 16
Modes "1024x768" "800x600" "640x480"
EndSubSection
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1600x1200" "1680x1050" "1400x1050" "1280x1024" "1280x960" "1152x864" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Section "Screen"
Identifier "Screen1"
Device "Videocard1"
Monitor "Monitor1"
DefaultDepth 24
Option "SecurityTypes" "VncAuth"
Option "UserPasswdVerifier" "VncAuth"
Option "PasswordFile" "/root/.vnc/passwd"
SubSection "Display"
Viewport 0 0
Depth 16
Modes "1024x768" "800x600" "640x480"
EndSubSection
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1600x1200" "1680x1050" "1400x1050" "1280x1024" "1280x960" "1152x864" "1024x768" "800x600" "640x480"
EndSubSection
EndSection
Results
TwinView shares one X session and it looks virtually one big screen. However, failed to have a optimal resolution since two monitor has different resolution and ether of them does not work in maximum resolution and it looks like big font or low resolution.
Dual monitor looks like you have two machines since it shows you different X session on each monitor. Mouse pointer moves one to another back and forth. It works fine and each monitor has menu bar separately. Best solution for two monitor with different resolution.
References
http://gentoo-wiki.com/HOWTO_Dual_Monitors
http://www.yolinux.com/TUTORIALS/LinuxAndDualMonitors.html
http://http.download.nvidia.com/XFree86/Linux-x86/1.0-8178/README/appendix-p.html
http://support.dell.com/support/edocs/video/P107426/en/usage.htm#display_modes
http://support.dell.com/support/edocs/monitors/2007WFP/en/index.htm
http://support2.jp.dell.com/docs/monitors/1707FPV/en/index.htm
Wednesday, October 10, 2007
RHEL4 Apache UserDir
On RHEL4, there is one more thing you need to do before your UserDir gets work.
http://www.astahost.com/info.php/configuring-apache-fedora-core-3-userdir_t3659.html
chcon -t httpd_sys_content_t -R $HOME/public_html
Now public_html works fine.
http://www.astahost.com/info.php/configuring-apache-fedora-core-3-userdir_t3659.html
chcon -t httpd_sys_content_t -R $HOME/public_html
Now public_html works fine.
Tuesday, March 20, 2007
Playing with SuSE
To config sendmail on SuSE 10.1
/etc/sysconfig/sendmail
/etc/sysconfig/mail
SMTPD_LISTEN_REMOTE="yes" # for incoming email
$/sbin/SuSEconfig
$service sendmail restart
/etc/sysconfig/sendmail
/etc/sysconfig/mail
SMTPD_LISTEN_REMOTE="yes" # for incoming email
$/sbin/SuSEconfig
$service sendmail restart
Subscribe to:
Comments (Atom)