erver上创建一个作业提交队列【什么叫作业提交队列?】。
在集群的所有节点上指定一个cluster
name作为property【指定同一的集群名称?】。这可以用qmgr命令做到。比如:
qmgr -c 'set node node properties=cluster-name'。
确保作业可以提交到节点上去。这可以通过使用qsub命令做到。比如:
echo
'sleep 30' | qsub -l nodes=3。
一、Torque安装(在master管理结点上【又称为主节点、服务器?】)
1. 解压安装包
tar -zxvf torque-2.3.0.tar.gz
2. 进入到解压后的文件夹
./configure –prefix=/usr/torque --with-default-server=keylab2
make
make install
3.
打包,必须是个普通用户【为什么必须是普通用户,针对普通用户产生的包,可用于所有普通用户】
1) [root@master torque-2.3.0]#./torque.setup
yexq
2) [root@master torque-2.3.0]# make packages
把产生的tpackages , torque-package-clients-linux-x86-64.sh,
torque-package-mom-linux-x86-64.sh拷贝到所有节点。
3) 客户端安装【提交任务的机器上安装,一般到主节点上提交任务】
在keylab2上执行
[root@master torque-2.3.0]#
./torque-package-server-linux-x86_64.sh --install
[root@master torque-2.3.0]#
./torque-package-clients-linux-x86_64.sh--install
[root@master torque-2.3.0]# ./torque-package-mom-linux-x86_64.sh
--install
4)加入运算节点、修改/etc/hosts文件
编辑/var/spool/torque/server_priv/nodes (需要自己建立)加入如下内容
keylab2 np=24
修改/etc/hosts文件
[root@keylab2 vasp]# cat /etc/hosts
192.168.1.100 keylab1
192.168.1.102 keylab2
root@xiaoqiugood:~# cat /etc/hosts
#
# hosts
This file describes a number ofhostname-to-address
#
mappings for the
TCP/IPsubsystem.
It is mostly
#
used at boot time,
when no nameservers are running.
#
On small systems,
this file canbe used instead of a
#
'named' nameserver.
Just add the names, addresses
#
and any aliases to
this file...
#
# By the way, Arnt Gulbrandsen says that 127.0.0.1
# should NEVER be named with the name ofthe machine.
It
causes problems
# for some (stupid) programs, irc andreputedly talk. :^)
#
# For loopbacking.
127.0.0.1
localhost
127.0.0.1
xiaoqiugood.xiaoqiugood.orgxiaoqiugood
# End of hosts.
修改服务器上的/etc/hosts文件,确保由主机名字到IP地址的正确翻译。
127.0.0.1 localhost localhost.localdomain
保留localhost域名供没有连网时使用
192.168.0.254 server1.example.com
server1
192.168.0.1 station1.example.com
station1
192.168.0.2 station2.example.com
station2
4)-I 配置 服务器名、mom进程配置文件
a)设置服务器名(主节点的名字):编辑/var/spool/torque/server_name文件,填写本机机器名(如keylab2)
b)设置MOM进程配置文件:编辑 /var/spool/torque/mom_priv/config文件,写入如下内容
$logevent0×1ff
$clienthost keylab2
4) –II环境变量设置及首次启动PBS(注意顺序)
LD_LIBRARY_PATH=/usr/torque/lib:$LD_LIBRARY_PATH
PATH=/usr/torque/bin: /usr/torque/sbin:$PATH
a)
启动mom进程:(任意路径下)执行命令
pbs_mom
b)启动调度器进程:执行命令
pbs_sched
root@keylab2 torque]# pbs_sched
pbs_sched: LOG_ERROR::Address already inuse (98) in main,
bind
c)启动OpenPBS服务器进程,创建pbs数据库
执行命令
pbs_server -t
create
其参数 -t
create 只在首次启动server进程时才需要
pbs_server: network: Address already in use
PBS_Server: LOG_ERROR::PBS_Server,init_network failed dis
4)-III 设置成开机自启动
[root@keylab2 vasp]# cat
/etc/rc.local
#!/bin/sh
#
# This script will be executed *after* allthe other init
scripts.
# You can put your own initialization stuffin here if you
don't
# want to do the full Sys V style initstuff.
touch /var/lock/subsys/local
/usr/torque/sbin/pbs_server -a true
/usr/torque/sbin/pbs_sched
/usr/torque/sbin/pbs_mom
4)-IV 将PBS做成系统服务,以便随时启动、关闭它
a编辑/etc/pbs.conf文件,填入如下内容:
#tecture and comment out the 'SAMPLEFILE' line below
#SAMPLE FILE
# MPD_SECRETWORD=123456
#secretword=yexiaoqiu123456
export pbs_home=/var/spool/torque
export pbs_exec=/usr/torque/
start_server=1
start_sched=1
start_mom=1
b)
将OpenPBS解压目录中的……/src/tools/init.d/pbs文件复制到/etc/init.d/pbs
c)
执行命令chkconfig
--add pbs
d)
用chkconfig
--list | grep pbs
命令查看pbs服务是否已被加入
e)
现在就可以使用如下命令启动、关闭pbs了:
i.
启动pbs:
service pbs start
ii.
关闭pbs:
service pbs stop
iii.
重启pbs:
service pbsrestart
有错误:
[root@keylab1 tools]#
service pbs start
Starting PBS
pbs_server: network: Address already in use
PBS_Server: LOG_ERROR::PBS_Server,init_network failed dis
PBS server
tail: cannot open `+2' for reading: No suchfile or directory
pbs_mom: LOG_ERROR::Resource temporarilyunavailable (11) in
pbs_mom, cannot lock '/var/spool/torque/mom_priv/mom.lock'- another
mom running
cannot lock'/var/spool/torque/mom_priv/mom.lock' - another mom
running
PBS mom
tail: cannot open `+2' for reading: No suchfile or directory
pbs_sched: LOG_ERROR::Address already inuse (98) in main,
bind
PBS sched
在torque源码包contrib目录中有一个文件
cp contrib/init.d/pbs_mom /etc/init.d/pbs_mom
chkconfig --add pbs_mom
在所有的机器(pbs服务器和计算节点)上都需要如此设置。
6)创建队列
qmgr -c 'set serverscheduling=true'
qmgr -c 'create queue batchqueue_type=execution'
qmgr -c 'set queue batch started=true'
qmgr -c 'set queue batchenabled=true'
qmgr -c 'set queue batchresources_default.nodes=1'
qmgr -c 'set queue batchresources_default.walltime=3600'
qmgr -c 'set serverdefault_queue=batch'
qmgr -c 'set serverscheduling=true'
qmgr -c 'create queue zdsys2queue_type=execution'
qmgr -c 'set queue zdsys2started=true'
qmgr -c 'set queue zdsys2enabled=true'
qmgr –c 'set queue zdsys2resources_max.ncpus =6'
qmgr -c 'set queue zdsys2resources_default.nodes=1'
qmgr -c 'set queue zdsys2resources_default.walltime=3600000'
qmgr -c 'set server default_queue=zdsys2'
qmgr -c 'set serverscheduling=true'
qmgr -c 'create queue super1queue_type=execution'
创建队列
qmgr -c 'set queue super1 started=true'设置队列
qmgr -c 'set queue super1 enabled=true'
qmgr -c 'set queue super1resources_max.ncpus = 24'
qmgr -c 'set queue super1resources_default.nodes=1'
qmgr -c 'set queue super1 resources_default.walltime=3600000'
qmgr -c 'set serverscheduling=true'
qmgr -c 'create queue super1queue_type=execution'
qmgr -c 'set queue super1 started=true'
qmgr -c 'set queue super1 enabled=true'
qmgr -c 'set queue super1resources_max.ncpus = 24'
qmgr -c 'set queue super1resources_default.nodes=1'
qmgr -c 'set queue super1 resources_default.walltime=3600000'
[root@master ~]# qmgr
create queue students
set queue students queue_type = Execution
set queue students Priority = 40
set queue students resources_max.cput =96:00:00
set queue students resources_min.cput =00:00:01
set queue students resources_default.cput =96:00:00
set queue students enabled = True
set queue students started = True
你可以试试
set queue long acl_user_enable = True
set queue long acl_users = test1
set queue long acl_users +=test2
这个是往指定的队列添加指定用户
set queue long resources_max.ncpus = N
set queue long resources_default.walltime = 100:00:00
set queue long max_user_run = N
set queue long max_queuable = N
set queue long max_user_queuable = N
set queue long max_running = N
上面的是对他一些资源进行限制。。具体含义去查查torque的手册吧,很详细的
What are the limits regarding
multiple processors and machines?
|
|
Currently, there is a chart that is displayed when
you login. If you failed to notice the chart or if it scrolled off
the screen, you may display it along with other login messages by
using the following argo command:
cat /etc/motd
At some point, that chart will be removed from
message of the day file
(motd). There are commands that you can use to
display queue limits:
- Maximum number of nodes a student may use in the student_short
queue:
qmgr -c 'list queue student_short' | grep
resources_max.nodect
- Maximum number of processors a student may request for the
job:
qmgr -c 'list queue zdsys1' | grep
resources_max.ncpus
- To see other policies (rules) of the student_short queue:
qmgr -c 'list queue student_short'
The samples use the student_short queue. You may substitute the
name of other queues (student_medium, student_long, staff,
dedicated) to get the corresponding settings for them:
- qmgr -c 'list queue staff' | grep
resources_max.nodect
- qmgr -c 'list queue staff' | grep
resources_max.ncpus
- qmgr -c 'list queue staff'
|
|
7) 验证配置
1.
pbsnodes // 结点信息
[root@keylab1 pbs-software]# pbsnodes
pbsnodes: Server has no node list MSG=nodelist is empty - check
'server_priv/nodes' file
似乎是执行这个pbs_server-t
create冲掉了nodes文件
重新编辑后 执行以下操作
[root@keylab1 pbs-software]# qterm
[root@keylab1 pbs-software]# pbs
pbs-config
pbsdsh
pbs_mom
pbs_sched
pbs_track
pbs_demux
pbs_iff
pbsnodes
pbs_server
[root@keylab1 pbs-software]# pbs
pbs-config
pbsdsh
pbs_mom
pbs_sched
pbs_track
pbs_demux
pbs_iff
pbsnodes
pbs_server
[root@keylab1 pbs-software]# pbs_server
[root@keylab1 pbs-software]#
pbsnodes
执行两次后显示
keylab1
state = down
np = 24
ntype = cluster
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
keylab1
state = free
np = 24
ntype = cluster
status =
rectime=1307431935,varattr=,jobs=,state=free,netload=2635063282,gres=,loadave=12.25,ncpus=24,physmem=66089076kb,availmem=229832984kb,totmem=204558660kb,idletime=310468,nusers=2,nsessions=2,sessions=46024726,uname=Linux
keylab1 2.6.32-71.el6.x86_64#1 SMP Wed Sep 1 01:33:01 EDT 2010
x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
2. qmgr-c 'print server'
//
队列信息
3. pbs任务提交脚本及查看任务zhuangt
B
只用于任务向量,表示任务向量已经开始执行
E
任务在运行后退出
H
任务被服务器或用户或者管理员阻塞
Q
任务正在排队中,等待被调度运行
R
任务正在运行
S
任务被服务器挂起,由于一个更高优先级的任务需要当前任务的资源
T
任务被转移到其它执行节点了
U
由于服务器繁忙,任务被挂起
W
任务在等待它所请求的执行时间的到来(qsub -a)
X
只用于子任务,表示子任务完成
再举一例,使用-a 选项指定任务开始执行时间:
Completed jobs will be reported in the C state and the exit status
is seen in the job attribute.
[root@keylab2 vasp]# pbsnodes
keylab2
state = free
np = 24
ntype = cluster
jobs = 0/5.keylab2, 1/5.keylab2,
2/5.keylab2, 3/5.keylab2, 4/5.keylab2,5/5.keylab2, 6/5.keylab2,
7/5.keylab2, 8/5.keylab2, 9/5.keylab2, 10/5.keylab2,11/5.keylab2,
12/5.keylab2, 13/5.keylab2, 14/5.keylab2,
15/5.keylab2,16/5.keylab2, 17/5.keylab2
status =
rectime=1307302753,varattr=,jobs=5.keylab2,state=free,netload=3171634,gres=,loadave=24.56,ncpus=24,physmem=66089076kb,availmem=21325715656kb,totmem=21858128692kb,idletime=0,nusers=1,nsessions=4,sessions=44116160
7100 7135,uname=Linux keylab2 2.6.32-71.el6.x86_64#1 SMP Wed Sep 1
01:33:01 EDT 2010 x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
[root@keylab2 vasp]# qmgr -c 'print server'
#
# Create queues and set their attributes.
#
#
# Create and define queue zdsys
#
create queue zdsys
set queue zdsys queue_type = Execution
set queue zdsys resources_default.nodes = 2
set queue zdsys resources_default.walltime= 1000:00:00
set queue zdsys enabled = True
set queue zdsys started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_hosts = keylab2
set server default_queue = zdsys
set server log_events = 511
set server mail_from = adm
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 6
set server next_job_number = 6
测试用脚本
[yexq@keylab2 zdsyspbs]$ cat testpbs
#!/bin/sh
#testpbs
#PBS -l nodes=1:ppn=1
#PBS -j oe
#PBS -V
echo This is a test
echo Today is `date`
echo This is `hostname`
echo The current working directory is 'pwd'
ls -alF /home
uptime
正式脚本
[yexq@keylab2 zdsyspbs]$ cat
pbstest4bb
#!/bin/bash
#PBS -q zdsys2
#PBS -l nodes=1:ppn=18
#PBS -j oe
#PBS -V
#PBS
-l walltime=1000:00:00
# go to work dir
cd $PBS_O_WORKDIR
echo
$PBS_O_WORKDIR
# setup mpd env (Of course use some othersecret word than
'dfadfs')
if [ ! -f ~/.mpd.conf ]; then
/bin/echo 'secretword=dfadfs'>> ~/.mpd.conf
/bin/chmod 600 ~/.mpd.conf
fi
# Intel MPI Home
MPI_HOME=/opt/mpich2
# setup Nums of Processor
NP=`cat $PBS_NODEFILE|wc -l`
echo 'Numbers of Processors:
$NP'
echo'---------------------------'
# Number of MPD
N_MPD=`cat $PBS_NODEFILE|uniq|wc -l`
echo 'started mpd Number: $N_MPD'
echo'---------------------------'
# setup mpi env (em64t)
$MPI_HOME/bin/mpdboot -r ssh -n $N_MPD -f$PBS_NODEFILE
# running program
nohup $MPI_HOME/bin/mpd &
cd /home/yexq/zdsyspbs/c225_opt2b
rm -rf opt-*
./opt.sh
wait
cd ../
# clean
$MPI_HOME/bin/mpdallexit
/////
# !/bin/bash
#PBS -q zdsys2
#PBS -l nodes=1:ppn=18
#PBS -j oe
#PBS -V
#PBS
-l walltime=1000:00:00
# go to work dir
cd $PBS_O_WORKDIR
echo
$PBS_O_WORKDIR
# setup mpd env (Of course use some othersecret word than
'dfadfs')
# Intel MPI Home
MPI_HOME=/opt/mpich214
# setup Nums of Processor
NP=`cat $PBS_NODEFILE|wc -l`
echo 'Numbers of Processors:
$NP'
echo'---------------------------'
# Number of MPD
N_MPD=`cat $PBS_NODEFILE|uniq|wc -l`
echo 'started mpd Number: $N_MPD'
echo'---------------------------'
# setup mpi env (em64t)
# running program
./run_scf
# clean
8) 异常情况处理
chmod –R 666 xxxx/
先[root@keylab2 vasp]# chmod a=rx ./vasp.5.2.mpich/vasp.5.2
而后
[yexq@keylab2 vasp.5.2]$ ./vasp
Error reading item 'IMAGES' from file INCAR.
chmod -vR 755 apps/
4、在node0x (x=1-9,计算结点上)
[root@node0x torque-2.3.0]#./torque-package-clients-linux-x86_64.sh
--install
[root@node0x torque-2.3.0]#./torque-package-mom-linux-x86_64.sh
--install
然后启动pbs_mom ,把pbs_mom写入/etc/rc.local
二、Torque PBS使用
1、创建用户在master的root下
useradd test
passwd test
输入test密码
到/var/yp下make一下
gmake[1]:Entering directory `/var/yp/(none)'
Updating passwd.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating passwd.byuid...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating group.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating group.bygid...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating hosts.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating hosts.byaddr...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating rpc.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating rpc.bynumber...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating services.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating services.byservicename...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating netid.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating protocols.bynumber...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating protocols.byname...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating netgroup...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating netgroup.byhost...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating netgroup.byuser...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating mail.aliases...
failed to send 'clear' to local ypserv:RPC: Program not
registeredUpdating shadow.byname... Ignored -> merged
withpasswd
gmake[1]:*** No rule to make target `/etc/publickey', needed by
`publickey.byname'.
Stop.
gmake[1]:Leaving directory `/var/yp/(none)'
make: *** [target]Error 2
2、配置普通用户的ssh
su test
ssh-keygen -t dsa
cd .ssh
cat id_pub.dsa >> authorized_keys
chmod 600 authorized_keys
3、编写作业脚本,见下文
4、启动mpd
mpdboot -n 10 -f mfa
mfa内容:
master:4
node01:4
….
node09:4
5、提交,查询,删除作业
提交作业:qsub pbsjob
[test1@master pbstest]$ qsub pbsjob
48.master
作业提交后会有一个作业号
查询作业:qstat
[test1@master pbstest]$ qstat
删除作业:qdel 作业号
[test1@master pbstest]$ qdel 48
6 PBS服务开启操作流程
我在Dawing上操作成功了!!
1)
在主节点上打开PBS服务
/etc/init.d/pbs_server
start
2)
在主节点和其他节点打开PBS客户端。主节点虽是服务端,但也可参加计算,因而要打开客服。依次执行如下:
/etc/init.d/pbs_mom
start
3)
在所有节点上打开调度器
/etc/init.d/maui.d
start
【?】
对于这些PBS的功能开启有几个相同的参量:
status
查看状态
restart
重启
stop
终止
start
开启
4)
接下来是检查是否可以提交作业
pbsnodes –a
返回free即表示可以提交作业。
5) 写脚本vim pbs_
fdtd_TE_xyPML_MPI_OpenMP
#!/bin/bash
#PBS -lnodes=5:ppn=4
规定使用的节点数nodes以及每个节点能跑多少核ppn
#PBS –Ntaskname
任取一作业任务名taskname
cd$PBS_O_WORKDIR
到工作目录下(此为PBS提供的环境变量)
mpirun -np 20 ./fdtd_TE_xyPML_MPI_OpenMP
执行mpirun一句可以用-machinefile或-p4pg 命令参量制定
6) 提交
qsub pbs_
fdtd_TE_xyPML_MPI_OpenMP
7)
可用qstat查看作业任务,具体参量参看下文。流程终结!
三、PBS常用命令和选项
!!3.1 基本脚本写法和选项
事例:
川大vasp.pb s
!/bin/sh
#PBS -N DiGGA
#PBS -l nodes=node42
#PBS -q fast
cd /home/iamp/work/vasp/DiGGA
/public/software/mpi/openmpi1.4.2-intel/bin/mpirun--mca btl
tcp,self -np 24 -hostfile $PBS_NODEFILE
/home/iamp/software/vas
p.4.6/vasp
PBS是Protable Batch
System的缩写,是一个任务管理系统。当多个用户使用同一个计算资源时,每个用户用PBS脚本提交自己的任务,由PBS对这些任务进行管理和资源的分配。下面是一个简单的PBS脚本:
#!/bin/bash
#PBS -l nodes=20
#PBS -N snaphu
#PBS -j oe
#PBS -l walltime=24:00:00
#PBS -l cput=1:00:00
#PBS -q dque
cd $PBS_O_WORKDIR
cat $PBS_NODEFILE $PBS_NODEFILE>NODEFILE
mpirun -np ./mpitest
将这个脚本保存成submit然后qsub
submit就将这个mpitest的任务提交给了系统。脚本中#PBS为脚本选项,用于设置一些参数。
#PBS
-l表示资源列表,用于设定特定任务所需的一些参数。这里的nodes表示并行环境下可以使用的节点数,而walltime表示任务最大时限,而cput
表示cpu时间的最大时限,运行时间和cpu使用时间超过对应的时限,任务就会以超时退出。这三个参数不是PBS脚本参数,而是并行环境所需的参数。
#PBS -N表示任务名称。
#PBS
-j表示系统输出,如果是oe,则标准错误输出(stderr)和标准输出(stdout)合并为stdout,如果是eo,则合并为stderr,如果没有设定或设定为n,则stderr和stdout分开。
#PBS
-q表示当前任务选用的队列。在并行环境下,一个系统中往往有多个队列,任务提交后,将在所选的队列中排除等候。系统中有哪些队列可以用qstat
-q查看。
PBS脚本文件由脚本选项和运行脚本两部分组成。
1)
PBS作业脚本选项(若无-C选项,则每项前面加‘#PBS’)
2)
运行脚本同LINUX下一般的运行脚本文件格式如下:
mpirun –np 进程数 ./可执行程序名
3.2 PBS命令与选项
PBS提供的4条命令用于作业管理
1. qsub 命令:用于提交作业脚本
命令格式:
qsub
[-a date_time]
[-e path] [-I]
[-lresource_list]
[-M user_list] [-N name]
[-S path_list]
[-uuser_list]
[-Wadditional_attributes]
例:# qsubaaa.pbs
提交某作业,系统将产生一个作业号
2. qstat 命令:用于查询作业状态信息
命令格式:
qstat [-f][-a][-i] [-n][-s] [-R][-Q][-q][-B][-u]
参数说明:
-f
jobid 列出指定作业的信息
-a
列出系统所有作业
-i
列出不在运行的作业
-n
列出分配给此作业的结点
-s
列出队列管理员与scheduler所提供的建议
-R
列出磁盘预留信息
-Q
操作符是destination id,指明请求的是队列状态
-q
列出队列状态,并以alternative形式显示
-au userid
列出指定用户的所有作业
-B
列出PBS Server信息
-r
列出所有正在运行的作业
-Qf queue
列出指定队列的信息
-u
若操作符为作业号,则列出其状态。
若操作符为destination id,则列出运行在其上的属于user_list中用户的作业状态。
例:# qstat -f 211
查询作业号为211的作业的具体信息。
3. qdel 命令:用于删除已提交的作业
命令格式:
qdel [-W 间隔时间] 作业号
例:# qdel -W 15211
15秒后删除作业号为211的作业
4. qmgr 命令:用于队列管理
qmgr -c'create queue batch
queue_type=execution'
qmgr -c'set queue batch
started=true'
qmgr -c'set queue batch
enabled=true'
qmgr -c'set queue batch
resources_default.nodes=1'
qmgr -c'set queue batch
resources_default.walltime=3600'
qmgr -c'set server
default_queue=batch'