" + + "#KillOnBadExit=0
" + + "#Licenses=foo*4,bar
" + +- "#MailProg=/bin/mail
" + ++ "#MailProg=/usr/bin/mail
" + + "#MaxJobCount=5000
" + + "#MaxStepCount=40000
" + + "#MaxTasksPerNode=128
" + diff -Nru slurm-llnl-2.2.7/debian/patches/mail-path-patch slurm-llnl-2.3.2/debian/patches/mail-path-patch --- slurm-llnl-2.2.7/debian/patches/mail-path-patch 2011-05-06 17:35:29.000000000 +0000 +++ slurm-llnl-2.3.2/debian/patches/mail-path-patch 1970-01-01 00:00:00.000000000 +0000 @@ -1,60 +0,0 @@ -Description: mail command absolute pathname - This patch changes the default command to /usr/bin/mail in various - configuration files and manual pages. -Author: Gennaro Oliva
" + - "#KillOnBadExit=0
" + - "#Licenses=foo*4,bar
" + -- "#MailProg=/bin/mail
" + -+ "#MailProg=/usr/bin/mail
" + - "#MaxJobCount=5000
" + - "#MaxTasksPerNode=128
" + - "MpiDefault=" + get_radio_value(document.config.mpi_default) + "
" + diff -Nru slurm-llnl-2.2.7/debian/patches/pamlibslurm slurm-llnl-2.3.2/debian/patches/pamlibslurm --- slurm-llnl-2.2.7/debian/patches/pamlibslurm 1970-01-01 00:00:00.000000000 +0000 +++ slurm-llnl-2.3.2/debian/patches/pamlibslurm 2011-12-28 12:02:37.000000000 +0000 @@ -0,0 +1,68 @@ +Description: Making pamslurm use libslurm shipped with the libslurm package + The pam_slurm module dlopen libslurm.so that is a link usually shipped + with the dev package (libslurm-dev), this patch modify the source to + open the libslur.so.version shipped with the libslurm package. + +--- +Origin: maintainer +Forwarded: by mail +Reviewed-By: Moe Jette
Accounting
- -NOTE: This documents accounting features available in SLURM version -1.3, which are far more extensive than those available in previous -releases.
+Accounting and Resource Limits
SLURM can be configured to collect accounting information for every job and job step executed. @@ -386,7 +382,7 @@ As the mysql user grant privileges to that user using a command such as:
-GRANT ALL ON StorageLoc.* TO 'StorageUser'@'StorageHost'; +
GRANT ALL ON StorageLoc.* TO 'StorageUser'@'StorageHost';
(The ticks are needed)
(You need to be root to do this. Also in the info for password @@ -394,7 +390,7 @@ prompt since the previous mysql statement did not end with a ';'. It assumes that you wish to input more info.)
-live example:
+Live example:
mysql@snowflake:~$ mysql @@ -664,8 +660,10 @@ if not we will refer to the account associated with the job. If the account doesn't have the limit set we will refer to the cluster's limits. -If the cluster doesn't have the limit set no limit will be enforced. -All of the above entities can include limits as described below... +If the cluster doesn't have the limit set no limit will be enforced.
+ +All of the above entities can include limits as described below and +in the Resource Limits document.
-
@@ -673,7 +671,6 @@
Essentially this is the amount of claim this association and it's
children have to the above system. Can also be the string "parent",
this means that the parent association is used for fairshare.
-
- GrpCPUMins= A hard limit of cpu minutes to be used by jobs
@@ -780,7 +777,7 @@
If an entity has existed for less than 1 day, the entity will be removed
completely. This is meant to clean up after typographic errors.
-
Last modified 27 January 2010
+Last modified 10 June 2010
diff -Nru slurm-llnl-2.2.7/doc/html/accounting_storageplugins.shtml slurm-llnl-2.3.2/doc/html/accounting_storageplugins.shtml --- slurm-llnl-2.2.7/doc/html/accounting_storageplugins.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/accounting_storageplugins.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -303,7 +303,7 @@ List acct_storage_p_modify_job(void *db_conn, uint32_t uid, acct_job_modify_cond_t *job_cond, acct_job_rec_t *job)Description:
-Used to modify two fields (the derived exit code and derived exit string) of an +Used to modify two fields (the derived exit code and the comment string) of an existing job in the storage type. Can only modify one job at a time.Arguments:
db_conn (input) connection to @@ -785,6 +785,24 @@ SLURM_ERROR on failure.+int clusteracct_storage_p_fini_ctld(void *db_conn, char *ip, +uint16_t port, char *cluster_nodes) +
Description:
+Used when a controller is turned off to tell the storage type the + slurmctld has gone away. +Arguments:
+db_conn (input) connection to +the storage type.
+ip (input) ip of connected slurmctld.
+port (input) port on host cluster is +running on the host is grabbed from the connection.
+cluster_nodes (input) name of all +nodes currently on the cluster.
+Returns:
+SLURM_SUCCESS on success, or
+SLURM_ERROR on failure. + +int jobacct_storage_p_job_start(void *db_conn, struct job_record *job_ptr)
Description:
Note that a job is about to begin execution or has just changed size. diff -Nru slurm-llnl-2.2.7/doc/html/big_sys.shtml slurm-llnl-2.3.2/doc/html/big_sys.shtml --- slurm-llnl-2.2.7/doc/html/big_sys.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/big_sys.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -72,7 +72,7 @@ on each compute node and use it for scheduling purposes, this entails extra overhead. Optimize performance by specifying the expected configuration using -the available parameters (RealMemory, Procs, and +the available parameters (RealMemory, CPUs, and TmpDisk). If the node is found to contain less resources than configured, it will be marked DOWN and not used. diff -Nru slurm-llnl-2.2.7/doc/html/bluegene.shtml slurm-llnl-2.3.2/doc/html/bluegene.shtml --- slurm-llnl-2.2.7/doc/html/bluegene.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/bluegene.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -6,40 +6,45 @@This document describes the unique features of SLURM on the IBM BlueGene systems. -You should be familiar with the SLURM's mode of operation on Linux clusters +You should be familiar with SLURM's mode of operation on Linux clusters before studying the relatively few differences in BlueGene operation described in this document.
BlueGene systems have several unique features making for a few differences in how SLURM operates there. -The BlueGene system consists of one or more base partitions or -midplanes connected in a three-dimensional torus. -Each base partition consists of 512 c-nodes each containing two processors; -one designed primarily for computations and the other primarily for managing communications. -The c-nodes can execute only one process and thus are unable to execute both -the user's jobs and SLURM's slurmd daemon. -Thus the slurmd daemon executes on one of the BlueGene Front End Nodes. -This single slurmd daemon provides (almost) all of the normal SLURM services +BlueGene systems consists of one or more base partitions or +midplanes connected in a three-dimensional (BlueGene/L and BlueGene/P +systems) or five-dimensional (BlueGene/Q) torus. +Each base partition typically includes 512 c-nodes or compute +nodes each containing two or more cores; +one core is typically designed primarily for managing communications while the +other cores are used primarily for computations. +Each c-node can execute only one process and thus are unable to execute +both the user's application plus SLURM's slurmd daemon. +Thus the slurmd daemon(s) executes on one or more of the BlueGene +Front End Nodes. +The slurmd daemons provide (almost) all of the normal SLURM services for every base partition on the system.
Internally SLURM treats each base partition as one node with -1024 processors, which keeps the number of entities being managed reasonable. +a processor count equal to the number of cores on the base partition, which +keeps the number of entities being managed by SLURM more reasonable. Since the current BlueGene software can sub-allocate a base partition -into blocks of 32 and/or 128 c-nodes, more than one user job can execute -on each base partition (subject to system administrator configuration). +into smaller blocks, more than one user job can execute on each base +partition (subject to system administrator configuration). In the case of +BlueGene/Q systems, more than one user job can also execute in each block. To effectively utilize this environment, SLURM tools present the user with -the view that each c-nodes is a separate node, so allocation requests -and status information use c-node counts (this is a new feature in -SLURM version 1.1). +the view that each c-node is a separate node, so allocation requests +and status information use c-node counts. Since the c-node count can be very large, the suffix "k" can be used -to represent multiples of 1024 (e.g. "2k" is equivalent to "2048").
+to represent multiples of 1024 or "m" for multiples of 1,048,576 (1024 x 1024). +For example, "2k" is equivalent to "2048".User Tools
-The normal set of SLURM user tools: sbatch, scancel, sinfo, squeue, and scontrol -provide all of the expected services except support for job steps. -SLURM performs resource allocation for the job, but initiation of tasks is performed -using the mpirun command. SLURM has no concept of a job step on BlueGene. +
The normal set of SLURM user tools: sbatch, scancel, sinfo, squeue, and +scontrol provide all of the expected services except support for job steps, +which is detailed later. Seven new sbatch options are available: --geometry (specify job size in each dimension), --no-rotate (disable rotation of geometry), @@ -52,25 +57,25 @@ --ramdisk-image (specify alternative ramdisk image for bluegene block. Default if not set, BGL only.) The --nodes option with a minimum and (optionally) maximum node count continues to be available. - Note that this is a c-node count.
-To reiterate: sbatch is used to submit a job script, -but mpirun is used to launch the parallel tasks. -Note that a SLURM batch job's default stdout and stderr file names are generated -using the SLURM job ID. -When the SLURM control daemon is restarted, SLURM job ID values can be repeated, -therefore it is recommended that batch jobs explicitly specify unique names for -stdout and stderr files using the srun options --output and --error -respectively. -While the salloc command may be used to create an interactive SLURM job, -it will be the responsibility of the user to insure that the bgblock -is ready for use before initiating any mpirun commands. -SLURM will assume this responsibility for batch jobs. -The script that you submit to SLURM can contain multiple invocations of mpirun as -well as any desired commands for pre- and post-processing. +
Task Launch on BlueGene/Q only
+ +Use SLURM's srun command to launch tasks (srun is a wrapper for IBM's +runjob command. +SLURM job step information including accounting functions as expected.
+ +Task Launch on BlueGene/L and BlueGene/P only
+ +SLURM performs resource allocation for the job, but initiation of tasks is +performed using the mpirun command. SLURM has no concept of a job step +on BlueGene/L or BlueGene/P systems. +To reiterate: salloc or sbatch are used to create a job allocation, but +mpirun is used to launch the parallel tasks. +The script that you submit to SLURM can contain multiple invocations of mpirun +as well as any desired commands for pre- and post-processing. The mpirun command will get its bgblock information from the -MPIRUN_PARTITION as set by SLURM. A sample script is shown below. +MPIRUN_PARTITION as set by SLURM. A sample script is shown below.
#!/bin/bash # pre-processing @@ -80,51 +85,50 @@ mpirun -exec /home/user/prog -cwd /home/user -args 124 # post-processing date -
- -Naming Convensions
-The naming of base partitions includes a three-digit suffix representing the its -coordinates in the X, Y and Z dimensions with a zero origin. -For example, "bg012" represents the base partition whose coordinate is at X=0, Y=1 and Z=2. In a system -configured with small blocks (any block less than a full base partition) there will be divisions -into the base partition notation. For example, if there were 64 psets in the -configuration, bg012[0-15] represents -the first quarter or first 16 ionodes of a midplane. In BlueGene/L -this would be 128 c-node block. To represent the first nodecard in the -second quarter or ionodes 16-19 the notation would be bg012[16-19], or -a 32 c-node block. -Since jobs must allocate consecutive base partitions in all three dimensions, we have developed -an abbreviated format for describing the base partitions in one of these three-dimensional blocks. -The base partition has a prefix determined from the system which is followed by the end-points -of the block enclosed in square-brackets and separated by an "x". -For example, "bg[620x731]" is used to represent the eight base partitions enclosed in a block -with end-points and bg620 and bg731 (bg620, bg621, bg630, bg631, bg720, bg721, -bg730 and bg731).
- --IMPORTANT: SLURM version 1.2 or higher can handle a bluegene system of -sizes up to 36x36x36. To try to keep with the 'three-digit suffix -representing the its coordinates in the X, Y and Z dimensions with a -zero origin', we now support A-Z as valid numbers. This makes it so -the prefix must always be lower case, and any letters in the -three-digit suffix must always be upper case. This schema -should be used in your slurm.conf file and in your bluegene.conf file -if you put a prefix there even though it is not necessary there. This -schema should also be used to specify midplanes or locations in -configure mode of smap. + +
Naming Conventions
+The naming of base partitions includes a numeric suffix representing the its +coordinates with a zero origin. The suffix contains three digits on BlueGene/L +and BlueGene/P systems, while four digits are required for the BlueGene/Q +systems. For example, "bgp012" represents the base partition whose coordinate +is at X=0, Y=1 and Z=2. +SLURM uses an abbreviated format for describing base partitions in which the +end-points of the block enclosed are in square-brackets and separated by an "x". +For example, "bgp[620x731]" is used to represent the eight base partitions +enclosed in a block with end-points and bgp620 and bgp731 (bgp620, bgp621, +bgp630, bgp631, bgp720, bgp721, bgp730 and bgp731).
+ +IMPORTANT: SLURM higher can support up to 36 elements in each +BlueGene dimension by supporting "A-Z" as valid numbers. SLURM requires the +prefix to be lower case and any letters in the suffix must always be upper +case. This schema must be used in both the slurm.conf and bluegene.conf +configuration files when specifying midplane/node names (the prefix is +optional). This schema should also be used to specify midplanes or locations +in configure mode of smap:
-
-valid: bgl[000xC44] bgl000 bglZZZ +valid: bgl[000xC44], bgl000, bglZZZ
-invalid: BGL[000xC44] BglC00 bglb00 Bglzzz +invalid: BGL[000xC44], BglC00, bglb00, BglzzzOne new tool provided is smap. -As of SLURM version 1.2, sview is -another new tool offering even more viewing and configuring options. -Smap is aware of system topography and provides a map of what base partitions -are allocated to jobs, partitions, etc. -See the smap man page for details. +
In a system configured with small blocks (any block less +than a full base partition) there will be divisions in the base partition +notation. On BlueGene/L and BlueGene/P systems, the base partition name may +be followed by a square bracket enclosing ID numbers of the IO nodes associated +with the block. For example, if there are 64 psets in a BlueGene/L +configuration, "bgl012[0-15]" represents the first quarter or first 16 IO nodes +of a midplane. In BlueGene/L this would be 128 c-node block. To represent +the first nodecard in the second quarter or IO nodes 16-19, the notation would +be "bgl012[16-19]", or a 32 c-node block. On BlueGene/Q systems, the specific +c-nodes would be identified in square brackets using their five digit +coordinates. For example "bgq0123[00000x11111]" would represent the 32 c-nodes +in midplane "bgq0123" having coordinates (within that midplane) from zero to +one in each of the five dimensions.
+ +Two topology-aware graphical user interfaces are provided: smap and +sview (sview provides more viewing and configuring options). +See each command's man page for details. A sample of smap output is provided below showing the location of five jobs. Note the format of the list of base partitions allocated to each job. Also note that idle (unassigned) base partitions are indicated by a period. @@ -226,14 +230,17 @@
The slurmctld daemon should execute on the system's service node. If an optional backup daemon is used, it must be in some location where it is capable of executing Bridge APIs. -One slurmd daemon should be configured to execute on one of the front end nodes. -That one slurmd daemon represents communications channel for every base partition. -You can use the scontrol command to drain individual nodes as desired and -return them to service.
- -The slurm.conf (configuration) file needs to have the value of InactiveLimit -set to zero or not specified (it defaults to a value of zero). -This is because there are no job steps and we don't want to purge jobs prematurely. +The slurmd daemons executes the user scripts and there must be at least one +front end node configured for this purpose. Multiple front end nodes may be +configured for slurmd use to improve performance and fault tolerance. +Each slurmd can execute jobs for every base partition and the work will be +distributed among the slurmd daemons to balance the workload. +You can use the scontrol command to drain individual compute nodes as desired +and return them to service.
+ +The slurm.conf (configuration) file needs to have the value of +InactiveLimit set to zero or not specified (it defaults to a value of zero). +This is because if there are no job steps, we don't want to purge jobs prematurely. The value of SelectType must be set to "select/bluegene" in order to have node selection performed using a system aware of the system's topography and interfaces. @@ -248,7 +255,7 @@ The prolog and epilog programs are used to insure proper synchronization between the slurmctld daemon, the user job, and MMCS. A multitude of other functions may also be placed into the prolog and -epilog as desired (e.g. enabling/disabling user logins, puring file systmes, +epilog as desired (e.g. enabling/disabling user logins, purging file systems, etc.). Sample prolog and epilog scripts follow.
@@ -257,8 +264,9 @@ # # Wait for bgblock to be ready for this job's use /usr/sbin/slurm_prolog +
- +#!/bin/bash # Sample BlueGene Epilog script # @@ -294,31 +302,47 @@
SLURM node and partition descriptions should make use of the naming conventions described above. For example, -"NodeName=bg[000x733] NodeAddr=frontend0 NodeHostname=frontend0 Procs=1024" +"NodeName=bg[000x733] CPUs=1024" is used in slurm.conf to define a BlueGene system with 128 midplanes -in an 8 by 4 by 4 matrix. +in an 8 by 4 by 4 matrix and each midplane is configured with 1024 processors +(cores). The node name prefix of "bg" defined by NodeName can be anything you want, but needs to be consistent throughout the slurm.conf file. -Note that the values of both NodeAddr and NodeHostname for all -128 base partitions is the name of the front-end node executing -the slurmd daemon. No computer is actually expected to a hostname of "bg000" and no -attempt will be made to route message traffic to this address.
+attempt will be made to route message traffic to this address. -While users are unable to initiate SLURM job steps on BlueGene systems, -this restriction does not apply to user root or SlurmUser. -Be advised that the one slurmd supporting all nodes is unable to manage a -large number of job steps, so this ability should be used only to verify normal -SLURM operation. +
Front end nodes used for executing the slurmd daemons must also be defined +in the slurm.conf file. +It is recommended that at least two front end nodes be dedicated to use by +the slurmd daemons for fault tolerance. +For example: +"FrontendName=frontend[00-03] State=UNKNOWN" +is used to define four front end nodes for running slurmd daemons.
+ ++# Portion of slurm.conf for BlueGene system +InactiveLimit=0 +SelectType=select/bluegene +Prolog=/usr/sbin/prolog +Epilog=/usr/sbin/epilog +# +FrontendName=frontend[00-01] State=UNKNOWN +NodeName=bg[000x733] CPUs=1024 State=UNKNOWN +
+ +While users are unable to initiate SLURM job steps on BlueGene/L or BlueGene/P +systems, this restriction does not apply to user root or SlurmUser. +Be advised that the slurmd daemon is unable to manage a large number of job +steps, so this ability should be used only to verify normal SLURM operation. If large numbers of job steps are initiated by slurmd, expect the daemon to fail due to lack of memory or other resources. -It is best to minimize other work on the front-end node executing slurmd +It is best to minimize other work on the front end nodes executing slurmd so as to maximize its performance and minimize other risk factors.
Bluegene.conf File Creation
In addition to the normal slurm.conf file, a new bluegene.conf configuration file is required with information pertinent -to the sytem. +to the system. Put bluegene.conf into the SLURM configuration directory with slurm.conf. A sample file is installed in bluegene.conf.example. @@ -326,7 +350,7 @@ configuration file for static partitioning. Note that smap -Dc can be run without the SLURM daemons active to establish the initial configuration. -Note that the defined bgblocks may not overlap (except for the +Note that the bgblocks defined using smap may not overlap (except for the full-system bgblock, which is implicitly created). See the smap man page for more information.
@@ -371,7 +395,7 @@ Dynamic partitioning was developed primarily for smaller BlueGene systems, but can be used on larger systems. Dynamic partitioning may introduce fragmentation of resources. -This fragementaiton may be severe since SLURM will run a job anywhere +This fragmentation may be severe since SLURM will run a job anywhere resources are available with little thought of the future. As with overlap partitioning, use dynamic partitioning with caution! @@ -383,17 +407,17 @@Blocks can be freed or set in an error state with scontrol, (i.e. "scontrol update BlockName=RMP0 state=error"). This will end any job on the block and set the state of the block to ERROR -making it so no job will run on the block. To set it back to a useable -state set the state to free (i.e. +making it so no job will run on the block. To set it back to a usable +state, set the state to free (i.e. "scontrol update BlockName=RMP0 state=free").
Alternatively, if only part of a base partition needs to be put into an error state which isn't already in a block of the size you -need, you can set a set of ionodes into an error state with scontrol, +need, you can set a collection of IO nodes into an error state using scontrol (i.e. "scontrol update subbpname=bg000[0-3] state=error"). This will end any job on the nodes listed, create a block there, and set the state of the block to ERROR making it so no job will run on the -block. To set it back to a useable state set the state to free (i.e. +block. To set it back to a usable state set the state to free (i.e. "scontrol update BlockName=RMP0 state=free" or "scontrol update subbpname=bg000[0-3] state=free"). This is helpful to allow other jobs to run on the unaffected nodes in @@ -411,14 +435,19 @@
Note that the Numpsets values defined in bluegene.conf is used only when SLURM creates bgblocks this -determines if the system is IO rich or not. For most bluegene/L +determines if the system is IO rich or not. For most BlueGene/L systems this value is either 8 (for IO poor systems) or 64 (for IO rich -systems). -
The Images can change during job start based on input from -the user. +systems).
+ +The Images file specifications identify which images are used when +booting a bgblock and the valid images are different for each BlueGene system +type (e.g. L, P and Q). Their values can change during job allocation based on +input from the user. If you change the bgblock layout, then slurmctld and slurmd should -both be cold-started (e.g. /etc/init.d/slurm startclean). -If you wish to modify the Numpsets values +both be cold-started (without preserving any state information, +"/etc/init.d/slurm startclean").
+ +If you wish to modify the Numpsets values for existing bgblocks, either modify them manually or destroy the bgblocks and let SLURM recreate them. Note that in addition to the bgblocks defined in bluegene.conf, an @@ -429,7 +458,7 @@ A sample bluegene.conf file is shown below.
############################################################################### -# Global specifications for BlueGene system +# Global specifications for a BlueGene/L system # # BlrtsImage: BlrtsImage used for creation of all bgblocks. # LinuxImage: LinuxImage used for creation of all bgblocks. @@ -531,7 +560,7 @@ # volume = 1x1x1 = 1 BPs=[000x000] Type=TORUS # 1x1x1 = 1 midplane BPs=[001x001] Type=SMALL 32CNBlocks=4 128CNBlocks=3 # 1x1x1 = 4-Nodecard sized - # cnode blocks 3-Base + # c-node blocks 3-Base # Partition Quarter sized # c-node blocks @@ -539,8 +568,8 @@The above bluegene.conf file defines multiple bgblocks to be created in a single midplane (see the "SMALL" option). -Using this mechanism, up to 32 independent jobs each consisting of 1 - 32 cnodes can be executed +Using this mechanism, up to 32 independent jobs each consisting of +32 c-nodes can be executed simultaneously on a one-rack BlueGene system. If defining bgblocks of Type=SMALL, the SLURM partition containing them as defined in slurm.conf must have the @@ -552,9 +581,10 @@ As in all SLURM configuration files, parameters and values are case insensitive.
-With a BlueGene/P system the image names are different. The - correct image names are CnloadImage, MloaderImage, and IoloadImage. - You can also use alternate images just the same as discribed above. +
The valid image names on a BlueGene/P system are CnloadImage, MloaderImage, +and IoloadImage. The only image name on BlueGene/Q systems is MloaderImage. +Alternate images may be specified as described above for all BlueGene system +types.
One more thing is required to support SLURM interactions with the DB2 database (at least as of the time this was written). @@ -601,9 +631,9 @@ A system administrator should address the problem before returning the base partitions to service.
-If you cold-start slurmctld (/etc/init.d/slurm startclean -or slurmctld -c) it is recommended that you also cold-start -the slurmd at the same time. +
If the slurmctld daemon is cold-started (/etc/init.d/slurm startclean +or slurmctld -c) it is recommended that the slurmd daemon(s) be +cold-started at the same time. Failure to do so may result in errors being reported by both slurmd and slurmctld due to bgblocks that previously existed being deleted.
@@ -614,20 +644,32 @@Resource Reservations
-This reservation mechanism for less than an entire midplane is still -under development.
+SLURM's advance reservation mechanism can accept a node count specification +as input rather than identification of specific nodes/midplanes. In that case, +SLURM may reserve nodes/midplanes which may not be formed into an appropriate +bgblock. Work is planned for SLURM version 2.4 to remedy this problem. Until +that time, identifying the specific nodes/midplanes to be included in an +advanced reservation may be necessary.
SLURM's advance reservation mechanism is designed to reserve resources at the level of whole nodes, which on a BlueGene systems would represent whole midplanes. In order to support advanced reservations with a finer grained resolution, you can configure one license per c-node on the system -and reserve c-nodes instead of entire midplanes. For example, in slurm.conf -specify something of this sort: "Licenses=cnode*512". Then create an -advanced reservation with a command like this:
+and reserve c-nodes instead of entire midplanes. Note that reserved licenses +are treated somewhat differently than reserved nodes. When nodes are reserved +then jobs using that reservation can use only those nodes. Reserved licenses +can only be used by jobs associated with that reservation, but licenses not +explicitly reserved are available to any job. + +
-"scontrol create reservation licenses="cnode*32" starttime=now duration=30:00 users=joe".For example, in slurm.conf specify something of this sort: +"Licenses=cnode*512". Then create an advanced reservation with a +command like this:
+"scontrol create reservation licenses="cnode*32" starttime=now duration=30:00 users=joe".
+Jobs run in this reservation will then have at least 32 c-nodes +available for their use, but could use more given an appropriate workload.There is also a job_submit/cnode plugin available for use that will -automatically set a job's license specification to match his c-node request +automatically set a job's license specification to match its c-node request (i.e. a command like
"sbatch -N32 my.sh" would automatically be translated to
"sbatch -N32 --licenses=cnode*32 my.sh" by the slurmctld daemon. @@ -639,7 +681,7 @@All of the testing and debugging guidance provided in Quick Start Administrator Guide apply to BlueGene systems. -One can start the slurmctld and slurmd in the foreground +One can start the slurmctld and slurmd daemons in the foreground with extensive debugging to establish basic functionality. Once running in production, the configured SlurmctldLog and SlurmdLog files will provide historical system information. @@ -659,18 +701,22 @@ This will define "HAVE_BG", "HAVE_BGL", and "HAVE_FRONT_END" in the config.h file. You can also emulate a BlueGene/P system with - the --enable-bgp-emulation option. +the --enable-bgp-emulation option. This will define "HAVE_BG", "HAVE_BGP", and "HAVE_FRONT_END" in the config.h file. +You can also emulate a BlueGene/Q system using +the --enable-bgq-emulation option. +This will define "HAVE_BG", "HAVE_BGQ", and "HAVE_FRONT_END" in the +config.h file. Then execute make normally. These variables will build the code as if it were running on an actual BlueGene computer, but avoid making calls to the -Bridge libary (that is controlled by the variable "HAVE_BG_FILES", +Bridge library (that is controlled by the variable "HAVE_BG_FILES", which is left undefined). You can use this to test configurations, scheduling logic, etc.
-Last modified 17 March 2009
+Last modified 16 August 2011
Binary files /tmp/jKTrgHjCbo/slurm-llnl-2.2.7/doc/html/bull.jpg and /tmp/eIE_aeUSMX/slurm-llnl-2.3.2/doc/html/bull.jpg differ diff -Nru slurm-llnl-2.2.7/doc/html/checkpoint_plugins.shtml slurm-llnl-2.3.2/doc/html/checkpoint_plugins.shtml --- slurm-llnl-2.2.7/doc/html/checkpoint_plugins.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/checkpoint_plugins.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -22,7 +22,6 @@ Berkeley Lab Checkpoint/Restart (BLCR) - none—No job checkpoint.
- ompi—OpenMPI checkpoint (requires OpenMPI version 1.3 or higher). -
- xlch—XLCH
The plugin_name and diff -Nru slurm-llnl-2.2.7/doc/html/configurator.html.in slurm-llnl-2.3.2/doc/html/configurator.html.in --- slurm-llnl-2.2.7/doc/html/configurator.html.in 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/configurator.html.in 2011-12-05 17:20:08.000000000 +0000 @@ -1,11 +1,11 @@ + +
CPU Management User and Administrator Guide
+ +Overview
+The purpose of this guide is to assist SLURM users and administrators in selecting configuration options +and composing command lines to manage the use of CPU resources by jobs, steps and tasks. The document +is divided into the following sections:
+-
+
- Overview +
- CPU Management Steps performed by SLURM +
- Getting Information about CPU usage by Jobs/Steps/Tasks +
- CPU Management and SLURM Accounting +
- CPU Management Examples +
CPU Management through user commands is constrained by the configuration parameters +chosen by the SLURM administrator. The interactions between different CPU management options are complex +and often difficult to predict. Some experimentation may be required to discover the exact combination +of options needed to produce a desired outcome. Users and administrators should refer to the man pages +for slurm.conf, cgroup.conf +salloc, +sbatch and srun for detailed explanations of each +option. The following html documents may also be useful:
+ +
+Consumable Resources in SLURM
+Sharing Consumable Resources
+Support for Multi-core/Multi-thread
+Architectures
+Plane distribution
This document describes SLURM CPU management for conventional Linux clusters only. For +information on Cray and IBM BlueGene systems, please refer to the appropriate documents.
+The information and examples in this document have been verified on SLURM version 2.3.0. Some +information may not be valid for previous SLURM versions.
+ +
CPU Management Steps performed by SLURM
+SLURM uses four basic steps to manage CPU resources for a job/step:
+-
+
- Step 1: Selection of Nodes +
- Step 2: Allocation of CPUs from the selected Nodes +
- Step 3: Distribution of Tasks to the selected Nodes +
- Step 4: Optional Distribution and Binding of Tasks to CPUs within a Node + +
Step 1: Selection of Nodes
+In Step 1, SLURM selects the set of nodes from which CPU resources are to be allocated to a job or
+job step. Node selection is therefore influenced by many of the configuration and command line options
+that control the allocation of CPUs (Step 2 below).
+If
+SelectType=select/linear is configured, all resources on the selected nodes will be allocated
+to the job/step. If SelectType=select/cons_res is configured,
+individual sockets, cores and threads may be allocated from the selected nodes as
+consumable resources. The consumable resource type is defined by
+SelectTypeParameters.
+
+
+Step 1 is performed by slurmctld and the select plugin.
+
+
|
+ slurm.conf + parameter + |
+
+ Possible values + |
+
+ Description + |
+
|
+ + NodeName + |
+
+ <name of the node> |
+
+ Defines + a node. This includes the number and layout of sockets, cores, + threads and processors (logical CPUs) on the node. + |
+
| + + | +
+ <name of the partition> |
+
+ Defines + a partition. Several parameters of the partition definition + affect the selection of nodes (e.g., Nodes, + Shared, MaxNodes) + |
+
| + + | +
+ 0 | 1 | 2 + |
+
+ Controls + how the information in a node definition is used. + |
+
| + + | +
+ + select/linear | select/cons_res + |
+
+ Controls + whether CPU resources are allocated to jobs and job steps in + units of whole nodes or as consumable resources (sockets, cores + or threads). + |
+
| + + | +
+ CR_CPU | CR_CPU_Memory | CR_Core |
+CR_Core_Memory | CR_Socket | CR_Socket_Memory |
+
+ Defines + the consumable resource type and controls other aspects of CPU + resource allocation by the select plugin. + |
+
+
|
+ + Command + line option + |
+
+ + Possible values + |
+ + Description + |
+
| + + | +
+ + <sockets[:cores[:threads]]> + |
+
+ Restricts + node selection to nodes with a specified layout of sockets, cores + and threads. + |
+
| + + | +
+ <list> + + |
+
+ Restricts + node selection to nodes with specified attributes + |
+
| + + | +
+ N/A + |
+
+ Restricts + node selection to contiguous nodes + |
+
| + + | +
+ <cores> + |
+
+ Restricts + node selection to nodes with at least the specified number of cores per socket + |
+
| + + | +
+ <ncpus> + |
+
+ Controls + the number of CPUs allocated per task + |
+
| + + | +
+ N/A + |
+
+ Prevents + sharing of allocated nodes with other jobs. Suballocates CPUs to job steps. + |
+
| + + | +
+ <node file> + |
+
+ File + containing a list of specific nodes to be selected for the job (salloc and sbatch only) + |
+
|
+ + --hint + |
+
+ compute_bound | + memory_bound | [no]multithread + |
+
+ Additional + controls on allocation of CPU resources + |
+
| + + | +
+ <n> + |
+
+ Controls + the minimum number of CPUs allocated per node + |
+
| + + | +
+ + <minnodes[-maxnodes]> + |
+
+ Controls + the minimum/maximum number of nodes allocated to the job + |
+
| + + | +
+ <number> + |
+
+ Controls + the number of tasks to be created for the job + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated core + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated socket + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated node + |
+
| + + | +
+ N/A + |
+
+ Allows + fewer CPUs to be allocated than the number of tasks + |
+
| + + | +
+ + <partition_names> + |
+
+ Controls + which partition is used for the job + |
+
| + + | +
+ N/A + |
+
+ Allows + sharing of allocated nodes with other jobs + |
+
| + + | +
+ <sockets> + |
+
+ Restricts + node selection to nodes with at least the specified number of sockets + |
+
| + + | +
+ <threads> + |
+
+ Restricts + node selection to nodes with at least the specified number of threads per core + |
+
| + + | +
+ + <host1,host2,... or filename> + |
+
+ List + of specific nodes to be allocated to the job + |
+
| + + | +
+ + <host1,host2,... or filename> + |
+
+ List + of specific nodes to be excluded from allocation to the job + |
+
| + + | +
+ N/A + |
+
+ Bypass + normal allocation (privileged option available to users + “SlurmUser” and “root” only) + |
+
+ +
Step 2: Allocation of CPUs from the selected Nodes
+In Step 2, SLURM allocates CPU resources to a job/step from the set of nodes selected
+in Step 1. CPU allocation is therefore influenced by the configuration and command line options
+that relate to node selection.
+If
+SelectType=select/linear is configured, all resources on the selected nodes will be allocated
+to the job/step. If SelectType=select/cons_res is configured,
+individual sockets, cores and threads may be allocated from the selected nodes as
+consumable resources. The consumable resource type is defined by
+SelectTypeParameters.
+
+
When using SelectType=select/cons_res,
+the default allocation method across nodes is block allocation (allocate all available CPUs in
+a node before using another node). The default allocation method within a node is cyclic
+allocation (allocate available CPUs in a round-robin fashion across the sockets within a node).
+Users may override the default behavior using the appropriate command
+line options described below. The choice of allocation methods may influence which specific
+CPUs are allocated to the job/step.
+
+Step 2 is performed by slurmctld and the select plugin.
+
+
|
+ slurm.conf + parameter + |
+
+ Possible values + |
+
+ Description + |
+
|
+ + NodeName + |
+
+ <name of the node> |
+
+ Defines + a node. This includes the number and layout of sockets, cores, + threads and processors (logical CPUs) on the node. + |
+
| + + | +
+ <name of the partition> |
+
+ Defines + a partition. Several parameters of the partition definition + affect the allocation of CPU resources to jobs (e.g., Nodes, + Shared, MaxNodes) + |
+
| + + | +
+ 0 | 1 | 2 + |
+
+ Controls + how the information in a node definition is used. + |
+
| + + | +
+ + select/linear | select/cons_res + |
+
+ Controls + whether CPU resources are allocated to jobs and job steps in + units of whole nodes or as consumable resources (sockets, cores + or threads). + |
+
| + + | +
+ CR_CPU | CR_CPU_Memory | CR_Core |
+CR_Core_Memory | CR_Socket | CR_Socket_Memory |
+
+ Defines + the consumable resource type and controls other aspects of CPU + resource allocation by the select plugin. + |
+
+
|
+ + Command + line option + |
+
+ + Possible values + |
+ + Description + |
+
| + + | +
+ + <sockets[:cores[:threads]]> + |
+
+ Restricts + node selection to nodes with a specified layout of sockets, cores + and threads. + |
+
| + + | +
+ <list> + + |
+
+ Restricts + node selection to nodes with specified attributes + |
+
| + + | +
+ N/A + |
+
+ Restricts + node selection to contiguous nodes + |
+
| + + | +
+ <cores> + |
+
+ Restricts + node selection to nodes with at least the specified number of cores per socket + |
+
| + + | +
+ <ncpus> + |
+
+ Controls + the number of CPUs allocated per task + |
+
| + + | +
+
+ block|cyclic |
+
+ The second specified distribution (after the ":") + can be used to override the default allocation method within nodes + |
+
| + + | +
+ N/A + |
+
+ Prevents + sharing of allocated nodes with other jobs + |
+
| + + | +
+ <node file> + |
+
+ File + containing a list of specific nodes to be selected for the job (salloc and sbatch only) + |
+
|
+ + --hint + |
+
+ compute_bound | + memory_bound | [no]multithread + |
+
+ Additional + controls on allocation of CPU resources + |
+
| + + | +
+ <n> + |
+
+ Controls + the minimum number of CPUs allocated per node + |
+
| + + | +
+ + <minnodes[-maxnodes]> + |
+
+ Controls + the minimum/maximum number of nodes allocated to the job + |
+
| + + | +
+ <number> + |
+
+ Controls + the number of tasks to be created for the job + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated core + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated socket + |
+
| + + | +
+ <number> + |
+
+ Controls + the maximum number of tasks per allocated node + |
+
| + + | +
+ N/A + |
+
+ Allows + fewer CPUs to be allocated than the number of tasks + |
+
| + + | +
+ + <partition_names> + |
+
+ Controls + which partition is used for the job + |
+
| + + | +
+ N/A + |
+
+ Allows + sharing of allocated nodes with other jobs + |
+
| + + | +
+ <sockets> + |
+
+ Restricts + node selection to nodes with at least the specified number of sockets + |
+
| + + | +
+ <threads> + |
+
+ Restricts + node selection to nodes with at least the specified number of threads per core + |
+
| + + | +
+ + <host1,host2,... or filename> + |
+
+ List + of specific nodes to be allocated to the job + |
+
| + + | +
+ + <host1,host2,... or filename> + |
+
+ List + of specific nodes to be excluded from allocation to the job + |
+
| + + | +
+ N/A + |
+
+ Bypass + normal allocation (privileged option available to users + “SlurmUser” and “root” only) + |
+
+ +
Step 3: Distribution of Tasks to the selected Nodes
+In Step 3, SLURM distributes tasks to the nodes that were selected for
+the job/step in Step 1. Each task is distributed to only one node, but more than one
+task may be distributed to each node. Unless overcommitment of CPUs to tasks is
+specified for the job, the number of tasks distributed to a node is
+constrained by the number of CPUs allocated on the node and the number of CPUs per
+task. If consumable resources is configured, or resource sharing is allowed, tasks from
+more than one job/step may run on the same node concurrently.
+
+Step 3 is performed by slurmctld.
+
+
|
+ slurm.conf + parameter + |
+
+ Possible values + |
+
+ Description + |
+
| + + | +
+ <number> + |
+
+ + Controls the maximum number of tasks that a job step can spawn on a single node + + |
+
+
|
+ + Command + line option + |
+
+ + Possible values + |
+ + Description + |
+
| + + | +
+
+ block|cyclic |
+
+ The first specified distribution (before the ":") + controls the sequence in which tasks are distributed to each of the selected nodes. Note that + this option does not affect the number of tasks distributed to each node, but only the sequence of + distribution. + |
+
| + + | +
+ <number> + + |
+
+ + Controls the maximum number of tasks per allocated core + |
+
| + + | +
+ <number> + |
+
+ + Controls the maximum number of tasks per allocated socket + |
+
| + + | +
+ <number> + |
+
+ + Controls the maximum number of tasks per allocated node + |
+
| + + | +
+ N/A + |
+
+ Controls + which node is used for a job step + |
+
+ +
Step 4: Optional Distribution and Binding of Tasks to CPUs within a Node
+In optional Step 4, SLURM distributes and binds each task to a specified subset of
+the allocated CPUs on the node to which the task was distributed in Step 3. Different
+tasks distributed to the same node may be bound to the same subset of CPUs or to
+different subsets. This step is known as task affinity or task/CPU binding.
+
+Step 4 is performed by slurmd and the task plugin.
+
+
|
+ slurm.conf + parameter + |
+
+ Possible values + |
+
+ Description + |
+
| + + | +
+ + task/none | task/affinity | task/cgroup + |
+
+ + Controls whether this step is enabled and which task plugin to use + + |
+
| + + | +
+ See man page + |
+
+ + For task/affinity, controls the binding unit (sockets, cores or threads) and the + binding method (sched or cpusets) + |
+
|
+ cgroup.conf + parameter + |
+
+ Possible values + |
+
+ Description + |
+
| + + | +
+ + yes|no + |
+
+ + Controls whether jobs are constrained to their allocated CPUs + + |
+
| + + | +
+ + yes|no + |
+
+ + Controls whether task-to-CPU binding is enabled + |
+
+
|
+ + Command + line option + |
+
+ + Possible values + |
+ + Description + |
+
| + + | +
+ + See man page + |
+
+ Controls binding of tasks to CPUs + |
+
| + + | +
+ <number> + + |
+
+ + Controls the maximum number of tasks per allocated core + |
+
| + + | +
+
+ block|cyclic |
+
+ + The second specified distribution (after the ":") controls the sequence in which tasks are + distributed to allocated CPUs within a node for binding of tasks to CPUs + |
+
+
Additional Notes on CPU Management Steps
+For consumable resources, it is important for users to understand the difference between +cpu allocation (Step 2) and task affinity/binding (Step 4). Exclusive (unshared) allocation +of CPUs as consumable resources limits the number of jobs/steps/tasks that +can use a node concurrently. But it does not limit the set of CPUs on the node that each +task distributed to the node can use. Unless some form of CPU/task binding is used +(e.g., a task or spank plugin), all tasks distributed to a node can use all of +the CPUs on the node, including CPUs not allocated to their job/step. This may have +unexpected adverse effects on performance, since it allows one job to use CPUs allocated +exclusively to another job. For this reason, it may not be advisable to configure +consumable resources without also configuring task affinity. Note that task affinity +can also be useful when select/linear (whole node allocation) is configured, to improve +performance by restricting each task to a particular socket or other subset of CPU +resources on a node.
++ +
Getting Information about CPU usage by Jobs/Steps/Tasks
+There is no easy way to generate a comprehensive set of CPU management information +for a job/step (allocation, distribution and binding). However, several +commands/options provide limited information about CPU usage. +
+|
+ Command/Option + |
+
+ Information + |
+
|
+ +scontrol show job option: +--details + |
+
+ +This option provides a list of the nodes selected for the job and the CPU ids allocated to the job on each +node. Note that the CPU ids reported by this command are SLURM abstract CPU ids, not Linux/hardware CPU ids +(as reported by, for example, /proc/cpuinfo). + + |
+
|
+ +Linux command: env + |
+
+
+Many SLURM environment variables provide information related to node and CPU usage:
+ |
+
|
+ +srun/salloc/sbatch option: +--cpu_bind=verbose + |
+
+ +This option provides a list of the CPU masks used by task affinity to bind tasks to CPUs. +Note that the CPU ids represented by these masks are Linux/hardware CPU ids, not SLURM +abstract CPU ids as reported by scontrol, etc. + + |
+
|
+ +srun/salloc/sbatch option: +-l + |
+
+ +This option adds the task id as a prefix to each line of output from a task sent to stdout/stderr. +This can be useful for distinguishing node-related and CPU-related information by task id +for multi-task jobs/steps. + + |
+
|
+
+Linux command: |
+
+ +Given a task's pid (or "self" if the command is executed by the task itself), this command +produces a list of the CPU ids bound to the task. This is the same information that is +provided by --cpu_bind=verbose, but in a more readable format. + + |
+
A Note on CPU Numbering
+The number and layout of logical CPUs known to SLURM is described in the node definitions in slurm.conf. This may +differ from the physical CPU layout on the actual hardware. For this reason, SLURM generates its own internal, or +"abstract", CPU numbers. These numbers may not match the physical, or "machine", CPU numbers known to Linux. +A utility to convert between SLURM abstract CPU numbers and physical machine CPU numbers is provided by SLURM. See +module src/slurmd/slurmd/get_mach_stat.c for details.
++ +
CPU Management and SLURM Accounting
+CPU management by SLURM users is subject to limits imposed by SLURM Accounting. Accounting limits may be applied on CPU +usage at the level of users, groups and clusters. For details, see the sacctmgr man page.
++ +
CPU Management Examples
+The following examples illustrate some scenarios for managing CPU +resources using SLURM. Many additional scenarios are possible. In +each example, it is assumed that all CPUs on each node are available +for allocation.
+-
+
- Example Node and Partition Configuration
+ - Example 1: Allocation of whole nodes
+ - Example 2: Simple allocation of cores as consumable resources +
- Example 3: Consumable resources with balanced allocation across nodes +
- Example 4: Consumable resources with minimization of resource fragmentation +
- Example 5: Consumable resources with cyclic distribution of tasks to nodes +
- Example 6: Consumable resources with default allocation and plane distribution of tasks to nodes +
- Example 7: Consumable resources with overcommitment of CPUs to tasks +
- Example 8: Consumable resources with resource sharing between jobs +
- Example 9: Consumable resources on multithreaded node, allocating only one thread per core +
- Example 10: Consumable resources with task affinity and core binding +
- Example 11: Consumable resources with task affinity and socket binding, Case 1 +
- Example 12: Consumable resources with task affinity and socket binding, Case 2 +
- Example 13: Consumable resources with task affinity and socket binding, Case 3 +
- Example 14: Consumable resources with task affinity and customized allocation and distribution +
- Example 15: Consumable resources with task affinity to optimize the performance of a multi-task, +multi-thread job +
- Example 16: Consumable resources with task cgroup and core binding +
+ +
Example Node and Partition Configuration
+For these examples, the SLURM cluster contains the following nodes:
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
+ n3 + |
+
|
+ Number + of Sockets + |
+
+ 2 + |
+
+ 2 + |
+
+ 2 + |
+
+ 2 + |
+
|
+ Number + of Cores per Socket + |
+
+ 4 + |
+
+ 4 + |
+
+ 4 + |
+
+ 4 + |
+
|
+ Total + Number of Cores + |
+
+ 8 + |
+
+ 8 + |
+
+ 8 + |
+
+ 8 + |
+
|
+ Number + of Threads (CPUs) per Core + |
+
+ 1 + |
+
+ 1 + |
+
+ 1 + |
+
+ 2 + |
+
|
+ Total + Number of CPUs + |
+
+ 8 + |
+
+ 8 + |
+
+ 8 + |
+
+ 16 + |
+
And the following partitions:
+|
+ PartitionName + |
+
+ regnodes + |
+
+ hypernode + |
+
|
+ Nodes + |
+
+ n0 + n1 n2 + |
+
+ n3 + |
+
|
+ Default + |
+
+ YES + |
+
+ - + |
+
These entities are defined in slurm.conf as follows:
+Nodename=n0 NodeAddr=node0 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 Procs=8 +Nodename=n1 NodeAddr=node1 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 Procs=8 State=IDLE +Nodename=n2 NodeAddr=node2 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 Procs=8 State=IDLE +Nodename=n3 NodeAddr=node3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=2 Procs=16 State=IDLE +PartitionName=regnodes Nodes=n0,n1,n2 Shared=YES Default=YES State=UP +PartitionName=hypernode Nodes=n3 State=UP ++
+ +
Example 1: Allocation of whole nodes
+Allocate a minimum of two whole nodes to a job.
+slurm.conf options:
+SelectType=select/linear ++
Command line:
+srun --nodes=2 ... ++
Comments:
+The SelectType=select/linear +configuration option specifies allocation in units of whole nodes. +The --nodes=2 srun option causes +SLURM to allocate at least 2 nodes to the job. ++ +
Example 2: Simple allocation of cores as consumable resources
+A job requires 6 CPUs (2 tasks and 3 CPUs per task with no overcommitment). Allocate the 6 CPUs as consumable resources +from a single node in the default partition.
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=1-1 --ntasks=2 --cpus-per-task=3 ... ++
Comments:
+The SelectType configuration options define cores as consumable resources. +The --nodes=1-1 srun option + restricts the job to a single node. The following table shows a possible pattern of allocation + for this job. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
|
+ Number + of Allocated CPUs + |
+
+ 6 + |
+
+ 0 + |
+
+ 0 + |
+
|
+ Number + of Tasks + |
+
+ 2 + |
+
+ 0 + |
+
+ 0 + |
+
+ +
Example 3: Consumable resources with balanced allocation across nodes
+A job requires 9 CPUs (3 tasks and 3 CPUs per task with no overcommitment). +Allocate 3 CPUs from each of the 3 nodes in the default partition. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=3-3 --ntasks=3 --cpus-per-task=3 ... ++
Comments:
+The options specify the following conditions for the job: 3 tasks, 3 unique CPUs + per task, using exactly 3 nodes. To satisfy these conditions, SLURM must + allocate 3 CPUs from each node. The following table shows the allocation + for this job. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
|
+ Number + of Allocated CPUs + |
+
+ 3 + |
+
+ 3 + |
+
+ 3 + |
+
|
+ Number + of Tasks + |
+
+ 1 + |
+
+ 1 + |
+
+ 1 + |
+
+ +
Example 4: Consumable resources with minimization of resource fragmentation
+A job requires 12 CPUs (12 tasks and 1 CPU per task with no overcommitment). Allocate +CPUs using the minimum number of nodes and the minimum number of sockets required for +the job in order to minimize fragmentation of allocated/unallocated CPUs in the cluster. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core,CR_CORE_DEFAULT_DIST_BLOCK ++
Command line:
+srun --ntasks=12 ... ++
Comments:
+The default allocation method across nodes is block. This minimizes the number of nodes + used for the job. The configuration option + CR_CORE_DEFAULT_DIST_BLOCK sets the default allocation method within a + node to block. This minimizes the number of sockets used for the job within a node. + The combination of these two methods causes SLURM to allocate the 12 CPUs using the + minimum required number of nodes (2 nodes) and sockets (3 sockets).The following + table shows a possible pattern of allocation for this job. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+ |||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 4 + |
+
+ 4 + |
+
+ 0 + |
+
+ 0 + |
+
+ 0 + |
+
|
+ Number of + Tasks + |
+
+ 8 + |
+
+ 4 + |
+
+ 0 + |
+ |||
+ +
Example 5: Consumable resources with cyclic distribution of tasks to nodes
+A job requires 12 CPUs (6 tasks and 2 CPUs per task with no overcommitment). Allocate +6 CPUs from each of 2 nodes in the default partition. Distribute tasks to nodes cyclically. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=2-2 --ntasks-per-node=3 --distribution=cyclic+
+--ntasks=6 --cpus-per-task=2 ... +
Comments:
+The options specify the following conditions for the job: 6 tasks, 2 unique CPUs per task, +using exactly 2 nodes, and with 3 tasks per node. To satisfy these conditions, SLURM +must allocate 6 CPUs from each of 2 nodes. The +--distribution=cyclic option causes the tasks to be distributed to the nodes in a +round-robin fashion. The following table shows a possible pattern of allocation and +distribution for this job. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
|
+ Number of + Allocated CPUs + |
+
+ 6 + |
+
+ 6 + |
+
+ 0 + |
+
|
+ Number of + Tasks + |
+
+ 3 + |
+
+ 3 + |
+
+ 0 + |
+
|
+ Distribution + of Tasks to Nodes, by Task id + |
+
+
+ 0 |
+
+
+ 1 |
+
+ - + |
+
+ +
Example 6: Consumable resources with default allocation and plane distribution of tasks to nodes
+A job requires 16 CPUs (8 tasks and 2 CPUs per task with no overcommitment). +Use all 3 nodes in the default partition. Distribute tasks to each node in blocks of two in a round-robin fashion. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=3-3 --distribution=plane=2 --ntasks=8 --cpus-per-task=2 ... ++
Comments:
+The options specify the following conditions for the job: 8 tasks, 2 unique CPUs +per task, using all 3 nodes in the partition. To satisfy these conditions using +the default allocation method across nodes (block), SLURM allocates 8 CPUs from +the first node, 6 CPUs from the second node and 2 CPUs from the third node. +The --distribution=plane=2 option causes SLURM +to distribute tasks in blocks of two to each of the nodes in a round-robin fashion, +subject to the number of CPUs allocated on each node. So, for example, only 1 task +is distributed to the third node because only 2 CPUs were allocated on that node and +each task requires 2 CPUs. The following table shows a possible pattern of allocation +and distribution for this job. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
|
+ Number of + Allocated CPUs + |
+
+ 8 + |
+
+ 6 + |
+
+ 2 + |
+
|
+ Number of + Tasks + |
+
+ 4 + |
+
+ 3 + |
+
+ 1 + |
+
|
+ Distribution + of Tasks to Nodes, by Task id + |
+
+ 0
+ 1 |
+
+ 2
+ 3 |
+
+ 4 |
+
+ +
Example 7: Consumable resources with overcommitment of CPUs to tasks
+A job has 20 tasks. Run the job in a single node. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=1-1 --ntasks=20 --overcommit ... ++
Comments:
+The +--overcommit option allows the job to +run in only one node by overcommitting CPUs to tasks.The following table shows + a possible pattern of allocation and distribution for this job. +
+ +|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+
|
+ Number of + Allocated CPUs + |
+
+ 8 + |
+
+ 0 + |
+
+ 0 + |
+
|
+ Number of + Tasks + |
+
+ 20 + |
+
+ 0 + |
+
+ 0 + |
+
|
+ Distribution + of Tasks to Nodes, by Task id + |
+
+ 0 + - 19 + |
+
+ - + |
+
+ - + |
+
+ +
Example 8: Consumable resources with resource sharing between jobs
+2 jobs each require 6 CPUs (6 tasks per job with no overcommitment). +Run both jobs simultaneously in a single node. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core ++
Command line:
+srun --nodes=1-1 --nodelist=n0 --ntasks=6 --share ... +srun --nodes=1-1 --nodelist=n0 --ntasks=6 --share ... ++
Comments:
+The --nodes=1-1 and -w n0 +srun options together restrict both jobs to node n0. The +Shared=YES option in the partition definition plus +the --share srun option allows the two +jobs to share CPUs on the node. +
+ +
Example 9: Consumable resources on multithreaded node, allocating only one thread per core
+A job requires 8 CPUs (8 tasks with no overcommitment). Run the job on node n3, +allocating only one thread per core. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_CPU ++
Command line:
+srun --partition=hypernode --ntasks=8 --hint=nomultithread ... ++
Comments:
+The CR_CPU configuration +option enables the allocation of only one thread per core. +The --hint=nomultithread +srun option causes SLURM to allocate only one thread from each core to +this job. The following table shows a possible pattern of allocation +for this job. +
+|
+ Nodename + |
+
+ n3 + |
+ |||||||||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ ||||||||||||||
|
+ Core id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+ ||||||||
|
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
+ 8 + |
+
+ 9 + |
+
+ 10 + |
+
+ 11 + |
+
+ 12 + |
+
+ 13 + |
+
+ 14 + |
+
+ 15 + |
+
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 4 + |
+ ||||||||||||||
|
+ Allocated + CPU ids + |
+
+ 0 + 2 4 6 + |
+
+ 8 + 10 12 14 + |
+ ||||||||||||||
+ +
Example 10: Consumable resources with task affinity and core binding
+A job requires 6 CPUs (6 tasks with no overcommitment). Run the job in a +single node in the default partition. Apply core binding to each task. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --nodes=1-1 --ntasks=6 --cpu_bind=cores ... ++
Comments:
+Using the default allocation method within nodes (cyclic), SLURM allocates +3 CPUs on each socket of 1 node. Using the default distribution method +within nodes (cyclic), SLURM distributes and binds each task to an allocated +core in a round-robin fashion across the sockets. The following table shows +a possible pattern of allocation, distribution and binding for this job. +For example, task id 2 is bound to CPU id 1. +
+
|
+ Nodename + |
+
+ n0 + |
+ ||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ |||||||
|
+ Number of + Allocated CPUs + |
+
+ 3 + |
+
+ 3 + |
+ |||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 + |
+
+ 4 5 6 + |
+ |||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task id + |
+
+ 0 + |
+
+ 2 + |
+
+ 4 + |
+
+ - + |
+
+ 1 + |
+
+ 3 + |
+
+ 5 + |
+
+ - + |
+ |
+ +
Example 11: Consumable resources with task affinity and socket binding, Case 1
+A job requires 6 CPUs (6 tasks with no overcommitment). Run the job in +a single node in the default partition. Apply socket binding to each task. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --nodes=1-1 --ntasks=6 --cpu_bind=sockets ... ++
Comments:
+Using the default allocation method within nodes (cyclic), SLURM allocates 3 +CPUs on each socket of 1 node. Using the default distribution method within nodes +(cyclic), SLURM distributes and binds each task to all of the allocated CPUs in +one socket in a round-robin fashion across the sockets. The following table shows +a possible pattern of allocation, distribution and binding for this job. For +example, task ids 1, 3 and 5 are all bound to CPU ids 4, 5 and 6. +
+|
+ Nodename + |
+
+ n0 + |
+ ||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ |||||||
|
+ Number of + Allocated CPUs + |
+
+ 3 + |
+
+ 3 + |
+ |||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 + |
+
+ 4 5 6 + |
+ |||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task ids + |
+
+ 0 + 2 4 + |
+
+ - + |
+
+ 1 + 3 5 + |
+
+ - + |
+ |||||
+ +
Example 12: Consumable resources with task affinity and socket binding, Case 2
+A job requires 6 CPUs (2 tasks with 3 cpus per task and no overcommitment). Run the job in +a single node in the default partition. Allocate cores using the block allocation method. +Distribute cores using the block distribution method. Apply socket binding to each task. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core,CR_CORE_DEFAULT_DIST_BLOCK +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --nodes=1-1 --ntasks=2 --cpus-per-task=3 --cpu_bind=sockets+
+--distribution=block:block ... +
Comments:
+Using the block allocation method, SLURM allocates 4 +CPUs on one socket and 2 CPUs on the other socket of one node. Using the block distribution method within +nodes, SLURM distributes 3 CPUs to each task. Applying socket binding, SLURM binds each task to all +allocated CPUs in all sockets in which the task has a distributed CPU. The following table shows +a possible pattern of allocation, distribution and binding for this job. In this example, using the +block allocation method CPU ids 0-3 are allocated on socket id 0 and CPU ids 4-5 are allocated on +socket id 1. Using the block distribution method, CPU ids 0-2 were distributed to task id 0, and CPU ids +3-5 were distributed to task id 1. Applying socket binding, task id 0 is therefore bound to the allocated +CPUs on socket 0, and task id 1 is bound to the allocated CPUs on both sockets. +
+|
+ Nodename + |
+
+ n0 + |
+ ||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ |||||||
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 2 + |
+ |||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 3 + |
+
+ 4 5 + |
+ |||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task ids + |
+
+ 0 1 + |
+
+ 1 + |
+
+ - + |
+ ||||||
+ +
Example 13: Consumable resources with task affinity and socket binding, Case 3
+A job requires 6 CPUs (2 tasks with 3 cpus per task and no overcommitment). Run the job in +a single node in the default partition. Allocate cores using the block allocation method. +Distribute cores using the cyclic distribution method. Apply socket binding to each task. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core,CR_CORE_DEFAULT_DIST_BLOCK +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --nodes=1-1 --ntasks=2 --cpus-per-task=3 --cpu_bind=sockets+
+--distribution=block:cyclic ... +
Comments:
+Using the block allocation method, SLURM allocates 4 +CPUs on one socket and 2 CPUs on the other socket of one node. Using the cyclic distribution method within +nodes, SLURM distributes 3 CPUs to each task. Applying socket binding, SLURM binds each task to all +allocated CPUs in all sockets in which the task has a distributed CPU. The following table shows +a possible pattern of allocation, distribution and binding for this job. In this example, using the +block allocation method CPU ids 0-3 are allocated on socket id 0 and CPU ids 4-5 are allocated on +socket id 1. Using the cyclic distribution method, CPU ids 0, 1 and 4 were distributed to task id 0, and CPU ids +2, 3 and 5 were distributed to task id 1. Applying socket binding, both tasks are therefore bound to the +allocated CPUs on both sockets. +
+|
+ Nodename + |
+
+ n0 + |
+ ||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ |||||||
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 2 + |
+ |||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 3 + |
+
+ 4 5 + |
+ |||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task ids + |
+
+ 0 1 + |
+
+ 0 1 + |
+
+ - + |
+ ||||||
+ +
Example 14: Consumable resources with task affinity and customized allocation and distribution
+A job requires 18 CPUs (18 tasks with no overcommitment). Run the job in the +default partition. Allocate 6 CPUs on each node using block allocation within +nodes. Use cyclic distribution of tasks to nodes and block distribution of +tasks for CPU binding. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core,CR_CORE_DEFAULT_DIST_BLOCK +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --nodes=3-3 --ntasks=18 --ntasks-per-node=6+
+--distribution=cyclic:block --cpu_bind=cores ... +
Comments:
+This example shows the use of task affinity with customized allocation of CPUs and +distribution of tasks across nodes and within nodes for binding. The srun options +specify the following conditions for the job: 18 tasks, 1 unique CPU per task, using +all 3 nodes in the partition, with 6 tasks per node. +The CR_CORE_DEFAULT_DIST_BLOCK +configuration option specifies block allocation within nodes. To satisfy these +conditions, SLURM allocates 6 CPUs on each node, with 4 CPUs allocated on one socket +and 2 CPUs on the other socket. The +--distribution=cyclic:block option specifies cyclic distribution of +tasks to nodes and block distribution of tasks to CPUs within nodes for binding. +The following table shows a possible pattern of allocation, distribution and binding +for this job. For example, task id 10 is bound to CPU id 3 on node n1. +
+|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+ ||||||||||||||||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+ |||||||||||||||||||
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 2 + |
+
+ 4 + |
+
+ 2 + |
+
+ 4 + |
+
+ 2 + |
+ |||||||||||||||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 3 4 5 + |
+
+ 0 1 2 3 4 5 + |
+
+ 0 1 2 3 4 5 + |
+ ||||||||||||||||||||||
|
+ Number of + Tasks + |
+
+ 6 + |
+
+ 6 + |
+
+ 6 + |
+ ||||||||||||||||||||||
|
+ Distribution + of Tasks to Nodes, by Task id + |
+
+ 0 |
+
+ 1 |
+
+ 2 |
+ ||||||||||||||||||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task id + |
+
+ 0 + |
+
+ 3 + |
+
+ 6 + |
+
+ 9 + |
+
+ 12 + |
+
+ 15 + |
+
+ - + |
+
+ - + |
+
+ 1 + |
+
+ 4 + |
+
+ 7 + |
+
+ 10 + |
+
+ 13 + |
+
+ 16 + |
+
+ - + |
+
+ - + |
+
+ 2 + |
+
+ 5 + |
+
+ 8 + |
+
+ 11 + |
+
+ 14 + |
+
+ 17 + |
+
+ - + |
+
+ - + |
+ |
+ +
Example 15: Consumable resources with task affinity to optimize the performance of a multi-task, +multi-thread job
+A job requires 9 CPUs (3 tasks and 3 CPUs per task with no overcommitment). Run +the job in the default partition, managing the CPUs to optimize the performance +of the job.
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core,CR_CORE_DEFAULT_DIST_BLOCK +TaskPlugin=task/affinity +TaskPluginParam=sched ++
Command line:
+srun --ntasks=3 --cpus-per-task=3 --ntasks-per-node=1 --cpu_bind=cores ... ++
Comments:
+To optimize the performance of this job, the user wishes to allocate 3 CPUs from each of +3 sockets and bind each task to the 3 CPUs in a single socket. The +SelectTypeParameters configuration option specifies +a consumable resource type of cores and block allocation within nodes. The +TaskPlugin +and TaskPluginParam +configuration options enable task affinity. The srun options specify the following +conditions for the job: 3 tasks, with 3 unique CPUs per task, with 1 task per node. To satisfy +these conditions, SLURM allocates 3 CPUs from one socket in each of the 3 nodes in the default partition. The +--cpu_bind=cores option causes SLURM to bind +each task to the 3 allocated CPUs on the node to which it is distributed. The +following table shows a possible pattern of allocation, distribution and binding +for this job. For example, task id 2 is bound to CPU ids 0, 1 and 2 on socket id 0 of node n2. +
+ +|
+ Nodename + |
+
+ n0 + |
+
+ n1 + |
+
+ n2 + |
+ ||||||||||||||||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+
+ 0 + |
+
+ 1 + |
+ |||||||||||||||||||
|
+ Number of + Allocated CPUs + |
+
+ 3 + |
+
+ 0 + |
+
+ 3 + |
+
+ 0 + |
+
+ 3 + |
+
+ 0 + |
+ |||||||||||||||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 + |
+
+ 0 1 2 + |
+
+ 0 1 2 + |
+ ||||||||||||||||||||||
|
+ Number of + Tasks + |
+
+ 1 + |
+
+ 1 + |
+
+ 1 + |
+ ||||||||||||||||||||||
|
+ Distribution + of Tasks to Nodes, by Task id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+ ||||||||||||||||||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task id + |
+
+ 0 + |
+
+ - + |
+
+ 1 + |
+
+ - + |
+
+ 2 + |
+
+ -- + |
+ |||||||||||||||||||
+ +
Example 16: Consumable resources with task cgroup and core binding
+A job requires 6 CPUs (6 tasks with no overcommitment). Run the job in a +single node in the default partition. Apply core binding to each task using the task/cgroup plugin. +
+slurm.conf options:
+SelectType=select/cons_res +SelectTypeParameters=CR_Core +TaskPlugin=task/cgroup ++
cgroup.conf options:
+ConstrainCores=yes +TaskAffinity=yes ++
Command line:
+srun --nodes=1-1 --ntasks=6 --cpu_bind=cores ... ++
Comments:
+The task/cgroup plugin currently supports only the block method for +allocating cores within nodes and distributing tasks to CPUs for binding. +The following table shows a possible pattern of allocation, distribution +and binding for this job. For example, task id 2 is bound to CPU id 2. +
+
|
+ Nodename + |
+
+ n0 + |
+ ||||||||
|
+ Socket id + |
+
+ 0 + |
+
+ 1 + |
+ |||||||
|
+ Number of + Allocated CPUs + |
+
+ 4 + |
+
+ 2 + |
+ |||||||
|
+ Allocated + CPU ids + |
+
+ 0 1 2 3 + |
+
+ 4 5 + |
+ |||||||
|
+ Binding of + Tasks to CPUs + |
+
+ CPU id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ 6 + |
+
+ 7 + |
+
|
+ Task id + |
+
+ 0 + |
+
+ 1 + |
+
+ 2 + |
+
+ 3 + |
+
+ 4 + |
+
+ 5 + |
+
+ - + |
+
+ - + |
+ |
Last modified 26 September 2011
+ + diff -Nru slurm-llnl-2.2.7/doc/html/cray.shtml slurm-llnl-2.3.2/doc/html/cray.shtml --- slurm-llnl-2.2.7/doc/html/cray.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/cray.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -2,144 +2,637 @@SLURM User and Administrator Guide for Cray systems
-NOTE: As of January 2009, the SLURM interface to Cray systems is incomplete. -User Guide
-This document describes the unique features of SLURM on -Cray computers. +
This document describes the unique features of SLURM on Cray computers. You should be familiar with the SLURM's mode of operation on Linux clusters -before studying the relatively few differences in Cray system -operation described in this document.
+before studying the differences in Cray system operation described in this +document. + +SLURM version 2.3 is designed to operate as a job scheduler over Cray's +Application Level Placement Scheduler (ALPS). +Use SLURM's sbatch or salloc commands to create a resource +allocation in ALPS. +Then use ALPS' aprun command to launch parallel jobs within the resource +allocation. +The resource allocation is terminated once the the batch script or the +salloc command terminates. +Alternately there is an aprun wrapper distributed with SLURM in +contribs/cray/srun which will translate srun options +into the equivalent aprun options. This wrapper will also execute +salloc as needed to create a job allocation in which to run the +aprun command. The srun script contains two new options: +--man will print a summary of the options including notes about which +srun options are not supported and --alps=" which can be used +to specify aprun options which lack an equivalent within srun. +For example, srun --alps="-a xt" -n 4 a.out. +Since aprun is used to launch tasks (the equivalent of a SLURM +job step), the job steps will not be visible using SLURM commands. +Other than SLURM's srun command being replaced by aprun +and the job steps not being visible, all other SLURM commands will operate +as expected. Note that in order to build and install the aprun wrapper +described above, execute "configure" with the --with-srun2aprun +option or add %_with_srun2aprun 1 to your ~/.rpmmacros file.
+ +Node naming and node geometry on Cray XT/XE systems
+SLURM node names will be of the form "nid#####" where "#####" is a five-digit sequence number. + Other information available about the node are it's XYZ coordinate in the node's NodeAddr + field and it's component label in the HostNodeName field. + The format of the component label is "c#-#c#s#n#" where the "#" fields represent in order: + cabinet, row, cage, blade or slot, and node. + For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.
+ +Cray XT/XE systems come with a 3D torus by default. On smaller systems the cabling in X dimension is + omitted, resulting in a two-dimensional torus (1 x Y x Z). On Gemini/XE systems, pairs of adjacent nodes + (nodes 0/1 and 2/3 on each blade) share one network interface each. This causes the same Y coordinate to + be assigned to those nodes, so that the number of distinct torus coordinates is half the number of total + nodes.
+The SLURM smap and sview tools can visualize node torus positions. Clicking on a particular + node shows its NodeAddr field, which is its (X,Y,Z) torus coordinate base-36 encoded as a 3-character + string. For example, a NodeAddr of '07A' corresponds to the coordinates X = 0, Y = 7, Z = 10. + The NodeAddr of a node can also be shown using 'scontrol show node nid#####'.
+ +Please note that the sbatch/salloc options "--geometry" and "--no-rotate" are BlueGene-specific + and have no impact on Cray systems. Topological node placement depends on what Cray makes available via the + ALPS_NIDORDER configuration option (see below).
+ +Specifying thread depth
+For threaded applications, use the --cpus-per-task/-c parameter of sbatch/salloc to set + the thread depth per node. This corresponds to mppdepth in PBS and to the aprun -d parameter. Please + note that SLURM does not set the OMP_NUM_THREADS environment variable. Hence, if an application spawns + 4 threads, an example script would look like
++ #SBATCH --comment="illustrate the use of thread depth and OMP_NUM_THREADS" + #SBATCH --ntasks=3 + #SBATCH -c 4 + export OMP_NUM_THREADS=4 + aprun -n 3 -d $OMP_NUM_THREADS ./my_exe ++ +
Specifying number of tasks per node
+SLURM uses the same default as ALPS, assigning each task to a single core/CPU. In order to + make more resources available per task, you can reduce the number of processing elements + per node (aprun -N parameter, mppnppn in PBS) with the + --ntasks-per-node option of sbatch/salloc. + This is in particular necessary when tasks require more memory than the per-CPU default.
+ +Specifying per-task memory
+In Cray terminology, a task is also called a "processing element" (PE), hence below we + refer to the per-task memory and "per-PE" memory interchangeably. The per-PE memory + requested through the batch system corresponds to the aprun -m parameter.
+ +Due to the implicit default assumption that 1 task runs per core/CPU, the default memory + available per task is the per-CPU share of node_memory / number_of_cores. For + example, on a XT5 system with 16000MB per 12-core node, the per-CPU share is 1333MB.
+ +If nothing else is specified, the --mem option to sbatch/salloc can only be used to + reduce the per-PE memory below the per-CPU share. This is also the only way that + the --mem-per-cpu option can be applied (besides, the --mem-per-cpu option + is ignored if the user forgets to set --ntasks/-n). + Thus, the preferred way of specifying memory is the more general --mem option.
+ +To increase the per-PE memory settable via the --mem option requires making + more per-task resources available using the --ntasks-per-node option to sbatch/salloc. + This allows --mem to request up to node_memory / ntasks_per_node MegaBytes.
+ +When --ntasks-per-node is 1, the entire node memory may be requested by the application. + Setting --ntasks-per-node to the number of cores per node yields the default per-CPU share + minimum value.
+ +For all cases in between these extremes, set --mem=per_task_memory and
++ --ntasks-per-node=floor(node_memory / per_task_memory) ++
whenever per_task_memory needs to be larger than the per-CPU share.
+ +Example: An application with 64 tasks needs 7500MB per task on a cluster with 32000MB and 24 cores + per node. Hence ntasks_per_node = floor(32000/7500) = 4.
++ #SBATCH --comment="requesting 7500MB per task on 32000MB/24-core nodes" + #SBATCH --ntasks=64 + #SBATCH --ntasks-per-node=4 + #SBATCH --mem=7500 ++
If you would like to fine-tune the memory limit of your application, you can set the same parameters in + a salloc session and then check directly, using
++ apstat -rvv -R $BASIL_RESERVATION_ID ++
to see how much memory has been requested.
-SLURM's primary mode of operation is designed for use on clusters with -nodes configured in a one-dimensional space. -Minor changes were required for the smap and sview tools -to map nodes in a three-dimensional space. -Some changes are also desirable to optimize job placement in three-dimensional -space.
- -SLURM has added an interface to Cray's Application Level Placement Scheduler -(ALPS). The ALPS aprun command must used for task launch rather than SLURM's -srun command. You should create a resource reservation using SLURM's -salloc or sbatch command and execute aprun from within -that allocation. +
Using aprun -B
+CLE 3.x allows a nice aprun shortcut via the -B option, which + reuses all the batch system parameters (--ntasks, --ntasks-per-node, + --cpus-per-task, --mem) at application launch, as if the corresponding + (-n, -N, -d, -m) parameters had been set; see the aprun(1) manpage + on CLE 3.x systems for details.
+ +Node ordering options
+SLURM honours the node ordering policy set for Cray's Application Level Placement Scheduler (ALPS). Node + ordering is a configurable system option (ALPS_NIDORDER in /etc/sysconfig/alps). The current + setting is reported by 'apstat -svv' (look for the line starting with "nid ordering option") and + can not be changed at runtime. The resulting, effective node ordering is revealed by 'apstat -no' + (if no special node ordering has been configured, 'apstat -no' shows the + same order as 'apstat -n').
+ +SLURM uses exactly the same order as 'apstat -no' when selecting + nodes for a job. With the --contiguous option to sbatch/salloc + you can request a contiguous (relative to the current ALPS nid ordering) set + of nodes. Note that on a busy system there is typically more fragmentation, + hence it may take longer (or even prove impossible) to allocate contiguous + sets of a larger size.
+ +Cray/ALPS node ordering is a topic of ongoing work, some information can be found in the CUG-2010 paper + "ALPS, Topology, and Performance" by Carl Albing and Mark Baker.
Administrator Guide
-Cray/ALPS configuration
+Install supporting rpms
+ +The build requires a few -devel RPMs listed below. You can obtain these from +SuSe/Novell. +
-
+
- CLE 2.x uses SuSe SLES 10 packages (rpms may be on the normal isos) +
- CLE 3.x uses Suse SLES 11 packages (rpms are on the SDK isos, there +are two SDK iso files for SDK) +
You can check by logging onto the boot node and running
++boot: # xtopview +default: # rpm -qa ++ +
The list of packages that should be installed is:
+-
+
- expat-2.0.xxx +
- libexpat-devel-2.0.xxx +
- cray-MySQL-devel-enterprise-5.0.64 (this should be on the Cray iso) +
For example, loading MySQL can be done like this:
++smw: # mkdir mnt +smw: # mount -o loop, ro xe-sles11sp1-trunk.201107070231a03.iso mnt +smw: # find mnt -name cray-MySQL-devel-enterprise\* +mnt/craydist/xt-packages/cray-MySQL-devel-enterprise-5.0.64.1.0000.2899.19.2.x86_64.rpm +smw: # scp mnt/craydist/xt-packages/cray-MySQL-devel-enterprise-5.0.64.1.0000.2899.19.2.x86_64 ++ +
Then switch to boot node and run:
++boot: # xtopview +default: # rpm -ivh /software/cray-MySQL-devel-enterprise-5.0.64.1.0000.2899.19.2.x86_64.rpm +default: # exit ++ +
All Cray-specific PrgEnv and compiler modules should be removed and root +privileges will be required to install these files.
+ +Create a build root
+ +The build is done on a normal service node, where you like +(e.g. /ufs/slurm/build would work). +Most scripts check for the environment variable LIBROOT. +You can either edit the scripts or export this variable. Easiest way:
+ ++login: # export LIBROOT=/ufs/slurm/build +login: # mkdir -vp $LIBROOT +login: # cd $LIBROOT ++ +
Install SLURM modulefile
+ +This file is distributed as part the SLURM tar-ball in +contribs/cray/opt_modulefiles_slurm. Install it as +/opt/modulefiles/slurm (or anywhere else in your module path). +It means that you can use Munge as soon as it is built.
++login: # scp ~/slurm/contribs/cray/opt_modulefiles_slurm root@boot:/rr/current/software/ ++ +
Build and install Munge
+ +Note the Munge installation process on Cray systems differs +somewhat from that described in the + +MUNGE Installation Guide.
+ +Munge is the authentication daemon and needed by SLURM. Download +munge-0.5.10.tar.bz2 or newer from + +http://code.google.com/p/munge/downloads/list. This is how one +can build on a login node and install it.
+
+login: # cd $LIBROOT
+login: # cp ~/slurm/contribs/cray/munge_build_script.sh $LIBROOT
+login: # mkdir -p ${LIBROOT}/munge/zip
+login: # curl -O http://munge.googlecode.com/files/munge-0.5.10.tar.bz2
+login: # cp munge-0.5.10.tar.bz2 ${LIBROOT}/munge/zip
+login: # chmod u+x ${LIBROOT}/munge/zip/munge_build_script.sh
+login: # ${LIBROOT}/munge/zip/munge_build_script.sh
+(generates lots of output and enerates a tar-ball called
+$LIBROOT/munge_build-.*YYYY-MM-DD.tar.gz)
+login: # scp munge_build-2011-07-12.tar.gz root@boot:/rr/current/software
+
+
+Install the tar-ball by on the boot node and build an encryption +key file executing: +
+boot: # xtopview +default: # tar -zxvf $LIBROOT/munge_build-*.tar.gz -C /rr/current / +default: # dd if=/dev/urandom bs=1 count=1024 >/opt/slurm/munge/etc/munge.key +default: # chmod go-rxw /opt/slurm/munge/etc/munge.key +default: # exit ++ +
Configure Munge
+ +The following steps apply to each login node and the sdb, where +
-
+
- The slurmd or slurmctld daemon will run and/or +
- Users will be submitting jobs +
+login: # mkdir --mode=0711 -vp /var/lib/munge +login: # mkdir --mode=0700 -vp /var/log/munge +login: # mkdir --mode=0755 -vp /var/run/munge +login: # module load slurm ++
+sdb: # mkdir --mode=0711 -vp /var/lib/munge +sdb: # mkdir --mode=0700 -vp /var/log/munge +sdb: # mkdir --mode=0755 -vp /var/run/munge ++ +
Start the munge daemon and test it.
++login: # munged --key-file /opt/slurm/munge/etc/munge.key +login: # munge -n +MUNGE:AwQDAAAEy341MRViY+LacxYlz+mchKk5NUAGrYLqKRUvYkrR+MJzHTgzSm1JALqJcunWGDU6k3vpveoDFLD7fLctee5+OoQ4dCeqyK8slfAFvF9DT5pccPg=: ++ +
When done, verify network connectivity by executing: +
-
+
- munge -n | ssh other-login-host /opt/slurm/munge/bin/unmunge +
If you decide to keep the installation, you may be interested in automating +the process using an init.d script distributed with the Munge. This +should be installed on all nodes running munge, e.g., 'xtopview -c login' and +'xtopview -n sdbNodeID' +
++boot: # xtopview -c login +login: # cp /software/etc_init_d_munge /etc/init.d/munge +login: # chmod u+x /etc/init.d/munge +login: # chkconfig munge on +login: # exit +boot: # xtopview -n 31 +node/31: # cp /software/etc_init_d_munge /etc/init.d/munge +node/31: # chmod u+x /etc/init.d/munge +node/31: # chkconfig munge on +node/31: # exit ++ +
Enable the Cray job service
+ +This is a common dependency on Cray systems. ALPS relies on the Cray job service to + generate cluster-unique job container IDs (PAGG IDs). These identifiers are used by + ALPS to track running (aprun) job steps. The default (session IDs) is not unique + across multiple login nodes. This standard procedure is described in chapter 9 of + S-2393 and takes only two + steps, both to be done on all 'login' class nodes (xtopview -c login):
+-
+
- make sure that the /etc/init.d/job service is enabled (chkconfig) and started +
- enable the pam_job.so module from /opt/cray/job/default in /etc/pam.d/common-session
+ (NB: the default pam_job.so is very verbose, a simpler and quieter variant is provided + in contribs/cray.)
+
The latter step is required only if you would like to run interactive + salloc sessions.
++boot: # xtopview -c login +login: # chkconfig job on +login: # emacs -nw /etc/pam.d/common-session +(uncomment the pam_job.so line) +session optional /opt/cray/job/default/lib64/security/pam_job.so +login: # exit +boot: # xtopview -n 31 +node/31:# chkconfig job on +node/31:# emacs -nw /etc/pam.d/common-session +(uncomment the pam_job.so line as shown above) ++ +
Build and Configure SLURM
-Node names must have a three-digit suffix describing their -zero-origin position in the X-, Y- and Z-dimension respectively (e.g. -"tux000" for X=0, Y=0, Z=0; "tux123" for X=1, Y=2, Z=3). -Rectangular prisms of nodes can be specified in SLURM commands and -configuration files using the system name prefix with the end-points -enclosed in square brackets and separated by an "x". -For example "tux[620x731]" is used to represent the eight nodes in a -block with endpoints at "tux620" and "tux731" (tux620, tux621, tux630, -tux631, tux720, tux721, tux730, tux731). -NOTE: We anticipate that Cray will provide node coordinate -information via the ALPS interface in the future, which may result -in a more flexible node naming convention.
- -In ALPS, configure each node to be scheduled using SLURM as type -BATCH.
- -SLURM configuration
- -Four variables must be defined in the config.h file: -APBASIL_LOC (location of the apbasil command), -HAVE_FRONT_END, HAVE_CRAY and HAVE_3D. -The apbasil command should automatically be found. -If that is not the case, please notify us of its location on your system -and we will add that to the search paths tested at configure time. -The other variable definitions can be initiated in several different -ways depending upon how SLURM is being built. -
-
-
- Execute the configure command with the option ---enable-cray-xt OR -
- Execute the rpmbuild command with the option ---with cray_xt OR -
- Add %with_cray_xt 1 to your ~/.rpmmacros file. -
One slurmd will be used to run all of the batch jobs on -the system. It is from here that users will execute aprun -commands to launch tasks. +
SLURM can be built and installed as on any other computer as described +Quick Start Administrator Guide. +An example of building and installing SLURM version 2.3.0 is shown below.
+ ++login: # mkdir build && cd build +login: # slurm/configure \ + --prefix=/opt/slurm/2.3.0 \ + --with-munge=/opt/slurm/munge/ \ + --with-mysql_config=/opt/cray/MySQL/5.0.64-1.0000.2899.20.2.gem/bin \ + --with-srun2aprun +login: # make -j +login: # mkdir install +login: # make DESTDIR=/tmp/slurm/build/install install +login: # make DESTDIR=/tmp/slurm/build/install install-contrib +login: # cd install +login: # tar czf slurm_opt.tar.gz opt +login: # scp slurm_opt.tar.gz boot:/rr/current/software ++ +
+boot: # xtopview +default: # tar xzf /software/slurm_opt.tar.gz -C / +default: # cd /opt/slurm/ +default: # ln -s 2.3.0 default ++ +
When building SLURM's slurm.conf configuration file, use the +NodeName parameter to specify all batch nodes to be scheduled. +If nodes are defined in ALPS, but not defined in the slurm.conf file, a +complete list of all batch nodes configured in ALPS will be logged by +the slurmctld daemon when it starts. +One would typically use this information to modify the slurm.conf file +and restart the slurmctld daemon. +Note that the NodeAddr and NodeHostName fields should not be +configured, but will be set by SLURM using data from ALPS. +NodeAddr be set to the node's XYZ coordinate and be used by SLURM's +smap and sview commands. +NodeHostName will be set to the node's component label. +The format of the component label is "c#-#c#s#n#" where the "#" fields +represent in order: cabinet, row, cate, blade or slot, and node. +For example "c0-1c2s5n3" is cabinet 0, row 1, cage 3, slot 5 and node 3.
+ +The slurmd daemons will not execute on the compute nodes, but will +execute on one or more front end nodes. +It is from here that batch scripts will execute aprun commands to +launch tasks. This is specified in the slurm.conf file by using the -NodeName field to identify the compute nodes and both the -NodeAddr and NodeHostname fields to identify the -computer when slurmd runs (normally some sort of front-end node) +FrontendName and optionally the FrontEndAddr fields as seen in the examples below.
-Next you need to select from two options for the resource selection -plugin (the SelectType option in SLURM's slurm.conf configuration -file): -
-
-
- select/cons_res - Performs a best-fit algorithm based upon a -one-dimensional space to allocate whole nodes, sockets, or cores to jobs -based upon other configuration parameters. -
- select/linear - Performs a best-fit algorithm based upon a -one-dimensional space to allocate whole nodes to jobs. -
In order for select/cons_res or select/linear to -allocate resources physically nearby in three-dimensional space, the -nodes be specified in SLURM's slurm.conf configuration file in -such a fashion that those nearby in slurm.conf (one-dimensional -space) are also nearby in the physical three-dimensional space. -If the definition of the nodes in SLURM's slurm.conf configuration -file are listed on one line (e.g. NodeName=tux[000x333]), -SLURM will automatically perform that conversion using a -Hilbert curve. -Otherwise you may construct your own node name ordering and list them -one node per line in slurm.conf. -Note that each node must be listed exactly once and consecutive -nodes should be nearby in three-dimensional space. -Also note that each node must be defined individually rather than using -a hostlist expression in order to preserve the ordering (there is no -problem using a hostlist expression in the partition specification after -the nodes have already been defined). -The open source code used by SLURM to generate the Hilbert curve is -included in the distribution at contribs/skilling.c in the event -that you wish to experiment with it to generate your own node ordering. -Two examples of SLURM configuration files are shown below:
- --# slurm.conf for Cray XT system of size 4x4x4 -# Parameters removed here -SelectType=select/linear -NodeName=DEFAULT Procs=8 RealMemory=2048 State=Unknown -NodeName=tux[000x333] NodeAddr=front_end NodeHostname=front_end -PartitionName=debug Nodes=tux[000x333] Default=Yes State=UP -- -
-# slurm.conf for Cray XT system of size 4x4x4 -# Parameters removed here -SelectType=select/linear -NodeName=DEFAULT Procs=8 RealMemory=2048 State=Unknown -NodeName=tux000 NodeAddr=front_end NodeHostname=front_end -NodeName=tux100 NodeAddr=front_end NodeHostname=front_end -NodeName=tux110 NodeAddr=front_end NodeHostname=front_end -NodeName=tux010 NodeAddr=front_end NodeHostname=front_end -NodeName=tux011 NodeAddr=front_end NodeHostname=front_end -NodeName=tux111 NodeAddr=front_end NodeHostname=front_end -NodeName=tux101 NodeAddr=front_end NodeHostname=front_end -NodeName=tux001 NodeAddr=front_end NodeHostname=front_end -PartitionName=debug Nodes=tux[000x111] Default=Yes State=UP -- -
In both of the examples above, the node names output by the -scontrol show nodes will be ordered as defined (sequentially -along the Hilbert curve or per the ordering in the slurm.conf file) -rather than in numeric order (e.g. "tux001" follows "tux101" rather -than "tux000"). -SLURM partitions should contain nodes which are defined sequentially -by that ordering for optimal performance.
+Note that SLURM will by default kill running jobs when a node goes DOWN, +while a DOWN node in ALPS only prevents new jobs from being scheduled on the +node. To help avoid confusion, we recommend that SlurmdTimeout in the +slurm.conf file be set to the same value as the suspectend +parameter in ALPS' nodehealth.conf file.
+ +You need to specify the appropriate resource selection plugin (the +SelectType option in SLURM's slurm.conf configuration file). +Configure SelectType to select/cray The select/cray +plugin provides an interface to ALPS plus issues calls to the +select/linear, which selects resources for jobs using a best-fit +algorithm to allocate whole nodes to jobs (rather than individual sockets, +cores or threads).
+ +Note that the system topology is based upon information gathered from +the ALPS database and is based upon the ALPS_NIDORDER configuration in +/etc/sysconfig/alps. Excerpts of a slurm.conf file for +use on a Cray systems follow:
+ +
+#---------------------------------------------------------------------
+# SLURM USER
+#---------------------------------------------------------------------
+# SLURM user on cray systems must be root
+# This requirement derives from Cray ALPS:
+# - ALPS reservations can only be created by the job owner or root
+# (confirmation may be done by other non-privileged users)
+# - Freeing a reservation always requires root privileges
+SlurmUser=root
+
+#---------------------------------------------------------------------
+# PLUGINS
+#---------------------------------------------------------------------
+# Network topology (handled internally by ALPS)
+TopologyPlugin=topology/none
+
+# Scheduling
+SchedulerType=sched/backfill
+
+# Node selection: use the special-purpose "select/cray" plugin.
+# Internally this uses select/linar, i.e. nodes are always allocated
+# in units of nodes (other allocation is currently not possible, since
+# ALPS does not yet allow to run more than 1 executable on the same
+# node, see aprun(1), section LIMITATIONS).
+#
+# Add CR_memory as parameter to support --mem/--mem-per-cpu.
+SelectType=select/cray
+SelectTypeParameters=CR_Memory
+
+# Proctrack plugin: only/default option is proctrack/sgi_job
+# ALPS requires cluster-unique job container IDs and thus the /etc/init.d/job
+# service needs to be started on all slurmd and login nodes, as described in
+# S-2393, chapter 9. Due to this requirement, ProctrackType=proctrack/sgi_job
+# is the default on Cray and need not be specified explicitly.
+
+#---------------------------------------------------------------------
+# PATHS
+#---------------------------------------------------------------------
+SlurmdSpoolDir=/ufs/slurm/spool
+StateSaveLocation=/ufs/slurm/spool/state
+
+# main logfile
+SlurmctldLogFile=/ufs/slurm/log/slurmctld.log
+# slurmd logfiles (using %h for hostname)
+SlurmdLogFile=/ufs/slurm/log/%h.log
+
+# PIDs
+SlurmctldPidFile=/var/run/slurmctld.pid
+SlurmdPidFile=/var/run/slurmd.pid
+
+#---------------------------------------------------------------------
+# COMPUTE NODES
+#---------------------------------------------------------------------
+# Return DOWN nodes to service when e.g. slurmd has been unresponsive
+ReturnToService=1
+
+# Configure the suspectend parameter in ALPS' nodehealth.conf file to the same
+# value as SlurmdTimeout for consistent behavior (e.g. "suspectend: 600")
+SlurmdTimeout=600
+
+# Controls how a node's configuration specifications in slurm.conf are
+# used.
+# 0 - use hardware configuration (must agree with slurm.conf)
+# 1 - use slurm.conf, nodes with fewer resources are marked DOWN
+# 2 - use slurm.conf, but do not mark nodes down as in (1)
+FastSchedule=2
+
+# Per-node configuration for PALU AMD G34 dual-socket "Magny Cours"
+# Compute Nodes. We deviate from slurm's idea of a physical socket
+# here, since the Magny Cours hosts two NUMA nodes each, which is
+# also visible in the ALPS inventory (4 Segments per node, each
+# containing 6 'Processors'/Cores).
+NodeName=DEFAULT Sockets=4 CoresPerSocket=6 ThreadsPerCore=1
+NodeName=DEFAULT RealMemory=32000 State=UNKNOWN
+
+# List the nodes of the compute partition below (service nodes are not
+# allowed to appear)
+NodeName=nid00[002-013,018-159,162-173,178-189]
+
+# Frontend nodes: these should not be available to user logins, but
+# have all filesystems mounted that are also
+# available on a login node (/scratch, /home, ...).
+FrontendName=palu[7-9]
+
+#---------------------------------------------------------------------
+# ENFORCING LIMITS
+#---------------------------------------------------------------------
+# Enforce the use of associations: {associations, limits, wckeys}
+AccountingStorageEnforce=limits
+
+# Do not propagate any resource limits from the user's environment to
+# the slurmd
+PropagateResourceLimits=NONE
+
+#---------------------------------------------------------------------
+# Resource limits for memory allocation:
+# * the Def/Max 'PerCPU' and 'PerNode' variants are mutually exclusive;
+# * use the 'PerNode' variant for both default and maximum value, since
+# - slurm will automatically adjust this value depending on
+# --ntasks-per-node
+# - if using a higher per-cpu value than possible, salloc will just
+# block.
+#--------------------------------------------------------------------
+# XXX replace both values below with your values from 'xtprocadmin -A'
+DefMemPerNode=32000
+MaxMemPerNode=32000
+
+#---------------------------------------------------------------------
+# PARTITIONS
+#---------------------------------------------------------------------
+# defaults common to all partitions
+PartitionName=DEFAULT Nodes=nid00[002-013,018-159,162-173,178-189]
+PartitionName=DEFAULT MaxNodes=178
+PartitionName=DEFAULT Shared=EXCLUSIVE State=UP DefaultTime=60
+
+# "User Support" partition with a higher priority
+PartitionName=usup Hidden=YES Priority=10 MaxTime=720 AllowGroups=staff
+
+# normal partition available to all users
+PartitionName=day Default=YES Priority=1 MaxTime=01:00:00
+
+
+SLURM supports an optional cray.conf file containing Cray-specific +configuration parameters. This file is NOT needed for production systems, +but is provided for advanced configurations. If used, cray.conf must be +located in the same directory as the slurm.conf file. Configuration +parameters supported by cray.conf are listed below.
+ +-
+
- apbasil +
- Fully qualified pathname to the apbasil command. +The default value is /usr/bin/apbasil. +
- apkill +
- Fully qualified pathname to the apkill command. +The default value is /usr/bin/apkill. +
- SDBdb +
- Name of the ALPS database. +The default value is XTAdmin. +
- SDBhost +
- Hostname of the database server. +The default value is based upon the contents of the 'my.cnf' file used to +store default database access information and that defaults to user 'sdb'. +
- SDBpass +
- Password used to access the ALPS database. +The default value is based upon the contents of the 'my.cnf' file used to +store default database access information and that defaults to user 'basic'. +
- SDBport +
- Port used to access the ALPS database. +The default value is 0. +
- SDBuser +
- Name of user used to access the ALPS database. +The default value is based upon the contents of the 'my.cnf' file used to +store default database access information and that defaults to user 'basic'. +
+# Example cray.conf file +apbasil=/opt/alps_simulator_40_r6768/apbasil.sh +SDBhost=localhost +SDBuser=alps_user +SDBdb=XT5istanbul ++ +
One additional configuration script can be used to insure that the slurmd +daemons execute with the highest resource limits possible, overriding default +limits on Suse systems. Depending upon what resource limits are propagated +from the user's environment, lower limits may apply to user jobs, but this +script will insure that higher limits are possible. Copy the file +contribs/cray/etc_sysconfig_slurm into /etc/sysconfig/slurm +for these limits to take effect. This script is executed from +/etc/init.d/slurm, which is typically executed to start the SLURM +daemons. An excerpt of contribs/cray/etc_sysconfig_slurmis shown +below.
+ ++# +# /etc/sysconfig/slurm for Cray XT/XE systems +# +# Cray is SuSe-based, which means that ulimits from +# /etc/security/limits.conf will get picked up any time SLURM is +# restarted e.g. via pdsh/ssh. Since SLURM respects configured limits, +# this can mean that for instance batch jobs get killed as a result +# of configuring CPU time limits. Set sane start limits here. +# +# Values were taken from pam-1.1.2 Debian package +ulimit -t unlimited # max amount of CPU time in seconds +ulimit -d unlimited # max size of a process's data segment in KB ++ +
SLURM's init.d script should also be installed to automatically +start SLURM daemons when nodes boot as shown below. Be sure to edit the script +as appropriate to reference the proper file location (modify the variable +PREFIX). + +
+login: # scp /home/crayadm/ben/slurm/etc/init.d.slurm boot:/rr/current/software/ ++ +
Now create the needed directories for logs and state files then start the +daemons on the sdb and login nodes as shown below.
+ ++sdb: # mkdir -p /ufs/slurm/log +sdb: # mkdir -p /ufs/slurm/spool +sdb: # /etc/init.d/slurm start ++ +
+login: # /etc/init.d/slurm start ++ +
Srun wrapper configuration
+ +The srun wrapper to aprun might require modification to run +as desired. Specifically the $aprun variable could be set to the +absolute pathname of that executable file. Without that modification, the +aprun command executed will depend upon the user's search path.
+ +In order to debug the srun wrapper, uncomment the line
++print "comment=$command\n" ++
If the srun wrapper is executed from +within an existing SLURM job allocation (i.e. within salloc or an +sbatch script), then it just executes the aprun command with +appropriate options. If executed without an allocation, the wrapper executes +salloc, which then executes the srun wrapper again. This second +execution of the srun wrapper is required in order to process environment +variables that are set by the salloc command based upon the resource +allocation.
-Last modified 9 January 2009
+Last modified 1 August 2011
diff -Nru slurm-llnl-2.2.7/doc/html/disclaimer.shtml slurm-llnl-2.3.2/doc/html/disclaimer.shtml --- slurm-llnl-2.2.7/doc/html/disclaimer.shtml 1970-01-01 00:00:00.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/disclaimer.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -0,0 +1,90 @@ + + +Legal Notices
+ +SLURM is free software; you can redistribute it and/or modify it under +the terms of the GNU General +Public License as published by the Free +Software Foundation; either version 2 of the License, or (at your option) +any later version.
+ +SLURM is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details.
+ +NO WARRANTY
+The following is an excerpt from the GNU General Public License.
+ +BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION.
+ +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES.
+ +Copyrights
+ +SLURM represents the collaborative efforts of roughly 100 people representing +roughtly 40 different organizations world-wide. A current list of contributors +can be found at the SLURM Team web page.
+ +While many organizations contributed code and/or documentation without
+including a copyright notice, the following copyright notices are found in
+SLURM's code and documenation files:
+Copyright (C) 2011 Trinity Centre for High Performance Computing
+Copyright (C) 2010-2011 SchedMD LLC
+Copyright (C) 2009 CEA/DAM/DIF
+Copyright (C) 2009-2011 Centro Svizzero di Calcolo Scientifico (CSCS)
+Copyright (C) 2008-2011 Lawrence Livermore National Security
+Copyright (C) 2008 Vijay Ramasubramanian
+Copyright (C) 2007-2008 Red Hat, Inc.
+Copyright (C) 2007-2009 National University of Defense Technology, China
+Copyright (C) 2007-2011 Bull
+Copyright (C) 2005-2008 Hewlett-Packard Development Company, L.P.
+Copyright (C) 2004-2009, Marcus Holland-Moritz
+Copyright (C) 2002-2007 The Regents of the University of California
+Copyright (C) 2002-2003 Linux NetworX
+Copyright (C) 2002 University of Chicago
+Copyright (C) 2001 Paul Marquess
+Copyright (C) 2000 Markus Friedl
+Copyright (C) 1999 Kenneth Albanowski
+Copyright (C) 1998 Todd C. Miller
+Copyright (C) 1996-2003 Maximum Entropy Data Consultants Ltd,
+Copyright (C) 1995 Tatu Ylonen
+Copyright (C) 1989-1994, 1996-1999, 2001 Free Software Foundation, Inc.
Much of the work was performed under the auspices of the U.S. Department of +Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. +This work was sponsored by an agency of the United States government. +Neither the United States Government nor Lawrence Livermore National +Security, LLC, nor any of their employees, makes any warranty, express +or implied, or assumes any liability or responsibility for the accuracy, +completeness, or usefulness of any information, apparatus, product, or +process disclosed, or represents that its use would not infringe privately +owned rights. References herein to any specific commercial products, process, +or services by trade names, trademark, manufacturer or otherwise does not +necessarily constitute or imply its endorsement, recommendation, or +favoring by the United States Government or the Lawrence Livermore National +Security, LLC. The views and opinions of authors expressed herein do not +necessarily state or reflect those of the United States government or +Lawrence Livermore National Security, LLC, and shall not be used for +advertising or product endorsement purposes.
+ +Last modified 27 June 2010
+ + + diff -Nru slurm-llnl-2.2.7/doc/html/dist_plane.shtml slurm-llnl-2.3.2/doc/html/dist_plane.shtml --- slurm-llnl-2.2.7/doc/html/dist_plane.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/dist_plane.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -17,7 +17,7 @@On One (1) node: srun -N 1-1 -n 21 -m plane=4 <...>.
The distribution results in a plane distribution with plane_size 21. -Even thought the user specified a plane_size of 4 the final plane +Even though the user specified a plane_size of 4 the final plane distribution results in a plane_size of 21.
@@ -114,7 +114,7 @@
On One (1) node: srun -N 1-1 -n 21 -m plane=4 --cpu_bind=core <...>. -Even thought the user specified a plane_size of 4 the final plane +Even though the user specified a plane_size of 4 the final plane distribution results in a plane distribution with plane_size=8.
diff -Nru slurm-llnl-2.2.7/doc/html/documentation.shtml slurm-llnl-2.3.2/doc/html/documentation.shtml --- slurm-llnl-2.2.7/doc/html/documentation.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/documentation.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -7,6 +7,8 @@
SLURM Users
- Quick Start User Guide +
- Man Pages +
- CPU Management User and Administrator Guide
- MPI Use Guide
- Support for Multi-core/Multi-threaded Architectures
- Multi-Cluster Operation @@ -23,6 +25,7 @@
- Quick Start Administrator Guide +
- CPU Management User and Administrator Guide
- Configuration Tool
- Troubleshooting Guide
- Large Cluster Administration Guide @@ -32,7 +35,7 @@
- Consumable Resources Guide
- Gang Scheduling -
- Generic Resource (Gres) Scheduling +
- Generic Resource (GRES) Scheduling
- High Throughput Computing Guide
- Multifactor Job Priority
- Preemption @@ -61,6 +64,12 @@
- Programmer Guide
- Application Programmer Interface (API) Guide +
- Design Information +
- Plugin Programmer Guide
- Plugin Interface Details
- MySQL (recommended) -
- PostgreSQL +
- PostgreSQL (Not fully functional)
- Debuggers and debugging tools @@ -68,7 +81,7 @@ is an implementation of Open Grid Forum DRMAA 1.0 (Distributed Resource Management Application API) specification for submission -and control of jobs to
- I/O Watchdog
@@ -118,15 +131,15 @@ - Quadrics MPI
- PAM Modules (pam_slurm)
+ - PAM Module (pam_slurm)
Pluggable Authentication Module (PAM) for restricting access to compute nodes where SLURM performs resource management. Access to the node is restricted to user root and users who have been allocated resources on that node. NOTE: pam_slurm is included within the SLURM distribution for version 2.1 or higher. For earlier SLURM versions, pam_slurm is available for download from
- -https://sourceforge.net/projects/slurm/
+ +http://www.schedmd.com/download/extras/pam_slurm-1.6.tar.bz2
SLURM's PAM module has also been packaged for Debian and Ubuntu @@ -176,20 +189,6 @@ repository with the following command:
svn checkout http://slurm-spank-plugins.googlecode.com/svn/trunk/ slurm-plugins - PAM Module (pam_slurm)
-Pluggable Authentication Module (PAM) for restricting access to compute nodes -where SLURM performs resource management. Access to the node is restricted to -user root and users who have been allocated resources on that node. -NOTE: pam_slurm is included within the SLURM distribution for version 2.1 -or higher. -For earlier SLURM versions, pam_slurm is available for download from
- -https://sourceforge.net/projects/slurm/
-SLURM's PAM module has also been packaged for -Debian and -Ubuntu -(both named libpam-slurm). - Sqlog
A set of scripts that leverages SLURM's job completion logging facility in provide information about what jobs were running at any point in the @@ -204,6 +203,6 @@ - Can I change my job's size after it has started running? +
- Why is my MPIHCH2 or MVAPICH2 job not running with +SLURM? Why does the DAKOTA program not run with SLURM?
- Is resource limit propagation useful on a homogeneous cluster? -
- Do I need to maintain synchronized clocks on the cluster?
- Why are "Invalid job credential" errors generated? @@ -300,9 +302,9 @@
- Job: ExcNodeList=NULL
- Job: Contiguous=0
- Job: Features=NULL -
- Job: MinProcs, MinMemory, and MinTmpDisk satisfied by all nodes in +
- Job: MinCPUs, MinMemory, and MinTmpDisk satisfied by all nodes in the partition -
- Job: MinProcs or MinNodes not to exceed partition's MaxNodes +
- Job: MinCPUs or MinNodes not to exceed partition's MaxNodes
- Support is not available on BlueGene or Cray system due to limitations +in the software underlying SLURM. +
- Job(s) changing size must not be in a suspended state, including jobs +suspended for gang scheduling. The jobs must be in a state of pending or +running. We plan to modify the gang scheduling logic in the future to +concurrently schedule a job to be used for expanding another job and the +job to be expanded. +
SLURM Administrators
-
@@ -84,6 +93,6 @@
Last modified 27 August 2010
+Last modified 31 May 2011
diff -Nru slurm-llnl-2.2.7/doc/html/download.shtml slurm-llnl-2.3.2/doc/html/download.shtml --- slurm-llnl-2.2.7/doc/html/download.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/download.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -1,15 +1,28 @@Download
-
-SLURM source can be downloaded from
-
-http://sourceforge.net/projects/slurm/
+
SLURM source can be downloaded from
+
+http://www.schedmd.com/#repos
SLURM has also been packaged for
Debian and
Ubuntu
(both named slurm-llnl).
A SLURM simulator +is available to assess various scheduling policies. +Under simulation jobs are not actually executed. Instead a job execution trace +from a real system or a synthetic trace are used.
+ + +Related software available from various sources include:
-
@@ -40,7 +53,7 @@
See our Accounting web page for more information.
@@ -88,8 +101,8 @@ plugin also requires the libelanhosts library and a corresponding /etc/elanhosts configuration file, used to map hostnames to Elan IDs. The libelanhosts source is available from - -https://sourceforge.net/projects/slurm/. + +http://www.schedmd.com/download/extras/libelanhosts-0.9-1.tgz.
-
-
-
Last modified 20 December 2010
+Last modified 24 May 2011
diff -Nru slurm-llnl-2.2.7/doc/html/faq.shtml slurm-llnl-2.3.2/doc/html/faq.shtml --- slurm-llnl-2.2.7/doc/html/faq.shtml 2011-06-10 16:57:27.000000000 +0000 +++ slurm-llnl-2.3.2/doc/html/faq.shtml 2011-12-05 17:20:08.000000000 +0000 @@ -44,6 +44,8 @@ the slurm-dev mailing list?For Administrators
@@ -86,7 +88,7 @@ core files?If the partitions specifications differ from those listed above, no jobs in that partition will be scheduled by the backfills scheduler. @@ -646,9 +648,22 @@
24. Can I change my job's size after it has started
running?
-Beginning with Slurm version 2.2 it is possible to decrease a job's size after
-it has started, but it is not currently possible to increase a job's size.
-Use the scontrol command to change a job's size either by specifying
+Support to decrease the size of a running job was added to SLURM version 2.2.
+The ability to increase the size of a running job was added to SLURM version 2.3.
+While the size of a pending job may be changed with few restrictions, several
+significant restrictions apply to changing the size of a running job as noted
+below:
+
-
+
Use the scontrol command to change a job's size either by specifying a new node count (NumNodes=) for the job or identify the specific nodes (NodeList=) that you want the job to retain. Any job steps running on the nodes which are reliquished by the job will be @@ -658,11 +673,13 @@ should either be removed or altered (e.g. SLURM_NNODES, SLURM_NODELIST and SLURM_NPROCS). The scontrol command will generate a script that can be executed to -reset local environment variables +reset local environment variables. You must retain the SLURM_JOBID environment variable in order for the srun command to gather information about the job's current state and -specify the desired node and/or task count in subsequent srun invocations. -An example is shown below. +specify the desired node and/or task count in subsequent srun invocations. +A new accounting record is generated when a job is resized showing the to have +been resubmitted and restarted at the new size. +An example is shown below.
#!/bin/bash
srun my_big_job
@@ -672,6 +689,114 @@
rm slurm_job_${SLURM_JOBID}_resize.*
+Increasing a job's size
+Directly increasing the size of a running job would adversely effect the
+scheduling of pending jobs.
+For the sake of fairness in job scheduling, expanding a running job requires
+the user to submit a new job, but specify the option
+--dependency=expand:<jobid>.
+This option tells SLURM that the job, when scheduled, can be used to expand
+the specified jobid.
+Other job options would be used to identify the required resources
+(e.g. task count, node count, node features, etc.).
+This new job's time limit will be automatically set to reflect the end time of
+the job being expanded.
+This new job's generic resources specification will be automatically set
+equal to that of the job being merged to. This is due to the current SLURM
+restriction of all nodes associated with a job needing to have the same
+generic resource specification (i.e. a job can not have one GPU on one
+node and two GPUs on another node), although this restriction may be removed
+in the future. This restriction can pose some problems when both jobs can be
+allocated resources on the same node, in which case the generic resources
+allocated to the new job will be released. If the jobs are allocated resources
+on different nodes, the generic resources associated with the resulting job
+allocation after the merge will be consistent as expected.
+Any licenses associated with the new job will be added to those available in
+the job being merged to.
+Note that partition and Quality Of Service (QOS) limits will be applied
+independently to the new job allocation so the expanded job may exceed size
+limits configured for an individual job.
After the new job is allocated resources, merge that job's allocation
+into that of the original job by executing:
+scontrol update jobid=<jobid> NumNodes=0
+The jobid above is that of the job to relinquish it's resources.
+To provides more control over when the job expansion occurs, the resources are
+not merged into the original job until explicitly requested.
+These resources will be transfered to the original job and the scontrol
+command will generate a script to reset variables in the second
+job's environment to reflect it's modified resource allocation (which would
+be no resources).
+One would normally exit this second job at this point, since it has no
+associated resources.
+In order to generate a script to modify the environment variables for the
+expanded job, execute:
+scontrol update jobid=<jobid> NumNodes=ALL
+Then execute the script generated.
+Note that this command does not change the original job's size, but only
+generates the script to change its environment variables.
+Until the environment variables are modified (e.g. the job's node count,
+CPU count, hostlist, etc.), any srun command will only consider the resources
+in the original resource allocation.
+Note that the original job may have active job steps at the time of it's
+expansion, but they will not be effected by the change.
+An example of the proceedure is shown below in which the original job
+allocation waits until the second resource allocation request can be
+satisfied. The job requesting additional resources could also use the sbatch
+command and permit the original job to continue execution at its initial size.
+Note that the development of additional user tools to manage SLURM resource
+allocations is planned in the future to make this process both simpler and
+more flexible.
+$ salloc -N4 bash +salloc: Granted job allocation 65542 +$ srun hostname +icrm1 +icrm2 +icrm3 +icrm4 + +$ salloc -N4 --dependency=expand:$SLURM_JOBID bash +salloc: Granted job allocation 65543 +$ scontrol update jobid=$SLURM_JOBID NumNodes=0 +To reset SLURM environment variables, execute + For bash or sh shells: . ./slurm_job_65543_resize.sh + For csh shells: source ./slurm_job_65543_resize.csh +$ exit +exit +salloc: Relinquishing job allocation 65543 + +$ scontrol update jobid=$SLURM_JOBID NumNodes=ALL +To reset SLURM environment variables, execute + For bash or sh shells: . ./slurm_job_65542_resize.sh + For csh shells: source ./slurm_job_65542_resize.csh +$ . ./slurm_job_$SLURM_JOBID_resize.sh + +$ srun hostname +icrm1 +icrm2 +icrm3 +icrm4 +icrm5 +icrm6 +icrm7 +icrm8 +$ exit +exit +salloc: Relinquishing job allocation 65542 ++ +
25. Why is my MPIHCH2 or MVAPICH2 job not running with
+SLURM? Why does the DAKOTA program not run with SLURM?
+The SLURM library used to support MPIHCH2 or MVAPICH2 references a variety of
+symbols. If those symbols resolve to functions or variables in your program
+rather than the appropriate library, the application will fail. In the case of
+DAKOTA, it contains a function named
+regcomp, which will get used rather than the POSIX regex functions.
+Rename DAKOTA's function and references from regcomp to something else to make
+it work properly.
Yes in SLURM version 1.2 or higher. In the slurm.conf file, set FastSchedule=2 and specify -any desired node resource specifications (Procs, Sockets, +any desired node resource specifications (CPUs, Sockets, CoresPerSocket, ThreadsPerCore, and/or TmpDisk). SLURM will use the resource specification for each node that is given in slurm.conf and will not check these specifications @@ -1269,7 +1394,7 @@ communication doesn't work. Solution is to either remove this line or set a different nodeaddr that is known by your other nodes. -
38. How can I stop SLURM from scheduling jobs? +
39. How can I stop SLURM from scheduling jobs? You can stop SLURM from scheduling jobs on a per partition basis by setting that partition's state to DOWN. Set its state UP to resume scheduling. For example: @@ -1278,7 +1403,7 @@ $ scontrol update PartitionName=bar State=UP
-39. Can I update multiple jobs with a
+ 40. Can I update multiple jobs with a
single scontrol command?
No, but you can probably use squeue to build the script taking
advantage of its filtering and formatting options. For example:
@@ -1286,7 +1411,7 @@
$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script
40. Can SLURM be used to run jobs on
+ 41. Can SLURM be used to run jobs on
Amazon's EC2?
Yes, here is a description of use SLURM use with
Amazon's EC2 courtesy of
@@ -1310,7 +1435,7 @@
all other instances. This way I have persistent home directories and a very
simple first-login script that configures the virtual cluster for me. 41. If a SLURM daemon core dumps, where can I find the
+ 42. If a SLURM daemon core dumps, where can I find the
core file?
For slurmctld the core file will be in the same directory as its
log files (SlurmctldLogFile) iif configured using an fully qualified
@@ -1326,7 +1451,7 @@
occurs. It will either be in spawned job's working directory on the same
location as that described above for the slurmd daemon. 42. How can TotalView be configured to operate with
+ 43. How can TotalView be configured to operate with
SLURM?
The following lines should also be added to the global .tvdrc file
for TotalView to operate with SLURM:
@@ -1344,6 +1469,6 @@
Last modified 6 January 2011 Last modified 4 September 2011
-
-
-
-
-
-
Operated by
-Lawrence Livermore National Security, LLC, for the
-
-Department of Energy's
-
-National Nuclear Security Administration