Option(s) define multiple jobs in a co-scheduled heterogeneous job. For more details about heterogeneous jobs see the document .br https://slurm.schedmd.com/heterogeneous_jobs.html .SH "DESCRIPTION" sbatch submits a batch script to Slurm. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with "#SBATCH" before any executable commands in the script. sbatch will stop processing further #SBATCH directives once the first non-comment non-whitespace line has been reached in the script. sbatch exits immediately after the script is successfully transferred to the Slurm controller and assigned a Slurm job ID. The batch script is not necessarily granted resources immediately, it may sit in the queue of pending jobs for some time before its required resources become available. By default both standard output and standard error are directed to a file of the name "slurm\-%j.out", where the "%j" is replaced with the job allocation number. The file will be generated on the first node of the job allocation. Other than the batch script itself, Slurm does no movement of user files. When the job allocation is finally granted for the batch script, Slurm runs a single copy of the batch script on the first node in the set of allocated nodes. The following document describes the influence of various options on the allocation of cpus to jobs and tasks. .br https://slurm.schedmd.com/cpu_management.html .SH "RETURN VALUE" sbatch will return 0 on success or error code on failure. .SH "SCRIPT PATH RESOLUTION" The batch script is resolved in the following order: .br 1. If script starts with ".", then path is constructed as: current working directory / script .br 2. If script starts with a "/", then path is considered absolute. .br 3. If script is in current working directory. .br 4. If script can be resolved through PATH. See \fBpath_resolution\fR(7). .br .P Current working directory is the calling process working directory unless the number. For example, "\-\-array=0\-15:4" is equivalent to "\-\-array=0,4,8,12". A maximum number of simultaneously running tasks from the job array may be specified using a "%" separator. For example "\-\-array=0\-15%4" will limit the number of simultaneously running tasks from this job array to 4. The minimum index value is 0. the maximum value is one less than the configuration parameter MaxArraySize. NOTE: currently, federated job arrays only run on the local cluster. .TP \fB\-A\fR, \fB\-\-account\fR=<\fIaccount\fR> Charge resources used by this job to specified account. The \fIaccount\fR is an arbitrary string. The account name may be changed after job submission using the \fBscontrol\fR command. .TP \fB\-\-acctg\-freq\fR Define the job accounting and profiling sampling intervals. This can be used to override the \fIJobAcctGatherFrequency\fR parameter in Slurm's configuration file, \fIslurm.conf\fR. The supported format is as follows: .RS .TP 12 \fB\-\-acctg\-freq=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling interval for the jobacct_gather plugin or a sampling interval for a profiling type by the acct_gather_profile plugin. Multiple, comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals may be specified. Supported datatypes are as follows: .RS .TP \fBtask=\fI<interval>\fR where \fI<interval>\fR is the task sampling interval in seconds for the jobacct_gather plugins and for task profiling by the acct_gather_profile plugin. NOTE: This frequency is used to monitor memory usage. If memory limits are enforced the highest frequency a user can request is what is configured in the slurm.conf file. They can not turn it off (=0) either. .TP \fBenergy=\fI<interval>\fR where \fI<interval>\fR is the sampling interval in seconds for energy profiling using the acct_gather_energy plugin .TP \fBnetwork=\fI<interval>\fR where \fI<interval>\fR is the sampling interval in seconds for infiniband profiling using the acct_gather_interconnect plugin. .TP \fBfilesystem=\fI<interval>\fR where \fI<interval>\fR is the sampling interval in seconds Smaller (non\-zero) values have a greater impact upon job performance, but a value of 30 seconds is not likely to be noticeable for applications having less than 10,000 tasks. .RE .TP \fB\-B\fR \fB\-\-extra\-node\-info\fR=<\fIsockets\fR[:\fIcores\fR[:\fIthreads\fR]]> Restrict node selection to nodes with at least the specified number of sockets, cores per socket and/or threads per core. NOTE: These options do not specify the resource allocation size. Each value specified is considered a minimum. An asterisk (*) can be used as a placeholder indicating that all available resources of that type are to be utilized. Values can also be specified as min-max. The individual levels can also be specified in separate options if desired: .nf \fB\-\-sockets\-per\-node\fR=<\fIsockets\fR> \fB\-\-cores\-per\-socket\fR=<\fIcores\fR> \fB\-\-threads\-per\-core\fR=<\fIthreads\fR> .fi If task/affinity plugin is enabled, then specifying an allocation in this manner also results in subsequently launched tasks being bound to threads if the \fB\-B\fR option specifies a thread count, otherwise an option of \fIcores\fR if a core count is specified, otherwise an option of \fIsockets\fR. If SelectType is configured to select/cons_res, it must have a parameter of CR_Core, CR_Core_Memory, CR_Socket, or CR_Socket_Memory for this option to be honored. If not specified, the scontrol show job will display 'ReqS:C:T=*:*:*'. This option applies to job allocations. \fBNOTE:\fR This option is mutually exclusive with \fB\-\-hint\fR, \fB\-\-threads\-per\-core\fR and \fB\-\-ntasks\-per\-core\fR. .TP \fB\-\-batch\fR=<\fIlist\fR> Nodes can have \fBfeatures\fR assigned to them by the Slurm administrator. Users can specify which of these \fBfeatures\fR are required by their batch script using this options. For example a job's allocation may include both Intel Haswell and KNL nodes with features "haswell" and "knl" respectively. On such a configuration the batch script would normally benefit by executing on a faster Haswell node. This would be specified using the option "\-\-batch=haswell". The specification can include AND and OR operators using the ampersand and vertical bar separators. For example: "\-\-batch=haswell|broadwell" or "\-\-batch=haswell|big_memory". The \-\-batch argument must be a subset of the job's \fB\-\-constraint\fR=<\fIlist\fR> argument (i.e. the job can not request only KNL nodes, but require the script to execute on a Haswell node). If the request can not be satisfied from the resources allocated to the job, the batch script will execute on the first node of the job allocation. .TP tell the controller to defer the allocation of the job until the specified time. Time may be of the form \fIHH:MM:SS\fR to run a job at a specific time of day (seconds are optional). (If that time is already past, the next day is assumed.) You may also specify \fImidnight\fR, \fInoon\fR, \fIfika\fR (3 PM) or \fIteatime\fR (4 PM) and you can have a time\-of\-day suffixed with \fIAM\fR or \fIPM\fR for running in the morning or the evening. You can also say what day the job will be run, by specifying a date of the form \fIMMDDYY\fR or \fIMM/DD/YY\fR \fIYYYY\-MM\-DD\fR. Combine date and time using the following format \fIYYYY\-MM\-DD[THH:MM[:SS]]\fR. You can also give times like \fInow + count time\-units\fR, where the time\-units can be \fIseconds\fR (default), \fIminutes\fR, \fIhours\fR, \fIdays\fR, or \fIweeks\fR and you can tell Slurm to run the job today with the keyword \fItoday\fR and to run the job tomorrow with the keyword \fItomorrow\fR. The value may be changed after job submission using the \fBscontrol\fR command. For example: .nf \-\-begin=16:00 \-\-begin=now+1hour \-\-begin=now+60 (seconds by default) \-\-begin=2010\-01\-20T12:34:00 .fi .RS .PP Notes on date/time specifications: \- Although the 'seconds' field of the HH:MM:SS time specification is allowed by the code, note that the poll time of the Slurm scheduler is not precise enough to guarantee dispatch of the job on the exact second. The job will be eligible to start on the next poll following the specified time. The exact poll interval depends on the Slurm scheduler (e.g., 60 seconds with the default sched/builtin). \- If no time (HH:MM:SS) is specified, the default is (00:00:00). \- If a date is specified without a year (e.g., MM/DD) then the current year is assumed, unless the combination of MM/DD and HH:MM:SS has already passed for that year, in which case the next year is used. .RE .TP \fB\-\-cluster\-constraint\fR=[!]<\fIlist\fR> Specifies features that a federated cluster must have to have a sibling job submitted to it. Slurm will attempt to submit a sibling job to a cluster if it has at least one of the specified features. If the "!" option is included, Slurm will attempt to submit a sibling job to a cluster that has none of the specified features. .TP \fB\-\-comment\fR=<\fIstring\fR> .RS .TP \fBSingle Name\fR Only nodes which have the specified feature will be used. For example, \fB\-\-constraint="intel"\fR .TP \fBNode Count\fR A request can specify the number of nodes needed with some feature by appending an asterisk and count after the feature name. For example, \fB\-\-nodes=16 \-\-constraint="graphics*4 ..."\fR indicates that the job requires 16 nodes and that at least four of those nodes must have the feature "graphics." .TP \fBAND\fR If only nodes with all of specified features will be used. The ampersand is used for an AND operator. For example, \fB\-\-constraint="intel&gpu"\fR .TP \fBOR\fR If only nodes with at least one of specified features will be used. The vertical bar is used for an OR operator. For example, \fB\-\-constraint="intel|amd"\fR .TP \fBMatching OR\fR If only one of a set of possible options should be used for all allocated nodes, then use the OR operator and enclose the options within square brackets. For example, \fB\-\-constraint="[rack1|rack2|rack3|rack4]"\fR might be used to specify that all nodes must be allocated on a single rack of the cluster, but any of those four racks can be used. .TP \fBMultiple Counts\fR Specific counts of multiple resources may be specified by using the AND operator and enclosing the options within square brackets. For example, \fB\-\-constraint="[rack1*2&rack2*4]"\fR might be used to specify that two nodes must be allocated from nodes with the feature of "rack1" and four nodes must be allocated from nodes with the feature "rack2". \fBNOTE:\fR This construct does not support multiple Intel KNL NUMA or MCDRAM modes. For example, while \fB\-\-constraint="[(knl&quad)*2&(knl&hemi)*4]"\fR is not supported, \fB\-\-constraint="[haswell*2&(knl&hemi)*4]"\fR is supported. Specification of multiple KNL modes requires the use of a heterogeneous job. .TP \fBBrackets\fR Brackets can be used to indicate that you are looking for a set of nodes with the different requirements contained within the brackets. For example, \fB--constraint="[(rack1|rack2)*1&(rack3)*2]"\fR will get you one node with either the "rack1" or "rack2" features and two nodes with the "rack3" feature. The same request without the brackets will try to find a single node that meets those requirements. .TP \fBParenthesis\fR .TP \fB\-\-cores\-per\-socket\fR=<\fIcores\fR> Restrict node selection to nodes with at least the specified number of cores per socket. See additional information under \fB\-B\fR option above when task/affinity plugin is enabled. .TP \fB\-\-cpu\-freq\fR =<\fIp1\fR[\-\fIp2\fR[:\fIp3\fR]]> Request that job steps initiated by srun commands inside this sbatch script be run at some requested frequency if possible, on the CPUs selected for the step on the compute node(s). \fBp1\fR can be [#### | low | medium | high | highm1] which will set the frequency scaling_speed to the corresponding value, and set the frequency scaling_governor to UserSpace. See below for definition of the values. \fBp1\fR can be [Conservative | OnDemand | Performance | PowerSave] which will set the scaling_governor to the corresponding value. The governor has to be in the list set by the slurm.conf option CpuFreqGovernors. When \fBp2\fR is present, p1 will be the minimum scaling frequency and p2 will be the maximum scaling frequency. \fBp2\fR can be [#### | medium | high | highm1] p2 must be greater than p1. \fBp3\fR can be [Conservative | OnDemand | Performance | PowerSave | UserSpace] which will set the governor to the corresponding value. If \fBp3\fR is UserSpace, the frequency scaling_speed will be set by a power or energy aware scheduling strategy to a value between p1 and p2 that lets the job run within the site's power goal. The job may be delayed if p1 is higher than a frequency that allows the job to run within the goal. If the current frequency is < min, it will be set to min. Likewise, if the current frequency is > max, it will be set to max. Acceptable values at present include: .RS .TP 14 \fB####\fR frequency in kilohertz .TP \fBLow\fR the lowest available frequency .TP \fBHigh\fR the highest available frequency .TP \fBHighM1\fR (high minus one) will select the next highest available frequency attempts to use the PowerSave CPU governor .TP \fBUserSpace\fR attempts to use the UserSpace CPU governor .TP .RE The following informational environment variable is set in the job step when \fB\-\-cpu\-freq\fR option is requested. .nf SLURM_CPU_FREQ_REQ .fi This environment variable can also be used to supply the value for the CPU frequency request if it is set when the 'srun' command is issued. The \fB\-\-cpu\-freq\fR on the command line will override the environment variable value. The form on the environment variable is the same as the command line. See the \fBENVIRONMENT VARIABLES\fR section for a description of the SLURM_CPU_FREQ_REQ variable. \fBNOTE\fR: This parameter is treated as a request, not a requirement. If the job step's node does not support setting the CPU frequency, or the requested value is outside the bounds of the legal frequencies, an error is logged, but the job step is allowed to continue. \fBNOTE\fR: Setting the frequency for just the CPUs of the job step implies that the tasks are confined to those CPUs. If task confinement (i.e., TaskPlugin=task/affinity or TaskPlugin=task/cgroup with the "ConstrainCores" option) is not configured, this parameter is ignored. \fBNOTE\fR: When the step completes, the frequency and governor of each selected CPU is reset to the previous values. \fBNOTE\fR: When submitting jobs with the \fB\-\-cpu\-freq\fR option with linuxproc as the ProctrackType can cause jobs to run too quickly before Accounting is able to poll for job information. As a result not all of accounting information will be present. .RE .TP \fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR> Advise Slurm that ensuing job steps will require \fIncpus\fR processors per allocated GPU. Not compatible with the \fB\-\-cpus\-per\-task\fR option. .TP \fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR> Advise the Slurm controller that ensuing job steps will require \fIncpus\fR number of processors per task. Without this option, the controller will just try to allocate one processor per task. .br HH:MM[:SS] [AM|PM] .br MMDD[YY] or MM/DD[/YY] or MM.DD[.YY] .br MM/DD[/YY]\-HH:MM[:SS] .br YYYY\-MM\-DD[THH:MM[:SS]]] .br now[+\fIcount\fR[seconds(default)|minutes|hours|days|weeks]] .TP \fB\-\-delay\-boot\fR=<\fIminutes\fR> Do not reboot nodes in order to satisfied this job's feature specification if the job has been eligible to run for less than this time period. If the job has waited for less than the specified period, it will use only nodes which already have the specified features. The argument is in units of minutes. A default value may be set by a system administrator using the \fBdelay_boot\fR option of the \fBSchedulerParameters\fR configuration parameter in the slurm.conf file, otherwise the default value is zero (no delay). .TP \fB\-d\fR, \fB\-\-dependency\fR=<\fIdependency_list\fR> Defer the start of this job until the specified dependencies have been satisfied completed. <\fIdependency_list\fR> is of the form <\fItype:job_id[:job_id][,type:job_id[:job_id]]\fR> or <\fItype:job_id[:job_id][?type:job_id[:job_id]]\fR>. All dependencies must be satisfied if the "," separator is used. Any dependency may be satisfied if the "?" separator is used. Only one separator may be used. Many jobs can share the same dependency and these jobs may even belong to different users. The value may be changed after job submission using the scontrol command. Dependencies on remote jobs are allowed in a federation. Once a job dependency fails due to the termination state of a preceding job, the dependent job will never be run, even if the preceding job is requeued and has a different termination state in a subsequent execution. .PD .RS .TP \fBafter:job_id[[+time][:jobid[+time]...]]\fR After the specified jobs start or are cancelled and 'time' in minutes from job start or cancellation happens, this job can begin execution. If no 'time' is given then there is no delay after start or cancellation. .TP \fBafterany:job_id[:jobid...]\fR This job can begin execution after the specified jobs have terminated. .TP \fBafterburstbuffer:job_id[:jobid...]\fR executed (ran to completion with an exit code of zero). .TP \fBexpand:job_id\fR Resources allocated to this job should be used to expand the specified job. The job to expand must share the same QOS (Quality of Service) and partition. Gang scheduling of resources in the partition is also not supported. "expand" is not allowed for jobs that didn't originate on the same cluster as the submitted job. .TP \fBsingleton\fR This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time. In a federation, a singleton dependency must be fulfilled on all clusters unless DependencyParameters=disable_remote_singleton is used in slurm.conf. .RE .TP \fB\-D\fR, \fB\-\-chdir\fR=<\fIdirectory\fR> Set the working directory of the batch script to \fIdirectory\fR before it is executed. The path can be specified as full path or relative path to the directory where the command is executed. .TP \fB\-e\fR, \fB\-\-error\fR=<\fIfilename pattern\fR> Instruct Slurm to connect the batch script's standard error directly to the file name specified in the "\fIfilename pattern\fR". By default both standard output and standard error are directed to the same file. For job arrays, the default file name is "slurm-%A_%a.out", "%A" is replaced by the job ID and "%a" with the array index. For other jobs, the default file name is "slurm-%j.out", where the "%j" is replaced by the job ID. See the \fBfilename pattern\fR section below for filename specification options. .TP \fB\-\-exclusive[=user|mcs]\fR The job allocation can not share nodes with other running jobs (or just other users with the "=user" option or with the "=mcs" option). The default shared/exclusive behavior depends on system configuration and the partition's \fBOverSubscribe\fR option takes precedence over the job's option. .TP \fB\-\-export\fR=<\fI[ALL,]environment variables\fR|\fIALL\fR|\fINONE\fR> Identify which environment variables from the submission environment are propagated to the launched application. Note that SLURM_* variables are always propagated. .RS .TP 10 \fB\-\-export\fR=\fIALL\fR .br Default mode if \fB\-\-export\fR is not specified. All of the users environment \fIALL\fR in the job script. .TP \fB\-\-export\fR=<\fI[ALL,]environment variables>\fR .br Exports all SLURM_* environment variables along with explicitly defined variables. Multiple environment variable names should be comma separated. Environment variable names may be specified to propagate the current value (e.g. "\-\-export=EDITOR") or specific values may be exported (e.g. "\-\-export=EDITOR=/bin/emacs"). If \fIALL\fR is specified, then all user environment variables will be loaded and will take precedence over any explicitly given environment variables. .RS 5 .TP 5 Example: \fB\-\-export\fR=EDITOR,ARG1=test .br In this example, the propagated environment will only contain the variable \fIEDITOR\fR from the user's environment, \fISLURM_*\fR environment variables, and \fIARG1\fR=test. .TP Example: \fB\-\-export\fR=ALL,EDITOR=/bin/emacs .br There are two possible outcomes for this example. If the caller has the \fIEDITOR\fR environment variable defined, then the job's environment will inherit the variable from the caller's environment. If the caller doesn't have an environment variable defined for \fIEDITOR\fR, then the job's environment will use the value given by \fB--export\fR. .RE .RE .TP \fB\-\-export\-file\fR=<\fIfilename\fR | \fIfd\fR> If a number between 3 and OPEN_MAX is specified as the argument to this option, a readable file descriptor will be assumed (STDIN and STDOUT are not supported as valid arguments). Otherwise a filename is assumed. Export environment variables defined in <\fIfilename\fR> or read from <\fIfd\fR> to the job's execution environment. The content is one or more environment variable definitions of the form NAME=value, each separated by a null character. This allows the use of special characters in environment definitions. .TP \fB\-F\fR, \fB\-\-nodefile\fR=<\fInode file\fR> Much like \-\-nodelist, but the list is contained in a file of name \fInode file\fR. The node names of the list may also span multiple lines in the file. Duplicate node names in the file will be ignored. The order of the node names in the list is not important; the node names will be sorted by Slurm. .TP \fB\-\-get\-user\-env\fR[=\fItimeout\fR][\fImode\fR] This option will tell sbatch to retrieve the login environment variables for the user specified in the \fB\-\-uid\fR option. "\-\-get\-user\-env=10L", and "\-\-get\-user\-env=S". .TP \fB\-\-gid\fR=<\fIgroup\fR> If \fBsbatch\fR is run as root, and the \fB\-\-gid\fR option is used, submit the job with \fIgroup\fR's group access permissions. \fIgroup\fR may be the group name or the numerical group ID. .TP \fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR> Specify the total number of GPUs required for the job. An optional GPU type specification can be supplied. For example "\-\-gpus=volta:3". Multiple options can be requested in a comma separated list, for example: "\-\-gpus=volta:3,kepler:1". See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and \fB\-\-gpus\-per\-task\fR options. .TP \fB\-\-gpu\-bind\fR=[verbose,]<\fItype\fR> Bind tasks to specific GPUs. By default every spawned task can access every GPU allocated to the job. If "verbose," is specified before <\fItype\fR>, then print out GPU binding information. Supported \fItype\fR options: .RS .TP 10 \fBclosest\fR Bind each task to the GPU(s) which are closest. In a NUMA environment, each task may be bound to more than one GPU (i.e. all GPUs in that NUMA environment). .TP \fBmap_gpu:<list>\fR Bind by setting GPU masks on tasks (or ranks) as specified where <list> is#!/bin/sh .br > srun hostname |sort .br > EOF .br sbatch: Submitted batch job 65541 .br .br $ cat slurm\-65541.out .br host1 .br host2 .br host3 .br host4 .LP To create a heterogeneous job with 3 components, each allocating a unique set This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>. .LP Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. .LP Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. .SH "SEE ALSO" .LP \fBsinfo\fR(1), \fBsattach\fR(1), \fBsalloc\fR(1), \fBsqueue\fR(1), \fBscancel\fR(1), \fBscontrol\fR(1), \fBslurm.conf\fR(5), \fBsched_setaffinity\fR (2), \fBnuma\fR (3)