cpuset man page on Archlinux

Man page or keyword search:  
man Server   11224 pages
apropos Keyword Search (all sections)
Output format
Archlinux logo
[printable version]

CPUSET(7)		   Linux Programmer's Manual		     CPUSET(7)

       cpuset - confine processes to processor and memory node subsets

       The  cpuset  filesystem	is a pseudo-filesystem interface to the kernel
       cpuset mechanism, which is used to control the processor placement  and
       memory placement of processes.  It is commonly mounted at /dev/cpuset.

       On systems with kernels compiled with built in support for cpusets, all
       processes are attached to a cpuset, and cpusets are always present.  If
       a  system supports cpusets, then it will have the entry nodev cpuset in
       the file /proc/filesystems.  By mounting the cpuset filesystem (see the
       EXAMPLE	section below), the administrator can configure the cpusets on
       a system to control the processor and memory placement of processes  on
       that  system.   By  default, if the cpuset configuration on a system is
       not modified or if the cpuset filesystem is not even mounted, then  the
       cpuset  mechanism, though present, has no affect on the system's behav‐

       A cpuset defines a list of CPUs and memory nodes.

       The CPUs of a system include all the logical processing units on	 which
       a  process can execute, including, if present, multiple processor cores
       within a package and Hyper-Threads within  a  processor	core.	Memory
       nodes  include all distinct banks of main memory; small and SMP systems
       typically have just one memory node that contains all the system's main
       memory,	while  NUMA  (non-uniform memory access) systems have multiple
       memory nodes.

       Cpusets are  represented	 as  directories  in  a	 hierarchical  pseudo-
       filesystem, where the top directory in the hierarchy (/dev/cpuset) rep‐
       resents the entire system (all online CPUs and memory  nodes)  and  any
       cpuset that is the child (descendant) of another parent cpuset contains
       a subset of that parent's CPUs and memory nodes.	 The  directories  and
       files representing cpusets have normal filesystem permissions.

       Every  process  in the system belongs to exactly one cpuset.  A process
       is confined to run only on the CPUs in the cpuset it belongs to, and to
       allocate	 memory	 only  on  the	memory	nodes  in that cpuset.	When a
       process fork(2)s, the child process is placed in the same cpuset as its
       parent.	 With  sufficient  privilege,  a process may be moved from one
       cpuset to another and the allowed CPUs and memory nodes of an  existing
       cpuset may be changed.

       When  the  system  begins  booting,  a  single  cpuset  is defined that
       includes all CPUs and memory nodes on the system, and all processes are
       in that cpuset.	During the boot process, or later during normal system
       operation, other cpusets may be created, as subdirectories of this  top
       cpuset,	under  the  control of the system administrator, and processes
       may be placed in these other cpusets.

       Cpusets are integrated with the sched_setaffinity(2) scheduling	affin‐
       ity  mechanism  and  the mbind(2) and set_mempolicy(2) memory-placement
       mechanisms in the kernel.  Neither of these mechanisms  let  a  process
       make  use of a CPU or memory node that is not allowed by that process's
       cpuset.	If changes to a process's cpuset placement conflict with these
       other  mechanisms,  then	 cpuset placement is enforced even if it means
       overriding these other mechanisms.  The kernel accomplishes this	 over‐
       riding  by  silently restricting the CPUs and memory nodes requested by
       these other mechanisms to  those	 allowed  by  the  invoking  process's
       cpuset.	 This  can  result in these other calls returning an error, if
       for example, such a call ends up requesting an empty  set  of  CPUs  or
       memory  nodes,  after  that  request  is	 restricted  to	 the  invoking
       process's cpuset.

       Typically, a cpuset is used to manage the CPU and memory-node  confine‐
       ment  for a set of cooperating processes such as a batch scheduler job,
       and these other mechanisms are used to manage the placement of individ‐
       ual processes or memory regions within that set or job.

       Each  directory	below  /dev/cpuset  represents a cpuset and contains a
       fixed set of pseudo-files describing the state of that cpuset.

       New cpusets are created using the mkdir(2) system call or the  mkdir(1)
       command.	  The  properties of a cpuset, such as its flags, allowed CPUs
       and memory nodes, and attached processes, are queried and  modified  by
       reading	or writing to the appropriate file in that cpuset's directory,
       as listed below.

       The pseudo-files in each cpuset	directory  are	automatically  created
       when the cpuset is created, as a result of the mkdir(2) invocation.  It
       is not possible to directly add or remove these pseudo-files.

       A cpuset directory that contains no child cpuset directories,  and  has
       no  attached  processes, can be removed using rmdir(2) or rmdir(1).  It
       is not necessary, or possible, to remove the  pseudo-files  inside  the
       directory before removing it.

       The pseudo-files in each cpuset directory are small text files that may
       be read and written using traditional shell utilities such  as  cat(1),
       and  echo(1),  or from a program by using file I/O library functions or
       system calls, such as open(2), read(2), write(2), and close(2).

       The pseudo-files in a cpuset directory represent internal kernel	 state
       and do not have any persistent image on disk.  Each of these per-cpuset
       files is listed and described below.

       tasks  List of the process IDs (PIDs) of the processes in that  cpuset.
	      The list is formatted as a series of ASCII decimal numbers, each
	      followed by a newline.  A process	 may  be  added	 to  a	cpuset
	      (automatically  removing it from the cpuset that previously con‐
	      tained it) by writing its PID to that cpuset's tasks file	 (with
	      or without a trailing newline.)

	      Warning:	only  one  PID	may  be written to the tasks file at a
	      time.  If a string is written that contains more than  one  PID,
	      only the first one will be used.

	      Flag  (0	or  1).	  If set (1), that cpuset will receive special
	      handling after it is released,  that  is,	 after	all  processes
	      cease  using  it	(i.e.,	terminate  or are moved to a different
	      cpuset) and all child cpuset directories have been removed.  See
	      the Notify On Release section, below.

	      List  of	the physical numbers of the CPUs on which processes in
	      that cpuset are allowed to execute.  See List Format below for a
	      description of the format of cpus.

	      The  CPUs	 allowed  to  a cpuset may be changed by writing a new
	      list to its cpus file.

	      Flag (0 or 1).  If set (1), the cpuset has exclusive use of  its
	      CPUs (no sibling or cousin cpuset may overlap CPUs).  By default
	      this is off (0).	Newly created cpusets also  initially  default
	      this to off (0).

	      Two  cpusets  are	 sibling cpusets if they share the same parent
	      cpuset in the /dev/cpuset hierarchy.   Two  cpusets  are	cousin
	      cpusets  if neither is the ancestor of the other.	 Regardless of
	      the cpu_exclusive setting, if one	 cpuset	 is  the  ancestor  of
	      another,	and  if both of these cpusets have nonempty cpus, then
	      their cpus must overlap, because the  cpus  of  any  cpuset  are
	      always a subset of the cpus of its parent cpuset.

	      List  of	memory	nodes  on  which  processes in this cpuset are
	      allowed to  allocate  memory.   See  List	 Format	 below	for  a
	      description of the format of mems.

	      Flag  (0 or 1).  If set (1), the cpuset has exclusive use of its
	      memory nodes (no sibling or cousin may overlap).	 Also  if  set
	      (1),  the	 cpuset	 is a Hardwall cpuset (see below.)  By default
	      this is off (0).	Newly created cpusets also  initially  default
	      this to off (0).

	      Regardless  of  the  mem_exclusive setting, if one cpuset is the
	      ancestor of another,  then  their	 memory	 nodes	must  overlap,
	      because  the  memory  nodes of any cpuset are always a subset of
	      the memory nodes of that cpuset's parent cpuset.

       cpuset.mem_hardwall (since Linux 2.6.26)
	      Flag (0 or 1).  If set (1), the cpuset is a Hardwall cpuset (see
	      below.)  Unlike mem_exclusive, there is no constraint on whether
	      cpusets marked mem_hardwall may have  overlapping	 memory	 nodes
	      with  sibling  or	 cousin	 cpusets.  By default this is off (0).
	      Newly created cpusets also initially default this to off (0).

       cpuset.memory_migrate (since Linux 2.6.16)
	      Flag (0 or 1).  If set (1), then memory  migration  is  enabled.
	      By  default  this is off (0).  See the Memory Migration section,

       cpuset.memory_pressure (since Linux 2.6.16)
	      A measure of how much memory  pressure  the  processes  in  this
	      cpuset  are  causing.   See  the Memory Pressure section, below.
	      Unless memory_pressure_enabled is enabled, always has value zero
	      (0).  This file is read-only.  See the WARNINGS section, below.

       cpuset.memory_pressure_enabled (since Linux 2.6.16)
	      Flag  (0	or  1).	 This file is present only in the root cpuset,
	      normally /dev/cpuset.  If set (1), the memory_pressure  calcula‐
	      tions  are  enabled  for	all cpusets in the system.  By default
	      this is off (0).	See the Memory Pressure section, below.

       cpuset.memory_spread_page (since Linux 2.6.17)
	      Flag (0 or 1).  If set (1),  pages  in  the  kernel  page	 cache
	      (filesystem buffers) are uniformly spread across the cpuset.  By
	      default this is off (0) in the top cpuset,  and  inherited  from
	      the  parent  cpuset  in  newly  created cpusets.	See the Memory
	      Spread section, below.

       cpuset.memory_spread_slab (since Linux 2.6.17)
	      Flag (0 or 1).  If set (1), the kernel slab caches for file  I/O
	      (directory and inode structures) are uniformly spread across the
	      cpuset.  By default this is off  (0)  in	the  top  cpuset,  and
	      inherited	 from the parent cpuset in newly created cpusets.  See
	      the Memory Spread section, below.

       cpuset.sched_load_balance (since Linux 2.6.24)
	      Flag (0 or 1).  If set (1, the default) the kernel will automat‐
	      ically  load  balance  processes in that cpuset over the allowed
	      CPUs in that cpuset.  If cleared (0) the kernel will avoid  load
	      balancing	 processes  in	this  cpuset, unless some other cpuset
	      with overlapping CPUs has its sched_load_balance flag set.   See
	      Scheduler Load Balancing, below, for further details.

       cpuset.sched_relax_domain_level (since Linux 2.6.26)
	      Integer,	 between   -1	and   a	 small	positive  value.   The
	      sched_relax_domain_level controls the width of the range of CPUs
	      over  which  the kernel scheduler performs immediate rebalancing
	      of runnable tasks across CPUs.  If  sched_load_balance  is  dis‐
	      abled,  then  the	 setting  of sched_relax_domain_level does not
	      matter, as no such load balancing is done.   If  sched_load_bal‐
	      ance   is	  enabled,   then   the	  higher   the	value  of  the
	      sched_relax_domain_level, the wider the range of CPUs over which
	      immediate	 load  balancing  is  attempted.   See Scheduler Relax
	      Domain Level, below, for further details.

       In  addition  to	 the  above  pseudo-files  in  each  directory	 below
       /dev/cpuset,  each  process has a pseudo-file, /proc/<pid>/cpuset, that
       displays the path of the process's cpuset  directory  relative  to  the
       root of the cpuset filesystem.

       Also the /proc/<pid>/status file for each process has four added lines,
       displaying the process's Cpus_allowed (on which CPUs it may  be	sched‐
       uled) and Mems_allowed (on which memory nodes it may obtain memory), in
       the two formats Mask Format and List Format (see below) as shown in the
       following example:

	      Cpus_allowed:   ffffffff,ffffffff,ffffffff,ffffffff
	      Cpus_allowed_list:     0-127
	      Mems_allowed:   ffffffff,ffffffff
	      Mems_allowed_list:     0-63

       The  "allowed"  fields  were  added in Linux 2.6.24; the "allowed_list"
       fields were added in Linux 2.6.26.

       In addition to controlling which cpus and mems a process is allowed  to
       use, cpusets provide the following extended capabilities.

   Exclusive cpusets
       If  a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset,
       other than a direct ancestor or descendant, may share any of  the  same
       CPUs or memory nodes.

       A  cpuset that is mem_exclusive restricts kernel allocations for buffer
       cache pages and other internal kernel data pages commonly shared by the
       kernel  across  multiple	 users.	 All cpusets, whether mem_exclusive or
       not, restrict allocations of memory for user space.  This enables  con‐
       figuring	 a  system  so	that several independent jobs can share common
       kernel data, while isolating each job's	user  allocation  in  its  own
       cpuset.	To do this, construct a large mem_exclusive cpuset to hold all
       the jobs, and construct child, non-mem_exclusive cpusets for each indi‐
       vidual  job.   Only  a  small amount of kernel memory, such as requests
       from interrupt handlers, is allowed to be placed on memory  nodes  out‐
       side even a mem_exclusive cpuset.

       A  cpuset  that	has  mem_exclusive  or	mem_hardwall set is a hardwall
       cpuset.	A hardwall cpuset restricts kernel allocations for page,  buf‐
       fer,  and  other	 data  commonly	 shared	 by the kernel across multiple
       users.  All cpusets, whether hardwall or not, restrict  allocations  of
       memory for user space.

       This  enables configuring a system so that several independent jobs can
       share common kernel data, such as  filesystem  pages,  while  isolating
       each  job's user allocation in its own cpuset.  To do this, construct a
       large hardwall cpuset to hold all the jobs, and construct child cpusets
       for each individual job which are not hardwall cpusets.

       Only  a	small amount of kernel memory, such as requests from interrupt
       handlers, is allowed to be taken outside even a hardwall cpuset.

   Notify on release
       If the notify_on_release flag is enabled (1) in a cpuset, then whenever
       the  last process in the cpuset leaves (exits or attaches to some other
       cpuset) and the last child cpuset of that cpuset is removed, the kernel
       will run the command /sbin/cpuset_release_agent, supplying the pathname
       (relative to the mount point of the cpuset filesystem) of the abandoned
       cpuset.	This enables automatic removal of abandoned cpusets.

       The  default  value  of	notify_on_release in the root cpuset at system
       boot is disabled (0).  The default value of other cpusets  at  creation
       is the current value of their parent's notify_on_release setting.

       The  command  /sbin/cpuset_release_agent	 is  invoked,  with  the  name
       (/dev/cpuset relative path) of the to-be-released cpuset in argv[1].

       The usual contents of the command /sbin/cpuset_release_agent is	simply
       the shell script:

	   rmdir /dev/cpuset/$1

       As with other flag values below, this flag can be changed by writing an
       ASCII number 0 or 1 (with optional trailing newline) into the file,  to
       clear or set the flag, respectively.

   Memory pressure
       The  memory_pressure  of	 a cpuset provides a simple per-cpuset running
       average of the rate that the processes in a cpuset  are	attempting  to
       free  up in-use memory on the nodes of the cpuset to satisfy additional
       memory requests.

       This enables batch managers that are monitoring jobs running  in	 dedi‐
       cated  cpusets to efficiently detect what level of memory pressure that
       job is causing.

       This is useful both on tightly managed systems running a	 wide  mix  of
       submitted jobs, which may choose to terminate or reprioritize jobs that
       are trying to use more memory than allowed on the nodes assigned	 them,
       and  with  tightly coupled, long-running, massively parallel scientific
       computing jobs that will dramatically fail to meet required performance
       goals if they start to use more memory than allowed to them.

       This  mechanism provides a very economical way for the batch manager to
       monitor a cpuset for signs of memory pressure.  It's up	to  the	 batch
       manager	or other user code to decide what action to take if it detects
       signs of memory pressure.

       Unless memory pressure calculation is enabled by	 setting  the  pseudo-
       file /dev/cpuset/cpuset.memory_pressure_enabled, it is not computed for
       any cpuset, and reads from any memory_pressure always return  zero,  as
       represented  by	the  ASCII  string  "0\n".   See the WARNINGS section,

       A per-cpuset, running average is employed for the following reasons:

       *  Because this meter is per-cpuset rather than per-process or per vir‐
	  tual	memory	region,	 the  system load imposed by a batch scheduler
	  monitoring this metric is sharply reduced on large systems,  because
	  a scan of the tasklist can be avoided on each set of queries.

       *  Because  this meter is a running average rather than an accumulating
	  counter, a batch scheduler can detect memory pressure with a	single
	  read,	 instead of having to read and accumulate results for a period
	  of time.

       *  Because this meter is per-cpuset rather than per-process, the	 batch
	  scheduler  can  obtain  the  key  information—memory	pressure  in a
	  cpuset—with a single read, rather than having to query  and  accumu‐
	  late results over all the (dynamically changing) set of processes in
	  the cpuset.

       The memory_pressure of a cpuset is calculated using a per-cpuset simple
       digital	filter	that is kept within the kernel.	 For each cpuset, this
       filter tracks the recent rate  at  which	 processes  attached  to  that
       cpuset enter the kernel direct reclaim code.

       The  kernel  direct  reclaim  code is entered whenever a process has to
       satisfy a memory page request by	 first	finding	 some  other  page  to
       repurpose,  due	to  lack  of any readily available already free pages.
       Dirty filesystem pages are repurposed by first writing  them  to	 disk.
       Unmodified  filesystem  buffer  pages are repurposed by simply dropping
       them, though if that page is needed again, it will have	to  be	reread
       from disk.

       The cpuset.memory_pressure file provides an integer number representing
       the recent (half-life of 10 seconds) rate  of  entries  to  the	direct
       reclaim	code caused by any process in the cpuset, in units of reclaims
       attempted per second, times 1000.

   Memory spread
       There are two Boolean flag files per cpuset that control where the ker‐
       nel  allocates  pages  for the filesystem buffers and related in-kernel
       data  structures.   They	 are  called   cpuset.memory_spread_page   and

       If  the	per-cpuset Boolean flag file cpuset.memory_spread_page is set,
       then the kernel will spread the filesystem buffers (page cache)	evenly
       over all the nodes that the faulting process is allowed to use, instead
       of preferring to put those pages on the node where the process is  run‐

       If  the	per-cpuset Boolean flag file cpuset.memory_spread_slab is set,
       then the kernel will spread some filesystem-related slab	 caches,  such
       as  those  for  inodes and directory entries, evenly over all the nodes
       that the faulting process is allowed to use, instead of	preferring  to
       put those pages on the node where the process is running.

       The  setting  of	 these	flags  does  not  affect the data segment (see
       brk(2)) or stack segment pages of a process.

       By default, both kinds of memory	 spreading  are	 off  and  the	kernel
       prefers	to  allocate  memory  pages  on	 the  node  local to where the
       requesting process is running.  If that node  is	 not  allowed  by  the
       process's  NUMA	memory	policy or cpuset configuration or if there are
       insufficient free memory pages on that node, then the kernel looks  for
       the nearest node that is allowed and has sufficient free memory.

       When  new  cpusets are created, they inherit the memory spread settings
       of their parent.

       Setting memory spreading causes allocations for the  affected  page  or
       slab  caches  to	 ignore the process's NUMA memory policy and be spread
       instead.	 However, the effect of	 these	changes	 in  memory  placement
       caused by cpuset-specified memory spreading is hidden from the mbind(2)
       or set_mempolicy(2) calls.  These two NUMA memory policy	 calls	always
       appear  to  behave  as  if  no  cpuset-specified memory spreading is in
       effect, even if it is.  If  cpuset  memory  spreading  is  subsequently
       turned  off,  the  NUMA	memory policy most recently specified by these
       calls is automatically reapplied.

       Both cpuset.memory_spread_page and cpuset.memory_spread_slab are	 Bool‐
       ean  flag files.	 By default they contain "0", meaning that the feature
       is off for that cpuset.	If a "1" is written to that file,  that	 turns
       the named feature on.

       Cpuset-specified	 memory	 spreading  behaves similarly to what is known
       (in other contexts) as round-robin or interleave memory placement.

       Cpuset-specified memory spreading can provide  substantial  performance
       improvements for jobs that:

       a) need	to  place  thread-local data on memory nodes close to the CPUs
	  which are running the threads that most frequently access that data;
	  but also

       b) need	to  access  large  filesystem data sets that must to be spread
	  across the several nodes in the job's cpuset in order to fit.

       Without this policy, the memory allocation  across  the	nodes  in  the
       job's  cpuset  can  become  very uneven, especially for jobs that might
       have just a single thread initializing or reading in the data set.

   Memory migration
       Normally,  under	 the  default  setting	 (disabled)   of   cpuset.mem‐
       ory_migrate,  once  a  page is allocated (given a physical page of main
       memory) then that page stays on whatever node it was allocated, so long
       as  it  remains allocated, even if the cpuset's memory-placement policy
       mems subsequently changes.

       When memory migration is enabled in a cpuset, if the  mems  setting  of
       the  cpuset  is	changed, then any memory page in use by any process in
       the cpuset that is on a memory node that is no longer allowed  will  be
       migrated to a memory node that is allowed.

       Furthermore,  if	 a  process is moved into a cpuset with memory_migrate
       enabled, any memory pages it uses that were on memory nodes allowed  in
       its  previous cpuset, but which are not allowed in its new cpuset, will
       be migrated to a memory node allowed in the new cpuset.

       The relative placement of a migrated page within	 the  cpuset  is  pre‐
       served  during these migration operations if possible.  For example, if
       the page was on the second valid node of the  prior  cpuset,  then  the
       page will be placed on the second valid node of the new cpuset, if pos‐

   Scheduler load balancing
       The kernel scheduler automatically load balances processes.  If one CPU
       is  underutilized,  the	kernel	will  look for processes on other more
       overloaded CPUs and move those  processes  to  the  underutilized  CPU,
       within  the  constraints	 of  such  placement mechanisms as cpusets and

       The algorithmic cost of load balancing and its  impact  on  key	shared
       kernel  data  structures	 such  as the process list increases more than
       linearly with the number of CPUs being balanced.	 For example, it costs
       more  to load balance across one large set of CPUs than it does to bal‐
       ance across two smaller sets of CPUs, each of  half  the	 size  of  the
       larger set.  (The precise relationship between the number of CPUs being
       balanced and the cost  of  load	balancing  depends  on	implementation
       details	of  the	 kernel	 process scheduler, which is subject to change
       over time, as improved kernel scheduler algorithms are implemented.)

       The per-cpuset flag sched_load_balance provides a mechanism to suppress
       this automatic scheduler load balancing in cases where it is not needed
       and suppressing it would have worthwhile performance benefits.

       By default, load balancing is done across all CPUs, except those marked
       isolated	 using the kernel boot time "isolcpus=" argument.  (See Sched‐
       uler Relax Domain Level, below, to change this default.)

       This default load balancing across all CPUs is not well suited  to  the
       following two situations:

       *  On  large systems, load balancing across many CPUs is expensive.  If
	  the system is managed using cpusets to  place	 independent  jobs  on
	  separate sets of CPUs, full load balancing is unnecessary.

       *  Systems  supporting  real-time  on some CPUs need to minimize system
	  overhead on those CPUs, including avoiding process load balancing if
	  that is not needed.

       When  the  per-cpuset  flag  sched_load_balance is enabled (the default
       setting), it requests load  balancing  across  all  the	CPUs  in  that
       cpuset's	 allowed CPUs, ensuring that load balancing can move a process
       (not otherwise pinned, as by sched_setaffinity(2)) from any CPU in that
       cpuset to any other.

       When  the  per-cpuset  flag  sched_load_balance	is  disabled, then the
       scheduler will avoid load balancing across the  CPUs  in	 that  cpuset,
       except  in  so  far as is necessary because some overlapping cpuset has
       sched_load_balance enabled.

       So, for example, if the top  cpuset  has	 the  flag  sched_load_balance
       enabled,	 then the scheduler will load balance across all CPUs, and the
       setting of the sched_load_balance flag in other cpusets has no  effect,
       as we're already fully load balancing.

       Therefore  in  the  above  two  situations, the flag sched_load_balance
       should be disabled in the top cpuset, and only  some  of	 the  smaller,
       child cpusets would have this flag enabled.

       When doing this, you don't usually want to leave any unpinned processes
       in the top cpuset that might use nontrivial amounts  of	CPU,  as  such
       processes  may  be  artificially	 constrained  to  some subset of CPUs,
       depending on  the  particulars  of  this	 flag  setting	in  descendant
       cpusets.	  Even	if  such  a process could use spare CPU cycles in some
       other CPUs, the kernel scheduler might not consider the possibility  of
       load balancing that process to the underused CPU.

       Of course, processes pinned to a particular CPU can be left in a cpuset
       that disables sched_load_balance as those processes aren't  going  any‐
       where else anyway.

   Scheduler relax domain level
       The  kernel  scheduler performs immediate load balancing whenever a CPU
       becomes free or another task becomes  runnable.	 This  load  balancing
       works  to  ensure  that	as many CPUs as possible are usefully employed
       running tasks.  The kernel also performs periodic  load	balancing  off
       the   software	clock	described   in	 time(7).    The   setting  of
       sched_relax_domain_level applies	 only  to  immediate  load  balancing.
       Regardless  of the sched_relax_domain_level setting, periodic load bal‐
       ancing is attempted over all  CPUs  (unless  disabled  by  turning  off
       sched_load_balance.)   In  any case, of course, tasks will be scheduled
       to  run	only  on  CPUs	allowed	 by  their  cpuset,  as	 modified   by
       sched_setaffinity(2) system calls.

       On  small  systems,  such as those with just a few CPUs, immediate load
       balancing is useful to improve system  interactivity  and  to  minimize
       wasteful	 idle  CPU cycles.  But on large systems, attempting immediate
       load balancing across a large number of CPUs can be more costly than it
       is  worth,  depending  on the particular performance characteristics of
       the job mix and the hardware.

       The   exact    meaning	 of    the    small    integer	  values    of
       sched_relax_domain_level will depend on internal implementation details
       of the kernel scheduler code and on the non-uniform architecture of the
       hardware.   Both	 of  these  will  evolve  over time and vary by system
       architecture and kernel version.

       As of this writing,  when  this	capability  was	 introduced  in	 Linux
       2.6.26,	on  certain  popular  architectures,  the  positive  values of
       sched_relax_domain_level have the following meanings.

       (1) Perform immediate load balancing across  Hyper-Thread  siblings  on
	   the same core.
       (2) Perform  immediate  load  balancing	across other cores in the same
       (3) Perform immediate load balancing across other CPUs on the same node
	   or blade.
       (4) Perform  immediate  load balancing across over several (implementa‐
	   tion detail) nodes [On NUMA systems].
       (5) Perform immediate load balancing across over all CPUs in system [On
	   NUMA systems].

       The  sched_relax_domain_level value of zero (0) always means don't per‐
       form immediate load balancing, hence that load balancing is  done  only
       periodically,  not  immediately when a CPU becomes available or another
       task becomes runnable.

       The sched_relax_domain_level value of minus one (-1) always  means  use
       the  system default value.  The system default value can vary by archi‐
       tecture and kernel version.  This system default value can  be  changed
       by kernel boot-time "relax_domain_level=" argument.

       In  the	case  of  multiple  overlapping cpusets which have conflicting
       sched_relax_domain_level values, then the highest such value applies to
       all  CPUs  in any of the overlapping cpusets.  In such cases, the value
       minus one (-1) is the lowest value, overridden by any other value,  and
       the value zero (0) is the next lowest value.

       The  following  formats	are  used to represent sets of CPUs and memory

   Mask format
       The Mask Format is used to represent CPU and memory-node bit  masks  in
       the /proc/<pid>/status file.

       This format displays each 32-bit word in hexadecimal (using ASCII char‐
       acters "0" - "9" and "a" - "f"); words are filled with  leading	zeros,
       if required.  For masks longer than one word, a comma separator is used
       between words.  Words are displayed in big-endian order, which has  the
       most  significant  bit first.  The hex digits within a word are also in
       big-endian order.

       The number of 32-bit words displayed is the minimum  number  needed  to
       display all bits of the bit mask, based on the size of the bit mask.

       Examples of the Mask Format:

	      00000001			      # just bit 0 set
	      40000000,00000000,00000000      # just bit 94 set
	      00000001,00000000,00000000      # just bit 64 set
	      000000ff,00000000		      # bits 32-39 set
	      00000000,000E3862		      # 1,5,6,11-13,17-19 set

       A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:


       The  first  "1" is for bit 64, the second for bit 32, the third for bit
       16, the fourth for bit 8, the fifth for bit 4, and the "7" is for  bits
       2, 1, and 0.

   List format
       The  List  Format for cpus and mems is a comma-separated list of CPU or
       memory-node numbers and ranges of numbers, in ASCII decimal.

       Examples of the List Format:

	      0-4,9	      # bits 0, 1, 2, 3, 4, and 9 set
	      0-2,7,12-14     # bits 0, 1, 2, 7, 12, 13, and 14 set

       The following rules apply to each cpuset:

       *  Its CPUs and memory nodes must be a (possibly equal) subset  of  its

       *  It can be marked cpu_exclusive only if its parent is.

       *  It can be marked mem_exclusive only if its parent is.

       *  If it is cpu_exclusive, its CPUs may not overlap any sibling.

       *  If it is memory_exclusive, its memory nodes may not overlap any sib‐

       The permissions of a cpuset are determined by the  permissions  of  the
       directories and pseudo-files in the cpuset filesystem, normally mounted
       at /dev/cpuset.

       For instance, a process can put itself in some other cpuset  (than  its
       current	one)  if  it  can  write the tasks file for that cpuset.  This
       requires execute permission on the encompassing directories  and	 write
       permission on the tasks file.

       An  additional  constraint  is  applied to requests to place some other
       process in a cpuset.  One process may not attach another	 to  a	cpuset
       unless  it  would  have	permission  to send that process a signal (see

       A process may create a child cpuset if it can access and write the par‐
       ent  cpuset  directory.	 It  can  modify the CPUs or memory nodes in a
       cpuset if it can access that cpuset's directory (execute permissions on
       the each of the parent directories) and write the corresponding cpus or
       mems file.

       There is one minor difference between the manner in which these permis‐
       sions are evaluated and the manner in which normal filesystem operation
       permissions are evaluated.  The kernel  interprets  relative  pathnames
       starting	 at  a	process's  current  working directory.	Even if one is
       operating on a cpuset file, relative pathnames are interpreted relative
       to  the	process's  current  working  directory,	 not  relative	to the
       process's current cpuset.  The only ways that cpuset paths relative  to
       a process's current cpuset can be used are if either the process's cur‐
       rent working directory is its cpuset (it first did a cd or chdir(2)  to
       its cpuset directory beneath /dev/cpuset, which is a bit unusual) or if
       some user code converts the relative cpuset path to a  full  filesystem

       In theory, this means that user code should specify cpusets using abso‐
       lute pathnames, which requires knowing the mount point  of  the	cpuset
       filesystem  (usually,  but not necessarily, /dev/cpuset).  In practice,
       all user level code that this author is aware of simply assumes that if
       the  cpuset  filesystem	is mounted, then it is mounted at /dev/cpuset.
       Furthermore, it is common practice for carefully written user  code  to
       verify  the  presence  of the pseudo-file /dev/cpuset/tasks in order to
       verify that the cpuset pseudo-filesystem is currently mounted.

   Enabling memory_pressure
       By default, the per-cpuset file cpuset.memory_pressure always  contains
       zero (0).  Unless this feature is enabled by writing "1" to the pseudo-
       file /dev/cpuset/cpuset.memory_pressure_enabled, the  kernel  does  not
       compute per-cpuset memory_pressure.

   Using the echo command
       When using the echo command at the shell prompt to change the values of
       cpuset files, beware that the built-in echo command in some shells does
       not  display  an	 error message if the write(2) system call fails.  For
       example, if the command:

	   echo 19 > cpuset.mems

       failed because memory node 19 was not allowed (perhaps the current sys‐
       tem  does  not  have a memory node 19), then the echo command might not
       display any error.  It is better to use the /bin/echo external  command
       to  change  cpuset file settings, as this command will display write(2)
       errors, as in the example:

	   /bin/echo 19 > cpuset.mems
	   /bin/echo: write error: Invalid argument

   Memory placement
       Not all allocations of system memory are constrained  by	 cpusets,  for
       the following reasons.

       If  hot-plug functionality is used to remove all the CPUs that are cur‐
       rently assigned to a cpuset, then the kernel will automatically	update
       the  cpus_allowed  of  all processes attached to CPUs in that cpuset to
       allow all CPUs.	When memory hot-plug functionality for removing memory
       nodes  is  available, a similar exception is expected to apply there as
       well.  In general, the kernel  prefers  to  violate  cpuset  placement,
       rather  than  starving  a  process that has had all its allowed CPUs or
       memory nodes taken offline.  User code should  reconfigure  cpusets  to
       refer  only  to online CPUs and memory nodes when using hot-plug to add
       or remove such resources.

       A few  kernel-critical,	internal  memory-allocation  requests,	marked
       GFP_ATOMIC,  must  be  satisfied immediately.  The kernel may drop some
       request or malfunction if one of these allocations  fail.   If  such  a
       request	cannot	be satisfied within the current process's cpuset, then
       we relax the cpuset, and look for memory anywhere we can find it.  It's
       better to violate the cpuset than stress the kernel.

       Allocations  of	memory requested by kernel drivers while processing an
       interrupt lack any relevant process context, and are  not  confined  by

   Renaming cpusets
       You  can	 use the rename(2) system call to rename cpusets.  Only simple
       renaming is supported; that is, changing the name of a cpuset directory
       is  permitted, but moving a directory into a different directory is not

       The Linux kernel implementation of cpusets sets errno  to  specify  the
       reason for a failed system call affecting cpusets.

       The  possible  errno  settings  and  their meaning when set on a failed
       cpuset call are as listed below.

       E2BIG  Attempted a write(2) on a special	 cpuset	 file  with  a	length
	      larger  than some kernel-determined upper limit on the length of
	      such writes.

       EACCES Attempted to write(2) the process ID (PID) of  a	process	 to  a
	      cpuset  tasks  file  when	 one  lacks  permission	 to  move that

       EACCES Attempted to add, using write(2), a CPU  or  memory  node	 to  a
	      cpuset, when that CPU or memory node was not already in its par‐

       EACCES Attempted	 to  set,  using  write(2),  cpuset.cpu_exclusive   or
	      cpuset.mem_exclusive  on	a  cpuset  whose parent lacks the same

       EACCES Attempted to write(2) a cpuset.memory_pressure file.

       EACCES Attempted to create a file in a cpuset directory.

       EBUSY  Attempted to remove, using rmdir(2), a cpuset with attached pro‐

       EBUSY  Attempted	 to  remove,  using  rmdir(2),	a  cpuset  with	 child

       EBUSY  Attempted to remove a CPU or memory node from a cpuset  that  is
	      also in a child of that cpuset.

       EEXIST Attempted	 to  create,  using  mkdir(2),	a  cpuset that already

       EEXIST Attempted to rename(2) a cpuset to a name that already exists.

       EFAULT Attempted to read(2) or write(2) a cpuset file  using  a	buffer
	      that is outside the writing processes accessible address space.

       EINVAL Attempted	 to  change  a	cpuset,	 using write(2), in a way that
	      would violate a cpu_exclusive or mem_exclusive attribute of that
	      cpuset or any of its siblings.

       EINVAL Attempted	 to  write(2) an empty cpuset.cpus or cpuset.mems list
	      to a cpuset which has attached processes or child cpusets.

       EINVAL Attempted to write(2) a cpuset.cpus or  cpuset.mems  list	 which
	      included	a  range with the second number smaller than the first

       EINVAL Attempted to write(2) a cpuset.cpus or  cpuset.mems  list	 which
	      included an invalid character in the string.

       EINVAL Attempted	 to write(2) a list to a cpuset.cpus file that did not
	      include any online CPUs.

       EINVAL Attempted to write(2) a list to a cpuset.mems file that did  not
	      include any online memory nodes.

       EINVAL Attempted to write(2) a list to a cpuset.mems file that included
	      a node that held no memory.

       EIO    Attempted to write(2) a string to a cpuset tasks file that  does
	      not begin with an ASCII decimal integer.

       EIO    Attempted to rename(2) a cpuset into a different directory.

	      Attempted to read(2) a /proc/<pid>/cpuset file for a cpuset path
	      that is longer than the kernel page size.

	      Attempted to create, using mkdir(2), a cpuset whose base	direc‐
	      tory name is longer than 255 characters.

	      Attempted	 to  create, using mkdir(2), a cpuset whose full path‐
	      name, including the mount point (typically "/dev/cpuset/")  pre‐
	      fix, is longer than 4095 characters.

       ENODEV The  cpuset was removed by another process at the same time as a
	      write(2) was attempted on one of the pseudo-files in the	cpuset

       ENOENT Attempted to create, using mkdir(2), a cpuset in a parent cpuset
	      that doesn't exist.

       ENOENT Attempted to access(2) or open(2) a nonexistent file in a cpuset

       ENOMEM Insufficient memory is available within the kernel; can occur on
	      a variety of system calls affecting cpusets,  but	 only  if  the
	      system is extremely short of memory.

       ENOSPC Attempted	 to  write(2)  the  process ID (PID) of a process to a
	      cpuset tasks file when the cpuset had an	empty  cpuset.cpus  or
	      empty cpuset.mems setting.

       ENOSPC Attempted	 to  write(2) an empty cpuset.cpus or cpuset.mems set‐
	      ting to a cpuset that has tasks attached.

	      Attempted to rename(2) a nonexistent cpuset.

       EPERM  Attempted to remove a file from a cpuset directory.

       ERANGE Specified a cpuset.cpus or cpuset.mems list to the kernel	 which
	      included	a  number  too	large for the kernel to set in its bit

       ESRCH  Attempted to write(2) the process	 ID  (PID)  of	a  nonexistent
	      process to a cpuset tasks file.

       Cpusets appeared in version 2.6.12 of the Linux kernel.

       Despite	its  name, the pid parameter is actually a thread ID, and each
       thread in a threaded group can be attached to a different cpuset.   The
       value  returned	from a call to gettid(2) can be passed in the argument

       cpuset.memory_pressure cpuset files can be  opened  for	writing,  cre‐
       ation,  or  truncation,	but  then the write(2) fails with errno set to
       EACCES, and the creation and truncation	options	 on  open(2)  have  no

       The  following examples demonstrate querying and setting cpuset options
       using shell commands.

   Creating and attaching to a cpuset.
       To create a new cpuset and attach the current command shell to it,  the
       steps are:

       1)  mkdir /dev/cpuset (if not already done)
       2)  mount -t cpuset none /dev/cpuset (if not already done)
       3)  Create the new cpuset using mkdir(1).
       4)  Assign CPUs and memory nodes to the new cpuset.
       5)  Attach the shell to the new cpuset.

       For  example,  the  following sequence of commands will set up a cpuset
       named "Charlie", containing just CPUs 2 and 3, and memory node  1,  and
       then attach the current shell to that cpuset.

	   $ mkdir /dev/cpuset
	   $ mount -t cpuset cpuset /dev/cpuset
	   $ cd /dev/cpuset
	   $ mkdir Charlie
	   $ cd Charlie
	   $ /bin/echo 2-3 > cpuset.cpus
	   $ /bin/echo 1 > cpuset.mems
	   $ /bin/echo $$ > tasks
	   # The current shell is now running in cpuset Charlie
	   # The next line should display '/Charlie'
	   $ cat /proc/self/cpuset

   Migrating a job to different memory nodes.
       To migrate a job (the set of processes attached to a cpuset) to differ‐
       ent CPUs and memory nodes in the system, including  moving  the	memory
       pages currently allocated to that job, perform the following steps.

       1)  Let's  say  we  want	 to move the job in cpuset alpha (CPUs 4-7 and
	   memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory nodes
       2)  First create the new cpuset beta.
       3)  Then allow CPUs 16-19 and memory nodes 8-9 in beta.
       4)  Then enable memory_migration in beta.
       5)  Then move each process from alpha to beta.

       The following sequence of commands accomplishes this.

	   $ cd /dev/cpuset
	   $ mkdir beta
	   $ cd beta
	   $ /bin/echo 16-19 > cpuset.cpus
	   $ /bin/echo 8-9 > cpuset.mems
	   $ /bin/echo 1 > cpuset.memory_migrate
	   $ while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

       The  above  should  move any processes in alpha to beta, and any memory
       held by these processes on  memory  nodes  2-3  to  memory  nodes  8-9,

       Notice that the last step of the above sequence did not do:

	   $ cp ../alpha/tasks tasks

       The  while loop, rather than the seemingly easier use of the cp(1) com‐
       mand, was necessary because only one process PID at a time may be writ‐
       ten to the tasks file.

       The  same  effect  (writing one PID at a time) as the while loop can be
       accomplished more efficiently, in fewer keystrokes and in  syntax  that
       works  on  any  shell,  but  alas  more	obscurely,  by	using  the  -u
       (unbuffered) option of sed(1):

	   $ sed -un p < ../alpha/tasks > tasks

       taskset(1),  get_mempolicy(2),  getcpu(2),  mbind(2),   sched_getaffin‐
       ity(2),	sched_setaffinity(2), sched_setscheduler(2), set_mempolicy(2),
       CPU_SET(3), proc(5), numa(7), migratepages(8), numactl(8)

       Documentation/cpusets.txt in the Linux kernel source tree

       This page is part of release 3.65 of the Linux  man-pages  project.   A
       description  of	the project, and information about reporting bugs, can
       be found at http://www.kernel.org/doc/man-pages/.

Linux				  2013-02-12			     CPUSET(7)

List of man pages available for Archlinux

Copyright (c) for man pages and the logo by the respective OS vendor.

For those who want to learn more, the polarhome community provides shell access and support.

[legal] [privacy] [GNU] [policy] [cookies] [netiquette] [sponsors] [FAQ]
Polarhome, production since 1999.
Member of Polarhome portal.
Based on Fawad Halim's script.
Vote for polarhome
Free Shell Accounts :: the biggest list on the net