Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 22
Текст из файла (страница 22)
If the property yarn.scheduler.capacity.<queue-path>.user-limit-factor is set to a value largerthan 1 (the default), then a single job is allowed to use more than its queue’s capacity.8. However, the Capacity Scheduler can perform work-preserving preemption, where the resource managerasks applications to return containers to balance capacity.88|Chapter 4: YARNExample 4-1.
A basic configuration file for the Capacity Scheduler<?xml version="1.0"?><configuration><property><name>yarn.scheduler.capacity.root.queues</name><value>prod,dev</value></property><property><name>yarn.scheduler.capacity.root.dev.queues</name><value>eng,science</value></property><property><name>yarn.scheduler.capacity.root.prod.capacity</name><value>40</value></property><property><name>yarn.scheduler.capacity.root.dev.capacity</name><value>60</value></property><property><name>yarn.scheduler.capacity.root.dev.maximum-capacity</name><value>75</value></property><property><name>yarn.scheduler.capacity.root.dev.eng.capacity</name><value>50</value></property><property><name>yarn.scheduler.capacity.root.dev.science.capacity</name><value>50</value></property></configuration>As you can see, the dev queue is further divided into eng and science queues of equalcapacity.
So that the dev queue does not use up all the cluster resources when the prodqueue is idle, it has its maximum capacity set to 75%. In other words, the prod queuealways has 25% of the cluster available for immediate use. Since no maximum capacitieshave been set for other queues, it’s possible for jobs in the eng or science queues to useall of the dev queue’s capacity (up to 75% of the cluster), or indeed for the prod queueto use the entire cluster.Beyond configuring queue hierarchies and capacities, there are settings to control themaximum number of resources a single user or application can be allocated, how manyapplications can be running at any one time, and ACLs on queues.
See the referencepage for details.Scheduling in YARN|89Queue placementThe way that you specify which queue an application is placed in is specific to theapplication. For example, in MapReduce, you set the property mapreduce.job.queuename to the name of the queue you want to use. If the queue does not exist, then you’llget an error at submission time. If no queue is specified, applications will be placed ina queue called default.For the Capacity Scheduler, the queue name should be the last partof the hierarchical name since the full hierarchical name is not rec‐ognized. So, for the preceding example configuration, prod and engare OK, but root.dev.eng and dev.eng do not work.Fair Scheduler ConfigurationThe Fair Scheduler attempts to allocate resources so that all running applications getthe same share of resources.
Figure 4-3 showed how fair sharing works for applicationsin the same queue; however, fair sharing actually works between queues, too, as we’llsee next.The terms queue and pool are used interchangeably in the context ofthe Fair Scheduler.To understand how resources are shared between queues, imagine two users A and B,each with their own queue (Figure 4-4). A starts a job, and it is allocated all the resourcesavailable since there is no demand from B.
Then B starts a job while A’s job is stillrunning, and after a while each job is using half of the resources, in the way we sawearlier. Now if B starts a second job while the other jobs are still running, it will shareits resources with B’s other job, so each of B’s jobs will have one-fourth of the resources,while A’s will continue to have half. The result is that resources are shared fairly betweenusers.90|Chapter 4: YARNFigure 4-4. Fair sharing between user queuesEnabling the Fair SchedulerThe scheduler in use is determined by the setting of yarn.resourcemanager.scheduler.class.
The Capacity Scheduler is used by default (although the Fair Scheduler isthe default in some Hadoop distributions, such as CDH), but this can be changed bysetting yarn.resourcemanager.scheduler.class in yarn-site.xml to the fully qualifiedclassname of the scheduler, org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.Queue configurationThe Fair Scheduler is configured using an allocation file named fair-scheduler.xml thatis loaded from the classpath.
(The name can be changed by setting the propertyyarn.scheduler.fair.allocation.file.) In the absence of an allocation file, the FairScheduler operates as described earlier: each application is placed in a queue namedafter the user and queues are created dynamically when users submit their first appli‐cations.Per-queue configuration is specified in the allocation file.
This allows configuration ofhierarchical queues like those supported by the Capacity Scheduler. For example, wecan define prod and dev queues like we did for the Capacity Scheduler using the allo‐cation file in Example 4-2.Example 4-2. An allocation file for the Fair Scheduler<?xml version="1.0"?><allocations><defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy><queue name="prod">Scheduling in YARN|91<weight>40</weight><schedulingPolicy>fifo</schedulingPolicy></queue><queue name="dev"><weight>60</weight><queue name="eng" /><queue name="science" /></queue><queuePlacementPolicy><rule name="specified" create="false" /><rule name="primaryGroup" create="false" /><rule name="default" queue="dev.eng" /></queuePlacementPolicy></allocations>The queue hierarchy is defined using nested queue elements.
All queues are children ofthe root queue, even if not actually nested in a root queue element. Here we subdividethe dev queue into a queue called eng and another called science.Queues can have weights, which are used in the fair share calculation. In this example,the cluster allocation is considered fair when it is divided into a 40:60 proportion be‐tween prod and dev.
The eng and science queues do not have weights specified, so theyare divided evenly. Weights are not quite the same as percentages, even though theexample uses numbers that add up to 100 for the sake of simplicity. We could havespecified weights of 2 and 3 for the prod and dev queues to achieve the same queueweighting.When setting weights, remember to consider the default queue anddynamically created queues (such as queues named after users). Theseare not specified in the allocation file, but still have weight 1.Queues can have different scheduling policies. The default policy for queues can be setin the top-level defaultQueueSchedulingPolicy element; if it is omitted, fair sched‐uling is used.
Despite its name, the Fair Scheduler also supports a FIFO (fifo) policyon queues, as well as Dominant Resource Fairness (drf), described later in the chapter.The policy for a particular queue can be overridden using the schedulingPolicy ele‐ment for that queue. In this case, the prod queue uses FIFO scheduling since we wanteach production job to run serially and complete in the shortest possible amount oftime. Note that fair sharing is still used to divide resources between the prod and devqueues, as well as between (and within) the eng and science queues.92|Chapter 4: YARNAlthough not shown in this allocation file, queues can be configured with minimumand maximum resources, and a maximum number of running applications.
(See thereference page for details.) The minimum resources setting is not a hard limit, but ratheris used by the scheduler to prioritize resource allocations. If two queues are below theirfair share, then the one that is furthest below its minimum is allocated resources first.The minimum resource setting is also used for preemption, discussed momentarily.Queue placementThe Fair Scheduler uses a rules-based system to determine which queue an applicationis placed in. In Example 4-2, the queuePlacementPolicy element contains a list of rules,each of which is tried in turn until a match occurs.
The first rule, specified, places anapplication in the queue it specified; if none is specified, or if the specified queue doesn’texist, then the rule doesn’t match and the next rule is tried. The primaryGroup rule triesto place an application in a queue with the name of the user’s primary Unix group; ifthere is no such queue, rather than creating it, the next rule is tried. The default ruleis a catch-all and always places the application in the dev.eng queue.The queuePlacementPolicy can be omitted entirely, in which case the default behavioris as if it had been specified with the following:<queuePlacementPolicy><rule name="specified" /><rule name="user" /></queuePlacementPolicy>In other words, unless the queue is explicitly specified, the user’s name is used for thequeue, creating it if necessary.Another simple queue placement policy is one where all applications are placed in thesame (default) queue.
This allows resources to be shared fairly between applications,rather than users. The definition is equivalent to this:<queuePlacementPolicy><rule name="default" /></queuePlacementPolicy>It’s also possible to set this policy without using an allocation file, by settingyarn.scheduler.fair.user-as-default-queue to false so that applications will beplaced in the default queue rather than a per-user queue. In addition,yarn.scheduler.fair.allow-undeclared-pools should be set to false so that userscan’t create queues on the fly.PreemptionWhen a job is submitted to an empty queue on a busy cluster, the job cannot start untilresources free up from jobs that are already running on the cluster.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
