Resource Management
- Guarantee completion in a reasonable time frame for critical workloads.
- Support reasonable cluster scheduling between groups of users based on fair allocation of resources per group.
- Prevent users from depriving other users access to the cluster.
For details on how to configure resource management, see the following topics:
Cloudera Manager Resource Management
Cloudera Manager provides two methods for allocating cluster resources to services: static and dynamic.
Static Allocation
Statically allocating resources using cgroups is configurable through a single static service pool wizard. You allocate services as a percentage of total resources, and the wizard configures the cgroups.
For example, the following figure illustrates static pools for HBase, HDFS, Impala, and YARN services that are respectively assigned 20%, 30%, 20%, and 30% of cluster resources.
Dynamic Allocation
You can dynamically apportion resources that are statically allocated to YARN and Impala by using dynamic resource pools.
- YARN - YARN manages the virtual cores, memory, running applications, maximum resources for undeclared children (for parent pools), and scheduling policy for each pool. In the preceding diagram, three dynamic resource pools—Dev, Product, and Mktg with weights 3, 2, and 1 respectively—are defined for YARN. If an application starts and is assigned to the Product pool, and other applications are using the Dev and Mktg pools, the Product resource pool receives 30% x 2/6 (or 10%) of the total cluster resources. If no applications are using the Dev and Mktg pools, the YARN Product pool is allocated 30% of the cluster resources.
- Impala - Impala manages memory for pools running queries and limits the number of running and queued queries in each pool.
The scenarios in which YARN manages resources map to the YARN scheduler policy. The scenarios in which Impala independently manages resources use Impala admission control.
<< Tuning YARN | ©2016 Cloudera, Inc. All rights reserved | Static Service Pools >> |
Terms and Conditions Privacy Policy |