Cloudera Enterprise 6.0.x | Other versions

Apache Crunch Guide

  Warning: As of CDH 6.0.0, Apache Crunch is deprecated, and will be removed in a future release. For more information, see Deprecated Items.

The Apache Crunch™ project develops and supports Java APIs that simplify the process of creating data pipelines on top of Apache Hadoop. The Crunch APIs are modeled after FlumeJava, which is the library that Google uses for building data pipelines on top of their own implementation of MapReduce.

The Apache Crunch Java library provides a framework for writing, testing, and running MapReduce pipelines. Its goal is to make pipelines that are composed of many user-defined functions simple to write, easy to test, and efficient to run. Running on top of Hadoop MapReduce and Apache Spark, the Apache Crunch library is a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce. The APIs are especially useful when processing data that does not fit naturally into relational model, such as time series, serialized object formats like protocol buffers or Avro records, and HBase rows and columns. For Scala users, there is the Scrunch API, which is built on top of the Java APIs and includes a REPL (read-eval-print loop) for creating MapReduce pipelines.

The following sections describe how to install Crunch:

Crunch Prerequisites

Cloudera supports Crunch on the same operating systems and Java versions as CDH. For more information, see CDH and Cloudera Manager Supported Operating Systems and Java Requirements.

Installing Crunch

  Warning: As of CDH 6.0.0, Apache Crunch is deprecated, and will be removed in a future release. For more information, see Deprecated Items.

As of CDH 6.0.0, Apache Crunch is no longer available as an RPM, Debian package, parcel, or tarball. To use Crunch with CDH 6, you must configure your Java or Scala project dependencies to include the Crunch libraries. You can do this using your build tools that support Maven or Ivy, or by including the JAR files from the repository in the classpath.

Maven Repository URL: https://repository.cloudera.com/artifactory/cloudera-repos/

Continue reading:

Adding Crunch Maven Dependencies

The following examples show how to add Apache Crunch dependencies to your projects:

  Important: Make sure that your dependencies (including Crunch, Hadoop, and Spark) use the same CDH version number as your CDH cluster. You can find the correct version string for your CDH version in the Cloudera Maven repository. Each version string contains the corresponding CDH version number in the format CDHX.Y.Z, where X.Y.Z is the CDH version number (such as CDH6.0.0). For an example, see the crunch-core project repository.
  • Using the Cloudera repository in a Maven project: 
    <repository>
  <id>cloudera</id>
  <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
  • Specifying the Crunch dependency in a Maven project with MapReduce 2 (YARN): 
        <dependency>
      <groupId>org.apache.crunch</groupId>
      <artifactId>crunch-core</artifactId>
      <version>0.11.0-cdh6.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>3.0.0-cdh6.0.0</version>
      <scope>provided</scope>
    </dependency>
  • Specifying the Crunch dependency in a Maven project with Spark 2 (YARN): 
        <dependency>
      <groupId>org.apache.crunch</groupId>
      <artifactId>crunch-core</artifactId>
      <version>0.11.0-cdh6.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.crunch</groupId>
      <artifactId>crunch-spark</artifactId>
      <version>0.11.0-cdh6.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.0.0-cdh6.0.0</version>
    </dependency>

Building Your Project

Because CDH 6 clusters do not contain the Crunch libraries, you must create a so-called fat or uber JAR of your project that also contains Crunch. Make sure that the uber JAR contains only Crunch dependencies, and not any other CDH dependencies.

Alternatively, you can download the Crunch JAR files from the Maven repository and add it to the classpath when you are running the job. Cloudera recommends using the uber JAR method.

The following example demonstrates using the Maven Shade plugin to create a JAR that contains only the Crunch dependencies:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <artifactSet>
                <includes>
                  <include>org.apache:crunch:*</include>
                </includes>
              </artifactSet>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Apache Crunch Documentation

For more information about Crunch, see the following documentation:
Page generated July 25, 2018.