DataWorks:Create an EMR Spark node

This node can directly reference an OSS resource using the OSS REF method. When you run the EMR node, DataWorks automatically loads the OSS resource specified in the code to the local environment. This method is commonly used in scenarios where EMR tasks require JAR dependencies or scripts.

1. Develop a JAR resource.

   1. Prepare code dependencies.

      You can access the EMR cluster and view the required code dependencies in the /usr/lib/emr/spark-current/jars/ path of the master node. The following example uses Spark 3.4.2. You must open an existing IntelliJ IDEA project and add Project Object Model (POM) dependencies and reference plug-ins.

      Add pom dependencies

      <dependencies>
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-core_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
              <!-- Apache Spark SQL -->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-sql_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
      </dependencies>

      Reference related plugins

      <build>
              <sourceDirectory>src/main/scala</sourceDirectory>
              <testSourceDirectory>src/test/scala</testSourceDirectory>
              <plugins>
                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-compiler-plugin</artifactId>
                      <version>3.7.0</version>
                      <configuration>
                          <source>1.8</source>
                          <target>1.8</target>
                      </configuration>
                  </plugin>
                  <plugin>
                      <artifactId>maven-assembly-plugin</artifactId>
                      <configuration>
                          <descriptorRefs>
                              <descriptorRef>jar-with-dependencies</descriptorRef>
                          </descriptorRefs>
                      </configuration>
                      <executions>
                          <execution>
                              <id>make-assembly</id>
                              <phase>package</phase>
                              <goals>
                                  <goal>single</goal>
                              </goals>
                          </execution>
                      </executions>
                  </plugin>
                  <plugin>
                      <groupId>net.alchim31.maven</groupId>
                      <artifactId>scala-maven-plugin</artifactId>
                      <version>3.2.2</version>
                      <configuration>
                          <recompileMode>incremental</recompileMode>
                      </configuration>
                      <executions>
                          <execution>
                              <goals>
                                  <goal>compile</goal>
                                  <goal>testCompile</goal>
                              </goals>
                              <configuration>
                                  <args>
                                      <arg>-dependencyfile</arg>
                                      <arg>${project.build.directory}/.scala_dependencies</arg>
                                  </args>
                              </configuration>
                          </execution>
                      </executions>
                  </plugin>
              </plugins>
          </build>

   2. Write sample code.

      package com.aliyun.emr.example.spark
      
      import org.apache.spark.sql.SparkSession
      
      object SparkMaxComputeDemo {
        def main(args: Array[String]): Unit = {
          // Create a SparkSession.
          val spark = SparkSession.builder()
            .appName("HelloDataWorks")
            .getOrCreate()
      
          // Print the Spark version.
          println(s"Spark version: ${spark.version}")
        }
      }

   3. Package the code into a JAR file.

      After you write and save the preceding code, package the code into a JAR file. In this example, a file named SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar is generated.

2. Upload the JAR resource.

   1. After you finish developing the code, log on to the OSS console. In the navigation pane on the left, click Buckets.

   2. Click the target bucket name to open the Objects page.

      In this example, the onaliyun-bucket-2 bucket is used.

   3. Click Create Directory to create a directory to store JAR resources.

      Set Directory Name to emr/jars.

   4. Upload the JAR resource to its storage directory.

      Go to the destination folder. Click Upload File. In the Files To Upload section, click Select Files and add the SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar file. Then, click Upload File.

3. Reference the JAR resource.

   1. Edit the code to reference the JAR resource.

      On the editing page of the EMR Spark node, edit the code to reference the JAR resource.

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo --master yarn ossref://onaliyun-bucket-2/emr/jars/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      Parameter description:

      Parameter	Description
      class	The full name of the main class to run.
      master	The running mode of the Spark application.
      ossref file path	Format: ossref://{endpoint}/{bucket}/{object} endpoint: The access endpoint for OSS. If the endpoint parameter is left empty, only a resource in an OSS bucket that resides in the same region as the current EMR cluster can be referenced. Bucket: A container that is used to store objects in OSS. Each Bucket has a unique name. You can log on to the OSS Management Console to view all Buckets under the current logon account. object: a file name or path that is stored in a Bucket.

   2. Run the EMR Spark node task.

      After you finish writing the code, click the icon and select the Serverless resource group that you created to run the task. After the task execution is complete, record the applicationIds value from the console, such as application_1730367929285_xxxx.

   3. View the result.

      Create an EMR Shell node and run the yarn logs -applicationId application_1730367929285_xxxx command to view the running results: