DataWorks:Use third-party packages and custom Python scripts in PyODPS nodes

Preparations

Before you start, understand the following two key concepts to determine your configuration method.

1. PyODPS 3 vs. PyODPS 2

   • PyODPS 3 (recommended): Based on the Python 3.7+ environment. Official support for Python 2 has ended. This topic uses PyODPS 3 as the primary example.

   • PyODPS 2: Based on the Python 2.7 environment.

2. Resource groups: serverless resource groups vs. exclusive resource group for scheduling

   • Serverless resource group (recommended): Elastic and maintenance-free, reliant on custom images to manage dependencies.

   • Exclusive resource group for scheduling: Requires manual maintenance of the resource group and installs dependencies through O&M Assistant, which has more limitations.

   Note

   In the DataWorks console, go to the Resource Group page in workspace details to view the type of resource group bound to your workspace.

   • General-purpose Type indicates you are using a serverless resource group.

   • Data Scheduling indicates you are using an exclusive resource group for scheduling.

Install third-party packages using a custom image

This tutorial provides an end-to-end example that shows how to create a custom environment containing the pendulum package and call it in a PyODPS 3 node to get and format the current time in a specific time zone.

Install third-party packages using O&M Assistant

1. Log on to the DataWorks Workspaces page, switch the region at the top, find the target workspace, and click Details in the Actions column.

2. In the left-side navigation pane, click Resource Group, find the bound exclusive resource group for scheduling. In the Actions column, click the icon and select O&M Assistant.

3. Select Create Command in the upper-left corner.

4. For Python 3 (PyODPS 3), keep the other default options. From the drop-down list, select the pendulum installation package.

5. On the O&M Assistant page, click Run Command in the Actions column. Once the command completes, you can use import pendulum directly in the corresponding PyODPS node.

Create a Python script

If you only want to call a function from another .py file that you wrote, follow these steps:

1. Create a Python resource:

   1. On the DataStudio page, right-click the Business Flow and choose Create Resource > MaxCompute > Python.

   2. In the Create Resource dialog box, enter a name for the resource (for example: my_utils.py) and click Create.

   3. Enter the following code:

      def say_hello(name):
          print(f"Hello, {name}! This is from my_utils module.")

   4. Save and submit the resource.

2. Create a PyODPS 3 node and reference the Python resource:

   • In the target business flow, right-click MaxCompute, select Create Node > PyODPS 3 and create the node.

   • In the code editor, reference the Python resource using ##@resource_reference{"my_utils.py"} as shown in the following code:

     ##@resource_reference{"my_utils.py"}
     import sys
     import os
     # Add the current directory of the resource to the Python interpreter's search path
     sys.path.append(os.path.dirname(os.path.abspath('my_utils.py')))
     # Import and use it like a normal module
     import my_utils
     my_utils.say_hello("DataWorks")

3. Run the node. You will see "Hello, DataWorks! This is from my_utils module." in the run logs.

FAQ

Q: Why does my custom image test hang during package installation?

A: This issue typically occurs due to network connectivity problems. Try these solutions:

• If your task environment requires third-party packages from the Internet, ensure the VPC bound to your serverless resource group has Internet access. For more information, see Enable public network access for the resource group.

• Try switching to a different Python package source, such as https://mirrors.aliyun.com/pypi/simple/.

Q: What should I do if importing a third-party package fails?

A: Troubleshoot as follows:

1. Confirm that the custom image has been published successfully.

2. Confirm that the task type supported by the image (PyODPS 2/3) matches the type of node you created.

3. Confirm that the custom image is correctly selected in the Properties of the PyODPS node.

   You cannot select shared resource groups.

4. Check if the installed package version is compatible with your Python version (for example, pendulum 2.0+ does not support Python 2).

References

• Basic operations

• Develop a PyODPS 3 task

• Develop a PyODPS 2 task

• Manage images

Appendix: Installation command reference

If you use the Script mode for a custom image or the Manual Installation mode in O&M Assistant to configure installation commands, refer to the following commands for installation:

• If you use a PyODPS 2 node, run the following command.

  pip install <package_to_install>

  Note

  After running the command, if prompted to upgrade the PIP version, run the command pip install --upgrade pip.

• If you use a PyODPS 3 node, run the following command.

  /home/tops/bin/pip3 install <package_to_install>

  Note

  • After running the command, if prompted to upgrade the PIP version, run the command /home/tops/bin/pip3 install --upgrade pip.

  • If the error /home/admin/usertools/tools/cmd-0.sh:line 3: /home/tops/bin/python3: No such file or directory occurs, submit a ticket to request permissions.