DataWorks:Data Development process guide

Personal Folder development (for personal testing, temporary queries, and cross-project code synchronization)

Files in the Personal Folder are visible only to the current account. You can use these files for personal testing or temporary queries, but you cannot configure scheduling for them or publish them to the production environment. The Personal Folder is visible across all your workspaces and supports cross-workspace synchronization. To schedule and publish a file, you must first submit it from the Personal Folder to a Project Folder. You can then configure scheduling and publish the file from that Project Folder. For more information, see Personal Folder.

1. In the navigation pane on the left of DataStudio, click to go to the Data Development folder.

2. In the Personal Folder section, you can click to create a folder, and then create files in it as needed.

3. To submit a file from the Personal Folder to the Project Folder in the workspace, click Submit To Project Folder at the top of the editing window. For the next steps, see Project Folder development (for production environments).

Project Folder development (for production environments)

Files in the Project Folder support collaborative team development. You can create different types of nodes and orchestrate their upstream and downstream dependencies. For more information, see Project Folder.

1. In the navigation pane on the left of DataStudio, click to go to the Data Development folder.

2. Create a project folder, nodes, and workflows.

   In the Project Folder section, you can click to create a folder, node, or workflow.

   • Folder: You can use folders to manage nodes and workflows.

   • Node: DataStudio supports a wide range of node types, such as Data Integration, Notebook, and MaxCompute SQL. For more information about the functions and differences of various nodes, see Node development.

   • Workflow: A workflow is a tool for automating the management of data processing. It provides a visual canvas that lets you integrate various types of subtask nodes by dragging and dropping them. This makes it easy to establish dependencies between tasks, accelerate the creation of data processing flows, and improve development efficiency. For more information, see Recurring workflow.

3. Node orchestration.

   • Node: For a standalone node, you must configure its upstream and downstream dependencies in the scheduling dependency settings.

     On the node editing page, click Scheduling Configuration in the right pane. Configure the Node Scheduling parameters to define the upstream and downstream dependencies for the node. Dependencies ensure that nodes run in the correct order. A descendant node runs only after its ancestor nodes run successfully. This ensures that the current node can retrieve the correct data at the right time.

   • Workflow: A workflow lets you visually orchestrate the upstream and downstream dependencies of nodes on a canvas. You can plan the orchestration as required.

4. Node development.

   DataStudio supports a wide range of node types. The configurable content varies by node type. For more information, see Node development to complete the node configuration.

   Note

   During node development, you can define variables using the ${Variable_Name} format. You can then assign constant values to the variables during the testing phase and dynamically assign values during scheduling configuration.

Manual Folder development (for one-time tasks)

You can create one-time tasks or manually triggered workflows in the Manual Folder for one-time data processing scenarios that do not require recurring scheduling.

1. In the navigation pane on the left of DataStudio, click to go to the Manual Folder.

2. Create development folders and nodes under One-time Task or Manually Triggered Workflow as needed. For more information, see One-time Task and Manually Triggered Workflow.

Scheduling configuration

After you debug the node, if it needs to be published to the production environment for recurring automatic scheduling, click Scheduling Configuration on the right side of the node editing page to configure its scheduling properties.

• Scheduling Parameters: Define the parameters used for node scheduling. DataWorks provides multiple assignment formats. If you defined variables using the ${Variable_Name} format during node development, you can use scheduling parameters to dynamically assign values to variables in scheduling scenarios.

• Scheduling Policy: Define scheduling properties for the node in the scheduling environment, other than the execution frequency and specific execution time.

• Scheduling Time: Define the execution frequency and specific execution time for the node in the scheduling environment.

• Scheduling Dependencies: Define the upstream and downstream dependencies for the task. Dependencies ensure that nodes run in the correct order. A descendant node starts only after its ancestor nodes run successfully. This process ensures that the current node retrieves data correctly and at the right time.

Node publishing

After you configure the scheduling properties of the node, click the Publish button at the top of the node editing page. The node is then published to the production environment and scheduled to run periodically. For more information, see Publish a node or workflow.

Click the Publish button in the toolbar and then click Start Publishing To Production. This action publishes the task according to the publishing check process.