DataWorks:Configure a batch synchronization task in the codeless UI

Step 1: Create a batch synchronization node

Step 2: Configure the data source and resource group

Step 3: Configure the source and destination

In the source and destination sections, configure the tables from which to read data and to which to write data. You can also specify the data range to synchronize.

• Source

Operation	Description
Configure the synchronization range	Some source types support data filtering. You can specify a condition (a WHERE clause without the `where` keyword) to filter source data. At runtime, the task synchronizes only the data that meets the condition. For more information, see Scenario: Configure a batch synchronization task for incremental data. To perform incremental synchronization, you can combine this filter condition with scheduling parameters to make it dynamic. For example, with gmt_create >= '${bizdate}', the task synchronizes only the new data from the current day each time it runs. You also need to assign a value to the variable defined here when you configure scheduling properties. For more information, see Supported formats of scheduling parameters. The method for configuring incremental synchronization varies by data source (plugin). If you do not configure a filter condition, the task synchronizes all data from the table by default.
Configure a shard key for a relational database	Define the field in the source data that will be used as the shard key. The synchronization task splits the data into multiple tasks based on this key for concurrent, batched data reading. We recommend using the table's primary key for `splitPk` because primary keys are usually distributed evenly. This helps prevent data hot spots in the created shards. Currently, `splitPk` only supports integer data for sharding. It does not support strings, floating-point numbers, dates, or other types. If you specify an unsupported type, the `splitPk` feature is ignored, and the task uses a single channel for synchronization. If you do not specify `splitPk`, or if the value is empty, the data synchronization task uses a single channel to sync the table data. Not all plugins support specifying a shard key to configure task sharding logic. The preceding information is for example only. See the documentation for your specific plugin. For more information, see Supported data sources and synchronization solutions.

• Data processing

  Important

  Data processing is a feature available in the new Data Development. If you are using a previous version, you must upgrade your workspace to use this feature. For information about how to upgrade, see DataStudio upgrade guide.

  Data processing lets you process data from the source table using methods such as string replacement, AI-assisted processing, and data vectorization before you write the processed data to the destination table.

  

  1. Click the switch to turn on data processing.

  2. In the Data Processing List, click Add Node and select a data processing type: String Replacement, AI-assisted Processing, or Data Vectorization. You can add multiple data processing nodes, which DataWorks will process sequentially.

  3. Configure the data processing rules as prompted. For AI-assisted processing and data vectorization, see Intelligent data processing.

     Note

     Data processing requires additional computing resources, which increases the resource overhead and runtime of the data synchronization task. To avoid affecting synchronization efficiency, keep the processing logic as simple as possible.

• Destination

  Operation	Description
  Configure statements to execute before and after synchronization	Some data sources support executing SQL statements on the destination before data is written (pre-sync) and after the data is written (post-sync). MySQL Writer supports `preSql` and `postSql` configuration items, which allow you to execute MySQL commands before or after data is written to MySQL. For example, you can configure the MySQL command truncate table tablename in the Pre-SQL Statement (preSql) configuration item to clear existing data from the table before synchronization.
  Define the write mode for conflicts	Define how to write data to the destination when conflicts, such as path or primary key conflicts, occur. This configuration varies based on the data source attributes and writer plugin support. For configuration details, see the documentation for the specific writer plugin.

Step 4: Configure field mappings

After you select the source and destination, you must specify the mapping between the source and destination columns. The task writes data from the source fields to the corresponding destination fields based on these mappings.

During synchronization, mismatched field types between the source and destination can generate dirty data and cause write failures. To set the tolerance for dirty data, refer to the Channel Control settings in the next step.

Mapping by name and mapping by row are supported. You can also perform the following operations:

• Assign values to destination fields: You can use Add a row to add constants, scheduling parameters, or built-in variables to the destination table, such as '123', '${scheduling_parameter}', or '#{built_in_variable}#'.

  Note

  When you configure scheduling in the next step, you can assign values to scheduling parameters. For more information about how to use scheduling parameters, see Supported formats of scheduling parameters.

• Add built-in variables: You can manually add built-in variables and map them to destination fields to output them to a downstream node.

  The available built-in variables for each plugin are as follows:

  Built-in variable	Description	Supported plugins
  '#{DATASOURCE_NAME_SRC}#'	Source data source name	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{DB_NAME_SRC}#'	Name of the database where the source table is located	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{SCHEMA_NAME_SRC}#'	Name of the schema where the source table is located	PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader
  '#{TABLE_NAME_SRC}#'	Source table name	MySQL Reader MySQL (sharded) Reader PolarDB Reader PolarDB (sharded) Reader PostgreSQL Reader PolarDB-O Reader PolarDB-O (sharded) Reader

• Edit Source Fields: Click Manually Edit Mapping to perform the following operations:

  • Use functions that are supported by the source database to process fields. For example, you can use `Max(id)` to synchronize only the maximum value.

  • Manually edit source fields if not all fields were pulled during the field mapping process.

  Note

  MaxCompute Reader does not support the use of functions.

Step 5: Configure the channel

You can use channel control to configure properties related to the data synchronization process. For more information about the parameters, see Relationship between concurrency and throttling for batch synchronization.

Step 6: Configure scheduling properties

Step 7: Test and publish the task

Limits

• Some data sources do not support the configuration of batch synchronization tasks in the codeless UI.

  After you select a data source, if a message is displayed indicating that the codeless UI is not supported, click the icon in the toolbar to switch to the code editor and continue to configure the task. For more information, see Configure a task in the code editor.

  

• The codeless UI is easy to use but does not support some advanced features. If you require more fine-grained configuration management, you can click the convert to script icon in the toolbar to switch to the code editor to configure the batch synchronization task.

References

• Supported data sources and synchronization solutions

• Offline Synchronization Features

• Code editor configuration