DataWorks:Doris data source

Parameter

Description

Required

Default value

datasource

The name of the data source. You can add a data source in the code editor. The value of this parameter must be the same as the name of the added data source.

Yes

None

table

The name of the source table from which you want to synchronize data. A data integration task can read data from only one table.

The following examples show the advanced usage of the table parameter to configure a range:

• You can configure a range to read data from sharded databases and tables. For example, 'table_[0-99]' specifies that data is read from 'table_0', 'table_1', 'table_2', to 'table_99'.

• If the numeric suffixes of your table names have the same length, such as 'table_000', 'table_001', 'table_002', to 'table_999', you can set the parameter to '"table": ["table_00[0-9]", "table_0[10-99]", "table_[100-999]"]'.

Yes

None

column

The columns in the source table from which you want to synchronize data. Use a JSON array to describe the field information. By default, all columns are used. Example: [ * ].

• Supports column selection: You can select specific columns to export.

• Supports column reordering: You can export columns in an order different from the table schema.

• Supports constant configuration: You must follow the Doris SQL syntax. Example: ["id","table","1","'mingya.wmy'","'null'","to_char(a+1)","2.3","true"] .

  • id is a regular column name.

  • table is a column name that is also a reserved word.

  • 1 is an integer constant.

  • 'mingya.wmy' is a string constant. Note that it must be enclosed in single quotation marks.

  • About null:

    • " " indicates an empty string.

    • null indicates a null value.

    • 'null' indicates the string "null".

  • to_char(a+1) is a function that calculates the length of a string.

  • 2.3 is a floating-point number.

  • true is a Boolean value.

• You must explicitly specify the columns for synchronization in the column parameter. The parameter cannot be empty.

Yes

None

splitFactor

The sharding factor. You can configure the number of shards for data synchronization. If you configure multiple concurrent threads, the data is sharded into a number of shards equal to Concurrency × splitFactor. For example, if Concurrency is 5 and splitFactor is 5, the data is sharded into 5 × 5 = 25 shards and processed by five concurrent threads.

No

5

splitPk

When Doris Reader extracts data, if you specify the splitPk parameter, the data is sharded based on the specified field. This enables concurrent data synchronization tasks and improves efficiency.

• We recommend that you use the primary key of the table for splitPk. This is because the values in the primary key column are usually evenly distributed, which prevents data hot spots in the shards.

• Currently, splitPk supports sharding only for integer data. It does not support strings, floating-point numbers, dates, or other data types. If you specify an unsupported data type, the splitPk feature is ignored, and data is synchronized in a single channel.

• If you do not specify the splitPk parameter, or if the value is empty, data is synchronized in a single channel.

No

None

where

The filter condition. In business scenarios, you may want to synchronize only the data from the current day. In this case, you can set the where condition to gmt_create>$bizdate.

• The where condition allows for efficient incremental data synchronization. If you do not specify a where clause, including omitting the key or value, all data is synchronized.

• You cannot set the where condition to limit 10. This does not comply with the Doris SQL WHERE clause constraints.

No

None

querySql (This parameter is available only in the code editor.)

In some business scenarios, the where parameter may not be sufficient to describe the filter conditions. You can use this parameter to customize the filter SQL statement. If you configure this parameter, the data synchronization system ignores the tables, columns, and splitPk parameters and uses the SQL statement specified by this parameter to filter data. For example, to join multiple tables before synchronizing data, use select a,b from table_a join table_b on table_a.id = table_b.id. When you configure the querySql parameter, Doris Reader ignores the table, column, where, and splitPk parameters. The querySql parameter has a higher priority than the table, column, where, and splitPk parameters. The datasource parameter is used to parse information such as the username and password.

No

None

Parameter

Description

Required

Default value

datasource

The name of the data source. You can add a data source in the code editor. The value of this parameter must be the same as the name of the added data source.

Yes

None

table

The name of the destination table to which you want to synchronize data.

Yes

None

column

The columns in the destination table to which you want to write data. Separate the column names with commas (,). Example: "column":["id","name","age"]. To write data to all columns in order, use (*). Example: "column":["*"].

Yes

None

preSql

The SQL statement that is executed before the data synchronization task is run. In the codeless UI, you can execute only one SQL statement. In the code editor, you can execute multiple SQL statements. For example, you can clear old data from a table before the task runs.

No

None

postSql

The SQL statement that is executed after the data synchronization task is run. In the codeless UI, you can execute only one SQL statement. In the code editor, you can execute multiple SQL statements. For example, you can add a timestamp after the task runs.

No

None

maxBatchRows

The maximum number of rows in each batch of imported data. This parameter and batchSize control the import size of each batch. When the data in a batch reaches either of the two thresholds, the import of that batch starts.

No

500000

batchSize

The maximum data volume of each batch of imported data. This parameter and maxBatchRows control the import size of each batch. When the data in a batch reaches either of the two thresholds, the import of that batch starts.

No

104857600

maxRetries

The number of retries after a batch import fails.

No

3

labelPrefix

The label prefix for each uploaded file. The final label is a globally unique label composed of labelPrefix + UUID. This ensures that data is not imported repeatedly.

No

datax_doris_writer_

loadProps

The request parameters for StreamLoad. This parameter is mainly used to configure the format of the imported data. By default, data is imported in CSV format. If you do not configure the loadProps parameter, the default CSV format is used. The column delimiter is \t and the row delimiter is \n. The configuration is as follows.

"loadProps": {
    "format":"csv",
    "column_separator": "\t",
    "line_delimiter": "\n"
}

If you want to import data in JSON format, use the following configuration.

"loadProps": {
    "format": "json"
}

No

None