DataWorks:OSS data source

Supported

Unsupported

• Supports reading text files in TXT format. The schema in the TXT file must be a two-dimensional table.

• Supports CSV-like files with custom separators.

• Supports ORC and PARQUET formats.

• Supports reading multiple data types, which are represented as String. Supports column cropping and column constants.

• Supports recursive reads and file name filtering.

• Supports text compression. The available compression formats are gzip, bzip2, and zip.

  Note

  A compressed package cannot contain multiple compressed files.

• Supports concurrent reads for multiple objects.

• Does not support multi-threaded concurrent reads for a single object (file).

• Technically, a single compressed object cannot be read concurrently by multiple threads.

Supported

Unsupported

• Supports writing only text files. BLOBs such as videos and images are not supported. The schema in the text file must be a two-dimensional table.

• Supports CSV-like files with custom separators.

• Supports ORC and PARQUET formats.

• Supports multi-threaded writes. Each thread writes to a different sub-file.

• Supports file scrolling. When a file exceeds a specific size, it switches to a new file.

• Does not support concurrent writes to a single file.

• OSS does not provide data types. OSS Writer writes all data to OSS objects as the STRING type.

• Writing to an OSS bucket with the Cold Archive storage class is not supported.

• A single object (file) cannot exceed 100 GB.

Type classification

Data Integration column configuration type

Integer

LONG

String

STRING

Floating-point

DOUBLE

Boolean

BOOLEAN

Date/Time

DATE

Parameter

Description

Required

Default value

datasource

The name of the data source. The code editor supports adding data sources. The value of this parameter must be the same as the name of the added data source.

Yes

None

Object

This parameter specifies one or more objects to synchronize from OSS. You can configure this parameter in three ways: explicit path, wildcard path, and dynamic parameter path.

1. Configuration methods

• Explicit path

  • Basic rule: The path starts from the root directory of the bucket and does not need to include the bucket name.

  • Specify a single file: Enter the full path of the file. For example: my_folder/my_file.txt.

  • Specify multiple objects: Use commas (,) to separate the paths of multiple files or folders. For example: folder_a/file1.txt, folder_a/file2.txt.

• Wildcard path

  • Use a wildcard character to match multiple files that follow a specific pattern.

  • *: Matches zero or more characters.

  • ?: Matches a single character.

  • Examples:

    • abc*[0-9].txt matches abc0.txt, abc10.txt, and abc_test_9.txt.

    • abc?.txt matches abc1.txt and abcX.txt.

• Dynamic parameter path

  • Embed scheduling parameters in the path to automate synchronization. When the task runs, the parameters are replaced with their actual values.

  • Example: If you set the path to raw_data/${bizdate}/abc.txt, the task can dynamically synchronize the folder for the corresponding data timestamp every day.

  • For more information about scheduling parameters, see Supported formats of scheduling parameters.

2. Concurrent read mechanism and performance

The configuration method directly affects the concurrent performance of data extraction:

• Single-threaded mode: When you specify only a single, uncompressed file, the task extracts data in single-threaded mode.

• Multi-threaded mode: When you specify multiple specific files or use a wildcard character to match multiple files, the task automatically enables multi-threaded concurrent reading to significantly improve extraction efficiency. You can configure the specific number of concurrent threads in Channel Control.

Yes

None

parquetSchema

This parameter is configured when you read data from OSS in Parquet format. It takes effect only when fileFormat is set to parquet. It specifies the data types stored in the Parquet file. After you specify parquetSchema, make sure that the overall configuration complies with JSON syntax.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

None

column

The list of fields to read. `type` specifies the data type of the source data. `index` specifies the column number in the text file, starting from 0. `value` specifies that the current column is a constant. The data for this column is not read from the source file but is automatically generated based on the value.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // Get an int field from the first column of the OSS text file.
    },
    {
       "type": "string",
       "value": "alibaba"  // Generate a string field "alibaba" from within OSS Reader as the current field.
    }

Yes

All data is read as the STRING type.

fileFormat

The format of the source OSS file. For example, csv or text. Both formats support custom separators.

Yes

csv

fieldDelimiter

The column delimiter for reading data.

Yes

,

lineDelimiter

The row delimiter for reading data.

No

None

compress

The compression type of the text file. By default, this parameter is not specified, which means no compression. Supported compression types are gzip, bzip2, and zip.

No

No compression

encoding

The encoding format of the files to read.

No

utf-8

nullFormat

In a text file, a standard string cannot be used to define a null pointer. Data synchronization provides nullFormat to define which strings can be interpreted as null. For example:

• If you set nullFormat:"null", which is a visible character, and the source data is `null`, Data Integration treats it as a null field.

• If you set nullFormat:"\u0001", which is an invisible character, and the source data is the string `\u0001`, Data Integration treats it as a null field.

• If you do not specify the "nullFormat" parameter, the source data is written to the destination as is, without any conversion.

No

None

skipHeader

For CSV-like files, the header might be a title and needs to be skipped. By default, the header is not skipped. The skipHeader parameter is not supported for compressed files.

No

false

csvReaderConfig

The parameter settings for reading CSV files. This is a Map type. CsvReader is used to read CSV files. Many configurations are available. If you do not configure this parameter, the default values are used.

No

None

Parameter

Description

Required

Default value

datasource

The name of the data source. The code editor supports adding data sources. The value of this parameter must be the same as the name of the added data source.

Yes

None

object

The name of the file to be written by OSS Writer. OSS uses file names to simulate directories. OSS has the following limits on object names:

• If you set "object": "datax", the written object name starts with `datax` and has a random string appended.

• If you set "object": "cdo/datax", the written object name starts with /cdo/datax and has a random string appended. The separator for simulating directories in OSS is a forward slash (/).

If you do not want a random UUID as a suffix, set "writeSingleObject" : "true". For more information, see the description of writeSingleObject.

Yes

None

ossBlockSize

The size of OSS blocks. The default block size is 16 MB. When the file is written in parquet or ORC format, you can add this parameter at the same level as the object parameter.

Because OSS multipart upload supports a maximum of 10,000 parts, the default single file size limit is 160 GB. If the number of parts exceeds the limit, you can increase the block size to support larger file uploads.

No

16

writeMode

The action to perform on the data before OSS Writer writes it:

• truncate: Clears all objects that match the object name prefix before writing. For example, if you set "object":"abc", all objects starting with `abc` will be cleared.

• append: No action is taken before writing. Data Integration OSS Writer writes directly using the object name and appends a random UUID suffix to ensure no file name conflicts. For example, if the object name you specify is `DI`, the actual written name is DI_****_****_****.

• nonConflict: If an object with a matching prefix is found at the specified path, an error is reported. For example, if you set "object":"abc" and an object named `abc123` exists, an error is reported.

Yes

None

writeSingleObject

Specifies whether to write data to a single file in OSS:

• true: Writes to a single file. When no data can be read, no empty file is generated.

• false: Writes to multiple files. When no data can be read, if a file header is configured, an empty file containing only the file header is output. Otherwise, only an empty file is output.

No

false

fileFormat

The format of the file to be written. The following formats are supported:

• csv: Only strict csv format is supported. If the data to be written includes a column delimiter, it will be escaped according to the csv escape syntax. The escape character is a double quotation mark (").

• text: Simply splits the data to be written using the column delimiter. No escaping is performed if the data to be written includes the column delimiter.

• parquet: If you use this file type, you must add the parquetSchema parameter to define the data types.

  Important

  • To write data in Parquet format, you need to switch to the code editor and configure parquetSchema. For a configuration example, see Appendix: Script demos and parameter descriptions.

  • If you do not configure parquetSchema, DataWorks will convert the data types based on the source field types according to a certain policy. For more information about the conversion policy, see Appendix: Conversion policy for Parquet data types.

• ORC: If you use this format, you need to switch to the code editor.

No

text

compress

The compression format of the data file written to OSS. This parameter must be configured in the code editor for the task.

No

None

fieldDelimiter

The column delimiter for writing data.

No

,

encoding

The encoding format of the file to be written.

No

utf-8

parquetSchema

This is a required parameter for writing to OSS in Parquet file format. It describes the structure of the object file. This parameter is effective only when fileFormat is set to parquet. The format is as follows.

message MessageTypeName {
Required/Optional, Data type, Column name;
......................;
}

The following describes the configuration items:

• MessageTypeName: Enter a name.

• Required/Optional: `required` indicates that the field cannot be empty. `optional` indicates that the field can be empty. We recommend that you set all fields to `optional`.

• Data type: Parquet files support BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and FIXED_LEN_BYTE_ARRAY types.

The following is an example.

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

None

nullFormat

In a text file, a standard string cannot be used to define a null pointer. The data synchronization system provides nullFormat to define which strings can be interpreted as null. For example, if you set nullFormat="null" and the source data is null, the data synchronization system treats it as a null field.

No

None

header

The header of the file written to OSS. For example, ["id", "name", "age"].

No

None

maxFileSize (Advanced configuration. Not supported in the codeless UI)

The maximum size of a single object file written to OSS. The default value is 10,000 × 10 MB. This is similar to controlling the size of a log file when printing log4j logs. When OSS performs multipart upload, the size of each part is 10 MB. This is also the minimum granularity for log rotation files, which means a maxFileSize less than 10 MB is treated as 10 MB. The maximum number of parts supported by each OSS InitiateMultipartUploadRequest is 10,000.

When rotation occurs, the object name is formed by appending suffixes like _1, _2, _3 to the original object prefix, which already includes a random UUID.

No

100,000

suffix (Advanced configuration. Not supported in the codeless UI)

The suffix of the file name generated when data synchronization writes data. For example, if you set suffix to .csv, the final file name is fileName****.csv.

No

None