DataX vs 腾讯云COS对象存储 -> StarRocks集群

2023-12-11 18:32 由扶苏公子x 发表于 #数据库

本文将介绍使用DataX读出Cos的Orc文件往StarRocks里面写。

需求: 需要将腾讯云cos上84TB的数据, 同步到StarRocks某个大表。正常每个分区数据量20~30亿，600GB。

工具：DataX
插件：hdfsreader、starrockswriter
对象存储COS：非融合

hdfsreader：https://cloud.tencent.com/document/product/436/43654
starrockswriter：https://docs.mirrorship.cn/zh/docs/loading/DataX-starrocks-writer

DataX

这里我使用的datax版本是 DataX (DATAX-OPENSOURCE-3.0)

[svccnetlhs@HOST datax]<231211 17:17:11>$ tree bin/ conf/
bin/
├── datax.py
├── dxprof.py
└── perftrace.py
conf/
├── core.json
└── logback.xml

0 directories, 5 files
[svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3
python3     python3.6   python3.6m  
[svccnetlhs@HOST datax]<231211 17:18:52>$ /bin/python3 bin/datax.py 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


Usage: datax.py [options] job-url-or-path

Options:
  -h, --help            show this help message and exit

  Product Env Options:
    Normal user use these options to set jvm parameters, job runtime mode
    etc. Make sure these options can be used in Product Env.

    -j <jvm parameters>, --jvm=<jvm parameters>
                        Set jvm parameters if necessary.
    --jobid=<job unique id>
                        Set job unique id when running by Distribute/Local
                        Mode.
    -m <job runtime mode>, --mode=<job runtime mode>
                        Set job runtime mode such as: standalone, local,
                        distribute. Default mode is standalone.
    -p <parameter used in job config>, --params=<parameter used in job config>
                        Set job parameter, eg: the source tableName you want
                        to set it by command, then you can use like this:
                        -p"-DtableName=your-table-name", if you have mutiple
                        parameters: -p"-DtableName=your-table-name
                        -DcolumnName=your-column-name".Note: you should config
                        in you job tableName with ${tableName}.
    -r <parameter used in view job config[reader] template>, --reader=<parameter used in view job config[reader] template>
                        View job config[reader] template, eg:
                        mysqlreader,streamreader
    -w <parameter used in view job config[writer] template>, --writer=<parameter used in view job config[writer] template>
                        View job config[writer] template, eg:
                        mysqlwriter,streamwriter

  Develop/Debug Options:
    Developer use these options to trace more details of DataX.

    -d, --debug         Set to remote debug mode.
    --loglevel=<log level>
                        Set log level such as: debug, info, all etc.
[svccnetlhs@HOST datax]<231211 17:19:06>$

DataX (HdfsReader) 插件

[svccnetlhs@HOST datax]<231211 17:23:29>$ ls
bin  conf  job  lib  log  log_perf  plugin  script  tmp
[svccnetlhs@HOST datax]<231211 17:23:29>$ 
[svccnetlhs@HOST datax]<231211 17:23:30>$ cd plugin/
[svccnetlhs@HOST plugin]<231211 17:23:32>$ ls
reader  writer
[svccnetlhs@HOST plugin]<231211 17:23:32>$ cd reader/
[svccnetlhs@HOST reader]<231211 17:23:36>$ ls
cassandrareader   datahubreader  ftpreader  hbase094xreader  hbase11xsqlreader  hdfsreader        loghubreader   mysqlreader         odpsreader      oraclereader  otsreader        postgresqlreader  sqlserverreader  streamreader    tsdbreader
clickhousereader  drdsreader     gdbreader  hbase11xreader   hbase20xsqlreader  kingbaseesreader  mongodbreader  oceanbasev10reader  opentsdbreader  ossreader     otsstreamreader  rdbmsreader       starrocksreader  tdenginereader  txtfilereader
[svccnetlhs@HOST reader]<231211 17:23:37>$ cd hdfsreader/
[svccnetlhs@HOST hdfsreader]<231211 17:23:39>$ ls
hdfsreader-0.0.1-SNAPSHOT.jar  libs  plugin_job_template.json  plugin.json
[svccnetlhs@HOST hdfsreader]<231211 17:23:40>$ 
[svccnetlhs@HOST hdfsreader]<231211 17:23:42>$ pwd
/home/svccnetlhs/chengken/starrocks/datax/plugin/reader/hdfsreader
[svccnetlhs@HOST hdfsreader]<231211 17:23:43>$ 
[svccnetlhs@HOST hdfsreader]<231211 17:23:44>$ cd libs/
[svccnetlhs@HOST libs]<231211 17:23:54>$ ls
activation-1.1.jar                     commons-beanutils-1.9.2.jar       curator-recipes-2.7.1.jar               hadoop-mapreduce-client-core-2.7.1.jar                  httpclient-4.1.2.jar           jetty-util-6.1.26.jar       parquet-hadoop-bundle-1.6.0rc3.jar
aircompressor-0.3.jar                  commons-beanutils-core-1.8.0.jar  datanucleus-api-jdo-3.2.6.jar           hadoop-yarn-api-2.7.1.jar                               httpcore-4.1.2.jar             jline-2.12.jar              pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
annotations-2.0.3.jar                  commons-cli-1.2.jar               datanucleus-core-3.2.10.jar             hadoop-yarn-common-2.7.1.jar                            jackson-core-asl-1.9.13.jar    jpam-1.1.jar                plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar
ant-1.9.1.jar                          commons-codec-1.4.jar             datanucleus-rdbms-3.2.9.jar             hadoop-yarn-server-applicationhistoryservice-2.6.0.jar  jackson-jaxrs-1.9.13.jar       jsch-0.1.42.jar             protobuf-java-2.5.0.jar
ant-launcher-1.9.1.jar                 commons-collections-3.2.1.jar     datax-common-0.0.1-SNAPSHOT.jar         hadoop-yarn-server-common-2.6.0.jar                     jackson-mapper-asl-1.9.13.jar  jsp-api-2.1.jar             servlet-api-2.5.jar
antlr-2.7.7.jar                        commons-compiler-2.7.6.jar        derby-10.11.1.1.jar                     hadoop-yarn-server-resourcemanager-2.6.0.jar            jackson-xc-1.9.13.jar          jsr305-3.0.0.jar            slf4j-api-1.7.10.jar
antlr-runtime-3.4.jar                  commons-compress-1.4.1.jar        eigenbase-properties-1.1.4.jar          hadoop-yarn-server-web-proxy-2.6.0.jar                  janino-2.7.6.jar               jta-1.1.jar                 slf4j-log4j12-1.7.10.jar
aopalliance-1.0.jar                    commons-configuration-1.6.jar     fastjson2-2.0.23.jar                    hamcrest-core-1.3.jar                                   javacsv-2.0.jar                leveldbjni-all-1.8.jar      snappy-java-1.0.4.1.jar
apache-curator-2.6.0.pom               commons-daemon-1.0.13.jar         geronimo-annotation_1.0_spec-1.1.1.jar  hive-ant-1.1.1.jar                                      javax.inject-1.jar             libfb303-0.9.2.jar          ST4-4.0.4.jar
apacheds-i18n-2.0.0-M15.jar            commons-dbcp-1.4.jar              geronimo-jaspic_1.0_spec-1.0.jar        hive-cli-1.1.1.jar                                      java-xmlbuilder-0.4.jar        libthrift-0.9.2.jar         stax-api-1.0.1.jar
apacheds-kerberos-codec-2.0.0-M15.jar  commons-digester-1.8.jar          geronimo-jta_1.1_spec-1.1.1.jar         hive-common-1.1.1.jar                                   jaxb-api-2.2.2.jar             log4j-1.2.17.jar            stax-api-1.0-2.jar
apache-log4j-extras-1.2.17.jar         commons-httpclient-3.1.jar        groovy-all-2.1.6.jar                    hive-exec-1.1.1.jar                                     jaxb-impl-2.2.3-1.jar          log4j-api-2.17.1.jar        stringtemplate-3.2.1.jar
api-asn1-api-1.0.0-M20.jar             commons-io-2.4.jar                gson-2.2.4.jar                          hive-hcatalog-core-1.1.1.jar                            jdo-api-3.0.1.jar              log4j-core-2.17.1.jar       velocity-1.5.jar
api-util-1.0.0-M20.jar                 commons-lang-2.6.jar              guava-11.0.2.jar                        hive-metastore-1.1.1.jar                                jersey-client-1.9.jar          logback-classic-1.0.13.jar  xercesImpl-2.9.1.jar
asm-3.1.jar                            commons-lang3-3.3.2.jar           guice-3.0.jar                           hive-serde-1.1.1.jar                                    jersey-core-1.9.jar            logback-core-1.0.13.jar     xml-apis-1.3.04.jar
asm-commons-3.1.jar                    commons-logging-1.1.3.jar         guice-servlet-3.0.jar                   hive-service-1.1.1.jar                                  jersey-guice-1.9.jar           lzo-core-1.0.5.jar          xmlenc-0.52.jar
asm-tree-3.1.jar                       commons-math3-3.1.1.jar           hadoop-aliyun-2.7.2.jar                 hive-shims-0.20S-1.1.1.jar                              jersey-json-1.9.jar            mail-1.4.1.jar              xz-1.0.jar
avro-1.7.4.jar                         commons-net-3.1.jar               hadoop-annotations-2.7.1.jar            hive-shims-0.23-1.1.1.jar                               jersey-server-1.9.jar          netty-3.6.2.Final.jar       zookeeper-3.4.6.jar
bonecp-0.8.0.RELEASE.jar               commons-pool-1.5.4.jar            hadoop-auth-2.7.1.jar                   hive-shims-1.1.1.jar                                    jets3t-0.9.0.jar               netty-all-4.0.23.Final.jar
calcite-avatica-1.0.0-incubating.jar   cos_api-bundle-5.6.137.2.jar      hadoop-common-2.7.1.jar                 hive-shims-common-1.1.1.jar                             jettison-1.1.jar               opencsv-2.3.jar
calcite-core-1.0.0-incubating.jar      curator-client-2.7.1.jar          hadoop-cos-3.1.0-8.3.2.jar              hive-shims-scheduler-1.1.1.jar                          jetty-6.1.26.jar               oro-2.0.8.jar
calcite-linq4j-1.0.0-incubating.jar    curator-framework-2.6.0.jar       hadoop-hdfs-2.7.1.jar                   htrace-core-3.1.0-incubating.jar                        jetty-all-7.6.0.v20120127.jar  paranamer-2.3.jar
[svccnetlhs@HOST libs]<231211 17:23:55>$

DataX (StarRocksWriter) 插件

[svccnetlhs@HOST datax]<231211 17:25:07>$ ls
bin  conf  job  lib  log  log_perf  plugin  script  tmp
[svccnetlhs@HOST datax]<231211 17:25:08>$ cd plugin/
[svccnetlhs@HOST plugin]<231211 17:25:11>$ ls
reader  writer
[svccnetlhs@HOST plugin]<231211 17:25:11>$ cd writer/
[svccnetlhs@HOST writer]<231211 17:25:13>$ ls
adbpgwriter      clickhousewriter  doriswriter          ftpwriter        hbase11xsqlwriter  hdfswriter          kuduwriter     mysqlwriter         ocswriter     oscarwriter  postgresqlwriter  sqlserverwriter  tdenginewriter
adswriter        databendwriter    drdswriter           gdbwriter        hbase11xwriter     hologresjdbcwriter  loghubwriter   neo4jwriter         odpswriter    osswriter    rdbmswriter       starrockswriter  tsdbwriter
cassandrawriter  datahubwriter     elasticsearchwriter  hbase094xwriter  hbase20xsqlwriter  kingbaseeswriter    mongodbwriter  oceanbasev10writer  oraclewriter  otswriter    selectdbwriter    streamwriter     txtfilewriter
[svccnetlhs@HOST writer]<231211 17:25:13>$ cd starrockswriter/
[svccnetlhs@HOST starrockswriter]<231211 17:25:15>$ ls
libs  plugin_job_template.json  plugin.json  starrockswriter-1.1.0.jar
[svccnetlhs@HOST starrockswriter]<231211 17:25:16>$ ls libs/
commons-codec-1.9.jar        commons-io-2.4.jar       commons-logging-1.1.1.jar  datax-common-0.0.1-SNAPSHOT.jar  fastjson2-2.0.23.jar  hamcrest-core-1.3.jar  httpcore-4.4.6.jar          logback-core-1.0.13.jar          plugin-rdbms-util-0.0.1-SNAPSHOT.jar
commons-collections-3.0.jar  commons-lang3-3.3.2.jar  commons-math3-3.1.1.jar    druid-1.0.15.jar                 guava-r05.jar         httpclient-4.5.3.jar   logback-classic-1.0.13.jar  mysql-connector-java-5.1.46.jar  slf4j-api-1.7.10.jar
[svccnetlhs@HOST starrockswriter]<231211 17:25:21>$

注: 两个datax插件在文件开头可以进行下载。

DataX JSON

众所周知，DataX的是基于数据抽取、数据转换和数据加载三个步骤来实现数据流的搬迁。

Datax设计理念：

Datax框架设计：

Datax工作流程：

连接JSON:

模板1：
{
    “content”: [
        {
            “reader”: {
                “name”: “hdfsreader”,
                “parameter”: {
                    “column”: [
                        {                                                             /************************************/
                            “name”: “ts”,                                             /************************************/
                            “type”: “string”,                                         /************************************/
                            “value”: “2023-11-14”                                     /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 0,                                               /************************************/
                            “name”: “local_id”,                                       /************************************/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /****1.由于cos文件中没有ts这个字段***/
                            “index”: 1,                                               /****这里我则使用value指定一个固定值*/
                            “name”: “encrypted_imei”,                                 /****value=2023-11-14代表当前path****/
                            “type”: “string”                                          /****的分区数据, ********************/
                        },                                                            /****此值在脚本中属于动态传参********/
                        {                                                             /************************************/
                            “index”: 2,                                               /****2.这里其他的字段使用了index*****/
                            “name”: “encrypted_idfa”,                                 /****下标的形式取到每个字段的值******/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 3,                                               /************************************/
                            “name”: “encrypted_mac”,                                  /************************************/
                            “type”: “string”                                          /************************************/
                        },                                                            /************************************/
                        {                                                             /************************************/
                            “index”: 4,                                               /************************************/
                            “name”: “encrypted_android_id”,                           /************************************/
                            “type”: “string”                                          /************************************/
                        }                                                             /************************************/
                    ],
                    “defaultFS”: “cosn: //桶名/”,
                    “encoding”: “UTF-8”,
                    “fieldDelimiter”: ",",
                    “fileType”: “orc”,
                    “hadoopConfig”: {
                        “fs.cosn.impl”: “org.apache.hadoop.fs.CosFileSystem”,
                        “fs.cosn.tmp.dir”: “本地临时路径（随便）”,
                        “fs.cosn.userinfo.region”: “ap-guangzhou”,
                        “fs.cosn.userinfo.secretId”: "",
                        “fs.cosn.userinfo.secretKey”: ""
                    },
                    “path”: "/sam/sam_dwd_user_action_cos_d/20231114/part-00011*"
                }
            },
            “writer”: {
                “name”: “starrockswriter”,
                “parameter”: {
                    “column”: [
                        “ts”,                                                        /******************************************/
                        “local_id”,                                                  /******************************************/
                        “encrypted_imei”,                                            /****StarRocks需要接收的字段名*************/
                        “encrypted_idfa”,                                            /******************************************/
                        “encrypted_mac”,                                             /******************************************/
                        “encrypted_android_id”                                       /******************************************/
                    ],
                    “database”: “StarRocks库名”,
                    “jdbcUrl”: “jdbc: mysql: //StarRocksFE_IP:9030/”,
                    “loadProps”: {
                        “max_filter_ratio”: 1
                    },
                    “loadUrl”: [
                        “StarRocksFE_IP:8030”,
                        “StarRocksFE_IP:8030”,
                        “StarRocksFE_IP:8030”
                    ],
                    “password”: “StarRocks密码”,
                    “postSql”: [
                        
                    ],
                    “preSql”: [
                        
                    ],
                    “table”: “StarRocks表名”,
                    “username”: “StarRocks用户”
                }
            }
        }
    ],
    “setting”: {
        “speed”: {
            “byte”: -1,                                                            /********channel调整为3，不限速**********/
            “channel”: 3                                                           /*********************************************/
        }
    }
}

模板2：
{
    "job": {
        "setting": {
          "speed": {
            "channel":3
          },
          "errorLimit": {}
        },
        "content": [{
            "reader": {
                "name": "hdfsreader",
                    "parameter": {
                        "path": "/sam/sam_dwd_user_action_cos_d/20231114/part-*",
                        "defaultFS": "cosn://*********/",
                        "column": [
                                {"name":"ts","type":"string","value":"2023-11-14"},
                                {"name":"import_ds_","type":"string","index":0},
                                {"name":"unique_action_id","type":"string","index":1},
                                {"name":"action_time","type":"string","index":2},
                                {"name":"report_time","type":"string","index":3},
                                {"name":"action_type","type":"string","index":4},
                                {"name":"ka_id","type":"string","index":5},
                                {"name":"action_session_id","type":"string","index":6},
                                {"name":"uuid","type":"string","index":7},
                                {"name":"wx_app_id","type":"string","index":8},
                                {"name":"wx_open_id","type":"string","index":9},
                                {"name":"wx_union_id","type":"string","index":10},
                                {"name":"external_user_id","type":"string","index":11},
                                {"name":"merber_id","type":"string","index":12},
                                {"name":"local_id","type":"string","index":13},
                                {"name":"encrypted_imei","type":"string","index":14},
                                {"name":"encrypted_idfa","type":"string","index":15},
                                {"name":"encrypted_mac","type":"string","index":16},
                                {"name":"encrypted_android_id","type":"string","index":17},
                                {"name":"encrypted_qq","type":"string","index":18},
                                {"name":"encrypted_phone","type":"string","index":19},
                                {"name":"encrypting_algorithm","type":"string","index":20},
                                {"name":"chan_id","type":"string","index":21},
                                {"name":"chan_refer_app_id","type":"string","index":22},
                                {"name":"chan_shop_id","type":"string","index":23},
                                {"name":"chan_shop_name","type":"string","index":24},
                                {"name":"client_type","type":"string","index":25},
                                {"name":"client_name","type":"string","index":26},
                                {"name":"client_version","type":"string","index":27},
                                {"name":"sdk_version","type":"string","index":28},
                                {"name":"device_model","type":"string","index":29},
                                {"name":"ip","type":"string","index":30},
                                {"name":"user_agent","type":"string","index":31},
                                {"name":"page_path","type":"string","index":32},
                                {"name":"page_name","type":"string","index":33},
                                {"name":"referrer","type":"string","index":34},
                                {"name":"address","type":"string","index":35},
                                {"name":"city","type":"string","index":36},
                                {"name":"province","type":"string","index":37},
                                {"name":"country","type":"string","index":38},
                                {"name":"latitude","type":"string","index":39},
                                {"name":"longitude","type":"string","index":40},
                                {"name":"json_properties","type":"string","index":41},
                                {"name":"fdate","type":"string","index":42},
                                {"name":"tag_id","type":"string","index":43},
                                {"name":"tag_name","type":"string","index":44},
                                {"name":"chan_custom_id","type":"string","index":45},
                                {"name":"etl_load_time","type":"string","index":46},
                                {"name":"event_name","type":"string","index":47}
                        ],
                        "fileType": "orc",
                        "encoding": "UTF-8",
                        "hadoopConfig": {
                            "fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem",
                            "fs.cosn.userinfo.region": "ap-guangzhou",
                            "fs.cosn.tmp.dir": "/u/chengken/starrocks/sam/data",
                            "fs.cosn.userinfo.secretId": "***************",
                            "fs.cosn.userinfo.secretKey": "**************",
                            "fs.cosn.read.ahead.block.size": 1048576,
                            "fs.cosn.read.ahead.queue.size": 2
                        },
                        "fieldDelimiter": ","
                    }
            },
            "writer": {
                 "name": "starrockswriter",
                 "parameter": {
                     "maxBatchRows":"5000000",
                     "maxBatchSize":"5368709120",
                     "username": "cndlopsns",
                     "password": "lizhenghua1.",
                     "database": "ods",
                     "table": "buckets_tmp_20231205__sams_cos",
                     "column": [
                            "ts",
                            "import_ds_",
                            "unique_action_id",
                            "action_time",
                            "report_time",
                            "action_type",
                            "ka_id",
                            "action_session_id",
                            "uuid",
                            "wx_app_id",
                            "wx_open_id",
                            "wx_union_id",
                            "external_user_id",
                            "merber_id",
                            "local_id",
                            "encrypted_imei",
                            "encrypted_idfa",
                            "encrypted_mac",
                            "encrypted_android_id",
                            "encrypted_qq",
                            "encrypted_phone",
                            "encrypting_algorithm",
                            "chan_id",
                            "chan_refer_app_id",
                            "chan_shop_id",
                            "chan_shop_name",
                            "client_type",
                            "client_name",
                            "client_version",
                            "sdk_version",
                            "device_model",
                            "ip",
                            "user_agent",
                            "page_path",
                            "page_name",
                            "referrer",
                            "address",
                            "city",
                            "province",
                            "country",
                            "latitude",
                            "longitude",
                            "json_properties",
                            "fdate",
                            "tag_id",
                            "tag_name",
                            "chan_custom_id",
                            "etl_load_time",
                            "event_name"
                  ],
                     "preSql": [],
                     "postSql": [], 
                     "jdbcUrl": "jdbc:mysql://***********:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
                     "loadUrl": ["****1:8030","****2:8030","****3:8030"],
                     "loadProps": {"max_filter_ratio":1}
                 }
             }
        }]
    }
}

启动

启动并顺利读到上游数据文件，然后异步写入StarRocks。

/usr/bin/python2.7 /home/hadoop/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/hadoop/datax/cos-starrocks1.json

Run: /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms16g -Xmx16g" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json
Output_____________

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2023-12-11 17:49:54.918 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-12-11 17:49:54.920 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-12-11 17:49:54.945 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-12-11 17:49:54.950 [main] INFO  Engine - the machine info  => 

    osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
    jvmInfo:    Oracle Corporation 1.8 25.112-b15
    cpu num:    16

    totalPhysicalMemory:    -0.00G
    freePhysicalMemory:    -0.00G
    maxFileDescriptorCount:    -1
    currentOpenFileDescriptorCount:    -1

    GC Names    [PS MarkSweep, PS Scavenge]

    MEMORY_NAME                    | allocation_size                | init_size                      
    PS Eden Space                  | 4,096.00MB                     | 4,096.00MB                     
    Code Cache                     | 240.00MB                       | 2.44MB                         
    Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
    PS Survivor Space              | 682.50MB                       | 682.50MB                       
    PS Old Gen                     | 10,923.00MB                    | 10,923.00MB                    
    Metaspace                      | -0.00MB                        | 0.00MB                         


2023-12-11 17:49:54.966 [main] INFO  Engine - 
{
    "setting":{
        "speed":{
            "channel":3
        },
        "errorLimit":{
            
        }
    },
    "content":[
        {
            "reader":{
                "name":"hdfsreader",
                "parameter":{
                    "path":"/sam/sam_dwd_user_action_cos_d/20231114/part-*",
                    "defaultFS":"cosn://*********/",
                    "column":[
                        {
                            "name":"ts",
                            "type":"string",
                            "value":"2023-11-14"
                        },
                        {
                            "name":"import_ds_",
                            "type":"string",
                            "index":0
                        },
                        {
                            "name":"unique_action_id",
                            "type":"string",
                            "index":1
                        },
                        {
                            "name":"action_time",
                            "type":"string",
                            "index":2
                        },
                        {
                            "name":"report_time",
                            "type":"string",
                            "index":3
                        },
                        {
                            "name":"action_type",
                            "type":"string",
                            "index":4
                        },
                        {
                            "name":"ka_id",
                            "type":"string",
                            "index":5
                        },
                        {
                            "name":"action_session_id",
                            "type":"string",
                            "index":6
                        },
                        {
                            "name":"uuid",
                            "type":"string",
                            "index":7
                        },
                        {
                            "name":"wx_app_id",
                            "type":"string",
                            "index":8
                        },
                        {
                            "name":"wx_open_id",
                            "type":"string",
                            "index":9
                        },
                        {
                            "name":"wx_union_id",
                            "type":"string",
                            "index":10
                        },
                        {
                            "name":"external_user_id",
                            "type":"string",
                            "index":11
                        },
                        {
                            "name":"merber_id",
                            "type":"string",
                            "index":12
                        },
                        {
                            "name":"local_id",
                            "type":"string",
                            "index":13
                        },
                        {
                            "name":"encrypted_imei",
                            "type":"string",
                            "index":14
                        },
                        {
                            "name":"encrypted_idfa",
                            "type":"string",
                            "index":15
                        },
                        {
                            "name":"encrypted_mac",
                            "type":"string",
                            "index":16
                        },
                        {
                            "name":"encrypted_android_id",
                            "type":"string",
                            "index":17
                        },
                        {
                            "name":"encrypted_qq",
                            "type":"string",
                            "index":18
                        },
                        {
                            "name":"encrypted_phone",
                            "type":"string",
                            "index":19
                        },
                        {
                            "name":"encrypting_algorithm",
                            "type":"string",
                            "index":20
                        },
                        {
                            "name":"chan_id",
                            "type":"string",
                            "index":21
                        },
                        {
                            "name":"chan_refer_app_id",
                            "type":"string",
                            "index":22
                        },
                        {
                            "name":"chan_shop_id",
                            "type":"string",
                            "index":23
                        },
                        {
                            "name":"chan_shop_name",
                            "type":"string",
                            "index":24
                        },
                        {
                            "name":"client_type",
                            "type":"string",
                            "index":25
                        },
                        {
                            "name":"client_name",
                            "type":"string",
                            "index":26
                        },
                        {
                            "name":"client_version",
                            "type":"string",
                            "index":27
                        },
                        {
                            "name":"sdk_version",
                            "type":"string",
                            "index":28
                        },
                        {
                            "name":"device_model",
                            "type":"string",
                            "index":29
                        },
                        {
                            "name":"ip",
                            "type":"string",
                            "index":30
                        },
                        {
                            "name":"user_agent",
                            "type":"string",
                            "index":31
                        },
                        {
                            "name":"page_path",
                            "type":"string",
                            "index":32
                        },
                        {
                            "name":"page_name",
                            "type":"string",
                            "index":33
                        },
                        {
                            "name":"referrer",
                            "type":"string",
                            "index":34
                        },
                        {
                            "name":"address",
                            "type":"string",
                            "index":35
                        },
                        {
                            "name":"city",
                            "type":"string",
                            "index":36
                        },
                        {
                            "name":"province",
                            "type":"string",
                            "index":37
                        },
                        {
                            "name":"country",
                            "type":"string",
                            "index":38
                        },
                        {
                            "name":"latitude",
                            "type":"string",
                            "index":39
                        },
                        {
                            "name":"longitude",
                            "type":"string",
                            "index":40
                        },
                        {
                            "name":"json_properties",
                            "type":"string",
                            "index":41
                        },
                        {
                            "name":"fdate",
                            "type":"string",
                            "index":42
                        },
                        {
                            "name":"tag_id",
                            "type":"string",
                            "index":43
                        },
                        {
                            "name":"tag_name",
                            "type":"string",
                            "index":44
                        },
                        {
                            "name":"chan_custom_id",
                            "type":"string",
                            "index":45
                        },
                        {
                            "name":"etl_load_time",
                            "type":"string",
                            "index":46
                        },
                        {
                            "name":"event_name",
                            "type":"string",
                            "index":47
                        }
                    ],
                    "fileType":"orc",
                    "encoding":"UTF-8",
                    "hadoopConfig":{
                        "fs.cosn.impl":"org.apache.hadoop.fs.CosFileSystem",
                        "fs.cosn.userinfo.region":"ap-guangzhou",
                        "fs.cosn.tmp.dir":"/u/chengken/starrocks/sam/data",
                        "fs.cosn.userinfo.secretId":"**********",
                        "fs.cosn.userinfo.secretKey":"****************",
                        "fs.cosn.read.ahead.block.size":1048576,
                        "fs.cosn.read.ahead.queue.size":2
                    },
                    "fieldDelimiter":","
                }
            },
            "writer":{
                "name":"starrockswriter",
                "parameter":{
                    "maxBatchRows":"5000000",
                    "maxBatchSize":"5368709120",
                    "username":"cndlopsns",
                    "password":"************",
                    "database":"ods",
                    "table":"buckets_tmp_20231205__sams_cos",
                    "column":[
                        "ts",
                        "import_ds_",
                        "unique_action_id",
                        "action_time",
                        "report_time",
                        "action_type",
                        "ka_id",
                        "action_session_id",
                        "uuid",
                        "wx_app_id",
                        "wx_open_id",
                        "wx_union_id",
                        "external_user_id",
                        "merber_id",
                        "local_id",
                        "encrypted_imei",
                        "encrypted_idfa",
                        "encrypted_mac",
                        "encrypted_android_id",
                        "encrypted_qq",
                        "encrypted_phone",
                        "encrypting_algorithm",
                        "chan_id",
                        "chan_refer_app_id",
                        "chan_shop_id",
                        "chan_shop_name",
                        "client_type",
                        "client_name",
                        "client_version",
                        "sdk_version",
                        "device_model",
                        "ip",
                        "user_agent",
                        "page_path",
                        "page_name",
                        "referrer",
                        "address",
                        "city",
                        "province",
                        "country",
                        "latitude",
                        "longitude",
                        "json_properties",
                        "fdate",
                        "tag_id",
                        "tag_name",
                        "chan_custom_id",
                        "etl_load_time",
                        "event_name"
                    ],
                    "preSql":[
                        
                    ],
                    "postSql":[
                        
                    ],
                    "jdbcUrl":"jdbc:mysql://192.168.1.121:9030/ods?useCursorFetch=true&tinyInt1isBit=false&query_timeout=36000&useUnicode=true&characterEncoding=utf8",
                    "loadUrl":[
                        "192.168.1.121:8030",
                        "192.168.1.122:8030",
                        "192.168.1.123:8030"
                    ],
                    "loadProps":{
                        "max_filter_ratio":1
                    }
                }
            }
        }
    ]
}

2023-12-11 17:49:54.983 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=true
2023-12-11 17:49:54.983 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-12-11 17:49:54.984 [main] INFO  JobContainer - Set jobId = 0
2023-12-11 17:49:54.994 [job-0] INFO  HdfsReader$Job - init() begin...
2023-12-11 17:49:55.250 [job-0] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":[]}
2023-12-11 17:49:55.250 [job-0] INFO  HdfsReader$Job - init() ok and end...
2023-12-11 17:49:55.279 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-12-11 17:49:55.279 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] do prepare work .
2023-12-11 17:49:55.279 [job-0] INFO  HdfsReader$Job - prepare(), start to getAllFiles...
2023-12-11 17:49:55.279 [job-0] INFO  HdfsReader$Job - get HDFS all files in path = [/sam/sam_dwd_user_action_cos_d/20231114/part-*]
十二月 11, 2023 5:49:55 下午 org.apache.hadoop.util.NativeCodeLoader <clinit>
警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-11 17:49:55.527 [job-0] INFO  RangerCredentialsClient - begin to init ranger client, impl []
2023-12-11 17:49:55.776 [job-0] INFO  CosNativeFileSystemStore - hadoop cos retry times: 200, cos client retry times: 5
log4j:WARN No appenders could be found for logger (com.qcloud.cos.thirdparty.org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2023-12-11 17:49:56.400 [job-0] INFO  CosFileSystem - The cos bucket is the normal bucket.
2023-12-11 17:49:56.418 [job-0] INFO  BufferPool - Initialize the buffer pool.
2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - fs.cosn.upload.buffer.size is set to -1, so the 'mapped_disk' buffer will be used by default.
2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - The type of the upload buffer pool is [MAPPED_DISK]. Buffer size:[-1]
2023-12-11 17:49:56.419 [job-0] INFO  BufferPool - tmp dir list
2023-12-11 17:49:56.796 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:56.953 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.014 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.192 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.244 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.392 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.443 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.586 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.645 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:57.782 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:57.837 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.007 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.066 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.233 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.292 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.444 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.510 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.694 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:58.749 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:58.975 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.030 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.181 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.232 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.393 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.449 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.602 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.656 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:49:59.836 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:49:59.888 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.038 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.106 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.325 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.382 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.551 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.606 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.769 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:00.826 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:00.969 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:01.021 [job-0] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.165 [job-0] INFO  HdfsReader$Job - [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]是[orc]类型的文件, 将该文件加入source files列表
2023-12-11 17:50:01.165 [job-0] INFO  HdfsReader$Job - 您即将读取的文件数为: [20], 列表为: [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00018-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00001-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00003-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00017-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00009-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00011-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00015-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00016-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00010-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00013-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00007-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00012-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00014-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00006-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00000-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00019-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc,cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00005-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
2023-12-11 17:50:01.166 [job-0] INFO  JobContainer - DataX Writer.Job [starrockswriter] do prepare work .
2023-12-11 17:50:01.168 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-12-11 17:50:01.168 [job-0] INFO  JobContainer - Job set Channel-Number to 3 channels.
2023-12-11 17:50:01.168 [job-0] INFO  HdfsReader$Job - split() begin...
2023-12-11 17:50:01.173 [job-0] INFO  JobContainer - DataX Reader.Job [hdfsreader] splits to [20] tasks.
2023-12-11 17:50:01.174 [job-0] INFO  JobContainer - DataX Writer.Job [starrockswriter] splits to [20] tasks.
2023-12-11 17:50:01.200 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-12-11 17:50:01.213 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-12-11 17:50:01.215 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-12-11 17:50:01.227 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [3] channels for [20] tasks.
2023-12-11 17:50:01.231 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to 209715200.
2023-12-11 17:50:01.232 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-12-11 17:50:01.240 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started
2023-12-11 17:50:01.243 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-12-11 17:50:01.244 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started
2023-12-11 17:50:01.288 [0-0-17-writer] INFO  HostUtils - IP 10.233.76.104 HOSTNAME pose-app-52211-pdc
2023-12-11 17:50:01.322 [0-0-17-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.324 [0-0-17-reader] INFO  Reader$Task - read start
2023-12-11 17:50:01.324 [0-0-17-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
2023-12-11 17:50:01.324 [0-0-17-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
2023-12-11 17:50:01.326 [0-0-0-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.326 [0-0-8-reader] INFO  HdfsReader$Job - hadoopConfig details:{"classLoader":{"URLs":["file:/u/chengken/datax/plugin/reader/hdfsreader/hdfsreader-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-impl-2.2.3-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-hcatalog-core-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-auth-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-linq4j-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-metastore-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jta-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/activation-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aircompressor-0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/aopalliance-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-configuration-1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-web-proxy-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-servlet-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-cli-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-avatica-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xz-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/parquet-hadoop-bundle-1.6.0rc3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-all-7.6.0.v20120127.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-httpclient-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0-2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-i18n-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-api-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/mail-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jaxb-api-2.2.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-annotations-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jta_1.1_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libthrift-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/oro-2.0.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-hdfs-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-jaxrs-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xercesImpl-2.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-logging-1.1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-serde-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-recipes-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/opencsv-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-json-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-service-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/avro-1.7.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.20S-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpclient-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-cli-1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xml-apis-1.3.04.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-digester-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/servlet-api-2.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/lzo-core-1.0.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/log4j-core-2.17.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javacsv-2.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/protobuf-java-2.5.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-resourcemanager-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-common-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang3-3.3.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/xmlenc-0.52.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-xc-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-ant-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apache-log4j-extras-1.2.17.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-util-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-0.23-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guava-11.0.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-core-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-scheduler-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-aliyun-2.7.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-dbcp-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compress-1.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-shims-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-io-2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/httpcore-4.1.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-mapreduce-client-core-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-core-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-core-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/janino-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jdo-api-3.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsp-api-2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/annotations-2.0.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/velocity-1.5.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-codec-1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-3.6.2.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-framework-2.6.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-tree-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jetty-6.1.26.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/calcite-core-1.0.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-core-1.8.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/groovy-all-2.1.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-yarn-api-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-server-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-api-jdo-3.2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/guice-3.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-rdbms-3.2.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/java-xmlbuilder-0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-common-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-client-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jersey-guice-1.9.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/libfb303-0.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/paranamer-2.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-runtime-3.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/zookeeper-3.4.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-annotation_1.0_spec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ST4-4.0.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jettison-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/datanucleus-core-3.2.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-asn1-api-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/geronimo-jaspic_1.0_spec-1.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/apacheds-kerberos-codec-2.0.0-M15.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hamcrest-core-1.3.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-commons-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/ant-launcher-1.9.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-exec-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-collections-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/asm-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stringtemplate-3.2.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hive-common-1.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jline-2.12.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/api-util-1.0.0-M20.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/stax-api-1.0.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-net-3.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/bonecp-0.8.0.RELEASE.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/gson-2.2.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-log4j12-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jets3t-0.9.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-lang-2.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/eigenbase-properties-1.1.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/antlr-2.7.7.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsch-0.1.42.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/leveldbjni-all-1.8.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/logback-classic-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jpam-1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/snappy-java-1.0.4.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jackson-mapper-asl-1.9.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-math3-3.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-pool-1.5.4.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/netty-all-4.0.23.Final.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-compiler-2.7.6.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/commons-daemon-1.0.13.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/javax.inject-1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/htrace-core-3.1.0-incubating.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/jsr305-3.0.0.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/slf4j-api-1.7.10.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/fastjson2-2.0.23.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/derby-10.11.1.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/curator-client-2.7.1.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/cos_api-bundle-5.6.137.2.jar","file:/u/chengken/datax/plugin/reader/hdfsreader/libs/hadoop-cos-3.1.0-8.3.2.jar"],"parent":{"URLs":["file:/u/chengken/datax/lib/commons-lang3-3.3.2.jar","file:/u/chengken/datax/lib/logback-core-1.0.13.jar","file:/u/chengken/datax/lib/commons-io-2.4.jar","file:/u/chengken/datax/lib/datax-common-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/hamcrest-core-1.3.jar","file:/u/chengken/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar","file:/u/chengken/datax/lib/logback-classic-1.0.13.jar","file:/u/chengken/datax/lib/commons-math3-3.1.1.jar","file:/u/chengken/datax/lib/slf4j-api-1.7.10.jar","file:/u/chengken/datax/lib/fastjson2-2.0.23.jar","file:/u/chengken/datax/lib/commons-logging-1.1.1.jar","file:/u/chengken/datax/lib/httpcore-4.4.13.jar","file:/u/chengken/datax/lib/commons-configuration-1.10.jar","file:/u/chengken/datax/lib/fluent-hc-4.5.jar","file:/u/chengken/datax/lib/commons-cli-1.2.jar","file:/u/chengken/datax/lib/janino-2.5.16.jar","file:/u/chengken/datax/lib/groovy-all-2.1.9.jar","file:/u/chengken/datax/lib/commons-codec-1.11.jar","file:/u/chengken/datax/lib/commons-collections-3.2.1.jar","file:/u/chengken/datax/lib/commons-lang-2.6.jar","file:/u/chengken/datax/lib/httpclient-4.5.13.jar","file:/u/chengken/datax/lib/commons-beanutils-1.9.2.jar","file:/u/chengken/datax/lib/datax-core-0.0.1-SNAPSHOT.jar","file:/home/svccnetlhs/chengken/starrocks/"],"parent":{"URLs":["file:/opt/software/jdk1.8.0_112/jre/lib/ext/jaccess.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunjce_provider.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunpkcs11.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/localedata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/nashorn.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/sunec.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/dnsns.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/zipfs.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/cldrdata.jar","file:/opt/software/jdk1.8.0_112/jre/lib/ext/jfxrt.jar"]}}},"finalParameters":["mapreduce.job.end-notification.max.retry.interval","mapreduce.job.end-notification.max.attempts"]}
2023-12-11 17:50:01.327 [0-0-0-reader] INFO  Reader$Task - read start
2023-12-11 17:50:01.327 [0-0-0-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
2023-12-11 17:50:01.328 [0-0-0-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
2023-12-11 17:50:01.328 [0-0-8-reader] INFO  Reader$Task - read start
2023-12-11 17:50:01.328 [0-0-8-reader] INFO  Reader$Task - reading file : [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc]
2023-12-11 17:50:01.328 [0-0-8-reader] INFO  HdfsReader$Job - Start Read orcfile [cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc].
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogBegin
信息: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
十二月 11, 2023 5:50:01 下午 org.apache.hadoop.conf.Configuration warnOnceIfDeprecated
信息: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
2023-12-11 17:50:01.646 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.776 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:01.778 [ORC_GET_SPLITS #1] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202185 duration=760 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.288 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202296 duration=871 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.369 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcInputFormat generateSplitsInfo
信息: FooterCacheHitRatio: 0/1
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.log.PerfLogger PerfLogEnd
信息: </PERFLOG method=OrcGetSplits start=1702288201425 end=1702288202433 duration=1008 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2023-12-11 17:50:02.492 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.647 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.784 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = null, max key = {originalTxn: 0, bucket: -1, row: 302079}
十二月 11, 2023 5:50:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 3, length: 9223372036854775807}
2023-12-11 17:50:02.867 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:11.235 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-12-11 17:50:21.236 [job-0] INFO  StandAloneJobContainerCommunicator - Total 297312 records, 558177719 bytes | Speed 53.23MB/s, 29731 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.023s |  All Task WaitReaderTime 25.333s | Percentage 0.00%
2023-12-11 17:50:30.385 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
十二月 11, 2023 5:50:30 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156410867, length: 9223372036854775807}
2023-12-11 17:50:30.731 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:30.907 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:31.114 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 599039}
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156011050, length: 9223372036854775807}
2023-12-11 17:50:31.237 [job-0] INFO  StandAloneJobContainerCommunicator - Total 598944 records, 1125524739 bytes | Speed 54.11MB/s, 30163 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.040s |  All Task WaitReaderTime 43.878s | Percentage 0.00%
2023-12-11 17:50:31.280 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 302079}, max key = {originalTxn: 0, bucket: -1, row: 604159}
十二月 11, 2023 5:50:31 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 156411995, length: 9223372036854775807}
2023-12-11 17:50:31.457 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:41.238 [job-0] INFO  StandAloneJobContainerCommunicator - Total 906144 records, 1702990600 bytes | Speed 55.07MB/s, 30720 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.056s |  All Task WaitReaderTime 62.147s | Percentage 0.00%
2023-12-11 17:50:51.241 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1203104 records, 2261515086 bytes | Speed 53.27MB/s, 29696 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.070s |  All Task WaitReaderTime 92.372s | Percentage 0.00%
2023-12-11 17:50:56.098 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:56 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 311140141, length: 9223372036854775807}
2023-12-11 17:50:56.469 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:57.564 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 604159}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:57 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 312563739, length: 9223372036854775807}
2023-12-11 17:50:57.971 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:50:58.447 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 599039}, max key = {originalTxn: 0, bucket: -1, row: 803839}
十二月 11, 2023 5:50:58 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 310368886, length: 9223372036854775807}
2023-12-11 17:50:58.782 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:01.242 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1703232 records, 3203376993 bytes | Speed 89.82MB/s, 50012 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.093s |  All Task WaitReaderTime 127.642s | Percentage 0.00%
2023-12-11 17:51:11.242 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1829984 records, 3440997771 bytes | Speed 22.66MB/s, 12675 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.101s |  All Task WaitReaderTime 138.648s | Percentage 0.00%
2023-12-11 17:51:15.910 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
十二月 11, 2023 5:51:16 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 417009783, length: 9223372036854775807}
2023-12-11 17:51:16.305 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:17.688 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1100799}
十二月 11, 2023 5:51:17 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415835058, length: 9223372036854775807}
2023-12-11 17:51:18.017 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:18.695 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 803839}, max key = {originalTxn: 0, bucket: -1, row: 1105919}
十二月 11, 2023 5:51:18 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 415904892, length: 9223372036854775807}
2023-12-11 17:51:19.066 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:21.243 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2342208 records, 4403928517 bytes | Speed 91.83MB/s, 51222 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.126s |  All Task WaitReaderTime 178.482s | Percentage 0.00%
2023-12-11 17:51:31.245 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2466496 records, 4638515258 bytes | Speed 22.37MB/s, 12428 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.132s |  All Task WaitReaderTime 190.077s | Percentage 0.00%
2023-12-11 17:51:41.246 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2967264 records, 5579213090 bytes | Speed 89.71MB/s, 50076 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.156s |  All Task WaitReaderTime 229.517s | Percentage 0.00%
2023-12-11 17:51:41.359 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
十二月 11, 2023 5:51:41 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 571347891, length: 9223372036854775807}
2023-12-11 17:51:41.747 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:41.998 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1100799}, max key = {originalTxn: 0, bucket: -1, row: 1300479}
十二月 11, 2023 5:51:42 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 570444727, length: 9223372036854775807}
2023-12-11 17:51:42.390 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:44.650 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1105919}, max key = {originalTxn: 0, bucket: -1, row: 1305599}
十二月 11, 2023 5:51:44 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 572342605, length: 9223372036854775807}
2023-12-11 17:51:44.978 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:51:51.248 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3307424 records, 6219717845 bytes | Speed 61.08MB/s, 34016 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.172s |  All Task WaitReaderTime 246.926s | Percentage 0.00%
2023-12-11 17:52:01.250 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3614624 records, 6797953036 bytes | Speed 55.14MB/s, 30720 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.187s |  All Task WaitReaderTime 279.975s | Percentage 0.00%
2023-12-11 17:52:02.446 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
十二月 11, 2023 5:52:02 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 674573070, length: 9223372036854775807}
2023-12-11 17:52:02.859 [0-0-8-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00008-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:03.369 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1300479}, max key = {originalTxn: 0, bucket: -1, row: 1597439}
十二月 11, 2023 5:52:03 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 673856234, length: 9223372036854775807}
2023-12-11 17:52:03.751 [0-0-0-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00002-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:04.567 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger <init>
信息: min key = {originalTxn: 0, bucket: -1, row: 1305599}, max key = {originalTxn: 0, bucket: -1, row: 1602559}
十二月 11, 2023 5:52:04 下午 org.apache.hadoop.hive.ql.io.orc.ReaderImpl rowsOptions
信息: Reading ORC rows from cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc with {include: null, offset: 675641495, length: 9223372036854775807}
2023-12-11 17:52:04.881 [0-0-17-reader] INFO  CosNFileSystem - Opening 'cosn://bucketsName/sam/sam_dwd_user_action_cos_d/20231114/part-00004-04661bb7-8d42-4e91-ae8e-47f996111f67-c000.snappy.orc' for reading
2023-12-11 17:52:11.251 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.202s |  All Task WaitReaderTime 297.463s | Percentage 0.00%
2023-12-11 17:52:21.252 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.215s |  All Task WaitReaderTime 331.832s | Percentage 0.00%
...

StarRocks数据顺利加载：

[Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > show partitions from ods.buckets_tmp_20231205__sams_cos;
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
| PartitionId | PartitionName | VisibleVersion | VisibleVersionTime  | VisibleVersionHash | State  | PartitionKey | Range                                                                      | DistributionKey | Buckets | ReplicationNum | StorageMedium | CooldownTime        | LastConsistencyCheckTime | DataSize | IsInMemory |
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
| 513180658   | p20231114     | 1              | 2023-12-11 09:49:41 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-14]; ..types: [DATE]; keys: [2023-11-15]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | 16.3GB   | false      |
| 504724038   | p20231115     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-15]; ..types: [DATE]; keys: [2023-11-16]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 504722860   | p20231116     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-11-16]; ..types: [DATE]; keys: [2023-11-17]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 504721682   | p20231206     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-06]; ..types: [DATE]; keys: [2023-12-07]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 504722271   | p20231207     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-07]; ..types: [DATE]; keys: [2023-12-08]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 504720504   | p20231208     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-08]; ..types: [DATE]; keys: [2023-12-09]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 504721093   | p20231209     | 1              | 2023-12-06 13:47:20 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-09]; ..types: [DATE]; keys: [2023-12-10]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 505174943   | p20231210     | 1              | 2023-12-07 00:14:47 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-10]; ..types: [DATE]; keys: [2023-12-11]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 506968779   | p20231211     | 1              | 2023-12-08 00:04:57 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-11]; ..types: [DATE]; keys: [2023-12-12]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 508936021   | p20231212     | 1              | 2023-12-09 00:11:09 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-12]; ..types: [DATE]; keys: [2023-12-13]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 510079030   | p20231213     | 1              | 2023-12-10 00:05:38 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-13]; ..types: [DATE]; keys: [2023-12-14]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
| 510795860   | p20231214     | 1              | 2023-12-11 00:07:15 | 0                  | NORMAL | ts           | [types: [DATE]; keys: [2023-12-14]; ..types: [DATE]; keys: [2023-12-15]; ) | uuid            | 196     | 2              | HDD           | 9999-12-31 15:59:59 | NULL                     | .000     | false      |
+-------------+---------------+----------------+---------------------+--------------------+--------+--------------+----------------------------------------------------------------------------+-----------------+---------+----------------+---------------+---------------------+--------------------------+----------+------------+
12 rows in set (0.002 sec)

[Mon Dec 11 09:56:57 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] >

查询正常：

[Mon Dec 11 09:58:20 2023]:['default_cluster:cndlopsns']>[192.168.1.121]:[ods] [ADHOC用户集群] > select * from ods.buckets_tmp_20231205__sams_cos where ts='2023-11-14' limit 2;
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
| ts         | uuid             | import_ds_ | unique_action_id                     | action_time   | report_time   | action_type          | ka_id    | action_session_id                                 | wx_app_id | wx_open_id | wx_union_id | external_user_id | merber_id         | local_id | encrypted_imei                       | encrypted_idfa | encrypted_mac | encrypted_android_id | encrypted_qq | encrypted_phone | encrypting_algorithm | chan_id | chan_refer_app_id | chan_shop_id             | chan_shop_name                                                                                                     | client_type  | client_name | client_version | sdk_version | device_model | ip   | user_agent | page_path    | page_name | referrer | address | city | province | country | latitude | longitude | json_properties                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | fdate    | chan_custom_id | etl_load_time | tag_id    | tag_name     | event_name |
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
| 2023-11-14 | 8325 | 2023111412 | ed55491f-da41-4976-8601-49c4405d994f | 1699936160765 | 1699936132584 | element              | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL        | 274012627        | 10742100529761108 | NULL     | 9090 | NULL           | NULL          | NULL                 | NULL         | NULL            | NULL                 | NULL    | NULL              | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL        | NULL           | NULL        | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL     | NULL    | NULL | NULL     | NULL    | NULL     | NULL      | *****                                                                                                                                                 | 20231114 | default        | NULL          | advantage | 普通会员     | NULL       |
| 2023-11-14 | 8614 | 2023111412 | f30f3672-1bad-478c-ab19-16b736b850ef | 1699936161032 | 1699936133093 | expose_sku_component | 10001042 | 1699936154050a1b33cce-f249-4112-8db2-288f1f0f6703 | NULL      | NULL       | NULL        | 274012627        | 10742100529761108 | NULL     | 9090 | NULL           | NULL          | NULL                 | NULL         | NULL            | NULL                 | NULL    | NULL              | 9991,6758,4834,6119,9996 | 苏州木渎DC                                     | sams-app-sdk | NULL        | NULL           | NULL        | NULL         | NULL | NULL       | HomeFragment | 首页      | NULL     | NULL    | NULL | NULL     | NULL    | NULL     | NULL      | *****                           | 20231114 | default        | NULL          | advantage | 普通会员     | NULL       |
+------------+------------------+------------+--------------------------------------+---------------+---------------+----------------------+----------+---------------------------------------------------+-----------+------------+-------------+------------------+-------------------+----------+--------------------------------------+----------------+---------------+----------------------+--------------+-----------------+----------------------+---------+-------------------+--------------------------+--------------------------------------------------------------------------------------------------------------------+--------------+-------------+----------------+-------------+--------------+------+------------+--------------+-----------+----------+---------+------+----------+---------+----------+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+----------------+---------------+-----------+--------------+------------+
2 rows in set (1.660 sec)

速率不断太慢，Datax也就这样了：

2023-12-11 17:52:11.251 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3906464 records, 7347149221 bytes | Speed 52.38MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.202s |  All Task WaitReaderTime 297.463s | Percentage 0.00%
2023-12-11 17:52:21.252 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4198304 records, 7896134390 bytes | Speed 52.36MB/s, 29184 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.215s |  All Task WaitReaderTime 331.832s | Percentage 0.00%

备注

之前有伙伴问到，为什么我的json里面字段用到“index”

"column": [
    {
        "name": "ts",
        "type": "string",
        "value": "2023-11-14"
    },
    {
        "name": "import_ds_",
        "type": "string",
        "index": 0
    },
    {
        "name": "unique_action_id",
        "type": "string",
        "index": 1
    },
    {
        "name": "action_time",
        "type": "string",
        "index": 2
    },
    ....
    ]

这个地方经过尝试，原有的column1，column2，column3...的方式测试行不通，因为我有个ts的字段需要造值。但如果使用

    {
        "name": "ts",
        "type": "string",
        "value": "2023-11-14"
    },
    {
        "name": "import_ds_",
        "type": "string",
    },
    ....

这种方式，则抛出：由于您配置了type, 则至少需要配置 index 或 value，这是一件令人头疼的事。

[svccnetlhs@HOST log]<231211 18:10:01>$ /usr/bin/python2.7 /u/chengken/datax/bin/datax.py --jvm="-Xms1G -Xmx4G" /home/svccnetlhs/chengken/starrocks/json/20231114_sam_dwd_user_action_cos__1702288181.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2023-12-11 18:10:30.213 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-12-11 18:10:30.215 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-12-11 18:10:30.226 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-12-11 18:10:30.231 [main] INFO  Engine - the machine info  => 

    osInfo:    Linux amd64 4.18.0-348.7.1.el8_5.x86_64
    jvmInfo:    Oracle Corporation 1.8 25.112-b15
    cpu num:    16

    totalPhysicalMemory:    -0.00G
    freePhysicalMemory:    -0.00G
    maxFileDescriptorCount:    -1
    currentOpenFileDescriptorCount:    -1

    GC Names    [PS MarkSweep, PS Scavenge]

    MEMORY_NAME                    | allocation_size                | init_size                      
    PS Eden Space                  | 1,280.00MB                     | 256.00MB                       
    Code Cache                     | 240.00MB                       | 2.44MB                         
    Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
    PS Survivor Space              | 42.50MB                        | 42.50MB                        
    PS Old Gen                     | 2,731.00MB                     | 683.00MB                       
    Metaspace                      | -0.00MB                        | 0.00MB                         



2023-12-11 18:10:30.258 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=true
2023-12-11 18:10:30.258 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-12-11 18:10:30.259 [main] INFO  JobContainer - Set jobId = 0
2023-12-11 18:10:30.269 [job-0] INFO  HdfsReader$Job - init() begin...
2023-12-11 18:10:30.274 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index].  - 由于您配置了type, 则至少需要配置 index 或 value
    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50) ~[hdfsreader-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na]
    at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-12-11 18:10:30.277 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-12-11 18:10:30.279 [job-0] ERROR Engine - 

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-06], Description:[没有 Index].  - 由于您配置了type, 则至少需要配置 index 或 value
    at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validateColumns(HdfsReader.java:150)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.validate(HdfsReader.java:111)
    at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Job.init(HdfsReader.java:50)
    at com.alibaba.datax.core.job.JobContainer.initJobReader(JobContainer.java:673)
    at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:303)
    at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
    at com.alibaba.datax.core.Engine.start(Engine.java:86)
    at com.alibaba.datax.core.Engine.entry(Engine.java:168)
    at com.alibaba.datax.core.Engine.main(Engine.java:201)

那么我是如何取得每个字段精准的index？

这里我用到了 orc-tools-1.8.0-uber.jar 这个包把orc里面的字段先解析出来，

下载：https://repo1.maven.org/maven2/org/apache/orc/orc-tools/

java -jar orc-tools-1.8.0-uber.jar meta <ORC文件>

成功解析orc文件的元数据字段信息，Type: struct：代表的就是字段列与下标顺序。

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Processing data file part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000 [length: 290734159]
Structure for part-01291-c9231ae0-6186-4b4a-83dc-c95521cf2b8d-c000
File Version: 0.12 with ORC_14 by ORC Java 1.6.14
Rows: 849920
Compression: SNAPPY
Compression size: 131072
Calendar: Julian/Gregorian
Type: struct<import_ds_:int,unique_action_id:string,action_time:bigint,report_time:bigint,action_type:string,ka_id:bigint,action_session_id:string,uuid:string,wx_app_id:string,wx_open_id:string,wx_union_id:string,external_user_id:string,merber_id:string,local_id:string,encrypted_imei:string,encrypted_idfa:string,encrypted_mac:string,encrypted_android_id:string,encrypted_qq:string,encrypted_phone:string,encrypting_algorithm:string,chan_id:string,chan_refer_app_id:string,chan_shop_id:string,chan_shop_name:string,client_type:string,client_name:string,client_version:string,sdk_version:string,device_model:string,ip:string,user_agent:string,page_path:string,page_name:string,referrer:string,address:string,city:string,province:string,country:string,latitude:string,longitude:string,json_properties:string,fdate:string,tag_id:string,tag_name:string,chan_custom_id:string,etl_load_time:string>

Stripe Statistics:

完。