You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2022/01/20 19:33:00 UTC
[jira] [Commented] (HUDI-3262) Integration test suite failure
[ https://issues.apache.org/jira/browse/HUDI-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479629#comment-17479629 ]
sivabalan narayanan commented on HUDI-3262:
-------------------------------------------
I tested integ test bundle for spark2 and its all good w/ latest master.
{code:java}
.
.
.
22/01/20 19:30:24 INFO DagScheduler: Finished executing 7cbb0f66-6b04-40b7-a471-ece73a2a8779
22/01/20 19:30:24 WARN DagScheduler: Executing node "second_insert" :: {"name":"95c3111e-30d6-4c6d-b55b-544d0e5dd9aa","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":10000,"config":"second_insert"}
22/01/20 19:30:24 INFO DagNode: Generating input data for node 95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:24 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:30:25 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/2/f48b4f6b-8957-4346-8c66-945b67063954.avro
22/01/20 19:30:25 INFO DagNode: Configs : {"name":"95c3111e-30d6-4c6d-b55b-544d0e5dd9aa","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":10000,"config":"second_insert"}
22/01/20 19:30:25 INFO DagNode: Inserting input data 95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:25 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector, checkpoint: Option{val=1} sourceLimit: 9223372036854775807 lastBatchId: 1 nextBatchId: 2
00:10 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:11 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:13 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:30:31 INFO DagScheduler: Finished executing 95c3111e-30d6-4c6d-b55b-544d0e5dd9aa
22/01/20 19:30:31 WARN DagScheduler: Executing node "third_insert" :: {"name":"965a7f10-cfaa-4086-9233-386ab5ca8086","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":300,"config":"third_insert"}
22/01/20 19:30:31 INFO DagNode: Generating input data for node 965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:31 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:30:31 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/3/d20d3a9b-dea3-431e-aa3f-ee203f716b6f.avro
22/01/20 19:30:31 INFO DagNode: Configs : {"name":"965a7f10-cfaa-4086-9233-386ab5ca8086","record_size":1000,"repeat_count":1,"num_partitions_insert":1,"num_records_insert":300,"config":"third_insert"}
22/01/20 19:30:31 INFO DagNode: Inserting input data 965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:31 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector, checkpoint: Option{val=2} sourceLimit: 9223372036854775807 lastBatchId: 2 nextBatchId: 3
00:15 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:16 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:17 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:30:36 INFO DagScheduler: Finished executing 965a7f10-cfaa-4086-9233-386ab5ca8086
22/01/20 19:30:36 WARN DagScheduler: Executing node "first_hive_sync" :: {"queue_name":"adhoc","engine":"mr","name":"427549c7-65f3-484d-8e85-7cf4adfb4b77","config":"first_hive_sync"}
22/01/20 19:30:36 INFO DagNode: Executing hive sync node
22/01/20 19:30:41 INFO DagScheduler: Finished executing 427549c7-65f3-484d-8e85-7cf4adfb4b77
22/01/20 19:30:41 WARN DagScheduler: Executing node "first_validate" :: {"name":"61f58646-16be-49ce-8695-bdf5c49f1f31","validate_hive":false,"config":"first_validate"}
22/01/20 19:30:41 WARN DagNode: Validation using data from input path /user/hive/warehouse/hudi-integ-test-suite/input/*/*
22/01/20 19:30:45 INFO ValidateDatasetNode: Validate data in target hudi path /user/hive/warehouse/hudi-integ-test-suite/output/*/*/*
22/01/20 19:31:01 INFO DagScheduler: Finished executing 61f58646-16be-49ce-8695-bdf5c49f1f31
22/01/20 19:31:01 WARN DagScheduler: Executing node "first_upsert" :: {"name":"dcbd7071-f044-4a3a-b245-af5b873e5860","record_size":1000,"repeat_count":1,"num_records_upsert":100,"num_partitions_insert":1,"num_records_insert":300,"num_partitions_upsert":1,"config":"first_upsert"}
22/01/20 19:31:01 INFO DagNode: Generating input data dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:01 INFO DFSHoodieDatasetInputReader: NumPartitions : Option{val=1}, NumFiles : Option{val=1}, numRecordsToUpdate : Option{val=100}, percentageRecordsPerFile : Optional.empty
22/01/20 19:31:01 INFO DFSHoodieDatasetInputReader: Finished generating updates
22/01/20 19:31:01 INFO DeltaGenerator: Repartitioning records into 1 partitions for updates
22/01/20 19:31:01 INFO DeltaGenerator: Repartitioning records done for updates
22/01/20 19:31:02 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:31:02 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1000
22/01/20 19:31:02 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/4/b3966bb1-e279-42e9-ba9f-fdd61afedc18.avro
22/01/20 19:31:02 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/4/8c455a6e-f514-4640-97a3-11a701fb3a55.avro
22/01/20 19:31:02 INFO DagNode: Configs : {"name":"dcbd7071-f044-4a3a-b245-af5b873e5860","record_size":1000,"repeat_count":1,"num_records_upsert":100,"num_partitions_insert":1,"num_records_insert":300,"num_partitions_upsert":1,"config":"first_upsert"}
22/01/20 19:31:02 INFO DagNode: Inserting input data dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:02 INFO DagNode: Upserting input data dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:02 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector, checkpoint: Option{val=3} sourceLimit: 9223372036854775807 lastBatchId: 3 nextBatchId: 4
00:51 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:52 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
00:54 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:31:12 INFO DagScheduler: Finished executing dcbd7071-f044-4a3a-b245-af5b873e5860
22/01/20 19:31:12 WARN DagScheduler: Executing node "first_delete" :: {"name":"053b3a65-737c-4afd-bb52-6e009a5a5179","num_partitions_delete":1,"num_records_delete":2000,"config":"first_delete"}
22/01/20 19:31:12 INFO DFSHoodieDatasetInputReader: NumPartitions : Option{val=1}, NumFiles : Option{val=1}, numRecordsToUpdate : Option{val=2000}, percentageRecordsPerFile : Optional.empty
22/01/20 19:31:13 INFO DFSHoodieDatasetInputReader: Finished generating updates
22/01/20 19:31:13 INFO DeltaGenerator: Repartitioning records for delete
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:13 WARN GenericRecordFullPayloadGenerator: The schema does not have any collections/complex fields. Cannot achieve minPayloadSize : 1024
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/3efade2a-70e2-454a-a8a9-5f253cd306df.avro
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/58d7c3fb-c5fa-4d9a-93db-fc00b18dbf9e.avro
22/01/20 19:31:14 INFO AvroFileDeltaInputWriter: New Avro File : hoodie-hdfs:/user/hive/warehouse/hudi-integ-test-suite/input/5/1c102439-a98a-45a7-98a7-44f0c167a4b6.avro
22/01/20 19:31:14 INFO DagNode: Configs : {"name":"053b3a65-737c-4afd-bb52-6e009a5a5179","num_partitions_delete":1,"num_records_delete":2000,"config":"first_delete"}
22/01/20 19:31:14 INFO DagNode: Inserting input data 053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:14 INFO DagNode: Deleting input data 053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:14 INFO HoodieTestSuiteJob: Using DFSTestSuitePathSelector, checkpoint: Option{val=4} sourceLimit: 9223372036854775807 lastBatchId: 4 nextBatchId: 5
01:04 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
01:05 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
01:08 WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled. Falling back to direct markers.
22/01/20 19:31:26 INFO DagScheduler: Finished executing 053b3a65-737c-4afd-bb52-6e009a5a5179
22/01/20 19:31:26 WARN DagScheduler: Executing node "second_hive_sync" :: {"queue_name":"adhoc","engine":"mr","name":"dcb5e6c7-a6a4-469d-9510-bfdacc240b04","config":"second_hive_sync"}
22/01/20 19:31:26 INFO DagNode: Executing hive sync node
22/01/20 19:31:27 INFO DagScheduler: Finished executing dcb5e6c7-a6a4-469d-9510-bfdacc240b04
22/01/20 19:31:27 WARN DagScheduler: Executing node "second_validate" :: {"delete_input_data":true,"name":"12a098f1-0a27-45c2-ad2c-86012d8f9106","validate_hive":false,"config":"second_validate"}
22/01/20 19:31:27 WARN DagNode: Validation using data from input path /user/hive/warehouse/hudi-integ-test-suite/input/*/*
22/01/20 19:31:27 INFO ValidateDatasetNode: Validate data in target hudi path /user/hive/warehouse/hudi-integ-test-suite/output/*/*/*
22/01/20 19:31:38 INFO DagScheduler: Finished executing 12a098f1-0a27-45c2-ad2c-86012d8f9106
22/01/20 19:31:38 INFO DagScheduler: Finished workloads for round num 1
22/01/20 19:31:38 INFO DagScheduler: Finished workloads
22/01/20 19:31:38 INFO HoodieTestSuiteJob: Finished scheduling all tasks, Time taken 88385{code}
> Integration test suite failure
> ------------------------------
>
> Key: HUDI-3262
> URL: https://issues.apache.org/jira/browse/HUDI-3262
> Project: Apache Hudi
> Issue Type: Bug
> Components: tests-ci
> Reporter: Raymond Xu
> Assignee: sivabalan narayanan
> Priority: Critical
> Labels: sev:normal
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> detailed in https://github.com/apache/hudi/issues/4621
--
This message was sent by Atlassian Jira
(v8.20.1#820001)