You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Kip Kohn (Jira)" <ji...@apache.org> on 2023/12/14 20:14:00 UTC
[jira] [Closed] (GOBBLIN-1972) Fix Iceberg-Distcp, which now erroneously shows the job finishing as "SUCCESSFUL", despite the commit step failing with an exception.
[ https://issues.apache.org/jira/browse/GOBBLIN-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kip Kohn closed GOBBLIN-1972.
-----------------------------
> Fix Iceberg-Distcp, which now erroneously shows the job finishing as "SUCCESSFUL", despite the commit step failing with an exception.
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1972
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1972
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Reporter: Kip Kohn
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Find and fix what appears to be a regression in `CopyDataPublisher` or `SafeDatasetCommit`...
> Here, the same Iceberg-Distcp `CommitStep` fails (`org.apache.gobblin.data.management.copy.iceberg.IcebergRegisterStep`), but with apache gobblin build `0.18.0-dev-493` it correctly shows "FAILED" yet with `0.18.0-dev-515` it claims "SUCCEEDED".
> BOTH display the same error:
> {code:java}
> ERROR [CopyDataPublisher] [Azkaban] Failed to publish tracking_db.MyEvent
> [then the exception stacktrace for the commit step, followed by...]
> ERROR [SafeDatasetCommit] [Azkaban] Failed to commit dataset
> - java.io.IOException: Not all datasets published successfully
> - at org.apache.gobblin.data.management.copy.publisher.CopyDataPublisher.publishData(CopyDataPublisher.java:160)
> - at org.apache.gobblin.publisher.DataPublisher.publish(DataPublisher.java:79)
> - at org.apache.gobblin.runtime.SafeDatasetCommit.commitDataset(SafeDatasetCommit.java:260)
> - at org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:168)
> [...]{code}
> but only "FAILED" continues with:
> {code:java}
> WARN [SafeDatasetCommit] [Azkaban] At least one task in CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent) did not get committed successfully. Setting dataset state to FAILED. java.io.IOException: Not all datasets published successfully
> - at org.apache.gobblin.data.management.copy.publisher.CopyDataPublisher.publishData(CopyDataPublisher.java:160)
> - at org.apache.gobblin.publisher.DataPublisher.publish(DataPublisher.java:79)
> - at org.apache.gobblin.runtime.SafeDatasetCommit.commitDataset(SafeDatasetCommit.java:260)
> - at org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:168)
> [...]
> INFO [AbstractJobLauncher] [Azkaban] Due to task failure, will report that no records or bytes were written for CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent){code}
> by contrast, "SUCCEEDED" announces:
> {code:java}
> INFO [SafeDatasetCommit] [Azkaban] Persisting dataset state for dataset CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent)
> [...]
> INFO [AbstractJobLauncher] [Azkaban] DatasetMetrics for 'CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent)' - (records: 0; bytes: 0) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)