You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/03/21 06:57:56 UTC
[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1432:
[HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
lamber-ken opened a new pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432
## What is the purpose of the pull request
More context, please go ahead with https://issues.apache.org/jira/browse/HUDI-716
Fix `Not an avro data file ...`
```
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Not an Avro data file
at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50)
at org.apache.hudi.common.util.AvroUtils.deserializeAvroMetadata(AvroUtils.java:147)
at org.apache.hudi.common.util.CleanerUtils.getCleanerPlan(CleanerUtils.java:87)
at org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:141) ... 24 more
```
With 0.5.1 onwards, hudi would first scan all files and store the plan (atomically) in .clean.requested and .clean.inflight before triggering the actual clean. If there are any intermittent failures, subsequent clean would read the cleaner plan again and retry.


## Verify this pull request
This pull request is a trivial rework without any test coverage.
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] tooptoop4 commented on issue #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-602258783
can this go into 0.5.2 release?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] hmatu commented on issue #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
hmatu commented on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-614241591
@lamber-ken thanks for the changes, I have been running into the same issue and worked! 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-602005682
hi @bvaradar, just `LOG.warn` when met exception, we can't be sure if zero byte files causes it every time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r400043115
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java
##########
@@ -485,6 +485,14 @@ public static void deleteOlderRollbackMetaFiles(FileSystem fs, String metaPath,
});
}
+ public static void deleteInstantFile(FileSystem fs, String metaPath, HoodieInstant instant) {
+ try {
+ fs.delete(new Path(metaPath, instant.getFileName()), false);
Review comment:
Add a Log message to not what is getting deleted ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar merged pull request #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
bvaradar merged pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on issue #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-605907112
hi @bvaradar, your suggestion is reasonable, all comments are addressed, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r400070949
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/FSUtils.java
##########
@@ -485,6 +485,14 @@ public static void deleteOlderRollbackMetaFiles(FileSystem fs, String metaPath,
});
}
+ public static void deleteInstantFile(FileSystem fs, String metaPath, HoodieInstant instant) {
+ try {
+ fs.delete(new Path(metaPath, instant.getFileName()), false);
Review comment:
reasonable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r400041838
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/UpgradesCommand.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FSUtils;
+import org.apache.log4j.Logger;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.stereotype.Component;
+
+import java.io.IOException;
+
+/**
+ * CLI command to during upgrading.
+ */
+@Component
+public class UpgradesCommand implements CommandMarker {
+ private static final Logger LOG = Logger.getLogger(UpgradesCommand.class);
+
+ @CliCommand(value = "cleanup corrupted files", help = "cleanup corrupted files")
+ public void cleanUpCorruptedFiles() {
Review comment:
minor : rename to removeCorruptedPendingCleanAction ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1432:
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-625984466
There was a follow on fix as well for this IIRC.. cc @lamber-ken @bvaradar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r396116123
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieCleanClient.java
##########
@@ -85,7 +85,11 @@ public HoodieCleanMetadata clean(String startCleanTime) throws HoodieIOException
// If there are inflight(failed) or previously requested clean operation, first perform them
table.getCleanTimeline().filterInflightsAndRequested().getInstants().forEach(hoodieInstant -> {
LOG.info("There were previously unfinished cleaner operations. Finishing Instant=" + hoodieInstant);
- runClean(table, hoodieInstant);
+ try {
+ runClean(table, hoodieInstant);
+ } catch (Exception e) {
+ LOG.warn("Failed to perform previous clean operation, instant: " + hoodieInstant, e);
Review comment:
@lamber-ken : Can we rollback the corrupted instant seamlessly in this case ?
Can you also add a CLI (called timeline upgrade) which would read the timeline and fix upgrade issues. For now, the CLI can just handle only known issue - to deal with corrupted inflight clean instants. We can have users run them before upgrade for now and also see if this can be done seamlessly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r396116882
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieCleanClient.java
##########
@@ -85,7 +85,11 @@ public HoodieCleanMetadata clean(String startCleanTime) throws HoodieIOException
// If there are inflight(failed) or previously requested clean operation, first perform them
table.getCleanTimeline().filterInflightsAndRequested().getInstants().forEach(hoodieInstant -> {
LOG.info("There were previously unfinished cleaner operations. Finishing Instant=" + hoodieInstant);
- runClean(table, hoodieInstant);
+ try {
+ runClean(table, hoodieInstant);
+ } catch (Exception e) {
+ LOG.warn("Failed to perform previous clean operation, instant: " + hoodieInstant, e);
Review comment:
Good idea 👍 , will update the pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r400042557
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/UpgradesCommand.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FSUtils;
+import org.apache.log4j.Logger;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.stereotype.Component;
+
+import java.io.IOException;
+
+/**
+ * CLI command to during upgrading.
+ */
+@Component
+public class UpgradesCommand implements CommandMarker {
Review comment:
On second thought, I think this method can be best present in existing RepairsCommand class instead of this new class. Let me know your thoughts ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken edited a comment on issue #1432:
[HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken edited a comment on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-602005682
hi @bvaradar, just `LOG.warn` when met exception, we can't be sure if zero byte files causes it every time.
if user see the `Not an Avro data file at` warn message, they can remove them manually. Also, will add this note to troubleshooting guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1432: [HUDI-716]
Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-602310355
@tooptoop4 unfortunately not... we froze code for 0.5.2 and been since only taking LICENSE/NOTICE related fixes..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] lamber-ken commented on a change in pull request
#1432: [HUDI-716] Exception: Not an Avro data file when running
HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
lamber-ken commented on a change in pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
URL: https://github.com/apache/incubator-hudi/pull/1432#discussion_r399455231
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieCleanClient.java
##########
@@ -85,7 +85,11 @@ public HoodieCleanMetadata clean(String startCleanTime) throws HoodieIOException
// If there are inflight(failed) or previously requested clean operation, first perform them
table.getCleanTimeline().filterInflightsAndRequested().getInstants().forEach(hoodieInstant -> {
LOG.info("There were previously unfinished cleaner operations. Finishing Instant=" + hoodieInstant);
- runClean(table, hoodieInstant);
+ try {
+ runClean(table, hoodieInstant);
+ } catch (Exception e) {
+ LOG.warn("Failed to perform previous clean operation, instant: " + hoodieInstant, e);
Review comment:
Sorry for delay, had updated the pr, any suggestion are welcome, thanks @bvaradar
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] tooptoop4 commented on pull request #1432: [HUDI-716] Exception: Not an Avro data file when running HoodieCleanClient.runClean
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on pull request #1432:
URL: https://github.com/apache/incubator-hudi/pull/1432#issuecomment-625857986
@bhasudha pls include in 0.5.3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org