You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/10 19:27:08 UTC
[GitHub] [hudi] ws-dohashi opened a new issue, #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
ws-dohashi opened a new issue, #5553:
URL: https://github.com/apache/hudi/issues/5553
**Describe the problem you faced**
Upgrading to 0.11.0 has caused consistent EOFExceptions on compaction for our Flink job. We have not seen this exception in 0.10.* versions.
**To Reproduce**
Steps to reproduce the behavior:
1. Use Flink to write and compact data in S3 ?
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.11.0
* Flink version : 1.14.2
* Hive version : 2.3.9
* Hadoop version : 2.10.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Our config/setup looks something like this:
```
val path = "$basePath/$databaseName/$hudiTableName"
conf.setString(FlinkOptions.PATH, path)
conf.setString(FlinkOptions.TABLE_NAME, hudiTableName)
conf.setString(FlinkOptions.TABLE_TYPE, "MERGE_ON_READ")
conf.setString(FlinkOptions.SOURCE_AVRO_SCHEMA, schema)
conf.setInteger(FlinkOptions.WRITE_TASKS, 4)
conf.setInteger(FlinkOptions.COMPACTION_TASKS, 4)
conf.setString(FlinkOptions.RECORD_KEY_FIELD, "primaryKey")
conf.setString(FlinkOptions.PRECOMBINE_FIELD, "primaryKey") // TODO: pick something more intelligibly
conf.setString(FlinkOptions.PARTITION_PATH_FIELD, "") // TODO: determine partition strategy
conf.setString(FlinkOptions.KEYGEN_TYPE, KeyGeneratorType.NON_PARTITION.name)
conf.setInteger(FlinkOptions.COMPACTION_DELTA_COMMITS, checkpointsPerCompaction)
....
dataStream.let { Pipelines.bootstrap(conf, rowType, 1, it) }
.let { Pipelines.hoodieStreamWrite(conf, 1, it) }
.let { Pipelines.compact(conf, it) }
```
**Stacktrace**
```
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 6 for operator stream_write (3/4)#0. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258)
... 22 more
Caused by: org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from s3://bucket/database/table/.hoodie/hoodie.properties
at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:254)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:125)
at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:78)
at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:668)
at org.apache.hudi.client.BaseHoodieClient.createMetaClient(BaseHoodieClient.java:138)
at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1448)
at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1496)
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:140)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:184)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219)
... 33 more
Caused by: java.io.IOException: Unexpected end of stream pos=5200, contentLength=5175
at [com.amazon.ws](http://com.amazon.ws/).emr.hadoop.fs.s3.S3FSInputStream.read(S3FSInputStream.java:297)
at [java.io](http://java.io/).BufferedInputStream.fill(BufferedInputStream.java:246)
at [java.io](http://java.io/).BufferedInputStream.read1(BufferedInputStream.java:286)
at [java.io](http://java.io/).BufferedInputStream.read(BufferedInputStream.java:345)
at [java.io](http://java.io/).DataInputStream.read(DataInputStream.java:149)
at [java.io](http://java.io/).DataInputStream.read(DataInputStream.java:100)
at java.util.Properties$LineReader.readLine(Properties.java:508)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:303)
at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:244)
... 50 more
Caused by: java.io.EOFException: Invalid position: 5200, exceeds the bounds of the stream: [0, 5175]
at [com.amazon.ws](http://com.amazon.ws/).emr.hadoop.fs.s3.S3FSInputStream.throwPositionOutOfBoundsException(S3FSInputStream.java:411)
at [com.amazon.ws](http://com.amazon.ws/).emr.hadoop.fs.s3.S3FSInputStream.retrieveInputStreamWithInfo(S3FSInputStream.java:385)
at [com.amazon.ws](http://com.amazon.ws/).emr.hadoop.fs.s3.S3FSInputStream.reopenStream(S3FSInputStream.java:378)
at [com.amazon.ws](http://com.amazon.ws/).emr.hadoop.fs.s3.S3FSInputStream.read(S3FSInputStream.java:260)
... 60 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138118194
Okey, our user feedback shows that the bug is fixed, maybe there are some other cases that cause this problem too, did you use the MOR table and what is your checkpoint interval ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
qianchutao commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138040724
i had the same problem, i used flink 1.14.0 stream write to hudi 0.11.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123160614
I think I found the cause. We set the checkpoint interval to 1 min and `COMPACTION_DELTA_COMMITS` to 2 commits for testing purposes. If the time used for compaction is more than the checkpoint interval, the exception will raise. Could you imagine what would happen in that case? @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1134143994
And i believe this exception is very probably caused by this PR: https://github.com/apache/hudi/commit/ed106f671e929d2a6277ae189242201a02a7b58f
cc @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
umehrot2 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1133216912
We are facing same issue via Spark SQL DML for merge on read tables:
```
Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: hudi_bootstrap_test_cow_non_hive_style_partitioned_9809f361_71e8_4053_baac_684db11f6b8e; line 1 pos 23;
'Project ['event_name]
+- 'Filter ('event_id = 4)
+- 'UnresolvedRelation [hudi_bootstrap_test_cow_non_hive_style_partitioned_9809f361_71e8_4053_baac_684db11f6b8e], [], false
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:123)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:302)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:301)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:301)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
```
When we disable metadata table we are able to get past this. This does need further investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1129870494
@xccui are we good to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
qianchutao commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138124161
> Okey, our user feedback shows that the bug is fixed, maybe there are some other cases that cause this problem too, did you use the MOR table and what is your checkpoint interval ?
Do you have a packed patch? I wanted to see if I could solve the problem in my development environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123137435
1. We manually appended the compaction operator to the pipeline with `Pipelines.compact(conf, it)`. Would get the exception every time after the first compaction finishes successfully.
2. Verified the length of the new `hoodie.properties` file, it's 5200.
3. This only happens after we migrated to version 0.11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123170449
I can give some details/background here,
the `CompactionPlanOperator` tries to schedule new compaction plan on each checkpoint complete event, in release 0.11, we tweak the strategy to not schedule new in the last compaction does not finish yet. While before 0.11, the new compaction is scheduled no matter whether the last compaction is on-going.
We also add a timed-out rollback strategy in release 0.11, the default timeout interval is 20 minutes. If a compaction does not finish within 20 minutes, we also try to rollback it, see: https://github.com/apache/hudi/blob/4258a715174e0a97271112148ab20ef9307e2bd8/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionPlanOperator.java#L91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1134270047
Have fired a fix for flink here: https://github.com/apache/hudi/pull/5660
https://issues.apache.org/jira/browse/HUDI-3782 and
https://issues.apache.org/jira/browse/HUDI-4138
In general, we should never make any side effect in the read code path of hoodie table config.
And hoodie table metadata writer.
I'm not sure how to fix this in Spark side, have two ways to fix on my mind:
1. make table config concurrency safe (not suggested because it is too heavy for a config)
2. make sure the metadata cleaning only happens once for the whole Job lifetime (still risky because there may be multiple jobs, but with very small probability).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123129848
On compaction do you mean this is a separate compaction job ? The error stacktrace indicates that the errors throws when hoodie.properties is read, kind of think this is very about the memory consumption, the input stream does not have enough memory to read the bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1133965682
It seems that Hudi tries to use the new length to read data from the old file stream, even though the file has been updated in S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1134102604
My insight was that there are some concurrency conflict when multiple threads try to read and modify the hoodie.properties file. After the refactoring from: https://github.com/apache/hudi/pull/4739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123146512
We did tweak the compaction strategy in 0.11 here: https://github.com/apache/hudi/blob/4258a715174e0a97271112148ab20ef9307e2bd8/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionPlanOperator.java#L110, to only compact one plan at a time and no scheduling if there is a pending compaction on-going.
But it seems not relative to the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138502573
See https://github.com/apache/hudi/pull/5660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ws-dohashi commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
ws-dohashi commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138816937
@danny0405 , looks like @xccui and I were getting a similar but unrelated error with the latest patch. We have since resolved it and we haven't seen the same `Unexpected end of stream` error since the patch- thanks for your assistance!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1123164002
Problem solved after we increase the checkpoint interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1129930460
> @xccui are we good to close this?
Hi @xushiyan, we triggered the exception a few times with "improper" checkpoint/compaction settings these days. I believe there's a bug that requires further investigation. But it's fine to close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ws-dohashi closed issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
ws-dohashi closed issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
URL: https://github.com/apache/hudi/issues/5553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138825163
Yes, sorry for the misleading and thanks for helping with this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qianchutao commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
qianchutao commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138123328
> Okey, our user feedback shows that the bug is fixed, maybe there are some other cases that cause this problem too, did you use the MOR table and what is your checkpoint interval ?
I'm using the MOR table, and the checkpoint interval is 120 seconds，My configuration is as follows
flink 1.14
hudi 0.11
hive 3.1.2
Use S3 as the data store
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xccui commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1138062379
> Have fired a fix for flink here: #5660
>
> https://issues.apache.org/jira/browse/HUDI-3782 and https://issues.apache.org/jira/browse/HUDI-4138 may cause this bug.
>
> The `HoodieTable#getMetadataWriter` is used by many async table service such as cleaning, compaction, clustering and so on, this method now would try to modify the table config each time it is called no matter whether metadata table is enabled/disabled.
>
> In general, we should never make any side effect in the read code path of hoodie table config. And hoodie table metadata writer.
>
> I'm not sure how to fix this on Spark side, have two ways to fix on my mind:
>
> 1. make table config concurrency safe (not suggested because it is too heavy for a config)
> 2. make sure the metadata cleaning only happens once for the whole Job lifetime (still risky because there may be multiple jobs, but with very small probability). I would suggest this way from my side.
Hi @danny0405, we tested our job with this patch applied but still got the same exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1134102712
cc @alexeykudinkin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5553: [SUPPORT] java.io.EOFException: Invalid position - exceeds the bounds of the stream
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5553:
URL: https://github.com/apache/hudi/issues/5553#issuecomment-1132900528
> with "improper" checkpoint/compaction settings these days
@xccui can you describe these with more details? we might file a bug if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org