You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/22 21:24:20 UTC
[GitHub] [hudi] stackfun opened a new issue #2368: [SUPPORT]
stackfun opened a new issue #2368:
URL: https://github.com/apache/hudi/issues/2368
**Describe the problem you faced**
When reading avro log files, the query for MOR tables fails in GCP.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a fairly large MOR Table on GCS (Seems to force the use of the buffered input stream)
2. Snapshot Query on MOR Table using spark datasource.
**Expected behavior**
Snapshot query returns successfully
**Environment Description**
* Hudi version : 0.6.0
* Dataproc 1.4
* Spark version : 2.4.5
* Hive version : 2.3.7
* Hadoop version : 2.9.2
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : no
**Stacktrace**
```
Caused by: java.io.EOFException: Invalid seek offset: position value (18631040) must be between 0 and 18631040 for 'gs:/******'
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.validatePosition(GoogleCloudStorageReadChannel.java:710)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.position(GoogleCloudStorageReadChannel.java:597)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.seek(GoogleHadoopFSInputStream.java:198)
at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:96)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
at org.apache.hudi.common.table.log.block.HoodieLogBlock.safeSeek(HoodieLogBlock.java:267)
at org.apache.hudi.common.table.log.block.HoodieLogBlock.readOrSkipContent(HoodieLogBlock.java:225)
at org.apache.hudi.common.table.log.HoodieLogFileReader.createCorruptBlock(HoodieLogFileReader.java:226)
at org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:146)
at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:346)
... 25 more
```
Here's the code in FSUtils.isGCSInputStream called by HoodieLogBlock.safeSeek
```java
public static boolean isGCSInputStream(FSDataInputStream inputStream) {
return inputStream.getClass().getCanonicalName().equals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream")
|| inputStream.getWrappedStream().getClass().getCanonicalName()
.equals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream");
}
```
In my callstack, the input stream's canonical name is `org.apache.hadoop.fs.FSDataInputStream` and the wrapped stream is `org.apache.hadoop.fs.BufferedFSInputStream` so this function fails to detect that the input stream is the `GoogleHadoopFSInputStream`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stackfun closed issue #2368: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
stackfun closed issue #2368:
URL: https://github.com/apache/hudi/issues/2368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stackfun commented on issue #2368: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
stackfun commented on issue #2368:
URL: https://github.com/apache/hudi/issues/2368#issuecomment-749783535
Please delete, accidentally created a duplicate. Sorry!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org