You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/17 02:06:47 UTC
[GitHub] [hudi] eric9204 opened a new issue, #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
eric9204 opened a new issue, #6964:
URL: https://github.com/apache/hudi/issues/6964
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
Error in querying hudi table stored in orc format with hive, and the query result is empty.
Steps to reproduce the behavior:
hoodie.datasource.write.operation=insert
hoodie.datasource.write.table.type=MERGE_ON_READ
hoodie.datasource.write.precombine.field=ts
hoodie.datasource.write.recordkey.field=id
hoodie.datasource.write.partitionpath.field=part
hoodie.table.name=ss_bucket_dsj_orc
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
hoodie.insert.shuffle.parallelism=8
hoodie.datasource.compaction.async.enable=true
hoodie.compact.inline.max.delta.commits=4
hoodie.index.type=BUCKET
hoodie.bucket.index.num.buckets=8
hoodie.bucket.index.hash.field=id
hoodie.storage.layout.partitioner.class=org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner
hoodie.storage.layout.type=BUCKET
hoodie.metadata.enable=true
hoodie.embed.timeline.server=false
path=/tmp/hudi/ss_bucket_dsj_orc
checkpointLocation=/tmp/hudi/ckp
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.username=ocdp
hoodie.datasource.hive_sync.database=default
hoodie.datasource.hive_sync.table=ss_bucket_dsj_orc
hoodie.datasource.hive_sync.password=ocdp
hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://10.9.2.8:10000
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.MultiPartKeysValueExtractor
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
`>= Hudi-0.11.1`
* Spark version :
Spark-3.1.1, Spark-3.1.2
* Hive version :
Hive-3.1.2, Hive-3.1.0
* Hadoop version :
Hadoop-3.3.0
* Storage (HDFS/S3/GCS..) :
Hdfs
* Running on Docker? (yes/no) :
No
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
2022-10-09T15:15:44,534 WARN [HiveServer2-Handler-Pool: Thread-22283855]: thrift.ThriftCLIService (:()) - Error fetching results:
java.lang.RuntimeException: java.lang.AssertionError: Index is not populated for 21
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:89) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_131]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) ~[hadoop-common-3.1.1.3.1.0.0-78.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at com.sun.proxy.$Proxy60.fetchResults(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:564) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-7
8]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3
.1.0.0-78]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3
.1.0.0-78]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647) ~[hive-e
xec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.AssertionError: Index is not populated for 21
at org.apache.orc.impl.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:923) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.orc.impl.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:1011) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1041) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1208) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1243) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:273) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:67) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:83) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:337) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:247) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1971) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:776) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:344) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2738) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:473) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
... 18 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6964:
URL: https://github.com/apache/hudi/issues/6964#issuecomment-1286482464
https://issues.apache.org/jira/browse/HUDI-4496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
URL: https://github.com/apache/hudi/issues/6964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6964:
URL: https://github.com/apache/hudi/issues/6964#issuecomment-1287878079
Please follow the jira for updates.
https://github.com/apache/hudi/pull/6227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6964: [SUPPORT] Error in querying hudi table stored in orc format with hive,Index is not populated for 10 (state=,code=0)
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6964:
URL: https://github.com/apache/hudi/issues/6964#issuecomment-1286482266
I don't think our ORC works w/ spark3.X. we are looking to fix it, but as of now, it has some gaps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org