You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/10 12:06:35 UTC
[GitHub] [hudi] aditiwari01 opened a new issue #2802: Hive read issues when different partition have different schemas.
aditiwari01 opened a new issue #2802:
URL: https://github.com/apache/hudi/issues/2802
Hive reads writer schema separately for each partition. If a schema has evolved and updates has not made for all partitions (i.e. for some partition last change was from older schema), they hive read for that partition would fail since non availability of new column in schema.
Concerned code: (Class: `AbstractRealtimeRecordReader`)
```
private void init() throws IOException {
Schema schemaFromLogFile = LogReaderUtils.readLatestSchemaFromLogFiles(split.getBasePath(), split.getDeltaLogPaths(), jobConf);
if (schemaFromLogFile == null) {
writerSchema = InputSplitUtils.getBaseFileSchema((FileSplit)split, jobConf);
LOG.info("Writer Schema From Parquet => " + writerSchema.getFields());
} else {
writerSchema = schemaFromLogFile;
LOG.info("Writer Schema From Log => " + writerSchema.toString(true));
}
```
I tried replacing this writer schema to get schema from TableSchemaResolver. This is working fine for me.
Does this look good? I haven't followed hive folw in detail yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-1002788785
@aditiwari01 : sorry, can you point me to the patch which you have put up on this regard. I can take a stab at it. I have filed a jira [here](https://issues.apache.org/jira/browse/HUDI-3130).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-1003784281
able to get hold of it, thanks. will close the github issue and follow up on the patch review. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-1002788785
@aditiwari01 : sorry, can you point me to the patch which you have put up on this regard. I can take a stab at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-819561416
Hi @n3nash
The issue I am facing is in the case when commit gets succeeded. Let me explain the issue with an example:
Commit1: insert key1 in partition1 and key2 in partition2 with schema1.
Commit2: update key2 in partition2 with schema2
In this case the schema2 is registered successfully after commit2. But the issue is that the `AbstractRealtimeRecordReader` is not reading the latest schema of the table. `AbstractRealtimeRecordReader` tries to read the latest schema for each partition from that last edited log/parquet file.
As for the given example, the last edit in partition1 was with schema1. Hence the above code snippet that I have pasted in first comment, takes inn schema1 as writerSchema, causing the issue.
Also I replace the said logic with taking writerSchema from TableSchemaResolver rather than log/parquet file and it solves the issue for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-946364855
@aditiwari01 : when you get a chance can you respond. Will close out in a week if we don't hear from you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-829195644
@n3nash Sorry for the delay. Attaching the complete stack trace for the issue faced.
Note: Here "newCol" was not present in the initial schema. I updated only one partition out of four with the new schema and when I try to do a select * on rt table the following error is thrown:
```
2021-04-29T17:49:00,849 WARN [HiveServer2-HttpHandler-Pool: Thread-61] thrift.ThriftCLIService: Error fetching results:
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hudi.exception.HoodieException: Field newcol not found in log schema. Query cannot proceed! Derived Schema Fields: [_hoodie_commit_time, _hoodie_partition_path, _hoodie_record_key, dim, _hoodie_commit_seqno, value, key, _hoodie_file_name, updatedat]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:465) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:309) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:911) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_172]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.1.jar:?]
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at com.sun.proxy.$Proxy57.fetchResults(Unknown Source) ~[?:?]
at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:561) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:787) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:207) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.Server.handle(Server.java:534) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) ~[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) ~[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) ~[jetty-io-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) ~[jetty-runner-9.3.20.v20170531.jar:9.3.20.v20170531]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
Caused by: java.io.IOException: org.apache.hudi.exception.HoodieException: Field newcol not found in log schema. Query cannot proceed! Derived Schema Fields: [_hoodie_commit_time, _hoodie_partition_path, _hoodie_record_key, dim, _hoodie_commit_seqno, value, key, _hoodie_file_name, updatedat]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2737) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:460) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
... 46 more
Caused by: org.apache.hudi.exception.HoodieException: Field newcol not found in log schema. Query cannot proceed! Derived Schema Fields: [_hoodie_commit_time, _hoodie_partition_path, _hoodie_record_key, dim, _hoodie_commit_seqno, value, key, _hoodie_file_name, updatedat]
at org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils.generateProjectionSchema(HoodieRealtimeRecordReaderUtils.java:138) ~[?:?]
at org.apache.hudi.hadoop.realtime.AbstractRealtimeRecordReader.init(AbstractRealtimeRecordReader.java:107) ~[?:?]
at org.apache.hudi.hadoop.realtime.AbstractRealtimeRecordReader.<init>(AbstractRealtimeRecordReader.java:67) ~[?:?]
at org.apache.hudi.hadoop.realtime.RealtimeCompactedRecordReader.<init>(RealtimeCompactedRecordReader.java:53) ~[?:?]
at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.constructRecordReader(HoodieRealtimeRecordReader.java:70) ~[?:?]
at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.<init>(HoodieRealtimeRecordReader.java:47) ~[?:?]
at org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat.getRecordReader(HoodieParquetRealtimeInputFormat.java:123) ~[?:?]
at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:776) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:344) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2737) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229) ~[hive-exec-3.1.2.jar:3.1.2.fk.10]
at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:460) ~[hive-service-3.1.2.fk.10.jar:3.1.2.fk.10]
... 46 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-1002574909
@nsivabalan I had to setup my env with latest hudi code. I have raised the PR with minor patch for the same. Kindly go through the change. I have tested it on local and on cluster as well. However, I. haven't added any unit tests for now. AFAIR when I initially worked on this I was unable to reproduce the issue via tests. Would be helpful if you can provide some pointers.
Also please let me know if there is some jira for the same. I'll tag in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-854401156
@aditiwari01 Sorry this slipped out of my radar, I will update here with my findings in the next couple of days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-820153027
@aditiwari01 Thanks for that explanation. Can you please add the failure message / exception that you see with the current code ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-826671636
Have quite a busy sprint. I'll try to get back. If not, I'll update by weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-905876095
@codope Thanks for the update. I might have lost my local setup and will try to test the latest code during the weekend.
As for why we're only updating a few partitions, we have lot of usecases wherein we're partitioning data based on timestamp. So it is very possible that after a schema update, older partitions never receive any update. But we'd like to continue reading the whole data as and when required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 edited a comment on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 edited a comment on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-819561416
Hi @n3nash
The issue I am facing is in the case when commit gets succeeded. Let me explain the issue with an example:
Commit1: insert key1 in partition1 and key2 in partition2 with schema1.
Commit2: update key2 in partition2 with schema2
In this case the schema2 is registered successfully after commit2. But the issue is that the `AbstractRealtimeRecordReader` is not reading the latest schema of the table. `AbstractRealtimeRecordReader` tries to read the latest schema for each partition from that last edited log/parquet file.
As for the given example, the last edit in partition1 was with schema1. Hence the above code snippet that I have pasted in first comment, takes inn schema1 as writerSchema, causing the issue. (when split belongs to partition1)
Also I replace the said logic with taking writerSchema from TableSchemaResolver rather than log/parquet file and it solves the issue for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-961627203
@codope : related to schema evolution. I remember you tried a similar usecase, testing out updating few partitions w/ a new filed added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-926144366
@aditiwari01 are you able to reconfirm this on 0.9.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-961627203
@codope : related to schema evolution. I remember you tried a similar usecase, testing out updating few partitions w/ a new filed added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-836220443
@n3nash Did you get a chance to look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #2802:
URL: https://github.com/apache/hudi/issues/2802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-818457749
@aditiwari01 What failure do you see ? Generally the following is the execution path
1. Provide new writer schema to Hudi
2. Write data with new writer schema to hudi log files
Here there are 2 possibilities
a. Commit this batch succeeds - In this case, the writerSchema for the `AbstractRealtimeRecordReader` will be this latest schema with which the latest data was written
b. Commit this batch fails - In this case, the writerSchema for the `AbstractRealtimeRecordReader` will be this latest schema by the previous successful commit with which the latest data for that commit was written. This failed commit will not have the latest schema or the data.
There shouldn't be a scenario where inside Hudi, a data record for the latest schema is written while the schema is not committed or vice-versa.
If you see some exceptions, please paste it here so I can understand your problem in more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-961627203
@codope : related to schema evolution. I remember you tried a similar usecase, testing out updating few partitions w/ a new filed added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-854401156
@aditiwari01 Sorry this slipped out of my radar, I will update here with my findings in the next couple of days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-905651663
Can you try with the latest master? There was a fix in this regard recently #3137
Also, the updates happening only to one of the few partitions intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-820157419
Actually we get Null pointer exceptionn when trying to get when we try to create readerSchema from writer schema and projectedFields, because the new column would present in projectedFields but not in writer schema.
I'm bit busy for next couple of days. Currently we're using the fix I've mentioned. Will try to rebuild with latest code and reproduce the issue and add the exact stack here in next few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-826577275
@aditiwari01 Any luck reproducing the issue and providing the stack trace ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-991915386
@aditiwari01 : sorry for dropping the ball on this. Can you put up the patch with the fix you have locally. we can help review and land it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2802: Hive read issues when different partition have different schemas.
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2802:
URL: https://github.com/apache/hudi/issues/2802#issuecomment-946511935
@nsivabalan This completely slipped my mind. Apologies for that.
I've tested the same with 0.10.0-SNAPSHOT and still facing the same issue.
I got some issue building 0.9.0 release version on my local so wasn't able to test on that. Let me kow if you guys want me to test on that as well.
FYI: We've been using the moinor patch mentioned above with 0.8.0 and that seems to be working fine for us. If that approach looks fine I can raise a PR for the same.
Let me know if you need any help reproducing it on your end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org