You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/21 14:42:29 UTC
[GitHub] [hudi] rajgowtham24 opened a new issue #4411: [SUPPORT] - Presto Issue
rajgowtham24 opened a new issue #4411:
URL: https://github.com/apache/hudi/issues/4411
Hi team,
We are using Hudi 0.5.0, Presto 0.230 in emr 6.0.0 version and we have used the below sample code for data ingestion. During the upgrade, we are facing error with Presto.
Below code executed in 0.5.0. Post the data ingestion we have manually updated the table location to **"s3://bucket_name/test/table_name/default"** and we are able to query the data in Presto 0.230 as well.
We are in progress of upgrading the emr version to 6.3, during the POC we have observed that the table ingested through emr 6.0 (Hudi 0.5.0, Presto 0.230) is not working in Presto query engine emr 6.3 (Presto 0.245.1 & Hudi 0.7.0). provided the error below. Could you please take a look at the below and let me know
Code Executed in 0.5.0
inputDF = spark.createDataFrame(
[
("100","Apache","Spark", "2015/01/01", "2015-01-01T13:51:39.340396Z","2015-01-02"),
("101","Apache","Beam","2015/01/01", "2015-01-01T12:14:58.597216Z","2015-01-02"),
("102","Apache","Hudi","2015/01/01", "2015-01-01T13:51:40.417052Z","2015-01-02"),
("103","GCP","DataProc","2015/01/01", "2015-01-01T13:51:40.519832Z","2015-01-02"),
("104","AWS","Glue","2015/01/02", "2015-01-01T12:15:00.512679Z","2015-01-03"),
("105","AWS","EMR","2015/01/02", "2015-01-01T13:51:42.248818Z","2015-01-03")
],
["id","provider","technology", "created_date", "last_update_time","load_date"]
)
hive_jdbcurl = "jdbc:hive2://localhost:10000"
hive_username = "hive"
hive_password = "xxx"
hudi_table_name="table_name"
landing_bucket = "s3://bucket_name/test"
hive_schema = "test"
inputDF.write.format("org.apache.hudi").option("hoodie.datasource.write.recordkey.field","id,provider").option("hoodie.datasource.write.precombine.field","last_update_time").option("hoodie.table.name",hudi_table_name).option("hoodie.datasource.write.storage.type","MERGE_ON_READ").option("hoodie.datasource.hive_sync.enable","true").option("hoodie.datasource.hive_sync.database",hive_schema).option("hoodie.datasource.hive_sync.table",hudi_table_name).option("hoodie.datasource.hive_sync.jdbcurl",hive_jdbcurl).option("hoodie.datasource.hive_sync.username",hive_username).option("hoodie.datasource.hive_sync.password",hive_password).option("hoodie.datasource.hive_sync.assume_date_partitioning","false").option("hoodie.datasource.hive_sync.partition_extractor_class","org.apache.hudi.hive.NonPartitionedExtractor").option("hoodie.datasource.write.keygenerator.class","org.apache.hudi.ComplexKeyGenerator").option("hoodie.compact.inline","true").option("hoodie.bulkinsert.shuffle.parallelism","10
0").option("hoodie.insert.shuffle.parallelism", "2").option("hoodie.upsert.shuffle.parallelism", "2").option("hoodie.datasource.write.operation","bulk_insert").mode("Overwrite").save(landing_bucket+hudi_table_name)
**Presto Error**
Hoodie table not found in path s3://bucket_name/test/table_name/default/.hoodie at com.facebook.presto.jdbc.PrestoResultSet.resultsException(PrestoResultSet.java:1841) at com.facebook.presto.jdbc.PrestoResultSet$ResultsPageIterator.computeNext(PrestoResultSet.java:1821) at com.facebook.presto.jdbc.PrestoResultSet$ResultsPageIterator.computeNext(PrestoResultSet.java:1760) at com.facebook.presto.jdbc.internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.facebook.presto.jdbc.internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.facebook.presto.jdbc.internal.guava.collect.TransformedIterator.hasNext(TransformedIterator.java:42) at com.facebook.presto.jdbc.internal.guava.collect.Iterators$ConcatenatedIterator.getTopMetaIterator(Iterators.java:1311) at com.facebook.presto.jdbc.internal.guava.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1327) at com.facebook.presto.jdbc.PrestoResultSet.next(PrestoResultSet.
java:146) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:259) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path s3://az-eu-azodh-opsit-s3-format-dev/f_poc/63_hudi_write_complex/default/.hoodie at org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:53) at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:128) at org.apa
che.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114) at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:100) at com.facebook.presto.hive.HudiDirectoryLister.<init>(HudiDirectoryLister.java:58) at com.facebook.presto.hive.StoragePartitionLoader.<init>(StoragePartitionLoader.java:138) at com.facebook.presto.hive.DelegatingPartitionLoader.<init>(DelegatingPartitionLoader.java:56) at com.facebook.presto.hive.BackgroundHiveSplitLoader.<init>(BackgroundHiveSplitLoader.java:112) at com.facebook.presto.hive.HiveSplitManager.getSplits(HiveSplitManager.java:298) at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorSplitManager.getSplits(ClassLoaderSafeConnectorSplitManager.java:62) at com.facebook.presto.split.SplitManager.getSplits(SplitManager.java:90) at com.facebook.presto.split.CloseableSplitSourceProvider.getSplits(CloseableSplitSourceProvider.java:53) at com.facebook.presto.sql.planner.SplitSourceFac
tory$Visitor.lambda$visitScanAndFilter$1(SplitSourceFactory.java:192) at com.facebook.presto.sql.planner.LazySplitSource.getDelegate(LazySplitSource.java:96) at com.facebook.presto.sql.planner.LazySplitSource.getConnectorId(LazySplitSource.java:48) at com.facebook.presto.execution.scheduler.SectionExecutionFactory.createStageScheduler(SectionExecutionFactory.java:281) at com.facebook.presto.execution.scheduler.SectionExecutionFactory.createStreamingLinkedStageExecutions(SectionExecutionFactory.java:243) at com.facebook.presto.execution.scheduler.SectionExecutionFactory.createStreamingLinkedStageExecutions(SectionExecutionFactory.java:221) at com.facebook.presto.execution.scheduler.SectionExecutionFactory.createSectionExecutions(SectionExecutionFactory.java:167) at com.facebook.presto.execution.scheduler.LegacySqlQueryScheduler.createStageExecutions(LegacySqlQueryScheduler.java:343) at com.facebook.presto.execution.scheduler.LegacySqlQueryScheduler.<init>(LegacySqlQueryScheduler.java
:233) at com.facebook.presto.execution.scheduler.LegacySqlQueryScheduler.createSqlQueryScheduler(LegacySqlQueryScheduler.java:164) at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:508) at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:381) at com.facebook.presto.$gen.Presto_0_245_1_amzn_0____20211208_121047_1.run(Unknown Source) at com.facebook.presto.execution.SqlQueryManager.createQuery(SqlQueryManager.java:254) at com.facebook.presto.dispatcher.LocalDispatchQuery.lambda$startExecution$5(LocalDispatchQuery.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #4411:
URL: https://github.com/apache/hudi/issues/4411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1025274983
> @xushiyan - in 0.5.0, NonpartitionedKeyGenerator doesn't support Composite Primary Keys, so we have used ComplexKeyGenerator and the querying works as expected through Presto 0.230 and not in Presto 0.245.1
@rajgowtham24 Ok do you mean the issue resolved with Presto 0.230 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
codope commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1001920497
@rajgowtham24 From the stacktrace, it seems like `.hoodie` directory is not present under the table basepath. How was the relocation done? Can you please check that the path contain .hoodie dir after updating the location?
Another problem I noticed with the stacktrace is that it contains a call from `HudiDirectoryLister`. This class is not present in Presto version 0.245.1. Can you check again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1017200245
@rajgowtham24 i think you need `NonpartitionedKeyGenerator` instead for non-partitioned table. `default/` is created when complex key gen fails to extract partition path properly. With `default/` in the directory, your table is unexpectedly "partitioned" now, and in your `hive_sync.partition_extractor_class` you used `NonPartitionedExtractor`, that's probably causing reading issue.
> @xushiyan - when we are using complexkey generator in 0.5.0, by default Hudi is creating the data under default partition(same is happening with 0.8.0 as well). So to query the table through Presto 0.230 we have to modify the table location s3://bucket_name/test/table_name/default.
>
> For partitioned table, without updating the table location we are able to query the table from Presto 0.230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1015048564
@rajgowtham24
> we have manually updated the table location to "s3://bucket_name/test/table_name/default"
so is `default/` the partition path in your table? I don't quite get why this is need. Were you not able to query the table with path = s3://bucket_name/test/table_name ? It's obviously causing problem as the stacktrace is complaining table not found, because the table path is overwritten to where the `.hoodie/` is not present. So the problem is about setting the correct table path. I suggest follow guide here https://hudi.apache.org/docs/query_engine_setup#prestodb and use the intended path.
Also please contact AWS support as this is under AWS environment, which we can't troubleshoot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rajgowtham24 commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
rajgowtham24 commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1016180998
@xushiyan - when we are using complexkey generator in 0.5.0, by default Hudi is creating the data under default partition(same is happening with 0.8.0 as well). So to query the table through Presto 0.230 we have to modify the table location s3://bucket_name/test/table_name/default.
For partitioned table, without updating the table location we are able to query the table from Presto 0.230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rajgowtham24 commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
rajgowtham24 commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1025699670
> > @xushiyan - in 0.5.0, NonpartitionedKeyGenerator doesn't support Composite Primary Keys, so we have used ComplexKeyGenerator and the querying works as expected through Presto 0.230 and not in Presto 0.245.1
>
> @rajgowtham24 Ok do you mean the issue resolved with Presto 0.230 ?
yes @xushiyan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1001766420
sure. Will be looking into github issues today and tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rajgowtham24 commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
rajgowtham24 commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1018298399
@xushiyan - in 0.5.0, NonpartitionedKeyGenerator doesn't support Composite Primary Keys, so we have used ComplexKeyGenerator and the querying works as expected through Presto 0.230 and not in Presto 0.245.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rajgowtham24 commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
rajgowtham24 commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1001947505
@codope - below is the folder structure post the data write from 0.5.0. Since Presto doesn't identify the parquet files under /default directory we have manually updated the table location for the table so that the query would work fine in Presto 0.230
s3://bucket_name/test/table_name/
-.hoodie_$folder$
-.hoodie/
-default_$folder$
-default/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rajgowtham24 commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
rajgowtham24 commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1001627701
@nsivabalan Hi Siva, when you get a chance, can you please take a look on the above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4411: [SUPPORT] - Presto Querying Issue in AWS EMR 6.3.1
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4411:
URL: https://github.com/apache/hudi/issues/4411#issuecomment-1026004640
@rajgowtham24 thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org