You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/10 10:17:43 UTC
[GitHub] [hudi] bram2000 opened a new issue, #7176: [SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
bram2000 opened a new issue, #7176:
URL: https://github.com/apache/hudi/issues/7176
**Describe the problem you faced**
We are using hudi on amazon EMR. We used to use hudi 0.7, which allowed us to use the following syntax to load specific directories of parquet files (in this example just two days):
```python
data_uris = ["s3://bucket/data_product/date=2022-11-01/*", "s3://bucket/data_product/date=2022-11-02/*"]
self.spark.read.format("hudi").option("hoodie.datasource.read.paths", ",".join(data_uris)).load()
```
After upgrading our EMR version we are now using hudi 0.11 but this syntax now fails with:
```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
```
**To Reproduce**
Steps to reproduce the behavior:
1. Attempt to load two different paths using `spark.read.format('hudi')`
**Expected behavior**
I can load specific directories of data.
**Environment Description**
* Hudi version : 0.11.0
* Spark version : 3.2.1
* Hive version : 3.1.3
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
An error was encountered:
An error occurred while calling o409.load.
: org.apache.hudi.exception.HoodieException: 'path' cannot be null, missing 'path' from table properties
at org.apache.hudi.Spark3DefaultSource.getTable(Spark3DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.getTableFromProvider(DataSourceV2Utils.scala:83)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:233)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pru007 commented on issue #7176: [SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
Posted by "pru007 (via GitHub)" <gi...@apache.org>.
pru007 commented on issue #7176:
URL: https://github.com/apache/hudi/issues/7176#issuecomment-1541343935
Hi, I am also facing this problem. We have hudi 0.11.0 installed. Any solution to this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #7176: [SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #7176:
URL: https://github.com/apache/hudi/issues/7176#issuecomment-1312694595
@bram2000 are the hudi versions (0.7 -> 0.11.0 ) both using hudi from amazon? as in the pre-installed hudi jars in EMR?
Have you ingested any new data into the table using hudi 0.11.0 ? if not, the table version is old and you probably need to perform a table upgrade using hudi-cli. see https://hudi.apache.org/docs/cli/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mandar-mw commented on issue #7176: [SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
Posted by "mandar-mw (via GitHub)" <gi...@apache.org>.
mandar-mw commented on issue #7176:
URL: https://github.com/apache/hudi/issues/7176#issuecomment-1541358423
@pru007 - IIRC we got around this after upgrading to 12.2, try doing that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] mandar-mw commented on issue #7176: [SUPPORT] Cannot load a list of specific files after upgrading to hudi 0.11
Posted by "mandar-mw (via GitHub)" <gi...@apache.org>.
mandar-mw commented on issue #7176:
URL: https://github.com/apache/hudi/issues/7176#issuecomment-1435753069
Is there a resolution for this issue? We too are running into this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org