You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TIMFANGZI (via GitHub)" <gi...@apache.org> on 2023/08/01 09:31:45 UTC
[GitHub] [hudi] TIMFANGZI opened a new issue, #9333: [SUPPORT] Failed to read S3 multi-path
TIMFANGZI opened a new issue, #9333:
URL: https://github.com/apache/hudi/issues/9333
**Describe the problem you faced**
In our project, we used to read several tables by the following code:
```
s3_path_list = []
path_A = "s3://my-bucket/table/table_A/*"
path_B = "s3://my-bucket/table/table_B/*"
s3_path_list.append(path_A)
s3_path_list.append(path_B)
multi_path = ",".join(s3_path_list)
df = spark.read.format('org.apache.hudi').option("hoodie.datasource.query.type", "read_optimized").option("hoodie.datasource.read.paths", multi_path).load()
```
table_A and table_B are the **two different tables** with the **same table schema** under the folder 'table', before upgrading the EMR, we can read successfully in this case, however, if fails with the following error after upgrading:
`pyspark.sql.utils.IllegalArgumentException: Wrong basePath xxxxx for the root path xxxxxxx`
After test, we find that if table_A and table_B are the partition tables under 'table', it can read successfully. But we still hope that we can read the different tables with the same schema in this way, do you have any suggestions?
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.13.0
* Spark version : 3.3.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1696013032
@TIMFANGZI Looks like a valid issue starting from 0.12. I confirmed this bug still exists in master.
Created a JIRA to track the fix for the same - https://issues.apache.org/jira/browse/HUDI-6764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TIMFANGZI commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "TIMFANGZI (via GitHub)" <gi...@apache.org>.
TIMFANGZI commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1667499683
> Nope. Hudi 0.12.0 Spark 3.2.3 Python 3.7.13 `pyspark.sql.utils.IllegalArgumentException: Wrong basePath file:/tmp/test3/A3 for the root path: file:/tmp/test3/B3/.hoodie`
>
> ```
> basePath = "/tmp/test3"
> tableA = "A3"
> tableB = "B3"
>
> spark.sql(f"""
> create table {tableA} (
> id int,
> name string,
> price double,
> ts long
> ) using hudi
> location '{basePath}/{tableA}'
> tblproperties (
> primaryKey ='id',
> preCombineField = 'ts'
> )""")
>
> spark.sql(f"""
> create table {tableB} (
> id int,
> name string,
> price double,
> ts long
> ) using hudi
> location '{basePath}/{tableB}'
> tblproperties (
> primaryKey ='id',
> preCombineField = 'ts'
> )""")
>
> spark.sql(f"insert into {tableA} values(1, 'a1', 10, 1000)")
> spark.sql(f"insert into {tableA} values(2, 'a2', 11, 1000)")
> spark.sql(f"insert into {tableB} values(3, 'a3', 12, 1000)")
> spark.sql(f"insert into {tableB} values(4, 'a4', 13, 1000)")
> path_A = f"{basePath}/{tableA}/*"
> path_B = f"{basePath}/{tableB}/*"
> multi_path = path_A + "," + path_B
> df = spark.read.format("hudi").option("hoodie.datasource.query.type", "read_optimized").option("hoodie.datasource.read.paths", multi_path).load()
> df.show(100,false)
> ```
The same error, it works well in Hudi 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Failed to read S3 multi-path [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #9333: [SUPPORT] Failed to read S3 multi-path
URL: https://github.com/apache/hudi/issues/9333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1668421215
Ok it works for me on 0.11.1 as well. Not sure why I thought you were upgrading from 12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1665838385
Nope. Hudi 0.12.0 Spark 3.2.3 Python 3.7.13
`pyspark.sql.utils.IllegalArgumentException: Wrong basePath file:/tmp/test3/A3 for the root path: file:/tmp/test3/B3/.hoodie`
```
basePath = "/tmp/test3"
tableA = "A3"
tableB = "B3"
spark.sql(f"""
create table {tableA} (
id int,
name string,
price double,
ts long
) using hudi
location '{basePath}/{tableA}'
tblproperties (
primaryKey ='id',
preCombineField = 'ts'
)""")
spark.sql(f"""
create table {tableB} (
id int,
name string,
price double,
ts long
) using hudi
location '{basePath}/{tableB}'
tblproperties (
primaryKey ='id',
preCombineField = 'ts'
)""")
spark.sql(f"insert into {tableA} values(1, 'a1', 10, 1000)")
spark.sql(f"insert into {tableA} values(2, 'a2', 11, 1000)")
spark.sql(f"insert into {tableB} values(3, 'a3', 12, 1000)")
spark.sql(f"insert into {tableB} values(4, 'a4', 13, 1000)")
path_A = f"{basePath}/{tableA}/*"
path_B = f"{basePath}/{tableB}/*"
multi_path = path_A + "," + path_B
df = spark.read.format("hudi").option("hoodie.datasource.query.type", "read_optimized").option("hoodie.datasource.read.paths", multi_path).load()
df.show(100,false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TIMFANGZI commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "TIMFANGZI (via GitHub)" <gi...@apache.org>.
TIMFANGZI commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1661535424
> Yeah, the multi paths read got lost, this is a regression ~
Is there any solution for it? We think this method is not only concise, but also efficient. We don't want to lost this featureš©
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1661517068
Yeah, the multi paths read got lost, this is a regression ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on issue #9333: [SUPPORT] Failed to read S3 multi-path
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1665803100
I tried the following with Hudi 0.12.0 Spark 3.2.3 and it failed with the exception `java.lang.IllegalArgumentException: Wrong basePath file:/tmp/test2/A2 for the root path: file:/tmp/test2/B2/.hoodie`
Is there anything I am missing from my setup?
```
val basePath = "/tmp/test2"
val tableA = "A2"
val tableB = "B2"
// Create source table
spark.sql(
s"""
| create table $tableA (
| id int,
| name string,
| price double,
| ts long
| ) using hudi
| location '$basePath/$tableA'
| tblproperties (
| primaryKey ='id',
| preCombineField = 'ts'
| )
""".stripMargin)
// Create target table
spark.sql(
s"""
|create table $tableB (
| id int,
| name string,
| price double,
| ts long
|) using hudi
| location '$basePath/$tableB'
| tblproperties (
| primaryKey ='id',
| preCombineField = 'ts'
| )
""".stripMargin)
// Insert data to source table
spark.sql(s"insert into $tableA values(1, 'a1', 10, 1000)")
spark.sql(s"insert into $tableA values(2, 'a2', 11, 1000)")
spark.sql(s"insert into $tableB values(3, 'a3', 12, 1000)")
spark.sql(s"insert into $tableB values(4, 'a4', 13, 1000)")
val path_A = s"$basePath/$tableA/*"
val path_B = s"$basePath/$tableB/*"
val multi_path = path_A + "," + path_B
val df = spark.read.format("hudi").option("hoodie.datasource.query.type", "read_optimized").option("hoodie.datasource.read.paths", multi_path).load()
df.show(100,false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org