You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/08/04 15:37:17 UTC

[GitHub] [hudi] jonvex commented on issue #9333: [SUPPORT] Failed to read S3 multi-path

jonvex commented on issue #9333:
URL: https://github.com/apache/hudi/issues/9333#issuecomment-1665803100

   I tried the following with Hudi 0.12.0 Spark 3.2.3 and it failed with the exception `java.lang.IllegalArgumentException: Wrong basePath file:/tmp/test2/A2 for the root path: file:/tmp/test2/B2/.hoodie`
   Is there anything I am missing from my setup?
   ```
         val basePath = "/tmp/test2"
         val tableA = "A2"
         val tableB = "B2"
         // Create source table
         spark.sql(
           s"""
              | create table $tableA (
              |  id int,
              |  name string,
              |  price double,
              |  ts long
              | ) using hudi
              | location '$basePath/$tableA'
              | tblproperties (
              |  primaryKey ='id',
              |  preCombineField = 'ts'
              | )
            """.stripMargin)
         // Create target table
         spark.sql(
           s"""
              |create table $tableB (
              |  id int,
              |  name string,
              |  price double,
              |  ts long
              |) using hudi
              | location '$basePath/$tableB'
              | tblproperties (
              |  primaryKey ='id',
              |  preCombineField = 'ts'
              | )
          """.stripMargin)
         // Insert data to source table
         spark.sql(s"insert into $tableA values(1, 'a1', 10, 1000)")
         spark.sql(s"insert into $tableA values(2, 'a2', 11, 1000)")
         spark.sql(s"insert into $tableB values(3, 'a3', 12, 1000)")
         spark.sql(s"insert into $tableB values(4, 'a4', 13, 1000)")
         val path_A = s"$basePath/$tableA/*"
         val path_B = s"$basePath/$tableB/*"
         val multi_path = path_A + "," + path_B
         val df = spark.read.format("hudi").option("hoodie.datasource.query.type", "read_optimized").option("hoodie.datasource.read.paths", multi_path).load()
         df.show(100,false)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org