You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "eshu (via GitHub)" <gi...@apache.org> on 2024/02/26 01:47:36 UTC

[I] [SUPPORT] [hudi]

eshu opened a new issue, #10754:
URL: https://github.com/apache/hudi/issues/10754

   When the partition column contains the slash character ("/"), Hudi could write the data incorrectly or do not read the back.
   
   Test (I use some helpers to write and read Hudi data, they write write data to the local FS and read it):
   ```scala
   class HudiPartitionPathTest extends AnyFlatSpec with Matchers with TestHudi {
     "Partition paths" should "be generated properly" in {
       val data = rows(
         (1, "one", "partition with space", ts"2024-02-26 08:25:05"),
         (2, "two", "partition with space", ts"2024-02-26 08:25:05"),
         (3, "three", "partition-with-dashes", ts"2024-02-26 08:25:05"),
         (4, "four", "partition-with-dashes", ts"2024-02-26 08:25:05"),
         (5, "five", "partition=", ts"2024-02-26 08:25:05"),
         (6, "six", "partition=", ts"2024-02-26 08:25:05"),
         (7, "seven", "partition%", ts"2024-02-26 08:25:05"),
         (8, "eight", "partition%", ts"2024-02-26 08:25:05"),
         (9, "nine", "partition/", ts"2024-02-26 08:25:05"),
         (10, "ten", "partition/", ts"2024-02-26 08:25:05"),
         (11, "eleven", "partition/slaanesh", ts"2024-02-26 08:25:05"),
         (12, "twelve", "partition/slaanesh", ts"2024-02-26 08:25:05")
       )
       val path = createHudiDataset(
         getClass.getName,
         data,
         schema("id" -> "int", "value" -> "string", "partition" -> "string")
       )
       val fsPartitionPaths = allFSPartitionPaths(new File(path), path.length + 1)
       println(fsPartitionPaths mkString "\n")
       val df = readHudiDataset(path)
       df show false
       val partitionPaths = df.select("_hoodie_partition_path").dropDuplicates.collect().map(_.getString(0)).toSet
       fsPartitionPaths shouldEqual partitionPaths
     }
   
     private val filter: FilenameFilter = (_, name) => !name.startsWith(".")
   
     def allFSPartitionPaths(dir: File, prefixLength: Int): Set[String] =
       (dir.listFiles(filter) foldLeft Set.empty[String]) { (paths, file) =>
         if (file.isFile) paths + file.getParent.substring(prefixLength)
         else paths | allFSPartitionPaths(file, prefixLength)
       }
   }
   ```
   
   The output is
   ```
   daas_date=partition
   daas_date=partition-with-dashes
   daas_date=partition with space
   daas_date=partition%
   daas_date=partition/slaanesh
   daas_date=partition=
   +-------------------+---------------------+------------------+-------------------------------+-----------------------------------------------------------------------+---+-----+---------------------+-------------------+
   |_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path         |_hoodie_file_name                                                      |id |value|daas_date            |daas_internal_ts   |
   +-------------------+---------------------+------------------+-------------------------------+-----------------------------------------------------------------------+---+-----+---------------------+-------------------+
   |20240226102735752  |20240226102735752_5_0|9                 |daas_date=partition/           |607b4c16-93c8-4a1f-9530-f4b6be57bc9c-0_5-4-14_20240226102735752.parquet|9  |nine |partition/           |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_5_1|10                |daas_date=partition/           |607b4c16-93c8-4a1f-9530-f4b6be57bc9c-0_5-4-14_20240226102735752.parquet|10 |ten  |partition/           |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_1_0|3                 |daas_date=partition-with-dashes|641c2e87-276f-48eb-9a1c-eac63fed00e2-0_1-4-10_20240226102735752.parquet|3  |three|partition-with-dashes|2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_1_1|4                 |daas_date=partition-with-dashes|641c2e87-276f-48eb-9a1c-eac63fed00e2-0_1-4-10_20240226102735752.parquet|4  |four |partition-with-dashes|2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_2_0|1                 |daas_date=partition with space |b94de450-3d40-490b-bbc7-7b1d15e5edef-0_2-4-11_20240226102735752.parquet|1  |one  |partition with space |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_2_1|2                 |daas_date=partition with space |b94de450-3d40-490b-bbc7-7b1d15e5edef-0_2-4-11_20240226102735752.parquet|2  |two  |partition with space |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_0_0|7                 |daas_date=partition%           |2101cdfe-74b3-4268-8504-c36ab3b59f89-0_0-4-9_20240226102735752.parquet |7  |seven|partition%           |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_0_1|8                 |daas_date=partition%           |2101cdfe-74b3-4268-8504-c36ab3b59f89-0_0-4-9_20240226102735752.parquet |8  |eight|partition%           |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_4_0|5                 |daas_date=partition=           |7f548d14-a985-42ba-82d2-fca4784c8906-0_4-4-13_20240226102735752.parquet|5  |five |partition=           |2024-02-26 08:25:05|
   |20240226102735752  |20240226102735752_4_1|6                 |daas_date=partition=           |7f548d14-a985-42ba-82d2-fca4784c8906-0_4-4-13_20240226102735752.parquet|6  |six  |partition=           |2024-02-26 08:25:05|
   +-------------------+---------------------+------------------+-------------------------------+-----------------------------------------------------------------------+---+-----+---------------------+-------------------+
   ```
   As you can see rows 11 and 12 was not read, and "partition" and "partition/" on the file system have the same path (I am not sure about the impact, but probably there could be performance issues).
   
   Maybe it would be great to quote some characters in partition paths?
   
   **Environment Description**
   
   * Hudi version :
   0.13.1
   
   * Storage (HDFS/S3/GCS..):
   Local FS
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #10754: [SUPPORT] Slashes in partition columns
URL: https://github.com/apache/hudi/issues/10754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "eshu (via GitHub)" <gi...@apache.org>.
eshu commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-2034546730

   @ad1happy2go It does not work in my example. Did you tried it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-2036902605

   Yes I tried this - https://github.com/apache/hudi/issues/10754#issuecomment-1979027421
   
   Can you try the same?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-1980114792

   Similar jira raised to fix this issue - https://issues.apache.org/jira/browse/HUDI-7484


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-1963209068

   you are right, we should enode the partition path for these special characters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-2050056653

   @eshu Any updates on the same?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Slashes in partition columns [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10754:
URL: https://github.com/apache/hudi/issues/10754#issuecomment-1979027421

   @eshu I tried to insert these values and at least read/write worked fine. I do understand in case of slash it created the inner sub folder. Were you able to make it work by encoding them. Let us know in case you need any other help here or Feel free to close if all good.
   
   ```
   columns = ["ts","uuid","rider","driver","fare","city"]
   data =[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san francisco"),
          (1695091554788,"e96c4396-3fad-413a-a942-4cb36106d721","rider-B","driver-L",27.70 ,"san-francisco"),
          (1695046462179,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-C","driver-M",33.90 ,"san_francisco%"),
          (1695516137016,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-C","driver-N",34.15,"sao/paulo")]
   
   spark = get_spark_session(spark_version="3.2", hudi_version="0.13.0")
   
   inserts = spark.createDataFrame(data).toDF(*columns)
   
   hudi_options = {
       'hoodie.table.name': tableName,
       'hoodie.datasource.write.recordkey.field' : 'uuid',
       'hoodie.datasource.write.precombine.field' : 'ts',
       'hoodie.datasource.write.partitionpath.field': 'city',
   }
   # Insert data
   inserts.write.format("hudi"). \
       options(**hudi_options). \
       mode("overwrite"). \
       save(basePath)
   
   spark.read.format("hudi").load(basePath).show()
   ```
   <img width="1455" alt="image" src="https://github.com/apache/hudi/assets/63430370/f8581dab-0ab5-4426-876f-1f6f1c99523b">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org