You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "jamb2024 (via GitHub)" <gi...@apache.org> on 2024/05/03 13:47:16 UTC

[I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

jamb2024 opened a new issue, #11144:
URL: https://github.com/apache/hudi/issues/11144

   Hi.
   I am developing a process to ingest data from my hdfs using Hudi. I want to partition the data using a custom keygenerator class where the partition key will be a tuple columnName@NumPartitions. Then, in my custom keygenerator using the function module to send the row to a partition or another.
   
   The initial load is the following:
   
   
   spark.read.option("mergeSchema","true").parquet("PATH").
   withColumn("_hoodie_is_deleted", lit(false)).
   write.format("hudi").
   	  option(OPERATION_OPT_KEY, "upsert").
   	  option(CDC_ENABLED.key(), "true").
   	  option(TABLE_NAME, tableName).
   	  option("hoodie.datasource.write.payload.class","CustomOverwriteWithLatestAvroPayload").
   	  option("hoodie.avro.schema.validate","false").
   	  option("hoodie.datasource.write.recordkey.field","CID").
   	  option("hoodie.datasource.write.precombine.field","sequential_total").
   	  option("hoodie.datasource.write.new.columns.nullable", "true"). 
   	  option("hoodie.datasource.write.reconcile.schema","true").
   	  option("hoodie.metadata.enable","false").
   	  option("hoodie.index.type","SIMPLE").
   	  option("hoodie.datasource.write.table.type","COPY_ON_WRITE").
   	  option("hoodie.datasource.write.keygenerator.class","CustomKeyGenerator").
   	  option("hoodie.datasource.write.partitionpath.field","CID@12").
   	  option("hoodie.datasource.write.drop.partition.columns","true").
   	  mode(Overwrite).
   	  save("/tmp/hudi2")
   
   I have added the property hoodie.datasource.write.drop.partition.columns because when I read the final path, hudi throws me the error: Cannot find columns: 'CID@12' in the schema
   But with this property, It does not work either. The error that appears is the following:
   
   org.apache.hudi.internal.schema.HoodieSchemaException: Failed to fetch schema from the table
     at org.apache.hudi.HoodieBaseRelation.$anonfun$x$2$10(HoodieBaseRelation.scala:179)
     at scala.Option.getOrElse(Option.scala:189)
     at org.apache.hudi.HoodieBaseRelation.x$2$lzycompute(HoodieBaseRelation.scala:175)
     at org.apache.hudi.HoodieBaseRelation.x$2(HoodieBaseRelation.scala:151)
     at org.apache.hudi.HoodieBaseRelation.internalSchemaOpt$lzycompute(HoodieBaseRelation.scala:151)
     at org.apache.hudi.HoodieBaseRelation.internalSchemaOpt(HoodieBaseRelation.scala:151)
     at org.apache.hudi.BaseFileOnlyRelation.<init>(BaseFileOnlyRelation.scala:69)
     at org.apache.hudi.DefaultSource$.resolveBaseFileOnlyRelation(DefaultSource.scala:321)
     at org.apache.hudi.DefaultSource$.createRelation(DefaultSource.scala:262)
     at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:118)
     at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:74)
     at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
     at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
     at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
     at scala.Option.getOrElse(Option.scala:189)
     at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
     at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
     ... 63 elided
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

Posted by "jamb2024 (via GitHub)" <gi...@apache.org>.

jamb2024 commented on issue #11144:
URL: https://github.com/apache/hudi/issues/11144#issuecomment-2096073258

   And is there any way of partitioning the data using a hash function of the row primary key to improve the performance for update rows. I have developed my custom BuiltinKeyGenerator overwriting the method getPartitionPath (I get the partitionPath, which is the primary key and I apply the operation % numBuckets) but the problem is that when I read the data, the value for the primary key column is the value of the operation instead of the real value.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #11144:
URL: https://github.com/apache/hudi/issues/11144#issuecomment-2095033520

   `hoodie.datasource.write.drop.partition.columns` is setup by default as false, which means the data file does not include the partition columns, the partition field you declared here should be a field name instead of a value `CID@12`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

Posted by "bhavya-ganatra (via GitHub)" <gi...@apache.org>.

bhavya-ganatra commented on issue #11144:
URL: https://github.com/apache/hudi/issues/11144#issuecomment-3612414076

Hi @danny0405 @codope , I had a similar doubt regarding the same. I have same field 'cid' as recordKey and partition field, and I have defined my own KeyGenerator - HashPartitionKeyGenerator(extends BuiltinKeyGenerator). I have override the `getPartitionPath` method to get the hash value of partition field.
When I am reading the data via spark, I am seeing that value of `cid` field is replaced with _hoodie_partition_path value.

Ex. cid: cid152
Result: _hoodie_record_key: "cid:cid152", "_hoodie_partition_path": "60", "cid":"60". Value of "cid" field should have been "cid152".

When I was using hudi 0.15, this wasn't the case. It was working as expected. Though, With hudi 0.15, HashPartitionKeyGenerator was extending BaseKeyGenerator.

I am not sure if there is a behaviour change around this area.
I have also tried setting hoodie.datasource.write.drop.partition.columns field to true, but this doesn't seem to be working for me. In hoodie.properties file, value of hoodie.datasource.write.drop.partition.columns is coming as false only. (Hudi 1.1.0)

I can also dynamically generate hash field and use that as partition field, but I just wanted to know if this is expectation or is there any workaround?
Thank you.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #11144:
URL: https://github.com/apache/hudi/issues/11144#issuecomment-2097231302

   The contract here is: the partition **field** shoud be in the table schema anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Error using the property hoodie.datasource.write.drop.partition.columns [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #11144:
URL: https://github.com/apache/hudi/issues/11144#issuecomment-3615064284

   > And is there any way of partitioning the data using a hash function of the row primary key to improve the performance for update rows
   
   We would deem this as a valid request of Spark, cc @yihua for taking care.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org