You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2023/06/12 08:09:07 UTC

[GitHub] [hudi] parisni opened a new issue, #8934: [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming

parisni opened a new issue, #8934:
URL: https://github.com/apache/hudi/issues/8934

   hudi 0.13.1
   
   All goes well util two cases: 
   
   # When only one row
   
   ```python
   from pyspark.sql.types import StructType, StructField, IntegerType, StringType, MapType, ArrayType, TimestampType
   
   tableName = 'test_hudi_spacial'
   basePath = "/tmp/test/{tableName}".format(tableName=tableName)
   from pyspark.sql.functions import expr
   
   df =spark.sql("select '1' as event_id, 'a' col1, 'b' col2, 'c' col3, '2' as ts, '3' as version, 'foo' as event_date")
   ##
   # Create a nested table
   ##
   hudi_options = {
       "hoodie.table.name": tableName,
       "hoodie.datasource.write.recordkey.field": "event_id",
       "hoodie.datasource.write.partitionpath.field": "version,event_date",
       "hoodie.datasource.write.table.name": tableName,
       "hoodie.datasource.write.operation": "bulk_insert",
       "hoodie.datasource.write.precombine.field": "ts",
       "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
       "hoodie.datasource.write.hive_style_partitioning": "true",
       "hoodie.datasource.hive_sync.enable": "false",
       "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
       "hoodie.metadata.enable": "false",
       # for spacial curve
       "hoodie.clustering.plan.strategy.sort.columns": "col1,col2,col3",
       "hoodie.layout.optimize.build.curve.sample.size": "1",
       "hoodie.layout.optimize.curve.build.method": "sample",
       "hoodie.layout.optimize.strategy": "hilbert",
       "hoodie.bulkinsert.user.defined.partitioner.class": "org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner",
   }
   (df.write.format("hudi").options(**hudi_options).mode("overwrite").save(basePath))
   ```
   raises
   ```
   : java.lang.ArithmeticException: / by zero
           at org.apache.spark.sql.hudi.execution.RangeSampleSort$.$anonfun$sortDataFrameBySample$11(RangeSample.scala:337)
           at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
           at scala.collection.Iterator.foreach(Iterator.scala:943)
           at scala.collection.Iterator.foreach$(Iterator.scala:943)
           at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
           at scala.collection.IterableLike.foreach(IterableLike.scala:74)
           at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
           at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
           at scala.collection.TraversableLike.map(TraversableLike.scala:286)
           at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
           at scala.collection.AbstractTraversable.map(Traversable.scala:108)
           at org.apache.spark.sql.hudi.execution.RangeSampleSort$.sortDataFrameBySample(RangeSample.scala:336)
           at org.apache.hudi.sort.SpaceCurveSortingHelper.orderDataFrameBySamplingValues(SpaceCurveSortingHelper.java:275)
           at org.apache.hudi.execution.bulkinsert.SpatialCurveSortPartitionerBase.reorder(SpatialCurveSortPartitionerBase.java:73)
           at org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner.repartitionRecords(RowSpatialCurveSortPartitioner.java:40)
           at org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner.repartitionRecords(RowSpatialCurveSortPartitioner.java:26)
           at org.apache.hudi.HoodieDatasetBulkInsertHelper$.prepareForBulkInsert(HoodieDatasetBulkInsertHelper.scala:125)
           at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:802)
           at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:322)
           at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
   ```
   
   # Now when empty dataframe
   
   ```python
   from pyspark.sql.types import StructType, StructField, IntegerType, StringType, MapType, ArrayType, TimestampType
   
   tableName = 'test_hudi_spacial'
   basePath = "/tmp/test/{tableName}".format(tableName=tableName)
   from pyspark.sql.functions import expr
   
   df =spark.sql("select '1' as event_id, 'a' col1, 'b' col2, 'c' col3, '2' as ts, '3' as version, 'foo' as event_date")
   ##
   # Create a nested table
   ##
   hudi_options = {
       "hoodie.table.name": tableName,
       "hoodie.datasource.write.recordkey.field": "event_id",
       "hoodie.datasource.write.partitionpath.field": "version,event_date",
       "hoodie.datasource.write.table.name": tableName,
       "hoodie.datasource.write.operation": "bulk_insert",
       "hoodie.datasource.write.precombine.field": "ts",
       "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
       "hoodie.datasource.write.hive_style_partitioning": "true",
       "hoodie.datasource.hive_sync.enable": "false",
       "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
       "hoodie.metadata.enable": "false",
       # for spacial curve
       "hoodie.clustering.plan.strategy.sort.columns": "col1,col2,col3",
       "hoodie.layout.optimize.build.curve.sample.size": "1",
       "hoodie.layout.optimize.curve.build.method": "sample",
       "hoodie.layout.optimize.strategy": "hilbert",
       "hoodie.bulkinsert.user.defined.partitioner.class": "org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner",
   }
   (df.limit(0).write.format("hudi").options(**hudi_options).mode("overwrite").save(basePath))
   ```
   
   ```
   : java.util.NoSuchElementException: next on empty iterator
           at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
           at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
           at scala.collection.IterableLike.head(IterableLike.scala:109)
           at scala.collection.IterableLike.head$(IterableLike.scala:108)
           at scala.collection.mutable.ArrayBuffer.scala$collection$IndexedSeqOptimized$$super$head(ArrayBuffer.scala:49)
           at scala.collection.IndexedSeqOptimized.head(IndexedSeqOptimized.scala:129)
           at scala.collection.IndexedSeqOptimized.head$(IndexedSeqOptimized.scala:129)
           at scala.collection.mutable.ArrayBuffer.head(ArrayBuffer.scala:49)
           at org.apache.spark.sql.hudi.execution.RangeSampleSort$.sortDataFrameBySample(RangeSample.scala:320)
           at org.apache.hudi.sort.SpaceCurveSortingHelper.orderDataFrameBySamplingValues(SpaceCurveSortingHelper.java:275)
           at org.apache.hudi.execution.bulkinsert.SpatialCurveSortPartitionerBase.reorder(SpatialCurveSortPartitionerBase.java:73)
           at org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner.repartitionRecords(RowSpatialCurveSortPartitioner.java:40)
           at org.apache.hudi.execution.bulkinsert.RowSpatialCurveSortPartitioner.repartitionRecords(RowSpatialCurveSortPartitioner.java:26)
           at org.apache.hudi.HoodieDatasetBulkInsertHelper$.prepareForBulkInsert(HoodieDatasetBulkInsertHelper.scala:125)
           at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:802)
           at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:322)
           at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
           at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
           at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
           at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
           at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
           at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
           at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
           at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
           at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
           at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on issue #8934: [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on issue #8934:
URL: https://github.com/apache/hudi/issues/8934#issuecomment-1586940941

   @parisni Nice catch!! It looks some edge cases we didn't notice before, will try to fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming [hudi]

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar closed issue #8934: [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming
URL: https://github.com/apache/hudi/issues/8934


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #8934: [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8934:
URL: https://github.com/apache/hudi/issues/8934#issuecomment-1588903046

   @parisni Thanks for pointing out. @boneanxs We can use this JIRA to fix if you haven't created yet. https://issues.apache.org/jira/browse/HUDI-6369


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] parisni commented on issue #8934: [SUPPORT] Spacial curve with sample strategy fails when 0 or 1 rows only is incoming

Posted by "parisni (via GitHub)" <gi...@apache.org>.
parisni commented on issue #8934:
URL: https://github.com/apache/hudi/issues/8934#issuecomment-1586806540

   @boneanxs  maybe ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org