You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/01 02:40:40 UTC

[GitHub] [spark] AngersZhuuuu opened a new pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

AngersZhuuuu opened a new pull request #32018:
URL: https://github.com/apache/spark/pull/32018


   ### What changes were proposed in this pull request?
   
   ExternalCatalogUtils.escapePathName  should support null
   
   ### Why are the changes needed?
   ExternalCatalogUtils.escapePathName  should support null
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811959345


   **[Test build #136821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136821/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811920107


   **[Test build #136816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136816/testReport)** for PR 32018 at commit [`03325c3`](https://github.com/apache/spark/commit/03325c3bb9f254222b2ac1f54375704e6b0e3b6f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812325437


   jenkins, retest this, please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812385399


   BTW, @AngersZhuuuu does the issue exist in 3.1/3.0/2.4? If so, please, backport the changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812600868


   Could you open separate PRs per each branch, please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812068245


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136821/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812380712


   +1, LGTM. Merging to master. The failed GA is a known issue.
   Thank you @AngersZhuuuu, and @cloud-fan @wangyum for your review. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811597991


   **[Test build #136787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136787/testReport)** for PR 32018 at commit [`960f2c1`](https://github.com/apache/spark/commit/960f2c10c94954e6810d3537166905b9e17f4728).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811959345


   **[Test build #136821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136821/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812496996


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136841/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811902232


   > DSv2 impl should handle `null` itself. So, we shouldn't replace it by `__HIVE_DEFAULT_PARTITION__`.
   
   I know Why I am confused, I test this in spark 3.0, this behavior has been change to keep consistence.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812496996


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136841/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811949039


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41398/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811718199


   
   > I am not sure that it is right fix. I guess, `null` should be replaced by `"__HIVE_DEFAULT_PARTITION__"` like:
   > 
   > https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala#L122-L127
   
   Looks like we should handle it in 
   https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala#L351-L355


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812073418


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136816/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812101280


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136817/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811968962


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41398/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811616047


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41370/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811719759


   What confused me is that 
   ```
   spark-sql> CREATE TABLE t(i STRING, c string) USING PARQUET PARTITIONED BY (c);
   Time taken: 2.12 seconds
   spark-sql> INSERT OVERWRITE t PARTITION (c=null) VALUES ('1');
   Time taken: 4.984 seconds
   spark-sql> desc formatted t partition(c=null);
   i	string	NULL
   c	string	NULL
   # Partition Information
   # col_name	data_type	comment
   c	string	NULL
   
   # Detailed Partition Information
   Database	default
   Table	t
   Partition Values	[c=null]
   Location	hdfs://tl0/user/hive/warehouse/t/c=null
   Serde Library	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
   InputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
   OutputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   Storage Properties	[path=hdfs://tl0/user/hive/warehouse/t, serialization.format=1]
   Partition Parameters	{rawDataSize=-1, numFiles=1, transient_lastDdlTime=1617244501, totalSize=396, COLUMN_STATS_ACCURATE=false, numRows=-1}
   Created Time	Thu Apr 01 10:35:01 SGT 2021
   Last Access	UNKNOWN
   Partition Statistics	396 bytes
   
   # Storage Information
   Location	hdfs://tl0/user/hive/warehouse/t
   Serde Library	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
   InputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
   OutputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   Time taken: 0.135 seconds, Fetched 25 row(s)
   ```
   
   The path can be c=null, for current code, which case the path will be null and which case it can be `__HIVE_DEFAULT_PARTITION__`? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811597991


   **[Test build #136787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136787/testReport)** for PR 32018 at commit [`960f2c1`](https://github.com/apache/spark/commit/960f2c10c94954e6810d3537166905b9e17f4728).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu edited a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811904168


   ping @MaxGekk Updated, current code should be ok since `InsertIntoHadoopFsRelationCommand` only used in DsV1 or convert from hive related command.
   
   and I have checked that `PartitioningUtils.getPathFragment` only used in `InsertIntoHadoopFsRelationCommand`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812325204


   GA are failing on Avro tests, for instance. And jenkins build failed on the latest commit. @AngersZhuuuu To continue with the fix, let's re-trigger tests. Also @cloud-fan could you look at this PR since you reviewed previous changes related to null part values.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811718199


   > I am not sure that it is right fix. I guess, `null` should be replaced by `"__HIVE_DEFAULT_PARTITION__"` like:
   > 
   > https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala#L122-L127
   
   
   
   > I am not sure that it is right fix. I guess, `null` should be replaced by `"__HIVE_DEFAULT_PARTITION__"` like:
   > 
   > https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala#L122-L127
   
   Looks like we should handle it in 
   https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala#L351-L355


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811904168


   ping @MaxGekk Updated, current code should be ok since `InsertIntoHadoopFsRelationCommand` only used in DsV1 or convert from hive related command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812485857


   **[Test build #136841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136841/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811731723


   > > which case the path will be null and which case it can be **HIVE_DEFAULT_PARTITION**?
   > 
   > The `null` partition value should be replaced to `__HIVE_DEFAULT_PARTITION__` if it goes via DSv1 and external catalog (Hive MetaStore doesn't accept null part values, and `null` part value in filesystem like `col0=null` is not compatible with other systems). v1 In-Memory catalog follows Hive external catalog as far as I know.
   > 
   > DSv2 impl should handle `null` itself. So, we shouldn't replace it by `__HIVE_DEFAULT_PARTITION__`.
   
   Thanks you a lot for clarify this problem. This make me confused for a long time. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811998761


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41401/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812352039






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811922221


   **[Test build #136817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136817/testReport)** for PR 32018 at commit [`1305fe5`](https://github.com/apache/spark/commit/1305fe558673b57fbcbf3df3780a65e2351ac95d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811969015


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41398/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812604849


   > Could you open separate PRs per each branch, please.
   
   OK, I will ping you when finish these things.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812068245


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136821/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811969015


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41398/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812072213


   **[Test build #136816 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136816/testReport)** for PR 32018 at commit [`03325c3`](https://github.com/apache/spark/commit/03325c3bb9f254222b2ac1f54375704e6b0e3b6f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812393166


   > BTW, @AngersZhuuuu does the issue exist in 3.1/3.0/2.4? If so, please, backport the changes.
   
   Need to check. I will update here after check this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811616047


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41370/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811696620


   **[Test build #136787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136787/testReport)** for PR 32018 at commit [`960f2c1`](https://github.com/apache/spark/commit/960f2c10c94954e6810d3537166905b9e17f4728).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812046522


   **[Test build #136821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136821/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812234168


   > LGTM, @AngersZhuuuu could you update PR's description.
   
   DOne


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811716908


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136787/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812541811


   LGTM2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on a change in pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32018:
URL: https://github.com/apache/spark/pull/32018#discussion_r605662843



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
##########
@@ -350,7 +350,12 @@ object PartitioningUtils {
    */
   def getPathFragment(spec: TablePartitionSpec, partitionSchema: StructType): String = {
     partitionSchema.map { field =>
-      escapePathName(field.name) + "=" + escapePathName(spec(field.name))
+      val value = if (spec(field.name) == null || spec(field.name).isEmpty) {

Review comment:
       Can we re-use existing function or if not, could you extract common code?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
##########
@@ -350,7 +350,12 @@ object PartitioningUtils {
    */
   def getPathFragment(spec: TablePartitionSpec, partitionSchema: StructType): String = {
     partitionSchema.map { field =>
-      escapePathName(field.name) + "=" + escapePathName(spec(field.name))
+      val value = if (spec(field.name) == null || spec(field.name).isEmpty) {
+        DEFAULT_PARTITION_NAME
+      } else {
+        escapePathName(spec(field.name))

Review comment:
       Probably, look up time to `spec` is not big deal but I would store `spec(field.name)` to a val.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811920107


   **[Test build #136816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136816/testReport)** for PR 32018 at commit [`03325c3`](https://github.com/apache/spark/commit/03325c3bb9f254222b2ac1f54375704e6b0e3b6f).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811949440






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811716908


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136787/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811922221


   **[Test build #136817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136817/testReport)** for PR 32018 at commit [`1305fe5`](https://github.com/apache/spark/commit/1305fe558673b57fbcbf3df3780a65e2351ac95d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812334201


   **[Test build #136841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136841/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811727039


   > which case the path will be null and which case it can be __HIVE_DEFAULT_PARTITION__?
   
   The `null` partition value should be replaced to `__HIVE_DEFAULT_PARTITION__` if it goes via DSv1 and external catalog (Hive  MetaStore doesn't accept null part values, and `null` part value in filesystem like `col0=null` is not compatible with other systems). v1 In-Memory catalog follows Hive external catalog as far as I know.
   
   DSv2 impl should handle `null` itself. So, we shouldn't replace it by  `__HIVE_DEFAULT_PARTITION__`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811995765


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41401/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812355320


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41419/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812598464


   > BTW, @AngersZhuuuu does the issue exist in 3.1/3.0/2.4? If so, please, backport the changes.
   
   Checked all branch, we need to backport to branch-3.0/branch-3.1. Should I raise separated pr or you can just merge to that branchs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AngersZhuuuu commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811600763


   ping @MaxGekk Since your pr make partition value support value as `null`, here I think we need to handle null value and  I meet this in https://github.com/apache/spark/pull/30057 's UT  https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136751/testReport/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] MaxGekk closed pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #32018:
URL: https://github.com/apache/spark/pull/32018


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812355320


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41419/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812334201


   **[Test build #136841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136841/testReport)** for PR 32018 at commit [`992001b`](https://github.com/apache/spark/commit/992001bcf3ea7569a492659d97fbde25a5f0c406).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811998715


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41401/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811998761


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41401/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812085515


   **[Test build #136817 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136817/testReport)** for PR 32018 at commit [`1305fe5`](https://github.com/apache/spark/commit/1305fe558673b57fbcbf3df3780a65e2351ac95d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812073418


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136816/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32018: [SPARK-34926][SQL] ExternalCatalogUtils.escapePathName should support null

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-811610688






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32018: [SPARK-34926][SQL] PartitioningUtils.getPathFragment() should respect partition value is null

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32018:
URL: https://github.com/apache/spark/pull/32018#issuecomment-812101280


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136817/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org