You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liancheng <gi...@git.apache.org> on 2014/10/01 15:18:45 UTC
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/2616
[SPARK-3007][SQL] WIP: adds dynamic partitioning support
PR #2226 was reverted because of it broke Jenkins builds for unknown reason. This debugging PR aims to fix the Jenkins build.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark dp-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2616.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2616
----
commit 6bb58808875e82740da64bbb7cbca4d22817dbce
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-12T17:27:54Z
Update HiveQl.scala
commit 1867e23f72f94c7161a485222c6a8e31814d20f4
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-12T17:29:36Z
Update SparkHadoopWriter.scala
commit adf02f13e8d93eadb2d032007e00cd9406c62439
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-12T17:31:01Z
Update InsertIntoHiveTable.scala
commit 6af73f46430c5e38d43af8ed288936e1b4ca2678
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-12T17:53:04Z
Update InsertIntoHiveTable.scala
commit 98cfb1fd8dab201f9411dd487e4a8b259216d9a3
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T06:29:21Z
Update HiveCompatibilitySuite.scala
commit 37c603b5001169d1aa21fa293f782666eedc338a
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T06:34:53Z
Update InsertIntoHiveTable.scala
commit d452eb322da5962875c8e96865f1cd63b7dddda0
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T06:36:58Z
Update HiveQuerySuite.scala
commit 051ba91e4952c463446450c745073ab4cf742b26
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T07:18:07Z
Update Cast.scala
commit 8ad173cd41e2abb93df681383cf15aa59e3484c5
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T07:44:07Z
Update InsertIntoHiveTable.scala
commit 3f91665c1e546881f045034023cace851e1949d4
Author: baishuo(白硕) <vc...@hotmail.com>
Date: 2014-08-18T07:57:20Z
Update Cast.scala
commit 8e7268cae0f4f9463c092b6c2833a86be56524e7
Author: baishuo <vc...@hotmail.com>
Date: 2014-08-19T16:01:49Z
update file after test
commit cd822f0471ddf51635e0c2a7054725ab155f5939
Author: baishuo <vc...@hotmail.com>
Date: 2014-08-19T17:14:53Z
do a little modify
commit b660e74574585a0572b5b21aba1ee42a87d89437
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-03T03:28:29Z
delete a empty else branch
commit caea6fb7db2a34e6e0949aabb843b7f728cab541
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-03T06:03:34Z
modify code to pass scala style checks
commit 207c6ace8774f6a89411e0cd929eaedb9f957a8d
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-03T08:22:19Z
modify for some bad indentation
commit 761ecf20365f2864e28b7cad2c0550b159724eb3
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-09T09:57:48Z
modify according micheal's advice
commit 997c990855d94dd2df2ab412566c438def97de61
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-10T02:51:20Z
use HiveConf.DEFAULTPARTITIONNAME to replace hive.exec.default.partition.name
commit b8216119605132bb70a8862e921b0861d5ec7f9f
Author: baishuo <vc...@hotmail.com>
Date: 2014-09-10T03:09:20Z
pass check style
commit d53daa5a263cf0b0fdf47111f1d8ba0c55d08b24
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-16T07:06:25Z
Refactors dynamic partitioning support
Conflicts:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
commit 6fb16d76089e07c4fc273a86c7f887949e904b57
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-17T01:32:12Z
Fixes typo in test name, regenerated golden answer files
commit c47470e4660058b06f9a7704bad0920e69b246fe
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-17T05:13:36Z
Refactors InsertIntoHiveTable to a Command
commit 922718165903bf12b3a2ee40241a60cb20d21886
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-17T05:41:56Z
Minor refactoring
commit 26632c366ad6c8255b50d5e8a41bc23cddbd396b
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-17T07:28:57Z
Adds more tests
commit 0eed349f5824ef3917af1e380bfb529f9875b0c1
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-18T01:08:01Z
Addresses @yhuai's comments
commit 9c6eb2db8de06db0d800ebefb37fc016ecd4c88c
Author: Cheng Lian <li...@gmail.com>
Date: 2014-09-22T04:49:31Z
Adds tests to verify dynamic partitioning folder layout
commit a132c800ecb785a0a5c36a969b824908eba2801c
Author: Cheng Lian <li...@gmail.com>
Date: 2014-10-01T13:13:59Z
Fixes output compression
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57925374
The reason why `_SUCCESS` is reserved is because semantics of `FileSystem.globStatus` got changed, and `Utilities.getFileStatusRecurse` relies on it to find out all partition data files.
Test code:
```scala
object GlobExperiments extends App {
val conf = new Configuration()
val fs = FileSystem.getLocal(conf)
fs.globStatus(new Path("/tmp/wh/*/*/*")).foreach { status =>
println(status.getPath)
}
}
```
Target directory structure:
```
/tmp/wh
├── dir0
│ ├── dir1
│ │ └── level2
│ └── level1
└── level0
```
Hadoop 2.4.1 result:
```
file:/tmp/wh/dir0/dir1/level2
```
Hadoop 1.0.4 resuet:
```
file:/tmp/wh/dir0/dir1/level2
file:/tmp/wh/dir0/level1
file:/tmp/wh/level0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57907458
@scwf Can you elaborate on what configurations you're using? Details like compilation flags, environment variables and building process can be helpful. I've been tracking this failure during the last a few days but couldn't reproduce it either locally or on Jenkins PR builder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57909183
So this bug can be triggered by lower versions of Hadoop, e.g. 1.0.3. I haven't validate the exact range yet.
In `Hive.loadDynamicPartitions`, Hive calls `o.a.h.h.q.e.Utilities.getFIleStatusRecurse` to glob the temporary directory for data files, it seems that lower versions of Hadoop doesn't filter out files like `_SUCCESS`, which causes the problem.
Within Hive, `loadDynamicPartitions` is only used in operations like `LOAD`. At the end of a normal insertion to a dynamically partitioned table, `FileSinkOperator` calls `Utilities.mvFileToFinalPath` to move the entire temporary directory to target location, thus doesn't have this problem.
`Utilities.mvFileToFinalPath` is more efficient than `Hive.loadDynamicPartitions` since it doesn't parses and validates partition specs. But it requires some internal Hive data structures like `DynamicPartitionCtx`. I'll try to see whether I can mock these data structures and use `mvFileToFinalPath` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57495064
Darn... it's because IDEA automatically removed the trailing space of the query string, which causes MD5 of the query doesn't match the golden answer file name any more, and generating the golden file needs `hadoop` executable...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57908927
using -Phive,hadoop-2.4 is ok in my local maching
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57499283
Reverted the accidental trailing space change. However, since this is really dangerous, fixed it in #2619.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57462060
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21111/consoleFull) for PR 2616 at commit [`a132c80`](https://github.com/apache/spark/commit/a132c800ecb785a0a5c36a969b824908eba2801c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57909534
Get it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57907663
@scwf Or could you please describe the steps to reproduce this failure from a newly checked out master branch? I guess once you can reproduce it, it happens deterministically.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57468653
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21115/consoleFull) for PR 2616 at commit [`f471c4b`](https://github.com/apache/spark/commit/f471c4b7ddece2267a7223879fcdcd93d7326e70).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57473611
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21115/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2616
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57902670
Hi, @liancheng, master branch test failed in my machine for all dynamic partition ,
[info] - dynamic_partition *** FAILED ***
[info] - Dynamic partition folder layout *** FAILED ***
[info] - dynamic_partition_skip_default *** FAILED ***
[info] - load_dyn_part1 *** FAILED ***
[info] - load_dyn_part10 *** FAILED ***
[info] - load_dyn_part11 *** FAILED ***
[info] - load_dyn_part12 *** FAILED ***
[info] - load_dyn_part13 *** FAILED ***
[info] - load_dyn_part14 *** FAILED ***
[info] - load_dyn_part14_win *** FAILED ***
[info] - load_dyn_part2 *** FAILED ***
[info] - load_dyn_part3 *** FAILED ***
[info] - load_dyn_part4 *** FAILED ***
[info] - load_dyn_part5 *** FAILED ***
[info] - load_dyn_part6 *** FAILED ***
[info] - load_dyn_part8 *** FAILED ***
[info] - load_dyn_part9 *** FAILED ***
[info] *** 17 TESTS FAILED ***
Detail log---
[info] - dynamic_partition *** FAILED ***
[info] Failed to execute query using catalyst:
[info] Error: get partition: Value for key partcol1 is null or empty
[info] org.apache.hadoop.hive.ql.metadata.HiveException: get partition: Value for key partcol1 is null or empty
[info] at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1585)
[info] at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1556)
[info] at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1189)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57745439
Tried merging but it failed :(
@kayousterhout what did you end up doing to merge this the first time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57909238
@scwf Thanks for all the information you provided offline :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57473604
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21115/consoleFull) for PR 2616 at commit [`f471c4b`](https://github.com/apache/spark/commit/f471c4b7ddece2267a7223879fcdcd93d7326e70).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57908014
Yes, i will use -Phive,hadoop-2.4 to see whether it has the peoblem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57481450
Jenkins failure due to:
> 07:35:12.235 ERROR org.apache.hadoop.hive.ql.exec.mr.ExecDriver: Exception: Cannot run program "/usr/bin/hadoop" (in directory "/home/jenkins/workspace/SparkPullRequestBuilder@2/sql/hive"): error=2, No such file or directory
I assume this is because of the scheduled restart for upgrade.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57909423
According to previous failed Jenkins builds ([1](https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/752/), [2](https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-SBT/753/), etc.), Hadoop 1.0.3 and 2.0 are vulnerable, 2.2 and above are OK. That explains why this PR together with #2226 always passes Jenkins -- the PR builder uses Hadoop 2.3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57733000
@marmbrus Let's try to merge this one to master and see whether Jenkins accepts it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57746732
Comment out the print statement in merge_pr that causes the failure.
On Thu, Oct 2, 2014 at 7:58 PM, Michael Armbrust <no...@github.com>
wrote:
> Tried merging but it failed :(
>
> @kayousterhout <https://github.com/kayousterhout> what did you end up
> doing to merge this the first time?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/2616#issuecomment-57745439>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57835365
Hmmm, still failing with:
```
subprocess.CalledProcessError: Command '[u'git', u'fetch', u'apache', u'master:PR_TOOL_MERGE_PR_2616_MASTER']' returned non-zero exit status 128
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57506903
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21131/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57499218
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21131/consoleFull) for PR 2616 at commit [`21935b6`](https://github.com/apache/spark/commit/21935b66a1be14add4c047334115eeb81b95dae9).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] Adds dynamic partitioning su...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57907852
Ah, just found out that I can reproduce it with `-Phive`, had been using `-Phive,hadoop-2.4` all the time and just couldn't reproduce this, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57470260
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21111/consoleFull) for PR 2616 at commit [`a132c80`](https://github.com/apache/spark/commit/a132c800ecb785a0a5c36a969b824908eba2801c).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57470273
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21111/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57481422
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/222/consoleFull) for PR 2616 at commit [`f471c4b`](https://github.com/apache/spark/commit/f471c4b7ddece2267a7223879fcdcd93d7326e70).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57487884
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/222/consoleFull) for PR 2616 at commit [`f471c4b`](https://github.com/apache/spark/commit/f471c4b7ddece2267a7223879fcdcd93d7326e70).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class PStatsParam(AccumulatorParam):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL] WIP: adds dynamic partitioni...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2616#issuecomment-57506897
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21131/consoleFull) for PR 2616 at commit [`21935b6`](https://github.com/apache/spark/commit/21935b66a1be14add4c047334115eeb81b95dae9).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org