You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by peterableda <gi...@git.apache.org> on 2016/02/22 23:47:36 UTC
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
GitHub user peterableda opened a pull request:
https://github.com/apache/spark/pull/11312
[SPARK-13438][STREAMING] Remove by default dash from filename
Spark generates the following schema with prefix = `/data/timestamp=`
```
/data/timestamp=-1455894364000/_SUCCESS
/data/timestamp=-1455894364000/part-00000
```
The above behaviour forces our hands when we would like to write to a schema with partitions.
This change removes the by default dash from the schema.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/peterableda/spark SPARK-13438
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11312.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11312
----
commit e3edaf2f6c31d3ddd5bdac1d4f0b0db16cc10013
Author: Peter Ableda <pe...@cloudera.com>
Date: 2016-02-22T22:32:15Z
Remove by default dash from filename
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by peterableda <gi...@git.apache.org>.
Github user peterableda commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189236579
@srowen The idea was to create an external Hive table on top of the output of the Spark streaming application.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by peterableda <gi...@git.apache.org>.
Github user peterableda commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189619833
Conceptually this is how Hive works:
Create a simple Hive table definition for a `test` table for the location `/data/`
```
CREATE EXTERNAL TABLE test (a string, b string)
PARTITIONED BY (timestamp STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/data/';
```
Create a file in the `/data/timestamp=1455894364000/` partition folder with the following content
```
this,is
test,data
```
When you try to query this hive table `select * from test` you will get the following data
```
1455894364000,this,is
1455894364000,test,data
```
My problem is that currently you can't create the partitions above because of that fixed dash between the prefix and the timestamp.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-188748676
Why does this have to do with partitioning?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by peterableda <gi...@git.apache.org>.
Github user peterableda commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189749198
You are right, the timestamp here refers to the processing time of the batch. I don't have a use-case currently where I want to capture this information in the data. I understand that this is an edge-case and as a break change you don't want to merge... :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-187424720
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189642299
I'm not sure if it was intended to work as a partitioning key. The timestamp in the dir name does not necessarily reflect any timestamp in the data; it's always wall-clock time with a loose connection to when the events happened or arrived. I don't think it's supposed to be interpreted this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-188501591
This will break existing user applications. Since you can rename them in your codes, I don't think it's worth.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189855131
I suspect that the "=" should never have been in the prefix because it is just misleading. I think it's possible to change that for 2.x, but am trying to weigh the value vs disruption. Not sure if we can get an opinion from @tdas ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-189241388
Yes, but what does the directory naming have to do with that? I know partitioning attaches some meaning to "x=y" directories but that isn't changed here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by peterableda <gi...@git.apache.org>.
Github user peterableda closed the pull request at:
https://github.com/apache/spark/pull/11312
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13438][STREAMING] Remove by default das...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11312#issuecomment-192979002
You can close this PR for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org