You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Jason Plurad <pl...@gmail.com> on 2017/05/24 19:45:41 UTC
[DISCUSS] Scala 2.11 and Spark 2.0 support
This question
<https://groups.google.com/d/msg/gremlin-users/rPguKVm3GVE/f1OdIFvBAwAJ> on
TinkerPop 3.3 on the Gremlin users list ultimately comes back to Scala 2.11
support
<https://groups.google.com/d/msg/janusgraph-users/UypKyqnWqzc/eaQUOR2IDAAJ>.
I can understand that some features slated for 3.3 might be far off (
TINKERPOP-1564 <https://issues.apache.org/jira/browse/TINKERPOP-1564>), but
I don't think we should hold up on updating core dependencies such as
Spark+Scala.
Azure Cosmos DB (based on TP 3.2.4) recently published
<https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector-graph>
about connecting
it to Spark, and it tells the developer to build TinkerPop master (TP
3.3.0-SNAPSHOT) because Azure HDInsight uses Spark 2.0.
We ran into a similar issue when TInkerPop 3.0 came out with the Hadoop 1
dependency rather than Hadoop 2, and I think a lot of users had already moved
to Hadoop 2
<https://groups.google.com/d/msg/gremlin-users/_9UD6STUUOE/iEr2qoxFAQAJ> by
then. Side note, I don't know what the schedule for Hadoop 3 is, but they
already have alpha versions available.
Spark 2.0 came out in July 2016, and it has been integrated on master
since September
2016
<https://groups.google.com/d/msg/gremlin-users/j7lDGg5pIo8/QNWaFC3ABQAJ>.
Spark 2 builds with Scala 2.11 by default. If we can't get the next
TinkerPop release until next year, TinkerPop's Spark+Scala dependencies
will be very far out of date.
Would it make sense to cherry pick Spark 2.0 support into the tp32 branch?
Re: [DISCUSS] Scala 2.11 and Spark 2.0 support
Posted by Robert Dale <ro...@gmail.com>.
RERO +1
Robert Dale
On Wed, May 24, 2017 at 4:17 PM, Stephen Mallette <sp...@gmail.com>
wrote:
> well - i guess i'm realizing that i was thinking a lot of big things for
> 3.3.0 but it's almost 6 months into the year and i don't think everything i
> wanted to see is going to get done. then you add in the Spark 2.0 stuff you
> already mentioned which is more reason to release. soooo....maybe it's best
> to clean up 3.3.0, tack in a few critical things that will carry 3.3.x for
> 12 months of dev and release sooner than later. there's always 3.4.0 for
> more big deal stuffs.
>
> On Wed, May 24, 2017 at 4:07 PM, Jason Plurad <pl...@gmail.com> wrote:
>
> > Thanks Stephen. If we can get 3.3 better mapped out, that would be great.
> > Your response on that other thread made it sound like 3.3 was a no-go for
> > this year.
> >
> > On Wed, May 24, 2017 at 3:59 PM, Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > Interesting you brought this up. I was going to start a discussion on
> it
> > > actually next week when I started another discussion about 3.2.5/3.1.7
> > > release. I think we should just talk about how to release 3.3.0. Let's
> > just
> > > figure out what things need to absolutely go in 3.3.0 and then aim for
> a
> > > release date.
> > >
> > > On Wed, May 24, 2017 at 3:45 PM, Jason Plurad <pl...@gmail.com>
> wrote:
> > >
> > > > This question
> > > > <https://groups.google.com/d/msg/gremlin-users/rPguKVm3GVE/
> > f1OdIFvBAwAJ>
> > > > on
> > > > TinkerPop 3.3 on the Gremlin users list ultimately comes back to
> Scala
> > > 2.11
> > > > support
> > > > <https://groups.google.com/d/msg/janusgraph-users/
> > > UypKyqnWqzc/eaQUOR2IDAAJ
> > > > >.
> > > > I can understand that some features slated for 3.3 might be far off (
> > > > TINKERPOP-1564 <https://issues.apache.org/jira/browse/TINKERPOP-1564
> > >),
> > > > but
> > > > I don't think we should hold up on updating core dependencies such as
> > > > Spark+Scala.
> > > >
> > > > Azure Cosmos DB (based on TP 3.2.4) recently published
> > > > <https://docs.microsoft.com/en-us/azure/cosmos-db/spark-
> > connector-graph>
> > > > about connecting
> > > > it to Spark, and it tells the developer to build TinkerPop master (TP
> > > > 3.3.0-SNAPSHOT) because Azure HDInsight uses Spark 2.0.
> > > >
> > > > We ran into a similar issue when TInkerPop 3.0 came out with the
> > Hadoop 1
> > > > dependency rather than Hadoop 2, and I think a lot of users had
> already
> > > > moved
> > > > to Hadoop 2
> > > > <https://groups.google.com/d/msg/gremlin-users/_9UD6STUUOE/
> > iEr2qoxFAQAJ>
> > > > by
> > > > then. Side note, I don't know what the schedule for Hadoop 3 is, but
> > they
> > > > already have alpha versions available.
> > > >
> > > > Spark 2.0 came out in July 2016, and it has been integrated on master
> > > > since September
> > > > 2016
> > > > <https://groups.google.com/d/msg/gremlin-users/j7lDGg5pIo8/
> > QNWaFC3ABQAJ
> > > >.
> > > > Spark 2 builds with Scala 2.11 by default. If we can't get the next
> > > > TinkerPop release until next year, TinkerPop's Spark+Scala
> dependencies
> > > > will be very far out of date.
> > > >
> > > > Would it make sense to cherry pick Spark 2.0 support into the tp32
> > > branch?
> > > >
> > >
> >
>
Re: [DISCUSS] Scala 2.11 and Spark 2.0 support
Posted by Stephen Mallette <sp...@gmail.com>.
well - i guess i'm realizing that i was thinking a lot of big things for
3.3.0 but it's almost 6 months into the year and i don't think everything i
wanted to see is going to get done. then you add in the Spark 2.0 stuff you
already mentioned which is more reason to release. soooo....maybe it's best
to clean up 3.3.0, tack in a few critical things that will carry 3.3.x for
12 months of dev and release sooner than later. there's always 3.4.0 for
more big deal stuffs.
On Wed, May 24, 2017 at 4:07 PM, Jason Plurad <pl...@gmail.com> wrote:
> Thanks Stephen. If we can get 3.3 better mapped out, that would be great.
> Your response on that other thread made it sound like 3.3 was a no-go for
> this year.
>
> On Wed, May 24, 2017 at 3:59 PM, Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > Interesting you brought this up. I was going to start a discussion on it
> > actually next week when I started another discussion about 3.2.5/3.1.7
> > release. I think we should just talk about how to release 3.3.0. Let's
> just
> > figure out what things need to absolutely go in 3.3.0 and then aim for a
> > release date.
> >
> > On Wed, May 24, 2017 at 3:45 PM, Jason Plurad <pl...@gmail.com> wrote:
> >
> > > This question
> > > <https://groups.google.com/d/msg/gremlin-users/rPguKVm3GVE/
> f1OdIFvBAwAJ>
> > > on
> > > TinkerPop 3.3 on the Gremlin users list ultimately comes back to Scala
> > 2.11
> > > support
> > > <https://groups.google.com/d/msg/janusgraph-users/
> > UypKyqnWqzc/eaQUOR2IDAAJ
> > > >.
> > > I can understand that some features slated for 3.3 might be far off (
> > > TINKERPOP-1564 <https://issues.apache.org/jira/browse/TINKERPOP-1564
> >),
> > > but
> > > I don't think we should hold up on updating core dependencies such as
> > > Spark+Scala.
> > >
> > > Azure Cosmos DB (based on TP 3.2.4) recently published
> > > <https://docs.microsoft.com/en-us/azure/cosmos-db/spark-
> connector-graph>
> > > about connecting
> > > it to Spark, and it tells the developer to build TinkerPop master (TP
> > > 3.3.0-SNAPSHOT) because Azure HDInsight uses Spark 2.0.
> > >
> > > We ran into a similar issue when TInkerPop 3.0 came out with the
> Hadoop 1
> > > dependency rather than Hadoop 2, and I think a lot of users had already
> > > moved
> > > to Hadoop 2
> > > <https://groups.google.com/d/msg/gremlin-users/_9UD6STUUOE/
> iEr2qoxFAQAJ>
> > > by
> > > then. Side note, I don't know what the schedule for Hadoop 3 is, but
> they
> > > already have alpha versions available.
> > >
> > > Spark 2.0 came out in July 2016, and it has been integrated on master
> > > since September
> > > 2016
> > > <https://groups.google.com/d/msg/gremlin-users/j7lDGg5pIo8/
> QNWaFC3ABQAJ
> > >.
> > > Spark 2 builds with Scala 2.11 by default. If we can't get the next
> > > TinkerPop release until next year, TinkerPop's Spark+Scala dependencies
> > > will be very far out of date.
> > >
> > > Would it make sense to cherry pick Spark 2.0 support into the tp32
> > branch?
> > >
> >
>
Re: [DISCUSS] Scala 2.11 and Spark 2.0 support
Posted by Jason Plurad <pl...@gmail.com>.
Thanks Stephen. If we can get 3.3 better mapped out, that would be great.
Your response on that other thread made it sound like 3.3 was a no-go for
this year.
On Wed, May 24, 2017 at 3:59 PM, Stephen Mallette <sp...@gmail.com>
wrote:
> Interesting you brought this up. I was going to start a discussion on it
> actually next week when I started another discussion about 3.2.5/3.1.7
> release. I think we should just talk about how to release 3.3.0. Let's just
> figure out what things need to absolutely go in 3.3.0 and then aim for a
> release date.
>
> On Wed, May 24, 2017 at 3:45 PM, Jason Plurad <pl...@gmail.com> wrote:
>
> > This question
> > <https://groups.google.com/d/msg/gremlin-users/rPguKVm3GVE/f1OdIFvBAwAJ>
> > on
> > TinkerPop 3.3 on the Gremlin users list ultimately comes back to Scala
> 2.11
> > support
> > <https://groups.google.com/d/msg/janusgraph-users/
> UypKyqnWqzc/eaQUOR2IDAAJ
> > >.
> > I can understand that some features slated for 3.3 might be far off (
> > TINKERPOP-1564 <https://issues.apache.org/jira/browse/TINKERPOP-1564>),
> > but
> > I don't think we should hold up on updating core dependencies such as
> > Spark+Scala.
> >
> > Azure Cosmos DB (based on TP 3.2.4) recently published
> > <https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector-graph>
> > about connecting
> > it to Spark, and it tells the developer to build TinkerPop master (TP
> > 3.3.0-SNAPSHOT) because Azure HDInsight uses Spark 2.0.
> >
> > We ran into a similar issue when TInkerPop 3.0 came out with the Hadoop 1
> > dependency rather than Hadoop 2, and I think a lot of users had already
> > moved
> > to Hadoop 2
> > <https://groups.google.com/d/msg/gremlin-users/_9UD6STUUOE/iEr2qoxFAQAJ>
> > by
> > then. Side note, I don't know what the schedule for Hadoop 3 is, but they
> > already have alpha versions available.
> >
> > Spark 2.0 came out in July 2016, and it has been integrated on master
> > since September
> > 2016
> > <https://groups.google.com/d/msg/gremlin-users/j7lDGg5pIo8/QNWaFC3ABQAJ
> >.
> > Spark 2 builds with Scala 2.11 by default. If we can't get the next
> > TinkerPop release until next year, TinkerPop's Spark+Scala dependencies
> > will be very far out of date.
> >
> > Would it make sense to cherry pick Spark 2.0 support into the tp32
> branch?
> >
>
Re: [DISCUSS] Scala 2.11 and Spark 2.0 support
Posted by Stephen Mallette <sp...@gmail.com>.
Interesting you brought this up. I was going to start a discussion on it
actually next week when I started another discussion about 3.2.5/3.1.7
release. I think we should just talk about how to release 3.3.0. Let's just
figure out what things need to absolutely go in 3.3.0 and then aim for a
release date.
On Wed, May 24, 2017 at 3:45 PM, Jason Plurad <pl...@gmail.com> wrote:
> This question
> <https://groups.google.com/d/msg/gremlin-users/rPguKVm3GVE/f1OdIFvBAwAJ>
> on
> TinkerPop 3.3 on the Gremlin users list ultimately comes back to Scala 2.11
> support
> <https://groups.google.com/d/msg/janusgraph-users/UypKyqnWqzc/eaQUOR2IDAAJ
> >.
> I can understand that some features slated for 3.3 might be far off (
> TINKERPOP-1564 <https://issues.apache.org/jira/browse/TINKERPOP-1564>),
> but
> I don't think we should hold up on updating core dependencies such as
> Spark+Scala.
>
> Azure Cosmos DB (based on TP 3.2.4) recently published
> <https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector-graph>
> about connecting
> it to Spark, and it tells the developer to build TinkerPop master (TP
> 3.3.0-SNAPSHOT) because Azure HDInsight uses Spark 2.0.
>
> We ran into a similar issue when TInkerPop 3.0 came out with the Hadoop 1
> dependency rather than Hadoop 2, and I think a lot of users had already
> moved
> to Hadoop 2
> <https://groups.google.com/d/msg/gremlin-users/_9UD6STUUOE/iEr2qoxFAQAJ>
> by
> then. Side note, I don't know what the schedule for Hadoop 3 is, but they
> already have alpha versions available.
>
> Spark 2.0 came out in July 2016, and it has been integrated on master
> since September
> 2016
> <https://groups.google.com/d/msg/gremlin-users/j7lDGg5pIo8/QNWaFC3ABQAJ>.
> Spark 2 builds with Scala 2.11 by default. If we can't get the next
> TinkerPop release until next year, TinkerPop's Spark+Scala dependencies
> will be very far out of date.
>
> Would it make sense to cherry pick Spark 2.0 support into the tp32 branch?
>