You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Stephen Boesch <ja...@gmail.com> on 2014/10/29 03:42:48 UTC
HiveShim not found when building in Intellij
I have run on the command line via maven and it is fine:
mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn -Phadoop-2.3
compile package install
But with the latest code Intellij builds do not work. Following is one of
26 similar errors:
Error:(173, 38) not found: value HiveShim
Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
^
Re: HiveShim not found when building in Intellij
Posted by Cheng Lian <li...@gmail.com>.
Hao Cheng had just written such a "from scratch" guide for building
Spark SQL in IDEA. Although it's written in Chinese, I think the
illustrations are already descriptive enough.
http://www.cnblogs.com/cccchhhh/articles/4058371.html
On 10/29/14 12:45 PM, Patrick Wendell wrote:
> Btw - we should have part of the official docs that describes a full
> "from scratch" build in IntelliJ including any gotchas. Then we can
> update it if there are build changes that alter it. I created this
> JIRA for it:
>
> https://issues.apache.org/jira/browse/SPARK-4128
>
> On Tue, Oct 28, 2014 at 9:42 PM, Stephen Boesch <ja...@gmail.com> wrote:
>> I am interested specifically in how to build (and hopefully run/debug..)
>> under Intellij. Your posts sound like command line maven - which has always
>> been working already.
>>
>> Do you have instructions for building in IJ?
>>
>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>>
>>> Yes, these two combinations work for me.
>>>
>>>
>>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>>> but expected to go to upstream soon (Spark-3720).
>>>>
>>>> Thanks.
>>>>
>>>> Zhan Zhang
>>>>
>>>>
>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>>
>>>>> Thanks Patrick for the heads up.
>>>>>
>>>>> I have not been successful to discover a combination of profiles (i.e.
>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>>> appreciated.
>>>>>
>>>>>
>>>>>
>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>>
>>>>>> Hey Stephen,
>>>>>>
>>>>>> In some cases in the maven build we now have pluggable source
>>>>>> directories based on profiles using the maven build helper plug-in.
>>>>>> This is necessary to support cross building against different Hive
>>>>>> versions, and there will be additional instances of this due to
>>>>>> supporting scala 2.11 and 2.10.
>>>>>>
>>>>>> In these cases, you may need to add source locations explicitly to
>>>>>> intellij if you want the entire project to compile there.
>>>>>>
>>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>>> use of the maven-build-plugin to add source directories.
>>>>>>
>>>>>> We should come up with a good set of instructions on how to import the
>>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>>> sure exactly what the correct sequence is.
>>>>>>
>>>>>> - Patrick
>>>>>>
>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>>> wrote:
>>>>>>> Hi Matei,
>>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>>> to
>>>>>>> add the hive profile: is it now??
>>>>>>>
>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>>> Open
>>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>>
>>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>>> projects
>>>>>> ,
>>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>>> (which
>>>>>> may
>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>>> "NoClassDefFoundError
>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>> serious
>>>>>>> problem .
>>>>>>>
>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>>
>>>>>>>> Hi Stephen,
>>>>>>>>
>>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>> Hive
>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>>
>>>>>>>> Matei
>>>>>>>>
>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>> wrote:
>>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>>
>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>> -Phadoop-2.3
>>>>>>>>> compile package install
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>> one of
>>>>>>>>> 26 similar errors:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>>
>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>> ^
>>>>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Patrick Wendell <pw...@gmail.com>.
Btw - we should have part of the official docs that describes a full
"from scratch" build in IntelliJ including any gotchas. Then we can
update it if there are build changes that alter it. I created this
JIRA for it:
https://issues.apache.org/jira/browse/SPARK-4128
On Tue, Oct 28, 2014 at 9:42 PM, Stephen Boesch <ja...@gmail.com> wrote:
> I am interested specifically in how to build (and hopefully run/debug..)
> under Intellij. Your posts sound like command line maven - which has always
> been working already.
>
> Do you have instructions for building in IJ?
>
> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>
>> Yes, these two combinations work for me.
>>
>>
>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>
>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>> but expected to go to upstream soon (Spark-3720).
>>>
>>> Thanks.
>>>
>>> Zhan Zhang
>>>
>>>
>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>
>>>> Thanks Patrick for the heads up.
>>>>
>>>> I have not been successful to discover a combination of profiles (i.e.
>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>> appreciated.
>>>>
>>>>
>>>>
>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>
>>>>> Hey Stephen,
>>>>>
>>>>> In some cases in the maven build we now have pluggable source
>>>>> directories based on profiles using the maven build helper plug-in.
>>>>> This is necessary to support cross building against different Hive
>>>>> versions, and there will be additional instances of this due to
>>>>> supporting scala 2.11 and 2.10.
>>>>>
>>>>> In these cases, you may need to add source locations explicitly to
>>>>> intellij if you want the entire project to compile there.
>>>>>
>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>> use of the maven-build-plugin to add source directories.
>>>>>
>>>>> We should come up with a good set of instructions on how to import the
>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>> sure exactly what the correct sequence is.
>>>>>
>>>>> - Patrick
>>>>>
>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Matei,
>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>> to
>>>>>> add the hive profile: is it now??
>>>>>>
>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>> Open
>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>
>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>> projects
>>>>>
>>>>> ,
>>>>>>
>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>> (which
>>>>>
>>>>> may
>>>>>>
>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>> "NoClassDefFoundError
>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>
>>>>> serious
>>>>>>
>>>>>> problem .
>>>>>>
>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>
>>>>>>> Hi Stephen,
>>>>>>>
>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>
>>>>> Hive
>>>>>>>
>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>
>>>>>>> Matei
>>>>>>>
>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>
>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>
>>>>> -Phadoop-2.3
>>>>>>>>
>>>>>>>> compile package install
>>>>>>>>
>>>>>>>>
>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>
>>>>> one of
>>>>>>>>
>>>>>>>> 26 similar errors:
>>>>>>>>
>>>>>>>>
>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>
>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>
>>>>>>>> ^
>>>>>>>
>>>>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Stephen Boesch <ja...@gmail.com>.
Thanks guys - adding the source root for the shim manually was the issue.
For some reason the other issue I was struggling with
(NoCLassDefFoundError on ThreadFactoryBuilder) also disappeared. I am able
to run tests now inside IJ. Woot
2014-10-28 22:13 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> Oops - I actually should have added v0.13.0 (i.e. to match whatever I
> did in the profile).
>
> On Tue, Oct 28, 2014 at 10:05 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > Cheng - to make it recognize the new HiveShim for 0.12 I had to click
> > on spark-hive under "packages" in the left pane, then go to "Open
> > Module Settings" - then explicitly add the v0.12.0/src/main/scala
> > folder to the sources by navigating to it and then <ctrl>+click to add
> > it as a source. Did you have to do this?
> >
> > On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> >> I just started a totally fresh IntelliJ project importing from our
> >> root pom. I used all the default options and I added "hadoop-2.4,
> >> hive, hive-0.13.1" profiles. I was able to run spark core tests from
> >> within IntelliJ. Didn't try anything beyond that, but FWIW this
> >> worked.
> >>
> >> - Patrick
> >>
> >> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com>
> wrote:
> >>> You may first open the root pom.xml file in IDEA, and then go for menu
> View
> >>> / Tool Windows / Maven Projects, then choose desired Maven profile
> >>> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 +
> hive
> >>> + hive-0.12.0). IDEA will ask you to re-import the Maven projects,
> confirm,
> >>> then it should be OK.
> >>>
> >>> I can debug within IDEA with this approach. However, you have to clean
> the
> >>> whole project before debugging Spark within IDEA if you compiled the
> project
> >>> outside IDEA. Haven't got time to investigate this annoying issue.
> >>>
> >>> Also, you can remove sub projects unrelated to your tasks to accelerate
> >>> compilation and/or avoid other IDEA build issues (e.g. Avro related
> Spark
> >>> streaming build failure in IDEA).
> >>>
> >>>
> >>> On 10/29/14 12:42 PM, Stephen Boesch wrote:
> >>>
> >>> I am interested specifically in how to build (and hopefully
> run/debug..)
> >>> under Intellij. Your posts sound like command line maven - which has
> always
> >>> been working already.
> >>>
> >>> Do you have instructions for building in IJ?
> >>>
> >>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
> >>>>
> >>>> Yes, these two combinations work for me.
> >>>>
> >>>>
> >>>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
> >>>>>
> >>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to
> enable
> >>>>> hive-0.12.0. Note that the thrift-server is not supported yet in
> hive-0.13,
> >>>>> but expected to go to upstream soon (Spark-3720).
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Zhan Zhang
> >>>>>
> >>>>>
> >>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Thanks Patrick for the heads up.
> >>>>>>
> >>>>>> I have not been successful to discover a combination of profiles
> (i.e.
> >>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij
> with
> >>>>>> maven. Anyone who knows how to handle this - a quick note here
> would be
> >>>>>> appreciated.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> >>>>>>
> >>>>>>> Hey Stephen,
> >>>>>>>
> >>>>>>> In some cases in the maven build we now have pluggable source
> >>>>>>> directories based on profiles using the maven build helper plug-in.
> >>>>>>> This is necessary to support cross building against different Hive
> >>>>>>> versions, and there will be additional instances of this due to
> >>>>>>> supporting scala 2.11 and 2.10.
> >>>>>>>
> >>>>>>> In these cases, you may need to add source locations explicitly to
> >>>>>>> intellij if you want the entire project to compile there.
> >>>>>>>
> >>>>>>> Unfortunately as long as we support cross-building like this, it
> will
> >>>>>>> be an issue. Intellij's maven support does not correctly detect our
> >>>>>>> use of the maven-build-plugin to add source directories.
> >>>>>>>
> >>>>>>> We should come up with a good set of instructions on how to import
> the
> >>>>>>> pom files + add the few extra source directories. Off hand I am not
> >>>>>>> sure exactly what the correct sequence is.
> >>>>>>>
> >>>>>>> - Patrick
> >>>>>>>
> >>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <javadba@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Matei,
> >>>>>>>> Until my latest pull from upstream/master it had not been
> necessary
> >>>>>>>> to
> >>>>>>>> add the hive profile: is it now??
> >>>>>>>>
> >>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been
> to
> >>>>>>>> Open
> >>>>>>>> the parent directory. IJ recognizes it as a maven project.
> >>>>>>>>
> >>>>>>>> There are several steps to do surgery on the yarn-parent / yarn
> >>>>>>>> projects
> >>>>>>>
> >>>>>>> ,
> >>>>>>>>
> >>>>>>>> then do a full rebuild. That was working until one week ago.
> >>>>>>>> Intellij/maven is presently broken in two ways: this hive shim
> >>>>>>>> (which
> >>>>>>>
> >>>>>>> may
> >>>>>>>>
> >>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
> >>>>>>>> "NoClassDefFoundError
> >>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite
> a
> >>>>>>>
> >>>>>>> serious
> >>>>>>>>
> >>>>>>>> problem .
> >>>>>>>>
> >>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <matei.zaharia@gmail.com
> >:
> >>>>>>>>
> >>>>>>>>> Hi Stephen,
> >>>>>>>>>
> >>>>>>>>> How did you generate your Maven workspace? You need to make sure
> the
> >>>>>>>
> >>>>>>> Hive
> >>>>>>>>>
> >>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
> >>>>>>>>>
> >>>>>>>>> Matei
> >>>>>>>>>
> >>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I have run on the command line via maven and it is fine:
> >>>>>>>>>>
> >>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
> >>>>>>>
> >>>>>>> -Phadoop-2.3
> >>>>>>>>>>
> >>>>>>>>>> compile package install
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> But with the latest code Intellij builds do not work. Following
> is
> >>>>>>>
> >>>>>>> one of
> >>>>>>>>>>
> >>>>>>>>>> 26 similar errors:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Error:(173, 38) not found: value HiveShim
> >>>>>>>>>>
> >>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
> >>>>>>>>>>
> >>>>>>>>>> ^
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>>
> >>>
> >>>
>
Re: HiveShim not found when building in Intellij
Posted by Patrick Wendell <pw...@gmail.com>.
Oops - I actually should have added v0.13.0 (i.e. to match whatever I
did in the profile).
On Tue, Oct 28, 2014 at 10:05 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Cheng - to make it recognize the new HiveShim for 0.12 I had to click
> on spark-hive under "packages" in the left pane, then go to "Open
> Module Settings" - then explicitly add the v0.12.0/src/main/scala
> folder to the sources by navigating to it and then <ctrl>+click to add
> it as a source. Did you have to do this?
>
> On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> I just started a totally fresh IntelliJ project importing from our
>> root pom. I used all the default options and I added "hadoop-2.4,
>> hive, hive-0.13.1" profiles. I was able to run spark core tests from
>> within IntelliJ. Didn't try anything beyond that, but FWIW this
>> worked.
>>
>> - Patrick
>>
>> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com> wrote:
>>> You may first open the root pom.xml file in IDEA, and then go for menu View
>>> / Tool Windows / Maven Projects, then choose desired Maven profile
>>> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive
>>> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm,
>>> then it should be OK.
>>>
>>> I can debug within IDEA with this approach. However, you have to clean the
>>> whole project before debugging Spark within IDEA if you compiled the project
>>> outside IDEA. Haven't got time to investigate this annoying issue.
>>>
>>> Also, you can remove sub projects unrelated to your tasks to accelerate
>>> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark
>>> streaming build failure in IDEA).
>>>
>>>
>>> On 10/29/14 12:42 PM, Stephen Boesch wrote:
>>>
>>> I am interested specifically in how to build (and hopefully run/debug..)
>>> under Intellij. Your posts sound like command line maven - which has always
>>> been working already.
>>>
>>> Do you have instructions for building in IJ?
>>>
>>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>>>>
>>>> Yes, these two combinations work for me.
>>>>
>>>>
>>>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>>>
>>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>>>> but expected to go to upstream soon (Spark-3720).
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Zhan Zhang
>>>>>
>>>>>
>>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Patrick for the heads up.
>>>>>>
>>>>>> I have not been successful to discover a combination of profiles (i.e.
>>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>>>> appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>>>
>>>>>>> Hey Stephen,
>>>>>>>
>>>>>>> In some cases in the maven build we now have pluggable source
>>>>>>> directories based on profiles using the maven build helper plug-in.
>>>>>>> This is necessary to support cross building against different Hive
>>>>>>> versions, and there will be additional instances of this due to
>>>>>>> supporting scala 2.11 and 2.10.
>>>>>>>
>>>>>>> In these cases, you may need to add source locations explicitly to
>>>>>>> intellij if you want the entire project to compile there.
>>>>>>>
>>>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>>>> use of the maven-build-plugin to add source directories.
>>>>>>>
>>>>>>> We should come up with a good set of instructions on how to import the
>>>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>>>> sure exactly what the correct sequence is.
>>>>>>>
>>>>>>> - Patrick
>>>>>>>
>>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Matei,
>>>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>>>> to
>>>>>>>> add the hive profile: is it now??
>>>>>>>>
>>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>>>> Open
>>>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>>>
>>>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>>>> projects
>>>>>>>
>>>>>>> ,
>>>>>>>>
>>>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>>>> (which
>>>>>>>
>>>>>>> may
>>>>>>>>
>>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>>>> "NoClassDefFoundError
>>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>>>
>>>>>>> serious
>>>>>>>>
>>>>>>>> problem .
>>>>>>>>
>>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi Stephen,
>>>>>>>>>
>>>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>>>
>>>>>>> Hive
>>>>>>>>>
>>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>>>
>>>>>>>>> Matei
>>>>>>>>>
>>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>>>
>>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>>>
>>>>>>> -Phadoop-2.3
>>>>>>>>>>
>>>>>>>>>> compile package install
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>>>
>>>>>>> one of
>>>>>>>>>>
>>>>>>>>>> 26 similar errors:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>>>
>>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>>>
>>>>>>>>>> ^
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Cheng Lian <li...@gmail.com>.
Hm, the shim source folder could be automatically recognized some time
before, although at a wrong directory level (sql/hive/v0.12.0/src
instead of sql/hive/v0.12.0/src/main/scala), it compiles.
Just tried against a fresh checkout, indeed need to add shim source
folder manually. Sorry for the confusion.
Cheng
On 10/29/14 1:05 PM, Patrick Wendell wrote:
> Cheng - to make it recognize the new HiveShim for 0.12 I had to click
> on spark-hive under "packages" in the left pane, then go to "Open
> Module Settings" - then explicitly add the v0.12.0/src/main/scala
> folder to the sources by navigating to it and then <ctrl>+click to add
> it as a source. Did you have to do this?
>
> On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> I just started a totally fresh IntelliJ project importing from our
>> root pom. I used all the default options and I added "hadoop-2.4,
>> hive, hive-0.13.1" profiles. I was able to run spark core tests from
>> within IntelliJ. Didn't try anything beyond that, but FWIW this
>> worked.
>>
>> - Patrick
>>
>> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com> wrote:
>>> You may first open the root pom.xml file in IDEA, and then go for menu View
>>> / Tool Windows / Maven Projects, then choose desired Maven profile
>>> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive
>>> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm,
>>> then it should be OK.
>>>
>>> I can debug within IDEA with this approach. However, you have to clean the
>>> whole project before debugging Spark within IDEA if you compiled the project
>>> outside IDEA. Haven't got time to investigate this annoying issue.
>>>
>>> Also, you can remove sub projects unrelated to your tasks to accelerate
>>> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark
>>> streaming build failure in IDEA).
>>>
>>>
>>> On 10/29/14 12:42 PM, Stephen Boesch wrote:
>>>
>>> I am interested specifically in how to build (and hopefully run/debug..)
>>> under Intellij. Your posts sound like command line maven - which has always
>>> been working already.
>>>
>>> Do you have instructions for building in IJ?
>>>
>>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>>>> Yes, these two combinations work for me.
>>>>
>>>>
>>>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>>>> but expected to go to upstream soon (Spark-3720).
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Zhan Zhang
>>>>>
>>>>>
>>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Patrick for the heads up.
>>>>>>
>>>>>> I have not been successful to discover a combination of profiles (i.e.
>>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>>>> appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>>>
>>>>>>> Hey Stephen,
>>>>>>>
>>>>>>> In some cases in the maven build we now have pluggable source
>>>>>>> directories based on profiles using the maven build helper plug-in.
>>>>>>> This is necessary to support cross building against different Hive
>>>>>>> versions, and there will be additional instances of this due to
>>>>>>> supporting scala 2.11 and 2.10.
>>>>>>>
>>>>>>> In these cases, you may need to add source locations explicitly to
>>>>>>> intellij if you want the entire project to compile there.
>>>>>>>
>>>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>>>> use of the maven-build-plugin to add source directories.
>>>>>>>
>>>>>>> We should come up with a good set of instructions on how to import the
>>>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>>>> sure exactly what the correct sequence is.
>>>>>>>
>>>>>>> - Patrick
>>>>>>>
>>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi Matei,
>>>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>>>> to
>>>>>>>> add the hive profile: is it now??
>>>>>>>>
>>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>>>> Open
>>>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>>>
>>>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>>>> projects
>>>>>>> ,
>>>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>>>> (which
>>>>>>> may
>>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>>>> "NoClassDefFoundError
>>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>>> serious
>>>>>>>> problem .
>>>>>>>>
>>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi Stephen,
>>>>>>>>>
>>>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>>> Hive
>>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>>>
>>>>>>>>> Matei
>>>>>>>>>
>>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>>>
>>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>>> -Phadoop-2.3
>>>>>>>>>> compile package install
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>>> one of
>>>>>>>>>> 26 similar errors:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>>>
>>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>>> ^
>>>>>>>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Patrick Wendell <pw...@gmail.com>.
Cheng - to make it recognize the new HiveShim for 0.12 I had to click
on spark-hive under "packages" in the left pane, then go to "Open
Module Settings" - then explicitly add the v0.12.0/src/main/scala
folder to the sources by navigating to it and then <ctrl>+click to add
it as a source. Did you have to do this?
On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I just started a totally fresh IntelliJ project importing from our
> root pom. I used all the default options and I added "hadoop-2.4,
> hive, hive-0.13.1" profiles. I was able to run spark core tests from
> within IntelliJ. Didn't try anything beyond that, but FWIW this
> worked.
>
> - Patrick
>
> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com> wrote:
>> You may first open the root pom.xml file in IDEA, and then go for menu View
>> / Tool Windows / Maven Projects, then choose desired Maven profile
>> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive
>> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm,
>> then it should be OK.
>>
>> I can debug within IDEA with this approach. However, you have to clean the
>> whole project before debugging Spark within IDEA if you compiled the project
>> outside IDEA. Haven't got time to investigate this annoying issue.
>>
>> Also, you can remove sub projects unrelated to your tasks to accelerate
>> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark
>> streaming build failure in IDEA).
>>
>>
>> On 10/29/14 12:42 PM, Stephen Boesch wrote:
>>
>> I am interested specifically in how to build (and hopefully run/debug..)
>> under Intellij. Your posts sound like command line maven - which has always
>> been working already.
>>
>> Do you have instructions for building in IJ?
>>
>> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>>>
>>> Yes, these two combinations work for me.
>>>
>>>
>>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>>
>>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>>> but expected to go to upstream soon (Spark-3720).
>>>>
>>>> Thanks.
>>>>
>>>> Zhan Zhang
>>>>
>>>>
>>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>>
>>>>> Thanks Patrick for the heads up.
>>>>>
>>>>> I have not been successful to discover a combination of profiles (i.e.
>>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>>> appreciated.
>>>>>
>>>>>
>>>>>
>>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>>
>>>>>> Hey Stephen,
>>>>>>
>>>>>> In some cases in the maven build we now have pluggable source
>>>>>> directories based on profiles using the maven build helper plug-in.
>>>>>> This is necessary to support cross building against different Hive
>>>>>> versions, and there will be additional instances of this due to
>>>>>> supporting scala 2.11 and 2.10.
>>>>>>
>>>>>> In these cases, you may need to add source locations explicitly to
>>>>>> intellij if you want the entire project to compile there.
>>>>>>
>>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>>> use of the maven-build-plugin to add source directories.
>>>>>>
>>>>>> We should come up with a good set of instructions on how to import the
>>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>>> sure exactly what the correct sequence is.
>>>>>>
>>>>>> - Patrick
>>>>>>
>>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Matei,
>>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>>> to
>>>>>>> add the hive profile: is it now??
>>>>>>>
>>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>>> Open
>>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>>
>>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>>> projects
>>>>>>
>>>>>> ,
>>>>>>>
>>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>>> (which
>>>>>>
>>>>>> may
>>>>>>>
>>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>>> "NoClassDefFoundError
>>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>>
>>>>>> serious
>>>>>>>
>>>>>>> problem .
>>>>>>>
>>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>>
>>>>>>>> Hi Stephen,
>>>>>>>>
>>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>>
>>>>>> Hive
>>>>>>>>
>>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>>
>>>>>>>> Matei
>>>>>>>>
>>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>>
>>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>>
>>>>>> -Phadoop-2.3
>>>>>>>>>
>>>>>>>>> compile package install
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>>
>>>>>> one of
>>>>>>>>>
>>>>>>>>> 26 similar errors:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>>
>>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>>
>>>>>>>>> ^
>>>>>>>>
>>>>>>>>
>>>>
>>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Stephen Boesch <ja...@gmail.com>.
I have selected the same options as Cheng LIang: hadoop-2.4, hive, hive
0.12.0 . After a full Rebuild in IJ I still see the HiveShim errors.
I really do not know what is different. I had pulled three hours ago from
github upstream master.
Just for kicks i am trying PW's combination which uses 0.13.1 now.. But it
appears there is something else going on here.
Patrick/ Cheng: did you build on the command line using Maven first? I do
that since in the past that had been required.
2014-10-28 21:57 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> I just started a totally fresh IntelliJ project importing from our
> root pom. I used all the default options and I added "hadoop-2.4,
> hive, hive-0.13.1" profiles. I was able to run spark core tests from
> within IntelliJ. Didn't try anything beyond that, but FWIW this
> worked.
>
> - Patrick
>
> On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com> wrote:
> > You may first open the root pom.xml file in IDEA, and then go for menu
> View
> > / Tool Windows / Maven Projects, then choose desired Maven profile
> > combination under the "Profiles" node (e.g. I usually use hadoop-2.4 +
> hive
> > + hive-0.12.0). IDEA will ask you to re-import the Maven projects,
> confirm,
> > then it should be OK.
> >
> > I can debug within IDEA with this approach. However, you have to clean
> the
> > whole project before debugging Spark within IDEA if you compiled the
> project
> > outside IDEA. Haven't got time to investigate this annoying issue.
> >
> > Also, you can remove sub projects unrelated to your tasks to accelerate
> > compilation and/or avoid other IDEA build issues (e.g. Avro related Spark
> > streaming build failure in IDEA).
> >
> >
> > On 10/29/14 12:42 PM, Stephen Boesch wrote:
> >
> > I am interested specifically in how to build (and hopefully run/debug..)
> > under Intellij. Your posts sound like command line maven - which has
> always
> > been working already.
> >
> > Do you have instructions for building in IJ?
> >
> > 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
> >>
> >> Yes, these two combinations work for me.
> >>
> >>
> >> On 10/29/14 12:32 PM, Zhan Zhang wrote:
> >>>
> >>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
> >>> hive-0.12.0. Note that the thrift-server is not supported yet in
> hive-0.13,
> >>> but expected to go to upstream soon (Spark-3720).
> >>>
> >>> Thanks.
> >>>
> >>> Zhan Zhang
> >>>
> >>>
> >>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >>>
> >>>> Thanks Patrick for the heads up.
> >>>>
> >>>> I have not been successful to discover a combination of profiles (i.e.
> >>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
> >>>> maven. Anyone who knows how to handle this - a quick note here would
> be
> >>>> appreciated.
> >>>>
> >>>>
> >>>>
> >>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> >>>>
> >>>>> Hey Stephen,
> >>>>>
> >>>>> In some cases in the maven build we now have pluggable source
> >>>>> directories based on profiles using the maven build helper plug-in.
> >>>>> This is necessary to support cross building against different Hive
> >>>>> versions, and there will be additional instances of this due to
> >>>>> supporting scala 2.11 and 2.10.
> >>>>>
> >>>>> In these cases, you may need to add source locations explicitly to
> >>>>> intellij if you want the entire project to compile there.
> >>>>>
> >>>>> Unfortunately as long as we support cross-building like this, it will
> >>>>> be an issue. Intellij's maven support does not correctly detect our
> >>>>> use of the maven-build-plugin to add source directories.
> >>>>>
> >>>>> We should come up with a good set of instructions on how to import
> the
> >>>>> pom files + add the few extra source directories. Off hand I am not
> >>>>> sure exactly what the correct sequence is.
> >>>>>
> >>>>> - Patrick
> >>>>>
> >>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Matei,
> >>>>>> Until my latest pull from upstream/master it had not been
> necessary
> >>>>>> to
> >>>>>> add the hive profile: is it now??
> >>>>>>
> >>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
> >>>>>> Open
> >>>>>> the parent directory. IJ recognizes it as a maven project.
> >>>>>>
> >>>>>> There are several steps to do surgery on the yarn-parent / yarn
> >>>>>> projects
> >>>>>
> >>>>> ,
> >>>>>>
> >>>>>> then do a full rebuild. That was working until one week ago.
> >>>>>> Intellij/maven is presently broken in two ways: this hive shim
> >>>>>> (which
> >>>>>
> >>>>> may
> >>>>>>
> >>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
> >>>>>> "NoClassDefFoundError
> >>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
> >>>>>
> >>>>> serious
> >>>>>>
> >>>>>> problem .
> >>>>>>
> >>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
> >>>>>>
> >>>>>>> Hi Stephen,
> >>>>>>>
> >>>>>>> How did you generate your Maven workspace? You need to make sure
> the
> >>>>>
> >>>>> Hive
> >>>>>>>
> >>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
> >>>>>>>
> >>>>>>> Matei
> >>>>>>>
> >>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
> >>>>>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> I have run on the command line via maven and it is fine:
> >>>>>>>>
> >>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
> >>>>>
> >>>>> -Phadoop-2.3
> >>>>>>>>
> >>>>>>>> compile package install
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> But with the latest code Intellij builds do not work. Following is
> >>>>>
> >>>>> one of
> >>>>>>>>
> >>>>>>>> 26 similar errors:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Error:(173, 38) not found: value HiveShim
> >>>>>>>>
> >>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
> >>>>>>>>
> >>>>>>>> ^
> >>>>>>>
> >>>>>>>
> >>>
> >>
> >
> >
>
Re: HiveShim not found when building in Intellij
Posted by Patrick Wendell <pw...@gmail.com>.
I just started a totally fresh IntelliJ project importing from our
root pom. I used all the default options and I added "hadoop-2.4,
hive, hive-0.13.1" profiles. I was able to run spark core tests from
within IntelliJ. Didn't try anything beyond that, but FWIW this
worked.
- Patrick
On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <li...@gmail.com> wrote:
> You may first open the root pom.xml file in IDEA, and then go for menu View
> / Tool Windows / Maven Projects, then choose desired Maven profile
> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive
> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm,
> then it should be OK.
>
> I can debug within IDEA with this approach. However, you have to clean the
> whole project before debugging Spark within IDEA if you compiled the project
> outside IDEA. Haven't got time to investigate this annoying issue.
>
> Also, you can remove sub projects unrelated to your tasks to accelerate
> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark
> streaming build failure in IDEA).
>
>
> On 10/29/14 12:42 PM, Stephen Boesch wrote:
>
> I am interested specifically in how to build (and hopefully run/debug..)
> under Intellij. Your posts sound like command line maven - which has always
> been working already.
>
> Do you have instructions for building in IJ?
>
> 2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
>>
>> Yes, these two combinations work for me.
>>
>>
>> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>>>
>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable
>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>>> but expected to go to upstream soon (Spark-3720).
>>>
>>> Thanks.
>>>
>>> Zhan Zhang
>>>
>>>
>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>
>>>> Thanks Patrick for the heads up.
>>>>
>>>> I have not been successful to discover a combination of profiles (i.e.
>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>>> maven. Anyone who knows how to handle this - a quick note here would be
>>>> appreciated.
>>>>
>>>>
>>>>
>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>>
>>>>> Hey Stephen,
>>>>>
>>>>> In some cases in the maven build we now have pluggable source
>>>>> directories based on profiles using the maven build helper plug-in.
>>>>> This is necessary to support cross building against different Hive
>>>>> versions, and there will be additional instances of this due to
>>>>> supporting scala 2.11 and 2.10.
>>>>>
>>>>> In these cases, you may need to add source locations explicitly to
>>>>> intellij if you want the entire project to compile there.
>>>>>
>>>>> Unfortunately as long as we support cross-building like this, it will
>>>>> be an issue. Intellij's maven support does not correctly detect our
>>>>> use of the maven-build-plugin to add source directories.
>>>>>
>>>>> We should come up with a good set of instructions on how to import the
>>>>> pom files + add the few extra source directories. Off hand I am not
>>>>> sure exactly what the correct sequence is.
>>>>>
>>>>> - Patrick
>>>>>
>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Matei,
>>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>>> to
>>>>>> add the hive profile: is it now??
>>>>>>
>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>>> Open
>>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>>
>>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>>> projects
>>>>>
>>>>> ,
>>>>>>
>>>>>> then do a full rebuild. That was working until one week ago.
>>>>>> Intellij/maven is presently broken in two ways: this hive shim
>>>>>> (which
>>>>>
>>>>> may
>>>>>>
>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>>> "NoClassDefFoundError
>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>
>>>>> serious
>>>>>>
>>>>>> problem .
>>>>>>
>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>>
>>>>>>> Hi Stephen,
>>>>>>>
>>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>
>>>>> Hive
>>>>>>>
>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>>
>>>>>>> Matei
>>>>>>>
>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>> I have run on the command line via maven and it is fine:
>>>>>>>>
>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>
>>>>> -Phadoop-2.3
>>>>>>>>
>>>>>>>> compile package install
>>>>>>>>
>>>>>>>>
>>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>
>>>>> one of
>>>>>>>>
>>>>>>>> 26 similar errors:
>>>>>>>>
>>>>>>>>
>>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>>
>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>>>
>>>>>>>> ^
>>>>>>>
>>>>>>>
>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Cheng Lian <li...@gmail.com>.
You may first open the root pom.xml file in IDEA, and then go for menu
View / Tool Windows / Maven Projects, then choose desired Maven profile
combination under the "Profiles" node (e.g. I usually use hadoop-2.4 +
hive + hive-0.12.0). IDEA will ask you to re-import the Maven projects,
confirm, then it should be OK.
I can debug within IDEA with this approach. However, you have to clean
the whole project before debugging Spark within IDEA if you compiled the
project outside IDEA. Haven't got time to investigate this annoying issue.
Also, you can remove sub projects unrelated to your tasks to accelerate
compilation and/or avoid other IDEA build issues (e.g. Avro related
Spark streaming build failure in IDEA).
On 10/29/14 12:42 PM, Stephen Boesch wrote:
> I am interested specifically in how to build (and hopefully
> run/debug..) under Intellij. Your posts sound like command line maven
> - which has always been working already.
>
> Do you have instructions for building in IJ?
>
> 2014-10-28 21:38 GMT-07:00 Cheng Lian <lian.cs.zju@gmail.com
> <ma...@gmail.com>>:
>
> Yes, these two combinations work for me.
>
>
> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>
> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0” is
> to enable hive-0.12.0. Note that the thrift-server is not
> supported yet in hive-0.13, but expected to go to upstream
> soon (Spark-3720).
>
> Thanks.
>
> Zhan Zhang
>
>
> On Oct 28, 2014, at 9:09 PM, Stephen Boesch
> <javadba@gmail.com <ma...@gmail.com>> wrote:
>
> Thanks Patrick for the heads up.
>
> I have not been successful to discover a combination of
> profiles (i.e.
> enabling hive or hive-0.12.0 or hive-13.0) that works in
> Intellij with
> maven. Anyone who knows how to handle this - a quick note
> here would be
> appreciated.
>
>
>
> 2014-10-28 20:20 GMT-07:00 Patrick Wendell
> <pwendell@gmail.com <ma...@gmail.com>>:
>
> Hey Stephen,
>
> In some cases in the maven build we now have pluggable
> source
> directories based on profiles using the maven build
> helper plug-in.
> This is necessary to support cross building against
> different Hive
> versions, and there will be additional instances of
> this due to
> supporting scala 2.11 and 2.10.
>
> In these cases, you may need to add source locations
> explicitly to
> intellij if you want the entire project to compile there.
>
> Unfortunately as long as we support cross-building
> like this, it will
> be an issue. Intellij's maven support does not
> correctly detect our
> use of the maven-build-plugin to add source directories.
>
> We should come up with a good set of instructions on
> how to import the
> pom files + add the few extra source directories. Off
> hand I am not
> sure exactly what the correct sequence is.
>
> - Patrick
>
> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch
> <javadba@gmail.com <ma...@gmail.com>> wrote:
>
> Hi Matei,
> Until my latest pull from upstream/master it had
> not been necessary to
> add the hive profile: is it now??
>
> I am not using sbt gen-idea. The way to open in
> intellij has been to Open
> the parent directory. IJ recognizes it as a maven
> project.
>
> There are several steps to do surgery on the
> yarn-parent / yarn projects
>
> ,
>
> then do a full rebuild. That was working until
> one week ago.
> Intellij/maven is presently broken in two ways:
> this hive shim (which
>
> may
>
> yet hopefully be a small/simple fix - let us see)
> and (2) the
> "NoClassDefFoundError
> on ThreadFactoryBuilder" from my prior emails -and
> which is quite a
>
> serious
>
> problem .
>
> 2014-10-28 19:46 GMT-07:00 Matei Zaharia
> <matei.zaharia@gmail.com
> <ma...@gmail.com>>:
>
> Hi Stephen,
>
> How did you generate your Maven workspace? You
> need to make sure the
>
> Hive
>
> profile is enabled for it. For example sbt/sbt
> -Phive gen-idea.
>
> Matei
>
> On Oct 28, 2014, at 7:42 PM, Stephen
> Boesch <javadba@gmail.com
> <ma...@gmail.com>>
>
> wrote:
>
> I have run on the command line via maven
> and it is fine:
>
> mvn -Dscalastyle.failOnViolation=false
> -DskipTests -Pyarn
>
> -Phadoop-2.3
>
> compile package install
>
>
> But with the latest code Intellij builds
> do not work. Following is
>
> one of
>
> 26 similar errors:
>
>
> Error:(173, 38) not found: value HiveShim
>
> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>
> ^
>
>
>
>
>
Re: HiveShim not found when building in Intellij
Posted by Stephen Boesch <ja...@gmail.com>.
I am interested specifically in how to build (and hopefully run/debug..)
under Intellij. Your posts sound like command line maven - which has
always been working already.
Do you have instructions for building in IJ?
2014-10-28 21:38 GMT-07:00 Cheng Lian <li...@gmail.com>:
> Yes, these two combinations work for me.
>
>
> On 10/29/14 12:32 PM, Zhan Zhang wrote:
>
>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0” is to enable
>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13,
>> but expected to go to upstream soon (Spark-3720).
>>
>> Thanks.
>>
>> Zhan Zhang
>>
>>
>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>
>> Thanks Patrick for the heads up.
>>>
>>> I have not been successful to discover a combination of profiles (i.e.
>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>>> maven. Anyone who knows how to handle this - a quick note here would be
>>> appreciated.
>>>
>>>
>>>
>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>>
>>> Hey Stephen,
>>>>
>>>> In some cases in the maven build we now have pluggable source
>>>> directories based on profiles using the maven build helper plug-in.
>>>> This is necessary to support cross building against different Hive
>>>> versions, and there will be additional instances of this due to
>>>> supporting scala 2.11 and 2.10.
>>>>
>>>> In these cases, you may need to add source locations explicitly to
>>>> intellij if you want the entire project to compile there.
>>>>
>>>> Unfortunately as long as we support cross-building like this, it will
>>>> be an issue. Intellij's maven support does not correctly detect our
>>>> use of the maven-build-plugin to add source directories.
>>>>
>>>> We should come up with a good set of instructions on how to import the
>>>> pom files + add the few extra source directories. Off hand I am not
>>>> sure exactly what the correct sequence is.
>>>>
>>>> - Patrick
>>>>
>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Matei,
>>>>> Until my latest pull from upstream/master it had not been necessary
>>>>> to
>>>>> add the hive profile: is it now??
>>>>>
>>>>> I am not using sbt gen-idea. The way to open in intellij has been to
>>>>> Open
>>>>> the parent directory. IJ recognizes it as a maven project.
>>>>>
>>>>> There are several steps to do surgery on the yarn-parent / yarn
>>>>> projects
>>>>>
>>>> ,
>>>>
>>>>> then do a full rebuild. That was working until one week ago.
>>>>> Intellij/maven is presently broken in two ways: this hive shim (which
>>>>>
>>>> may
>>>>
>>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>>> "NoClassDefFoundError
>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>>>>
>>>> serious
>>>>
>>>>> problem .
>>>>>
>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>>
>>>>> Hi Stephen,
>>>>>>
>>>>>> How did you generate your Maven workspace? You need to make sure the
>>>>>>
>>>>> Hive
>>>>
>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>>
>>>>>> Matei
>>>>>>
>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>> I have run on the command line via maven and it is fine:
>>>>>>>
>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>>>>>>
>>>>>> -Phadoop-2.3
>>>>
>>>>> compile package install
>>>>>>>
>>>>>>>
>>>>>>> But with the latest code Intellij builds do not work. Following is
>>>>>>>
>>>>>> one of
>>>>
>>>>> 26 similar errors:
>>>>>>>
>>>>>>>
>>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>>
>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>>
>>>>>>> ^
>>>>>>>
>>>>>>
>>>>>>
>>
>
Re: HiveShim not found when building in Intellij
Posted by Cheng Lian <li...@gmail.com>.
Yes, these two combinations work for me.
On 10/29/14 12:32 PM, Zhan Zhang wrote:
> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0” is to enable hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13, but expected to go to upstream soon (Spark-3720).
>
> Thanks.
>
> Zhan Zhang
>
>
>
> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
>> Thanks Patrick for the heads up.
>>
>> I have not been successful to discover a combination of profiles (i.e.
>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
>> maven. Anyone who knows how to handle this - a quick note here would be
>> appreciated.
>>
>>
>>
>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>>
>>> Hey Stephen,
>>>
>>> In some cases in the maven build we now have pluggable source
>>> directories based on profiles using the maven build helper plug-in.
>>> This is necessary to support cross building against different Hive
>>> versions, and there will be additional instances of this due to
>>> supporting scala 2.11 and 2.10.
>>>
>>> In these cases, you may need to add source locations explicitly to
>>> intellij if you want the entire project to compile there.
>>>
>>> Unfortunately as long as we support cross-building like this, it will
>>> be an issue. Intellij's maven support does not correctly detect our
>>> use of the maven-build-plugin to add source directories.
>>>
>>> We should come up with a good set of instructions on how to import the
>>> pom files + add the few extra source directories. Off hand I am not
>>> sure exactly what the correct sequence is.
>>>
>>> - Patrick
>>>
>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>>> Hi Matei,
>>>> Until my latest pull from upstream/master it had not been necessary to
>>>> add the hive profile: is it now??
>>>>
>>>> I am not using sbt gen-idea. The way to open in intellij has been to Open
>>>> the parent directory. IJ recognizes it as a maven project.
>>>>
>>>> There are several steps to do surgery on the yarn-parent / yarn projects
>>> ,
>>>> then do a full rebuild. That was working until one week ago.
>>>> Intellij/maven is presently broken in two ways: this hive shim (which
>>> may
>>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>>> "NoClassDefFoundError
>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>>> serious
>>>> problem .
>>>>
>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>>
>>>>> Hi Stephen,
>>>>>
>>>>> How did you generate your Maven workspace? You need to make sure the
>>> Hive
>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>>
>>>>> Matei
>>>>>
>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>>> wrote:
>>>>>> I have run on the command line via maven and it is fine:
>>>>>>
>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>>> -Phadoop-2.3
>>>>>> compile package install
>>>>>>
>>>>>>
>>>>>> But with the latest code Intellij builds do not work. Following is
>>> one of
>>>>>> 26 similar errors:
>>>>>>
>>>>>>
>>>>>> Error:(173, 38) not found: value HiveShim
>>>>>>
>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>>> ^
>>>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Zhan Zhang <zz...@hortonworks.com>.
-Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0” is to enable hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13, but expected to go to upstream soon (Spark-3720).
Thanks.
Zhan Zhang
On Oct 28, 2014, at 9:09 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Thanks Patrick for the heads up.
>
> I have not been successful to discover a combination of profiles (i.e.
> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
> maven. Anyone who knows how to handle this - a quick note here would be
> appreciated.
>
>
>
> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
>
>> Hey Stephen,
>>
>> In some cases in the maven build we now have pluggable source
>> directories based on profiles using the maven build helper plug-in.
>> This is necessary to support cross building against different Hive
>> versions, and there will be additional instances of this due to
>> supporting scala 2.11 and 2.10.
>>
>> In these cases, you may need to add source locations explicitly to
>> intellij if you want the entire project to compile there.
>>
>> Unfortunately as long as we support cross-building like this, it will
>> be an issue. Intellij's maven support does not correctly detect our
>> use of the maven-build-plugin to add source directories.
>>
>> We should come up with a good set of instructions on how to import the
>> pom files + add the few extra source directories. Off hand I am not
>> sure exactly what the correct sequence is.
>>
>> - Patrick
>>
>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>> Hi Matei,
>>> Until my latest pull from upstream/master it had not been necessary to
>>> add the hive profile: is it now??
>>>
>>> I am not using sbt gen-idea. The way to open in intellij has been to Open
>>> the parent directory. IJ recognizes it as a maven project.
>>>
>>> There are several steps to do surgery on the yarn-parent / yarn projects
>> ,
>>> then do a full rebuild. That was working until one week ago.
>>> Intellij/maven is presently broken in two ways: this hive shim (which
>> may
>>> yet hopefully be a small/simple fix - let us see) and (2) the
>>> "NoClassDefFoundError
>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a
>> serious
>>> problem .
>>>
>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>>>
>>>> Hi Stephen,
>>>>
>>>> How did you generate your Maven workspace? You need to make sure the
>> Hive
>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>>>
>>>> Matei
>>>>
>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
>> wrote:
>>>>>
>>>>> I have run on the command line via maven and it is fine:
>>>>>
>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
>> -Phadoop-2.3
>>>>> compile package install
>>>>>
>>>>>
>>>>> But with the latest code Intellij builds do not work. Following is
>> one of
>>>>> 26 similar errors:
>>>>>
>>>>>
>>>>> Error:(173, 38) not found: value HiveShim
>>>>>
>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>>>>> ^
>>>>
>>>>
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Stephen Boesch <ja...@gmail.com>.
Thanks Patrick for the heads up.
I have not been successful to discover a combination of profiles (i.e.
enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with
maven. Anyone who knows how to handle this - a quick note here would be
appreciated.
2014-10-28 20:20 GMT-07:00 Patrick Wendell <pw...@gmail.com>:
> Hey Stephen,
>
> In some cases in the maven build we now have pluggable source
> directories based on profiles using the maven build helper plug-in.
> This is necessary to support cross building against different Hive
> versions, and there will be additional instances of this due to
> supporting scala 2.11 and 2.10.
>
> In these cases, you may need to add source locations explicitly to
> intellij if you want the entire project to compile there.
>
> Unfortunately as long as we support cross-building like this, it will
> be an issue. Intellij's maven support does not correctly detect our
> use of the maven-build-plugin to add source directories.
>
> We should come up with a good set of instructions on how to import the
> pom files + add the few extra source directories. Off hand I am not
> sure exactly what the correct sequence is.
>
> - Patrick
>
> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
> > Hi Matei,
> > Until my latest pull from upstream/master it had not been necessary to
> > add the hive profile: is it now??
> >
> > I am not using sbt gen-idea. The way to open in intellij has been to Open
> > the parent directory. IJ recognizes it as a maven project.
> >
> > There are several steps to do surgery on the yarn-parent / yarn projects
> ,
> > then do a full rebuild. That was working until one week ago.
> > Intellij/maven is presently broken in two ways: this hive shim (which
> may
> > yet hopefully be a small/simple fix - let us see) and (2) the
> > "NoClassDefFoundError
> > on ThreadFactoryBuilder" from my prior emails -and which is quite a
> serious
> > problem .
> >
> > 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
> >
> >> Hi Stephen,
> >>
> >> How did you generate your Maven workspace? You need to make sure the
> Hive
> >> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
> >>
> >> Matei
> >>
> >> > On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >> >
> >> > I have run on the command line via maven and it is fine:
> >> >
> >> > mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn
> -Phadoop-2.3
> >> > compile package install
> >> >
> >> >
> >> > But with the latest code Intellij builds do not work. Following is
> one of
> >> > 26 similar errors:
> >> >
> >> >
> >> > Error:(173, 38) not found: value HiveShim
> >> >
> >> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
> >> > ^
> >>
> >>
>
Re: HiveShim not found when building in Intellij
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Stephen,
In some cases in the maven build we now have pluggable source
directories based on profiles using the maven build helper plug-in.
This is necessary to support cross building against different Hive
versions, and there will be additional instances of this due to
supporting scala 2.11 and 2.10.
In these cases, you may need to add source locations explicitly to
intellij if you want the entire project to compile there.
Unfortunately as long as we support cross-building like this, it will
be an issue. Intellij's maven support does not correctly detect our
use of the maven-build-plugin to add source directories.
We should come up with a good set of instructions on how to import the
pom files + add the few extra source directories. Off hand I am not
sure exactly what the correct sequence is.
- Patrick
On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Matei,
> Until my latest pull from upstream/master it had not been necessary to
> add the hive profile: is it now??
>
> I am not using sbt gen-idea. The way to open in intellij has been to Open
> the parent directory. IJ recognizes it as a maven project.
>
> There are several steps to do surgery on the yarn-parent / yarn projects ,
> then do a full rebuild. That was working until one week ago.
> Intellij/maven is presently broken in two ways: this hive shim (which may
> yet hopefully be a small/simple fix - let us see) and (2) the
> "NoClassDefFoundError
> on ThreadFactoryBuilder" from my prior emails -and which is quite a serious
> problem .
>
> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
>
>> Hi Stephen,
>>
>> How did you generate your Maven workspace? You need to make sure the Hive
>> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>>
>> Matei
>>
>> > On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com> wrote:
>> >
>> > I have run on the command line via maven and it is fine:
>> >
>> > mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn -Phadoop-2.3
>> > compile package install
>> >
>> >
>> > But with the latest code Intellij builds do not work. Following is one of
>> > 26 similar errors:
>> >
>> >
>> > Error:(173, 38) not found: value HiveShim
>> >
>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
>> > ^
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: HiveShim not found when building in Intellij
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Matei,
Until my latest pull from upstream/master it had not been necessary to
add the hive profile: is it now??
I am not using sbt gen-idea. The way to open in intellij has been to Open
the parent directory. IJ recognizes it as a maven project.
There are several steps to do surgery on the yarn-parent / yarn projects ,
then do a full rebuild. That was working until one week ago.
Intellij/maven is presently broken in two ways: this hive shim (which may
yet hopefully be a small/simple fix - let us see) and (2) the
"NoClassDefFoundError
on ThreadFactoryBuilder" from my prior emails -and which is quite a serious
problem .
2014-10-28 19:46 GMT-07:00 Matei Zaharia <ma...@gmail.com>:
> Hi Stephen,
>
> How did you generate your Maven workspace? You need to make sure the Hive
> profile is enabled for it. For example sbt/sbt -Phive gen-idea.
>
> Matei
>
> > On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com> wrote:
> >
> > I have run on the command line via maven and it is fine:
> >
> > mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn -Phadoop-2.3
> > compile package install
> >
> >
> > But with the latest code Intellij builds do not work. Following is one of
> > 26 similar errors:
> >
> >
> > Error:(173, 38) not found: value HiveShim
> >
> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
> > ^
>
>
Re: HiveShim not found when building in Intellij
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Stephen,
How did you generate your Maven workspace? You need to make sure the Hive profile is enabled for it. For example sbt/sbt -Phive gen-idea.
Matei
> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> I have run on the command line via maven and it is fine:
>
> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn -Phadoop-2.3
> compile package install
>
>
> But with the latest code Intellij builds do not work. Following is one of
> 26 similar errors:
>
>
> Error:(173, 38) not found: value HiveShim
> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
> ^
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org