You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Judy Nash <ju...@exchange.microsoft.com> on 2015/02/10 07:02:37 UTC
New Metrics Sink class not packaged in spark-assembly jar
Hello,
Working on SPARK-5708<https://issues.apache.org/jira/browse/SPARK-5708> - Add Slf4jSink to Spark Metrics Sink.
Wrote a new Slf4jSink class (see patch attached), but the new class is not packaged as part of spark-assembly jar.
Do I need to update build config somewhere to have this packaged?
Current packaged class:
[cid:image001.png@01D044B4.1B17A1C0]
Thought I must have missed something basic but can't figure out why.
Thanks!
Judy
RE: New Metrics Sink class not packaged in spark-assembly jar
Posted by Judy Nash <ju...@exchange.microsoft.com>.
Thanks Patrick! That was the issue.
Built the jars on windows env with mvn and forgot to run make-distributions.ps1 afterward, so was looking at old jars.
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Monday, February 9, 2015 10:43 PM
To: Judy Nash
Cc: dev@spark.apache.org
Subject: Re: New Metrics Sink class not packaged in spark-assembly jar
Actually, to correct myself, the assembly jar is in assembly/target/scala-2.11 (I think).
On Mon, Feb 9, 2015 at 10:42 PM, Patrick Wendell <pw...@gmail.com>> wrote:
Hi Judy,
If you have added source files in the sink/ source folder, they should appear in the assembly jar when you build. One thing I noticed is that you are looking inside the "/dist" folder. That only gets populated if you run "make-distribution". The normal development process is just to do "mvn package" and then look at the assembly jar that is contained in core/target.
- Patrick
On Mon, Feb 9, 2015 at 10:02 PM, Judy Nash <ju...@exchange.microsoft.com>> wrote:
Hello,
Working on SPARK-5708<https://issues.apache.org/jira/browse/SPARK-5708> - Add Slf4jSink to Spark Metrics Sink.
Wrote a new Slf4jSink class (see patch attached), but the new class is not packaged as part of spark-assembly jar.
Do I need to update build config somewhere to have this packaged?
Current packaged class:
[cid:image001.png@01D044BC.8FE515C0]
Thought I must have missed something basic but can't figure out why.
Thanks!
Judy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: dev-help@spark.apache.org<ma...@spark.apache.org>
Re: New Metrics Sink class not packaged in spark-assembly jar
Posted by Patrick Wendell <pw...@gmail.com>.
Actually, to correct myself, the assembly jar is in
assembly/target/scala-2.11 (I think).
On Mon, Feb 9, 2015 at 10:42 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Hi Judy,
>
> If you have added source files in the sink/ source folder, they should
> appear in the assembly jar when you build. One thing I noticed is that you
> are looking inside the "/dist" folder. That only gets populated if you run
> "make-distribution". The normal development process is just to do "mvn
> package" and then look at the assembly jar that is contained in core/target.
>
> - Patrick
>
> On Mon, Feb 9, 2015 at 10:02 PM, Judy Nash <
> judynash@exchange.microsoft.com> wrote:
>
>> Hello,
>>
>>
>>
>> Working on SPARK-5708 <https://issues.apache.org/jira/browse/SPARK-5708>
>> - Add Slf4jSink to Spark Metrics Sink.
>>
>>
>>
>> Wrote a new Slf4jSink class (see patch attached), but the new class is
>> not packaged as part of spark-assembly jar.
>>
>>
>>
>> Do I need to update build config somewhere to have this packaged?
>>
>>
>>
>> Current packaged class:
>>
>>
>>
>> Thought I must have missed something basic but can't figure out why.
>>
>>
>>
>> Thanks!
>>
>> Judy
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
>
Re: New Metrics Sink class not packaged in spark-assembly jar
Posted by Patrick Wendell <pw...@gmail.com>.
Hi Judy,
If you have added source files in the sink/ source folder, they should
appear in the assembly jar when you build. One thing I noticed is that you
are looking inside the "/dist" folder. That only gets populated if you run
"make-distribution". The normal development process is just to do "mvn
package" and then look at the assembly jar that is contained in core/target.
- Patrick
On Mon, Feb 9, 2015 at 10:02 PM, Judy Nash <ju...@exchange.microsoft.com>
wrote:
> Hello,
>
>
>
> Working on SPARK-5708 <https://issues.apache.org/jira/browse/SPARK-5708>
> - Add Slf4jSink to Spark Metrics Sink.
>
>
>
> Wrote a new Slf4jSink class (see patch attached), but the new class is not
> packaged as part of spark-assembly jar.
>
>
>
> Do I need to update build config somewhere to have this packaged?
>
>
>
> Current packaged class:
>
>
>
> Thought I must have missed something basic but can't figure out why.
>
>
>
> Thanks!
>
> Judy
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>