You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jason White <ja...@shopify.com> on 2017/11/27 18:45:32 UTC
OutputMetrics empty for DF writes - any hints?
I'd like to use the SparkListenerInterface to listen for some metrics for
monitoring/logging/metadata purposes. The first ones I'm interested in
hooking into are recordsWritten and bytesWritten as a measure of throughput.
I'm using PySpark to write Parquet files from DataFrames.
I'm able to extract a rich set of metrics this way, but for some reason the
two that I want are always 0. This mirrors what I see in the Spark
Application Master - the # records written field is always missing.
I've filed a JIRA already for this issue:
https://issues.apache.org/jira/browse/SPARK-22605
I _think_ how this works is that inside the ResultTask.runTask method, the
rdd.iterator call is incrementing the bytes read & records read via
RDD.getOrCompute. Where would the equivalent be for the records written
metrics?
These metrics are populated properly if I save the data as an RDD via
df.rdd.saveAsTextFile, so the code path exists somewhere. Any hints as to
where I should be looking?
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: OutputMetrics empty for DF writes - any hints?
Posted by Jason White <ja...@shopify.com>.
It should be in the first email in this chain.
On Tue, Dec 12, 2017, 7:10 PM Ryan Blue <rb...@netflix.com> wrote:
> Great. What's the JIRA issue?
>
> On Mon, Dec 11, 2017 at 8:12 PM, Jason White <ja...@shopify.com>
> wrote:
>
>> Yes, the fix has been merged at should make it into the 2.3 release.
>>
>> On Mon, Dec 11, 2017, 5:50 PM Ryan Blue <rb...@netflix.com> wrote:
>>
>>> Is anyone currently working on this? I just fixed it in our Spark build
>>> and can contribute the fix if there isn't already a PR for it.
>>>
>>> On Mon, Nov 27, 2017 at 12:59 PM, Jason White <ja...@shopify.com>
>>> wrote:
>>>
>>>> It doesn't look like the insert command has any metrics in it. I don't
>>>> see
>>>> any commands with metrics, but I could be missing something.
>>>
>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: OutputMetrics empty for DF writes - any hints?
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Great. What's the JIRA issue?
On Mon, Dec 11, 2017 at 8:12 PM, Jason White <ja...@shopify.com>
wrote:
> Yes, the fix has been merged at should make it into the 2.3 release.
>
> On Mon, Dec 11, 2017, 5:50 PM Ryan Blue <rb...@netflix.com> wrote:
>
>> Is anyone currently working on this? I just fixed it in our Spark build
>> and can contribute the fix if there isn't already a PR for it.
>>
>> On Mon, Nov 27, 2017 at 12:59 PM, Jason White <ja...@shopify.com>
>> wrote:
>>
>>> It doesn't look like the insert command has any metrics in it. I don't
>>> see
>>> any commands with metrics, but I could be missing something.
>>
>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
--
Ryan Blue
Software Engineer
Netflix
Re: OutputMetrics empty for DF writes - any hints?
Posted by Jason White <ja...@shopify.com>.
Yes, the fix has been merged at should make it into the 2.3 release.
On Mon, Dec 11, 2017, 5:50 PM Ryan Blue <rb...@netflix.com> wrote:
> Is anyone currently working on this? I just fixed it in our Spark build
> and can contribute the fix if there isn't already a PR for it.
>
> On Mon, Nov 27, 2017 at 12:59 PM, Jason White <ja...@shopify.com>
> wrote:
>
>> It doesn't look like the insert command has any metrics in it. I don't see
>> any commands with metrics, but I could be missing something.
>
>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: OutputMetrics empty for DF writes - any hints?
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Is anyone currently working on this? I just fixed it in our Spark build and
can contribute the fix if there isn't already a PR for it.
On Mon, Nov 27, 2017 at 12:59 PM, Jason White <ja...@shopify.com>
wrote:
> It doesn't look like the insert command has any metrics in it. I don't see
> any commands with metrics, but I could be missing something.
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Ryan Blue
Software Engineer
Netflix
Re: OutputMetrics empty for DF writes - any hints?
Posted by Jason White <ja...@shopify.com>.
It doesn't look like the insert command has any metrics in it. I don't see
any commands with metrics, but I could be missing something.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: OutputMetrics empty for DF writes - any hints?
Posted by Reynold Xin <rx...@databricks.com>.
Is this due to the insert command not having metrics? It's a problem we
should fix.
On Mon, Nov 27, 2017 at 10:45 AM, Jason White <ja...@shopify.com>
wrote:
> I'd like to use the SparkListenerInterface to listen for some metrics for
> monitoring/logging/metadata purposes. The first ones I'm interested in
> hooking into are recordsWritten and bytesWritten as a measure of
> throughput.
> I'm using PySpark to write Parquet files from DataFrames.
>
> I'm able to extract a rich set of metrics this way, but for some reason the
> two that I want are always 0. This mirrors what I see in the Spark
> Application Master - the # records written field is always missing.
>
> I've filed a JIRA already for this issue:
> https://issues.apache.org/jira/browse/SPARK-22605
>
> I _think_ how this works is that inside the ResultTask.runTask method, the
> rdd.iterator call is incrementing the bytes read & records read via
> RDD.getOrCompute. Where would the equivalent be for the records written
> metrics?
>
> These metrics are populated properly if I save the data as an RDD via
> df.rdd.saveAsTextFile, so the code path exists somewhere. Any hints as to
> where I should be looking?
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: OutputMetrics empty for DF writes - any hints?
Posted by Jason White <ja...@shopify.com>.
I think the difference lies somewhere in here:
- RDD writes are done with SparkHadoopMapReduceWriter.executeTask, which
calls outputMetrics.setRecordsWritten
- DF writes are done with InsertIntoHadoopFsRelationCommand.run ? Which I'm
not entirely sure how it works.
executeTask appears to be run on the worker, writing a single RDD partition
out in a single Spark task. I can grok how it works. I'm not entirely sure
where the rubber hits the road for InsertIntoHadoopFsRelationCommand.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org