You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by AssafMendelson <as...@rsa.com> on 2017/03/13 12:59:36 UTC

Adding metrics to spark datasource

Hi,
I am building a data source so I can convert a custom source to dataframe.
I have been going over examples such as JDBC and noticed that JDBC does the following:

val inputMetrics = context.taskMetrics().inputMetrics

and whenever a new record is added:


inputMetrics.incRecordsRead(1)

however, incRecordsRead is spark private.
I was wondering: how can I get standard metrics through a data source?

Thanks,
                Assaf.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Adding-metrics-to-spark-datasource-tp28487.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.