You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2024/05/16 03:25:41 UTC
[PR] [WIP][SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
jiwen624 opened a new pull request, #46604:
URL: https://github.com/apache/spark/pull/46604
### What changes were proposed in this pull request?
Working on it...
### Why are the changes needed?
As mentioned in the Jira ticket: https://issues.apache.org/jira/browse/SPARK-48298
Currently, the StatsdSink in Spark supports UDP mode only, which is the default mode of StatsD. However, in real production environments, we often find that a more reliable transmission of metrics is needed to avoid metrics lose in high-traffic systems.
TCP mode is already supported by Statsd: https://github.com/statsd/statsd/blob/master/docs/server.md
Prometheus' statsd_exporter: https://github.com/prometheus/statsd_exporter
and also many other Statsd-based metrics proxies/receivers.
### Does this PR introduce _any_ user-facing change?
Yes.
The following new config options are added to `conf/metrics.properties.template`:
`*.sink.statsd.protocol`
`*.sink.statsd.connTimeoutMs`
A new error condition is defined in error-conditions.json for protocol configuration error.
### How was this patch tested?
Unit tests.
Manually tests with metric configurations sending metrics to a Netcat TCP/UDP server
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #46604:
URL: https://github.com/apache/spark/pull/46604#discussion_r1612041718
##########
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala:
##########
@@ -161,3 +162,77 @@ private[spark] class StatsdReporter(
}
}
+private trait DataSender {
+ def start(): DataSender
+ def localHostPort: (String, Int)
+ def send(data: String): Unit
+ def close(): Unit
+}
+
+private class UDPDataSender(
+ val host: String,
+ val port: Int) extends DataSender {
+ private val socket = new DatagramSocket
+ private val address = new InetSocketAddress(host, port)
+
+ override def start(): DataSender = {
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = data.getBytes(UTF_8)
+ val packet = new DatagramPacket(bytes, bytes.length, address)
+ socket.send(packet)
+ }
+
+ override def close(): Unit = {
+ socket.close()
+ }
+}
+
+private class TCPDataSender(
+ val host: String,
+ val port: Int,
+ val connTimeoutMs: Int) extends DataSender {
+ private val socket = new Socket
+ private var dataOut: Option[DataOutputStream] = None
+
+ override def start(): DataSender = {
+ socket.connect(new InetSocketAddress(host, port), connTimeoutMs)
+ dataOut = Some(new DataOutputStream(socket.getOutputStream))
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = (data + "\n").getBytes(UTF_8)
+ dataOut.foreach(_.write(bytes))
+ }
+
+ override def close(): Unit = {
+ dataOut.foreach(_.close())
+ }
+}
+
+private[spark] object DataSenderType extends Enumeration {
+ type DataSenderType = Value
+ val UDP, TCP = Value
+}
+
+private object DataSender {
+ def get(host: String, port: Int, protocol: DataSenderType, connTimeoutMs: Int): DataSender = {
+ val ds = protocol match {
+ case TCP => new TCPDataSender(host, port, connTimeoutMs)
Review Comment:
I'm not using TCP long connections here. The consideration is:
- For metrics we do not heavily use the connection, it's usually a single-shot of a few metrics in a period of a few seconds, 10s by default. It's very costly even overwhelming for the Statsd server side to keep a large number of long-lived connections, given that in the production environments we usually have a ton of Spark applications/drivers/executors.
- The drawback to consider is for a short TCP connection, the port cannot be reused for a short while due to TCP_TIMEWAIT, which is a tradeoff and it somewhat depends on what the `interval` is.
- Also there is a cost of initializing and destroying a TCP connection.
- Simplicity. To maintain a long connection, we may have to handle a lot more failure scenarios, e.g., detect & reconnect due to a server-side glitch or transient network issues, and possibly a retry mechanism, and perhaps a retry with backoff and jitter, etc. Not sure if this complexity is expected though given that the current UDP mode is just fire-and-forget.
- Other considerations:
- Short-lived TCP connection is better for load-balancing & automatic failover, etc when, let's say, the Statsd service is hosted by multiple hosts behind a load balancer (e.g., a DNS, pods behind a Service in Kubernetes). This can be another long story itself to talk about.
So I don't see an ideal solution for all scenarios (this is probably why we are offering both UDP and TCP modes), we might provide both options (short-lived and long-lived) for TCP mode in case it's worth it.
Please feel free to discuss if you have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46604:
URL: https://github.com/apache/spark/pull/46604#issuecomment-2323086424
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #46604:
URL: https://github.com/apache/spark/pull/46604#issuecomment-2114019176
Hi @cloud-fan @dongjoon-hyun @yaooqinn Could you take a look when you get a chance or perhaps pull other people in for review? Thank you🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #46604: [SPARK-48298][Core] Add TCP mode to StatsD sink
URL: https://github.com/apache/spark/pull/46604
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #46604:
URL: https://github.com/apache/spark/pull/46604#discussion_r1612041718
##########
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala:
##########
@@ -161,3 +162,77 @@ private[spark] class StatsdReporter(
}
}
+private trait DataSender {
+ def start(): DataSender
+ def localHostPort: (String, Int)
+ def send(data: String): Unit
+ def close(): Unit
+}
+
+private class UDPDataSender(
+ val host: String,
+ val port: Int) extends DataSender {
+ private val socket = new DatagramSocket
+ private val address = new InetSocketAddress(host, port)
+
+ override def start(): DataSender = {
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = data.getBytes(UTF_8)
+ val packet = new DatagramPacket(bytes, bytes.length, address)
+ socket.send(packet)
+ }
+
+ override def close(): Unit = {
+ socket.close()
+ }
+}
+
+private class TCPDataSender(
+ val host: String,
+ val port: Int,
+ val connTimeoutMs: Int) extends DataSender {
+ private val socket = new Socket
+ private var dataOut: Option[DataOutputStream] = None
+
+ override def start(): DataSender = {
+ socket.connect(new InetSocketAddress(host, port), connTimeoutMs)
+ dataOut = Some(new DataOutputStream(socket.getOutputStream))
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = (data + "\n").getBytes(UTF_8)
+ dataOut.foreach(_.write(bytes))
+ }
+
+ override def close(): Unit = {
+ dataOut.foreach(_.close())
+ }
+}
+
+private[spark] object DataSenderType extends Enumeration {
+ type DataSenderType = Value
+ val UDP, TCP = Value
+}
+
+private object DataSender {
+ def get(host: String, port: Int, protocol: DataSenderType, connTimeoutMs: Int): DataSender = {
+ val ds = protocol match {
+ case TCP => new TCPDataSender(host, port, connTimeoutMs)
Review Comment:
I'm not using TCP long connections here. The consideration is:
- For metrics we do not heavily use the connection, it's usually a single-shot of a few metrics in a period of a few seconds, 10s by default. It's very costly even overwhelming for the Statsd server side to keep a large number of long-lived connections, given that in the production environments we usually have a ton of Spark applications/drivers/executors.
- The drawback to consider is for a short TCP connection, the port cannot be reused for a short while due to TCP_TIMEWAIT, which is a tradeoff and it somewhat depends on what the `interval` is.
- Also there is a cost of initializing and destroying a TCP connection.
- Simplicity. To maintain a long connection, we may have to handle a lot more failure scenarios, e.g., detect & reconnect due to a server-side glitch or transient network issues, and possibly a retry mechanism, and perhaps a retry with backoff and jitter, etc. Not sure if this complexity is expected though given that the current UDP mode is just fire-and-forget.
- Other considerations:
- Short-lived TCP connection is better for load-balancing & automatic failover, etc when, let's say, the Statsd service is hosted by multiple hosts behind a load balancer. This can be another long story itself to talk about but may be too far away from this topic.
So I don't see an ideal solution for all scenarios (this is probably why we are offering both UDP and TCP modes), we might provide both options (short-lived and long-lived) for TCP mode in case it's worth it.
Please feel free to discuss if you have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #46604:
URL: https://github.com/apache/spark/pull/46604#discussion_r1612041718
##########
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala:
##########
@@ -161,3 +162,77 @@ private[spark] class StatsdReporter(
}
}
+private trait DataSender {
+ def start(): DataSender
+ def localHostPort: (String, Int)
+ def send(data: String): Unit
+ def close(): Unit
+}
+
+private class UDPDataSender(
+ val host: String,
+ val port: Int) extends DataSender {
+ private val socket = new DatagramSocket
+ private val address = new InetSocketAddress(host, port)
+
+ override def start(): DataSender = {
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = data.getBytes(UTF_8)
+ val packet = new DatagramPacket(bytes, bytes.length, address)
+ socket.send(packet)
+ }
+
+ override def close(): Unit = {
+ socket.close()
+ }
+}
+
+private class TCPDataSender(
+ val host: String,
+ val port: Int,
+ val connTimeoutMs: Int) extends DataSender {
+ private val socket = new Socket
+ private var dataOut: Option[DataOutputStream] = None
+
+ override def start(): DataSender = {
+ socket.connect(new InetSocketAddress(host, port), connTimeoutMs)
+ dataOut = Some(new DataOutputStream(socket.getOutputStream))
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = (data + "\n").getBytes(UTF_8)
+ dataOut.foreach(_.write(bytes))
+ }
+
+ override def close(): Unit = {
+ dataOut.foreach(_.close())
+ }
+}
+
+private[spark] object DataSenderType extends Enumeration {
+ type DataSenderType = Value
+ val UDP, TCP = Value
+}
+
+private object DataSender {
+ def get(host: String, port: Int, protocol: DataSenderType, connTimeoutMs: Int): DataSender = {
+ val ds = protocol match {
+ case TCP => new TCPDataSender(host, port, connTimeoutMs)
Review Comment:
I'm not using TCP long connections here. The consideration is:
- For metrics we do not heavily use the connection, it's usually a single-shot of a few metrics in a period of a few seconds, 10s by default. It's very costly even overwhelming for the Statsd server side to keep a large number of long-lived connections, given that in the production environments we usually have a ton of Spark applications/drivers/executors.
- The drawback to consider is for a short TCP connection, the port cannot be reused for a short while due to TCP_TIMEWAIT, which is a tradeoff and it somewhat depends on what the `interval` is.
- Also there is a cost of initializing and destroying a TCP connection.
- Simplicity. To maintain a long connection, we may have to handle a lot more failure scenarios, e.g., detect & reconnect due to a server-side glitch or transient network issues, and possibly a retry mechanism, and perhaps a retry with backoff and jitter, etc. Not sure if this complexity is expected though given that the current UDP mode is just fire-and-forget.
So I don't see an ideal solution for all scenarios, please feel free to discuss if you have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #46604:
URL: https://github.com/apache/spark/pull/46604#discussion_r1612041718
##########
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala:
##########
@@ -161,3 +162,77 @@ private[spark] class StatsdReporter(
}
}
+private trait DataSender {
+ def start(): DataSender
+ def localHostPort: (String, Int)
+ def send(data: String): Unit
+ def close(): Unit
+}
+
+private class UDPDataSender(
+ val host: String,
+ val port: Int) extends DataSender {
+ private val socket = new DatagramSocket
+ private val address = new InetSocketAddress(host, port)
+
+ override def start(): DataSender = {
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = data.getBytes(UTF_8)
+ val packet = new DatagramPacket(bytes, bytes.length, address)
+ socket.send(packet)
+ }
+
+ override def close(): Unit = {
+ socket.close()
+ }
+}
+
+private class TCPDataSender(
+ val host: String,
+ val port: Int,
+ val connTimeoutMs: Int) extends DataSender {
+ private val socket = new Socket
+ private var dataOut: Option[DataOutputStream] = None
+
+ override def start(): DataSender = {
+ socket.connect(new InetSocketAddress(host, port), connTimeoutMs)
+ dataOut = Some(new DataOutputStream(socket.getOutputStream))
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = (data + "\n").getBytes(UTF_8)
+ dataOut.foreach(_.write(bytes))
+ }
+
+ override def close(): Unit = {
+ dataOut.foreach(_.close())
+ }
+}
+
+private[spark] object DataSenderType extends Enumeration {
+ type DataSenderType = Value
+ val UDP, TCP = Value
+}
+
+private object DataSender {
+ def get(host: String, port: Int, protocol: DataSenderType, connTimeoutMs: Int): DataSender = {
+ val ds = protocol match {
+ case TCP => new TCPDataSender(host, port, connTimeoutMs)
Review Comment:
I'm not using TCP long connections here. The consideration is:
- For metrics we do not heavily use the connection, it's usually a single-shot of a few metrics in a period of a few seconds, 10s by default. It's very costly even overwhelming for the Statsd server side to keep a large number of long-lived connections, given that in the production environments we usually have a ton of Spark applications/drivers/executors.
- The drawback to consider is for a short TCP connection, the port cannot be reused for a short while due to TCP_TIMEWAIT, which is a tradeoff and it somewhat depends on what the `interval` is.
- Simplicity. To maintain a long connection, we may have to handle a lot more failure scenarios, e.g., detect & reconnect due to a server-side glitch or transient network issues, and possibly a retry mechanism, and perhaps a retry with backoff and jitter, etc. Not sure if this complexity is expected though given that the current UDP mode is just fire-and-forget.
So I don't see an ideal solution for all scenarios, please feel free to discuss if you have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48298][Core] Add TCP mode to StatsD sink [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on code in PR #46604:
URL: https://github.com/apache/spark/pull/46604#discussion_r1612041718
##########
core/src/main/scala/org/apache/spark/metrics/sink/StatsdReporter.scala:
##########
@@ -161,3 +162,77 @@ private[spark] class StatsdReporter(
}
}
+private trait DataSender {
+ def start(): DataSender
+ def localHostPort: (String, Int)
+ def send(data: String): Unit
+ def close(): Unit
+}
+
+private class UDPDataSender(
+ val host: String,
+ val port: Int) extends DataSender {
+ private val socket = new DatagramSocket
+ private val address = new InetSocketAddress(host, port)
+
+ override def start(): DataSender = {
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = data.getBytes(UTF_8)
+ val packet = new DatagramPacket(bytes, bytes.length, address)
+ socket.send(packet)
+ }
+
+ override def close(): Unit = {
+ socket.close()
+ }
+}
+
+private class TCPDataSender(
+ val host: String,
+ val port: Int,
+ val connTimeoutMs: Int) extends DataSender {
+ private val socket = new Socket
+ private var dataOut: Option[DataOutputStream] = None
+
+ override def start(): DataSender = {
+ socket.connect(new InetSocketAddress(host, port), connTimeoutMs)
+ dataOut = Some(new DataOutputStream(socket.getOutputStream))
+ this
+ }
+
+ override def localHostPort: (String, Int) = {
+ (Try(socket.getLocalAddress).map(_.getHostAddress).getOrElse(null), socket.getLocalPort)
+ }
+
+ override def send(data: String): Unit = {
+ val bytes = (data + "\n").getBytes(UTF_8)
+ dataOut.foreach(_.write(bytes))
+ }
+
+ override def close(): Unit = {
+ dataOut.foreach(_.close())
+ }
+}
+
+private[spark] object DataSenderType extends Enumeration {
+ type DataSenderType = Value
+ val UDP, TCP = Value
+}
+
+private object DataSender {
+ def get(host: String, port: Int, protocol: DataSenderType, connTimeoutMs: Int): DataSender = {
+ val ds = protocol match {
+ case TCP => new TCPDataSender(host, port, connTimeoutMs)
Review Comment:
I'm not using TCP long connections here. The consideration is:
- For metrics we do not heavily use the connection, it's usually a single-shot of a few metrics in a period of a few seconds, 10s by default. It's very costly even overwhelming for the Statsd server side to keep a large number of long-lived connections, given that in the production environments we usually have a ton of Spark applications/drivers/executors.
- The drawback to consider is for a short TCP connection, the port cannot be reused for a short while due to TCP_TIMEWAIT, which is a tradeoff and it somewhat depends on what the `interval` is.
- Also there is a cost of initializing and destroying a TCP connection.
- Simplicity. To maintain a long connection, we may have to handle a lot more failure scenarios, e.g., detect & reconnect due to a server-side glitch or transient network issues, and possibly a retry mechanism, and perhaps a retry with backoff and jitter, etc. Not sure if this complexity is expected though given that the current UDP mode is just fire-and-forget.
- Other considerations:
- Short-lived TCP connection is better for load-balancing & automatic failover, etc when, let's say, the Statsd service is hosted by multiple hosts behind a load balancer. This can be another long story itself to talk about but may be too far away from this topic.
So I don't see an ideal solution for all scenarios, and this is probably why we are offering both UDP and TCP modes. Please feel free to discuss if you have different opinions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org