You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/23 08:42:33 UTC
[GitHub] [hudi] YuweiXiao opened a new issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
YuweiXiao opened a new issue #5107:
URL: https://github.com/apache/hudi/issues/5107
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
When doing hudi datasource writing benchmark, we observed a large amount CPU time is spent in converting dataframe into RDD (`HoodieSparkUtils::createRdd`). By looking at the profiling flame graph, we found around 80% of the reading time (source -> dataframe -> RDD) is spent in constructing internal variables of `AvroSerializer`.
```
df.mapPartitions ( rows => {
val convert = new AvroSerializer()
rows.map( r => convert(r))
})
```
The above code is the pseudo code version of the current `createRdd` implementation. At first glance, we thought the variable `convert` is initialized once for each data partition, which should not cost too much. However, looking at its source code, it actually maintains a lambda function with some variables initialized inside. So for each input row, we have to do an almost full initialization of `AvroSerializer`.
Because `AvroSerializer` resides in spark-avro lib, it is not easy to directly optimize it in Hudi codebase. I am wondering if there is any workarounds to this, e.g., other way to convert df -> RDD, or re-implement a better version of `AvroSerializer` in Hudi codebase.
**To Reproduce**
NA
**Expected behavior**
`AvroSerializer` is initialized once for each data partition, or even once in driver to serialize to executors.
**Environment Description**
* Hudi version : master
* Spark version : spark3.2.0
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : Aliyun OSS
* Running on Docker? (yes/no) : no
**Additional context**

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1079650016
@YuweiXiao Thank you for discussing this issue, have you test on newest master branch.
I found the situation very even worse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qjqqyy commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
qjqqyy commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077384564
@YuweiXiao Hi, allow me to rephrase. This is what's actually going on in git master (removing the adapter abstraction)
```scala
df.mapPartitions { rows =>
rows.map { row =>
new AvroSerializer().serialize(row)
}
}
```
there could be extra initialization cost for each invocation of `serialize` but it seems to be *recreating* the AvroSerializer for each row, which I suspect is the main culprit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1076950122
@qjqqyy Hi, I was also referring to the master branch code (removing the adapter abstraction to simplify the context). Because the implementation of `AvroSerializer ` itself is a lambda (named converter in the source code), there is still initialization cost (for internal variables in the lambda) for each row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1079734532
Thanks for flagging this @YuweiXiao, great catch!
To summarize what the issue is here: it is unfortunately a very sneaky one and it occurred accidentally during the refactoring of AvroSerializer/Deserializer hierarchy in Hudi.
Crux of the issue is that converter initializes AvroSerializer/Deserializer upon _every_ invocation of it, b/c it's done w/in the returned lambda itself (it also has a side-effect of pulling whole `SparkAdapter` into the closure):
```
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] =
record => sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
.deserialize(record)
.map(_.asInstanceOf[InternalRow])
```
Instead it should have been
```
def createAvroToInternalRowConverter(rootAvroType: Schema, rootCatalystType: StructType): GenericRecord => Option[InternalRow] = {
val deserilizer = sparkAdapter.createAvroDeserializer(rootAvroType, rootCatalystType)
record =>
deserializer.deserialize(record).map(_.asInstanceOf[InternalRow]) }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs edited a comment on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
boneanxs edited a comment on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077280474
Yes, I also notice `AvroSerializer` is a very time consuming method(clustering's poor performance also because of this, it needs to serialize to avro twice), I'm afraid it is not easy to optimize this in hudi side. Maybe one solution is hudi fully support using Dataset not RDD, which can avoid using avroSerializer to parse data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1079713821
@YuweiXiao @qjqqyy @boneanxs @danny0405 thank you folks for raising and the discussion. @xiarixiaoyao has kindly helped verified the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qjqqyy edited a comment on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
qjqqyy edited a comment on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1076683168
On current git master, the pseudocode for `HoodieSparkUtils::createRdd` actually looks like this
```scala
df.mapPartitions { rows =>
val convert: InternalRow => GenericRecord = { row =>
sparkAdapter.createAvroSerializer(???).serialize(row)
}
rows.map(convert)
}
```
which seems to be why an `AvroSerializer` is created for each row.
I think this patch will cause `AvroSerializer` to only be initialized once for each partition, can you help try it?
```patch
diff --git a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
index 69005cd75..9c63295d6 100644
--- a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
+++ b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
@@ -76,7 +76,8 @@ object AvroConversionUtils {
* @return converter accepting Catalyst payload (in the form of [[InternalRow]]) and transforming it into an Avro one
*/
def createInternalRowToAvroConverter(rootCatalystType: StructType, rootAvroType: Schema, nullable: Boolean): InternalRow => GenericRecord = {
- row => sparkAdapter.createAvroSerializer(rootCatalystType, rootAvroType, nullable)
+ val serializer = sparkAdapter.createAvroSerializer(rootCatalystType, rootAvroType, nullable)
+ row => serializer
.serialize(row)
.asInstanceOf[GenericRecord]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
boneanxs commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077280474
Yes, I also notice `AvroSerializer` is a very time consuming method(clustering's poor performance also because of this, it needs to serialize to avro twice), I don't think it is easy to optimize this in hudi side. Maybe after hudi fully support using Dataset not RDD can solve this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077466306
> @boneanxs True, full support of Dataset is the long term solution. In my experiment, optimizing the usage of `AvroSerializer` could save 80% costs of the source data reading. But the optimization requires modification of the `AvroSerializer` source code in the spark side.
>
> @qjqqyy Yes, each row will initialize `AvroSerializer` (variables in the lambda named `converter`)
Can we copy a `AvroSerializer` currently on Hudi side and just use that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #5107:
URL: https://github.com/apache/hudi/issues/5107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qjqqyy commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
qjqqyy commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1078622236
@YuweiXiao both jobs were upserting into empty directories, the dataframes were also cached.
The timing shown is only for the shuffle write stage while Tagging, which includes the `InternalRow => GenericRecord` conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1078579124
@qjqqyy Tagging also includes the indexing costs. Could you check again and make sure you clean up the table between tests (or use INSERT rather than UPSERT)? In my previous experiments, I did the same change but the cost still remained. Because even we initialize `AvroSerializer ` once, the internal lambda function will be call for each row, and the local variables used in the lambda will be initialized for each row.
@danny0405 Let me try this. I did meet some NonSerializable Exception in my previous experiments, so I guess it may not be easy to directly copy it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qjqqyy commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
qjqqyy commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1076683168
On current git master, the pseudocode for `HoodieSparkUtils::createRdd` actually looks like this
```scala
df.mapPartitions { rows =>
val convert: InternalRow => GenericRecord = { row =>
sparkAdapter.createAvroSerializer(???).serialize(row)
}
convert(rows)
}
```
which seems to be why an `AvroSerializer` is created for each row.
I think this patch will cause `AvroSerializer` to only be initialized once for each partition, can you help try it?
```patch
diff --git a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
index 69005cd75..9c63295d6 100644
--- a/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
+++ b/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala
@@ -76,7 +76,8 @@ object AvroConversionUtils {
* @return converter accepting Catalyst payload (in the form of [[InternalRow]]) and transforming it into an Avro one
*/
def createInternalRowToAvroConverter(rootCatalystType: StructType, rootAvroType: Schema, nullable: Boolean): InternalRow => GenericRecord = {
- row => sparkAdapter.createAvroSerializer(rootCatalystType, rootAvroType, nullable)
+ val serializer = sparkAdapter.createAvroSerializer(rootCatalystType, rootAvroType, nullable)
+ row => serializer
.serialize(row)
.asInstanceOf[GenericRecord]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] qjqqyy commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
qjqqyy commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077508202
> @qjqqyy Yes, each row will initialize `AvroSerializer` (variables in the lambda named `converter`)
the reason why `converter` is re-initialized for each row is because a `new AvroSerializer()` (explicit object creation) is happening for each row.
I did some tests, exact same dataset.
#### git master
<img width="1224" alt="Screenshot 2022-03-24 at 7 02 57 PM" src="https://user-images.githubusercontent.com/8439769/159903075-067f3271-1ac0-4919-be25-8429d515fbae.png">
#### with the patch from earlier comment
<img width="1218" alt="Screenshot 2022-03-24 at 7 03 28 PM" src="https://user-images.githubusercontent.com/8439769/159903248-06f45a7a-301e-40b3-a2ca-36cdf988766f.png">
Seems like it's a regression introduced in #4789
* before: 1 invocation of `new AvroSerializer()` for each partition
* after: 1 invocation of `new AvroSerializer()` for each row
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on issue #5107: [SUPPORT] High performance costs of AvroSerializer in Datasource writing
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #5107:
URL: https://github.com/apache/hudi/issues/5107#issuecomment-1077404381
@boneanxs
True, full support of Dataset is the long term solution. In my experiment, optimizing the usage of `AvroSerializer` could save 80% costs of the source data reading. But the optimization requires modification of the `AvroSerializer` source code in the spark side.
@qjqqyy Yes, each row will initialize `AvroSerializer` (variables in the lambda named `converter`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org