[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #114: [FLINK-27096] Optimize VectorAssembler performance

[GitBox <gi...@apache.org>]

yunfengzhou-hub commented on code in PR #114:
URL: https://github.com/apache/flink-ml/pull/114#discussion_r904423389


##########
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/vectorassembler/VectorAssembler.java:
##########
@@ -80,24 +82,68 @@ public Table[] transform(Table... inputs) {
         return new Table[] {outputTable};
     }
 
-    private static class AssemblerFunc implements FlatMapFunction<Row, Row> {
+    private static class AssemblerFunc extends RichFlatMapFunction<Row, Row> {
         private final String[] inputCols;
         private final String handleInvalid;
 
+        /** The indices for assembling vectors. */
+        private transient IntArrayList indices;

Review Comment:
   Why should we use `IntArrayList`, instead of `List<Integer>` and `Integer[]`? Would it bring performance improvement compared with these options?
   
   How about implementing it in two loops? In the first loop, we only extract the vector/numbers, and calculate their total size. Then we allocate the integer and double arrays according to the calculated size, and assign the values in the second loop.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org