You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "Isso-W (via GitHub)" <gi...@apache.org> on 2025/06/19 19:46:57 UTC
[PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Isso-W opened a new pull request, #2275:
URL: https://github.com/apache/systemds/pull/2275
This pull request introduces a new framework for column decoding in Apache SystemDS, with the addition of a base class `ColumnDecoder` and several specialized implementations (`ColumnDecoderBin`, `ColumnDecoderComposite`, `ColumnDecoderRecode`,`ColumnDecoderPassthrough` and `ColumnDecoderDummycode`). These changes provide a flexible and extensible structure for decoding encoded data in matrix-to-frame transformations. Below are the most important changes grouped by theme:
### Core Framework for Column Decoding
* Added `ColumnDecoder` as an abstract base class to define the structure for decoding operations, including methods for decoding (`columnDecode`), handling sub-range decoding (`subRangeDecoder`), and metadata initialization (`initMetaData`). It also implements `Externalizable` for efficient serialization.
### Current Issues
* `ColumnDecoderDummycode` ist not supported yet, as well as the test case `ColumnDecoderMixedMethodsTest`
* There are still some issues with `DecoderDummyCode`, it do not work together with `DecoderRecode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "Isso-W (via GitHub)" <gi...@apache.org>.
Isso-W commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-3172826827
Sure @phaniarnab, here is the plot.
Full decode means the time cost for whole decode function in dml
Pure decoder means the time cost for decoder part in decoder function in Java code
And there is an additional comparison for these both.
<img width="596" height="356" alt="FTbench pure decoder" src="https://github.com/user-attachments/assets/b29ffe8b-7b9b-4f2a-8c89-73932d137a1f" />
<img width="634" height="359" alt="FTbench full decode" src="https://github.com/user-attachments/assets/9ba9e926-4749-47df-be0d-c69ca4cdccf2" />
<img width="590" height="347" alt="FTbench compare" src="https://github.com/user-attachments/assets/099fee8f-10b8-486d-b360-272f324ed728" />
I also created a new test using part of FTbench T9 by running DML, I can also put that test with responding json in PR to make the plot reproduciable. B.t.w. will this PR be merged into main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "phaniarnab (via GitHub)" <gi...@apache.org>.
phaniarnab commented on code in PR #2275:
URL: https://github.com/apache/systemds/pull/2275#discussion_r2182386638
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderDummycode.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.sysds.common.Types;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.frame.data.columns.ColumnMetadata;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+import org.apache.sysds.runtime.util.UtilFunctions;
+
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class ColumnDecoderDummycode extends ColumnDecoder {
+
+ private static final long serialVersionUID = 4758831042891032129L;
+
+ private int[] _clPos = null;
+ private int[] _cuPos = null;
+ // category index for dedicated single-column decoders (-1 if not used)
+ private int _category = -1;
+
+ protected ColumnDecoderDummycode(Types.ValueType[] schema, int[] colList) {
+ super(schema, colList);
+ }
+
+ @Override
+ public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+ out.ensureAllocatedColumns(in.getNumRows());
+ columnDecode(in, out, 0, in.getNumRows());
+ return out;
+ }
+
+ @Override
+ public void columnDecode(MatrixBlock in, FrameBlock out, int rl, int ru) {
+ if(_category >= 0) {
+ int col = _colList[0] - 1;
+ Object val = UtilFunctions.doubleToObject(out.getSchema()[col], _category);
+ for(int i = rl; i < ru; i++)
+ if(in.get(i, _clPos[0]-1) == 1)
+ synchronized(out) { out.set(i, col, val); }
+ }
+ else {
+ for( int i=rl; i<ru; i++ )
+ for( int j=0; j<_colList.length; j++ )
+ for( int k=_clPos[j]; k<_cuPos[j]; k++ )
+ if( in.get(i, k-1) != 0 ) {
+ int col = _colList[j] - 1;
+ Object val = UtilFunctions.doubleToObject(out.getSchema()[col], k-_clPos[j]+1);
+ synchronized(out) { out.set(i, col, val); }
+ }
+ }
Review Comment:
A column decoder should work on a single column that is provided.
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderRecode.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+import org.apache.sysds.runtime.matrix.data.Pair;
+import org.apache.sysds.runtime.transform.TfUtils;
+import org.apache.sysds.runtime.transform.encode.ColumnEncoderRecode;
+import org.apache.sysds.runtime.util.UtilFunctions;
+
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.*;
Review Comment:
Avoid all import. Only import the required classes.
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderBin.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.frame.data.columns.Array;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+import org.apache.sysds.runtime.util.UtilFunctions;
+
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+
+public class ColumnDecoderBin extends ColumnDecoder {
+ private static final long serialVersionUID = -3784249774608228805L;
+
+ private int[] _numBins;
+ private double[][] _binMins = null;
+ private double[][] _binMaxs = null;
+
+ public ColumnDecoderBin() {
+ super(null, null);
+ }
+
+ protected ColumnDecoderBin(ValueType[] schema, int[] binCols) {
+ super(schema, binCols);
+ }
+
+
+ //@Override
+ //public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+//
+ // long b1 = System.nanoTime();
+ // out.ensureAllocatedColumns(in.getNumRows());
+ // for (int i = 0; i < in.getNumRows(); i++) {
+ // for (int j = 0; j < _colList.length; j++) {
+ // double val = in.get(i, j);
+ // if (!Double.isNaN(val)) {
+ // int key = (int) Math.round(val);
+ // double bmin = _binMins[j][key - 1];
+ // double bmax = _binMaxs[j][key - 1];
+ // double oval = bmin + (bmax - bmin) / 2 + (val - key) * (bmax - bmin);
+ // out.getColumn(_colList[j] - 1).set(i, oval);
+ // } else {
+ // out.getColumn(_colList[j] - 1).set(i, val);
+ // }
+ // }
+ // }
+ // //columnDecode(in, out, 0, in.getNumRows());
+ // long b2 = System.nanoTime();
+ // System.out.println(this.getClass() + "time: " + (b2 - b1) / 1e6 + " ms");
+ // return out;
+ //}
+
+ @Override
+ public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+ long b1 = System.nanoTime();
+ out.ensureAllocatedColumns(in.getNumRows());
+
+ final int outColIndex = _colList[0] - 1;
+ final double[] binMins = _binMins[0];
+ final double[] binMaxs = _binMaxs[0];
+ final int nRows = in.getNumRows();
+ Array<?> a = out.getColumn(0);
+ for (int i = 0; i < nRows; i++) {
+ double val = in.get(i, 0);
+ double decoded;
+ if (!Double.isNaN(val)) {
+ int key = (int) Math.round(val);
+ double bmin = binMins[key - 1];
+ double bmax = binMaxs[key - 1];
+ decoded = bmin + (bmax - bmin) / 2
+ + (val - key) * (bmax - bmin);
+ a.set(i, decoded);
+ } else {
+ a.set(i, val);
+ }
+ }
+ long b2 = System.nanoTime();
+ System.out.println(this.getClass() +": "+ (b2 - b1) / 1e6 + " ms");
+ return out;
+ }
+
+
+ @Override
+ public void columnDecode(MatrixBlock in, FrameBlock out, int rl, int ru) {
+ for (int i = rl; i < ru; i++) {
+ for (int j = 0; j < _colList.length; j++) {
+ double val = in.get(i, j);
+ if (!Double.isNaN(val)) {
+ int key = (int) Math.round(val);
+ double bmin = _binMins[j][key - 1];
+ double bmax = _binMaxs[j][key - 1];
+ double oval = bmin + (bmax - bmin) / 2 + (val - key) * (bmax - bmin);
+ out.getColumn(_colList[j] - 1).set(i, oval);
+ } else {
+ out.getColumn(_colList[j] - 1).set(i, val);
+ }
+ }
Review Comment:
I don't understand why you are iterating all columns in a column decoder.
##########
src/main/java/org/apache/sysds/runtime/transform/decode/DecoderRecode.java:
##########
@@ -71,12 +71,13 @@ public FrameBlock decode(MatrixBlock in, FrameBlock out) {
@Override
public void decode(MatrixBlock in, FrameBlock out, int rl, int ru) {
- if( _onOut ) { //recode on output (after dummy)
+ if( _onOut ) { //recode on output (after dummy)
for( int i=rl; i<ru; i++ ) {
for( int j=0; j<_colList.length; j++ ) {
int colID = _colList[j];
double val = UtilFunctions.objectToDouble(
out.getSchema()[colID-1], out.get(i, colID-1));
+
Review Comment:
Remove these empty lines that you added.
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderBin.java:
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.frame.data.columns.Array;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+import org.apache.sysds.runtime.util.UtilFunctions;
+
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+
+public class ColumnDecoderBin extends ColumnDecoder {
+ private static final long serialVersionUID = -3784249774608228805L;
+
+ private int[] _numBins;
+ private double[][] _binMins = null;
+ private double[][] _binMaxs = null;
+
+ public ColumnDecoderBin() {
+ super(null, null);
+ }
+
+ protected ColumnDecoderBin(ValueType[] schema, int[] binCols) {
+ super(schema, binCols);
+ }
+
+
+ //@Override
+ //public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+//
+ // long b1 = System.nanoTime();
+ // out.ensureAllocatedColumns(in.getNumRows());
+ // for (int i = 0; i < in.getNumRows(); i++) {
+ // for (int j = 0; j < _colList.length; j++) {
+ // double val = in.get(i, j);
+ // if (!Double.isNaN(val)) {
+ // int key = (int) Math.round(val);
+ // double bmin = _binMins[j][key - 1];
+ // double bmax = _binMaxs[j][key - 1];
+ // double oval = bmin + (bmax - bmin) / 2 + (val - key) * (bmax - bmin);
+ // out.getColumn(_colList[j] - 1).set(i, oval);
+ // } else {
+ // out.getColumn(_colList[j] - 1).set(i, val);
+ // }
+ // }
+ // }
+ // //columnDecode(in, out, 0, in.getNumRows());
+ // long b2 = System.nanoTime();
+ // System.out.println(this.getClass() + "time: " + (b2 - b1) / 1e6 + " ms");
+ // return out;
+ //}
+
+ @Override
+ public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+ long b1 = System.nanoTime();
+ out.ensureAllocatedColumns(in.getNumRows());
+
+ final int outColIndex = _colList[0] - 1;
Review Comment:
Why the outColIndex is always _colList[0] - 1?
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoderRecode.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+import org.apache.sysds.runtime.matrix.data.Pair;
+import org.apache.sysds.runtime.transform.TfUtils;
+import org.apache.sysds.runtime.transform.encode.ColumnEncoderRecode;
+import org.apache.sysds.runtime.util.UtilFunctions;
+
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.*;
+
+public class ColumnDecoderRecode extends ColumnDecoder {
+
+ private static final long serialVersionUID = -3784249774608228805L;
+
+ private HashMap<Long, Object>[] _rcMaps = null;
+ private Object[][] _rcMapsDirect = null;
+ private boolean _onOut = false;
+
+ public ColumnDecoderRecode() {
+ super(null, null);
+ }
+
+ protected ColumnDecoderRecode(ValueType[] schema, boolean onOut, int[] rcCols) {
+ super(schema, rcCols);
+ _onOut = onOut;
+ }
+
+ @Override
+ public FrameBlock columnDecode(MatrixBlock in, FrameBlock out) {
+
+ long r1 = System.nanoTime();
+ // TODO
+ out.ensureAllocatedColumns(in.getNumRows());
+ columnDecode(in, out, 0, in.getNumRows());
+ long r2 = System.nanoTime();
+ System.out.println(this.getClass() + "time: " + (r2 - r1) / 1e6 + " ms");
+ return out;
+ }
+
+ @Override
+ public void columnDecode(MatrixBlock in, FrameBlock out, int rl, int ru) {
+ // TODO
+ if( _onOut ) { //recode on output (after dummy)
+ for( int i=rl; i<ru; i++ ) {
+ for( int j=0; j<_colList.length; j++ ) {
+ int colID = _colList[j];
+ double val = UtilFunctions.objectToDouble(
+ out.getSchema()[colID-1], out.get(i, colID-1));
+ long key = UtilFunctions.toLong(val);
+ out.set(i, colID-1, getRcMapValue(j, key));
+ }
Review Comment:
Why are you iterating all the columns? A column decoder should be called for each column.
##########
src/main/java/org/apache/sysds/runtime/transform/decode/ColumnDecoder.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.transform.decode;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
+
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+
+public abstract class ColumnDecoder implements Externalizable {
+ protected static final Log LOG = LogFactory.getLog(Decoder.class.getName());
+ private static final long serialVersionUID = -1732411001366177787L;
+
+ protected ValueType[] _schema;
+ protected int[] _colList;
+ protected String[] _colnames = null;
+ protected ColumnDecoder(ValueType[] schema, int[] colList) {
+ _schema = schema;
+ _colList = colList;
+ }
Review Comment:
Why a column list? A column encoder should work on a single column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-2991145451
## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2275?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `41.71779%` with `285 lines` in your changes missing coverage. Please review.
> Project coverage is 72.85%. Comparing base [(`bc2993e`)](https://app.codecov.io/gh/apache/systemds/commit/bc2993ef285cbeb440d1b970ad263e6b40be699e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`61f6d39`)](https://app.codecov.io/gh/apache/systemds/commit/61f6d39dc662ec7858e5d605b095221c6609b4d4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 8 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/systemds/pull/2275?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ime/transform/decode/ColumnDecoderPassThrough.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderPassThrough.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyUGFzc1Rocm91Z2guamF2YQ==) | 17.14% | [57 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ntime/transform/decode/ColumnDecoderDummycode.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderDummycode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyRHVtbXljb2RlLmphdmE=) | 33.33% | [47 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...sds/runtime/transform/decode/ColumnDecoderBin.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderBin.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyQmluLmphdmE=) | 46.25% | [38 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ntime/transform/decode/ColumnDecoderComposite.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderComposite.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyQ29tcG9zaXRlLmphdmE=) | 47.14% | [36 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../runtime/transform/decode/ColumnDecoderRecode.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderRecode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyUmVjb2RlLmphdmE=) | 61.05% | [31 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../sysds/runtime/transform/decode/ColumnDecoder.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyLmphdmE=) | 21.95% | [32 Missing :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...runtime/transform/decode/ColumnDecoderFactory.java](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&filepath=src%2Fmain%2Fjava%2Forg%2Fapache%2Fsysds%2Fruntime%2Ftransform%2Fdecode%2FColumnDecoderFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3N5c2RzL3J1bnRpbWUvdHJhbnNmb3JtL2RlY29kZS9Db2x1bW5EZWNvZGVyRmFjdG9yeS5qYXZh) | 51.72% | [27 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/systemds/pull/2275?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2275 +/- ##
============================================
- Coverage 72.95% 72.85% -0.10%
- Complexity 46062 46159 +97
============================================
Files 1479 1486 +7
Lines 172575 173144 +569
Branches 33776 33898 +122
============================================
+ Hits 125898 126149 +251
- Misses 37191 37481 +290
- Partials 9486 9514 +28
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/systemds/pull/2275?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "phaniarnab (via GitHub)" <gi...@apache.org>.
phaniarnab commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-3001917093
Thank you for the patch. I will take a look into this next week @Isso-W.
Meanwhile, please add the missing license headers to the new files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "phaniarnab (via GitHub)" <gi...@apache.org>.
phaniarnab commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-3172518366
@Isso-W, can you please post your plots of FTBench (decoding) here for others to see?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "phaniarnab (via GitHub)" <gi...@apache.org>.
phaniarnab commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-3031679780
@Isso-W, please address the comments. And fix the tests as well. Your tests are failing in the transform package, which you should be able to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SystemDS-#3524] Multi-threading of transformdecode/[SystemDS-#3521] Improved Feature Transformations [systemds]
Posted by "phaniarnab (via GitHub)" <gi...@apache.org>.
phaniarnab commented on PR #2275:
URL: https://github.com/apache/systemds/pull/2275#issuecomment-3095583225
The latest changes look good @Isso-W.
Please make sure all the unit tests pass and we get a green tick.
As discussed, for the remaining time, focus on extending the FTBench benchmark with decoding scripts and report performance improvement by multithreading for those use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org