You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by GitBox <gi...@apache.org> on 2020/04/09 20:31:34 UTC

[GitHub] [systemml] Shafaq-Siddiqi opened a new pull request #883: Builtin "is_correct" for validation of data against a given schema

Shafaq-Siddiqi opened a new pull request #883: Builtin "is_correct" for validation of data against a given schema
URL: https://github.com/apache/systemml/pull/883
 
 
   The built-in will validate the data in frame against a given schema. All non-conforming values will be converted into missing values (null in this case).
   usage: is_correct(Frame[String] data_frame, Frame[String] schema); schema is a row vector 
   ncol(data_frame) = ncol(schema).  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [systemml] j143 commented on a change in pull request #883: Builtin "is_correct" for validation of data against a given schema

Posted by GitBox <gi...@apache.org>.
j143 commented on a change in pull request #883: Builtin "is_correct" for validation of data against a given schema
URL: https://github.com/apache/systemml/pull/883#discussion_r406581861
 
 

 ##########
 File path: src/main/java/org/apache/sysds/runtime/instructions/SPInstructionParser.java
 ##########
 @@ -39,55 +39,10 @@
 import org.apache.sysds.lops.WeightedUnaryMMR;
 import org.apache.sysds.runtime.DMLRuntimeException;
 import org.apache.sysds.runtime.instructions.cp.CPOperand;
-import org.apache.sysds.runtime.instructions.spark.AggregateTernarySPInstruction;
-import org.apache.sysds.runtime.instructions.spark.AggregateUnarySPInstruction;
-import org.apache.sysds.runtime.instructions.spark.AppendGAlignedSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.AppendGSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.AppendMSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.AppendRSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.BinUaggChainSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.BinarySPInstruction;
-import org.apache.sysds.runtime.instructions.spark.BuiltinNarySPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CSVReblockSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CastSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CentralMomentSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CheckpointSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CompressionSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CovarianceSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CpmmSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CtableSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CumulativeAggregateSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.CumulativeOffsetSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.DnnSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.IndexingSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.MapmmChainSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.MapmmSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.MatrixReshapeSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.MultiReturnParameterizedBuiltinSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.PMapmmSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.ParameterizedBuiltinSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.PmmSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.QuantilePickSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.QuantileSortSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.QuaternarySPInstruction;
-import org.apache.sysds.runtime.instructions.spark.RandSPInstruction;
-import org.apache.sysds.runtime.instructions.spark.ReblockSPInstruction;
 
 Review comment:
   A change for imports can be done in a separate PR with `[MINOR]` tag. Otherwise this would impede the PR progress.
   
   So, for this PR only these _deletions_ be reverted. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [systemml] asfgit closed pull request #883: Builtin "is_correct" for validation of data against a given schema

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #883: Builtin "is_correct" for validation of data against a given schema
URL: https://github.com/apache/systemml/pull/883
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [systemml] mboehm7 commented on issue #883: Builtin "is_correct" for validation of data against a given schema

Posted by GitBox <gi...@apache.org>.
mboehm7 commented on issue #883: Builtin "is_correct" for validation of data against a given schema
URL: https://github.com/apache/systemml/pull/883#issuecomment-612163888
 
 
   LGTM - thanks @Shafaq-Siddiqi, that's very useful for extending our data cleaning primitives. During the merge, I just changed the name to `dropInvalid`, fixed some formatting issues (e.g., place of java annotations) and created consistency between CP and Spark instructions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services