You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by GitBox <gi...@apache.org> on 2020/11/27 07:46:39 UTC
[GitHub] [systemds] haubitzer opened a new pull request #1117: [WIP] Builtin function statsNA
haubitzer opened a new pull request #1117:
URL: https://github.com/apache/systemds/pull/1117
**Work in progress**
* no test implemented yet
* open question marked with "TODO"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531556439
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
+
+m_statsNA = function(Matrix[Double] X, Integer bins = 4, Boolean print_result = TRUE)
+ return(Integer length_series,
+ Double number_nans,
+ Double percentage_nans,
+ Double number_nan_gaps,
+ Double average_size_nan_gaps,
+ Integer longest_nan_gap,
+ Integer most_frequent_nan_gap,
+ Integer most_weighty_nan_gap
+ )
+{
+ #TODO Question: Do we need to check if the input matrix is a vector? and if yes in with direction should it go?
+ if(ncol(X) != 1) {
+ stop("CONFUSION MATRIX: expect a matrix with only one column");
+ }
+
+ # Count total entries
+ # TODO Question: Why returns length a double and not an integer?
+ length_series = length(X);
+
+ # TODO Question: What should happen when the input matrix is empty?
Review comment:
Printing an informative message and terminating the program seems fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531564514
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
+
+m_statsNA = function(Matrix[Double] X, Integer bins = 4, Boolean print_result = TRUE)
+ return(Integer length_series,
+ Double number_nans,
+ Double percentage_nans,
+ Double number_nan_gaps,
+ Double average_size_nan_gaps,
+ Integer longest_nan_gap,
+ Integer most_frequent_nan_gap,
+ Integer most_weighty_nan_gap
+ )
+{
+ #TODO Question: Do we need to check if the input matrix is a vector? and if yes in with direction should it go?
+ if(ncol(X) != 1) {
+ stop("CONFUSION MATRIX: expect a matrix with only one column");
+ }
+
+ # Count total entries
+ # TODO Question: Why returns length a double and not an integer?
+ length_series = length(X);
+
+ # TODO Question: What should happen when the input matrix is empty?
+ if (length_series == 0) {
+ stop("EMPTY MATRIX")
+ }
+
+ # TODO Question: What should happen when the number of bins is higher than length of the input matrix?
+ if (length_series < bins) {
+ bins = length_series;
+ }
+
+ # Count NaNs
+ # TODO Question: Is there a style guide for private or return variables?
+ # TODO Question: Why didn't I find any documentation for "is.nan" ?
Review comment:
New built-ins are added iteratively into the documentation.
is.na() or is.nan() will return a boolean mask matrix where the NaNs are represented as ones and other values as zeros.
the following line of code could be written as,
p_position_nans = is.na(X)
p_position_nans = is.nan(X)
is.na() looks for both NA and NaN values while is.nan() specifically looks for NaNs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531554010
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
Review comment:
Please feel free to add your built-in in our markdown files which will be updated on the website.
systemds\docs\site\builtins-reference.md
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531566877
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
+
+m_statsNA = function(Matrix[Double] X, Integer bins = 4, Boolean print_result = TRUE)
+ return(Integer length_series,
+ Double number_nans,
+ Double percentage_nans,
+ Double number_nan_gaps,
+ Double average_size_nan_gaps,
+ Integer longest_nan_gap,
+ Integer most_frequent_nan_gap,
+ Integer most_weighty_nan_gap
+ )
+{
+ #TODO Question: Do we need to check if the input matrix is a vector? and if yes in with direction should it go?
+ if(ncol(X) != 1) {
+ stop("CONFUSION MATRIX: expect a matrix with only one column");
+ }
+
+ # Count total entries
+ # TODO Question: Why returns length a double and not an integer?
+ length_series = length(X);
+
+ # TODO Question: What should happen when the input matrix is empty?
+ if (length_series == 0) {
+ stop("EMPTY MATRIX")
+ }
+
+ # TODO Question: What should happen when the number of bins is higher than length of the input matrix?
+ if (length_series < bins) {
+ bins = length_series;
+ }
+
+ # Count NaNs
+ # TODO Question: Is there a style guide for private or return variables?
+ # TODO Question: Why didn't I find any documentation for "is.nan" ?
+ p_position_nans = replace(target= X * 0, pattern = NaN, replacement = 1)
+ number_nans = sum(p_position_nans);
+
+ # Calculate percentage of NaNs
+ percentage_nans = number_nans / length_series;
+
+ # Create Vector with numbers of gas
+ p_gaps_vector = matrix(0, length_series, 1);
+ p_length_of_gap = 0;
+ for (i in 1:length_series) {
+ #TODO Question: Is there a way to compare these two values without as.scalar?
Review comment:
the comparison of two scalar values will only yield a scalar output, otherwise, you will get a matrix output which is not accepted inside the if clause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi edited a comment on pull request #1117: Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi edited a comment on pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#issuecomment-761576945
LGTM. Thank you @haubitzer and Ismael Ibrahim for your contribution. During the merger, I fixed the file headers, code formatting, and changed the output type from single eight variables to a column vector, and did some minor refactoring in the script. Ismael Ibrahim, could you please add your commit email address as a secondary email address in your Git account? Otherwise, you won't be visible in our contributors' list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on pull request #1117: Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#issuecomment-761576945
LGTM. Thank you @haubitzer and Ismael Ibrahim for your contribution. During the merger, I fixed the file headers, code formatting, and changed the output type from single eight variables to a column vector, and did some minor refactoring in the script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531568712
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
+
+m_statsNA = function(Matrix[Double] X, Integer bins = 4, Boolean print_result = TRUE)
+ return(Integer length_series,
+ Double number_nans,
+ Double percentage_nans,
+ Double number_nan_gaps,
+ Double average_size_nan_gaps,
+ Integer longest_nan_gap,
+ Integer most_frequent_nan_gap,
+ Integer most_weighty_nan_gap
+ )
+{
+ #TODO Question: Do we need to check if the input matrix is a vector? and if yes in with direction should it go?
+ if(ncol(X) != 1) {
+ stop("CONFUSION MATRIX: expect a matrix with only one column");
+ }
+
+ # Count total entries
+ # TODO Question: Why returns length a double and not an integer?
+ length_series = length(X);
+
+ # TODO Question: What should happen when the input matrix is empty?
+ if (length_series == 0) {
+ stop("EMPTY MATRIX")
+ }
+
+ # TODO Question: What should happen when the number of bins is higher than length of the input matrix?
+ if (length_series < bins) {
+ bins = length_series;
+ }
+
+ # Count NaNs
+ # TODO Question: Is there a style guide for private or return variables?
+ # TODO Question: Why didn't I find any documentation for "is.nan" ?
+ p_position_nans = replace(target= X * 0, pattern = NaN, replacement = 1)
+ number_nans = sum(p_position_nans);
+
+ # Calculate percentage of NaNs
+ percentage_nans = number_nans / length_series;
+
+ # Create Vector with numbers of gas
+ p_gaps_vector = matrix(0, length_series, 1);
+ p_length_of_gap = 0;
+ for (i in 1:length_series) {
+ #TODO Question: Is there a way to compare these two values without as.scalar?
+ if (as.scalar(p_position_nans[i,1]) == 1) {
+ p_length_of_gap += 1;
+ } else if (p_length_of_gap != 0){
+ p_gaps_vector[p_length_of_gap, 1] += 1;
+ p_length_of_gap = 0;
+ }
+ }
+ # The last element can also be a NaN but the loop will not update our vector map, so this workaround is needed.
+ if(p_length_of_gap > 0) {
+ p_gaps_vector[p_length_of_gap, 1] += 1;
+ }
+
+
+ # Count number of gaps
+ number_nan_gaps = sum(p_gaps_vector);
+
+ # Calculate average gap size
+ # WTF 1/0 is Infinity and 0/0 is NaN
+ average_size_nan_gaps = number_nans / number_nan_gaps
+
+ # Find longest gap
+ longest_nan_gap = NaN
+ # TODO Question: Why is there no "break" keyword?
+ # TODO Question: refactor into a while loop
+ for(i in 1:length_series){
+ if (as.scalar(p_gaps_vector[i,1]) > 0) {
+ longest_nan_gap = i;
+ }
+ }
+
+ # TODO Question: is the compiler smart enough that these two for loops can be merged? Does this even matter in our case?
+ # Find most frequent gap size
+ most_frequent_nan_gap = 0;
+ p_most_frequent_nan_gap_max_number = 0;
+ # TODO Question: Is there something like argmax? https://numpy.org/doc/stable/reference/generated/numpy.argmax.html
Review comment:
rowIndexMax(X) returns for each row the column number with the largest value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi commented on a change in pull request #1117: [WIP] Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi commented on a change in pull request #1117:
URL: https://github.com/apache/systemds/pull/1117#discussion_r531558217
##########
File path: scripts/builtin/statsNA.dml
##########
@@ -0,0 +1,212 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+# TODO Question: Is there a tool/template to generate the function documentation? Who updates the documentation on the website?
+
+m_statsNA = function(Matrix[Double] X, Integer bins = 4, Boolean print_result = TRUE)
+ return(Integer length_series,
+ Double number_nans,
+ Double percentage_nans,
+ Double number_nan_gaps,
+ Double average_size_nan_gaps,
+ Integer longest_nan_gap,
+ Integer most_frequent_nan_gap,
+ Integer most_weighty_nan_gap
+ )
+{
+ #TODO Question: Do we need to check if the input matrix is a vector? and if yes in with direction should it go?
+ if(ncol(X) != 1) {
+ stop("CONFUSION MATRIX: expect a matrix with only one column");
+ }
+
+ # Count total entries
+ # TODO Question: Why returns length a double and not an integer?
+ length_series = length(X);
+
+ # TODO Question: What should happen when the input matrix is empty?
+ if (length_series == 0) {
+ stop("EMPTY MATRIX")
+ }
+
+ # TODO Question: What should happen when the number of bins is higher than length of the input matrix?
Review comment:
use the default bin values in this case and also print a meaningful message something like "number of bins exceeds the length of series, using default bin value = 4"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [systemds] Shafaq-Siddiqi closed pull request #1117: Builtin function statsNA
Posted by GitBox <gi...@apache.org>.
Shafaq-Siddiqi closed pull request #1117:
URL: https://github.com/apache/systemds/pull/1117
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org