You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2022/04/19 13:02:59 UTC

[GitHub] [incubator-mxnet] bgawrych opened a new pull request, #21004: broadcast_like CPU optimization

bgawrych opened a new pull request, #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004

   ## Description ##
   ![image](https://user-images.githubusercontent.com/59644968/164009454-b838269b-20d2-4626-94a6-8c66f88d109d.png)
   
   ~1150 shapes examined:
   [broadcast_like.xlsx](https://github.com/apache/incubator-mxnet/files/8513171/broadcast_like.xlsx)
   
   Average speedup: -23%
   Worst: 68%
   Best: -97%
   
   Average speedup if better: -41%
   Average slowdown if worse: 4%
   
   
   
   quantized bert large before optimization:
   ![image](https://user-images.githubusercontent.com/59644968/164009525-5c49e36f-d3f4-448b-89d7-f368c77166aa.png)
   
   after optimization:
   ![image](https://user-images.githubusercontent.com/59644968/164009559-e44e1bf7-9d8b-4413-8bcd-3dda30b423f5.png)
   
   Over 2x speedup on broadcast_like
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1109828088

   @mxnet-bot run ci [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r861494669


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {
+      if (i < dst_shape.ndim()) {
+        in_shape[i]  = src_shape[i];
+        out_shape[i] = src_shape[i];

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1106521089

   Jenkins CI successfully triggered : [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1102625920

   Hey @bgawrych , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [miscellaneous, centos-cpu, unix-cpu, windows-cpu, clang, centos-gpu, unix-gpu, windows-gpu, edge, website, sanity]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1111034000

   Jenkins CI successfully triggered : [centos-cpu, unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1134232652

   Jenkins CI successfully triggered : [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r861494439


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {
+      if (i < dst_shape.ndim()) {
+        in_shape[i]  = src_shape[i];
+        out_shape[i] = src_shape[i];
+      } else {
+        in_shape[i]  = 1;

Review Comment:
   thanks, it was leftover from original function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1105403270

   Jenkins CI successfully triggered : [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1134232611

   @mxnet-bot run ci [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych merged pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych merged PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bartekkuncer commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r860712317


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }

Review Comment:
   You can simplify this one:
   ```suggestion
       preaxis_dims[ax] = src_shape[0];
       for (int i =1; i < axis; i++) {
         preaxis_dims[ax] *= src_shape[i];
       }
   ```
   You can do similar thing above as well.



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }

Review Comment:
   You can simplify this one:
   ```suggestion
       preaxis_dims[ax] = src_shape[0];
       for (int i = 1; i < axis; i++) {
         preaxis_dims[ax] *= src_shape[i];
       }
   ```
   You can do similar thing above as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1106520995

   @mxnet-bot run ci [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1104975597

   Jenkins CI successfully triggered : [unix-gpu, centos-gpu, website]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r875539150


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,79 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+
+    preaxis_dims[ax] = src_shape[0];
+    for (int i = 1; i < axis; i++) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis' elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+
+    const int ndim = dst_shape.ndim() == 2 ? 2 : MXNET_SPECIAL_MAX_NDIM;
+    Kernel<broadcast_kernel_cpu<mshadow_op::identity>, cpu>::Launch(
+        s, src_shape.Size(), src, dst, aux_data, req[0], ndim);
+
+  } else {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    IType* dst = static_cast<IType*>(outputs[0].dptr_);
+    // broadcast axis independently with result reusage
+    const int omp_threads = mxnet::engine::OpenMP::Get()->GetRecommendedOMPThreadCount();
+    for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+      index_t axis     = aux_data.axes[ax];
+      size_t bcast_dim = dst_shape[axis];
+
+#pragma omp parallel num_threads(omp_threads)
+      {
+        // start from the end to avoid overwriting values when src == dst
+        for (int i = preaxis_dims[ax] - 1; i >= 0; i--) {
+#pragma omp for
+          for (int j = bcast_dim - 1; j >= 0; j--) {
+#pragma GCC diagnostic push
+#if __GNUC__ >= 8
+#pragma GCC diagnostic ignored "-Wclass-memaccess"
+#endif
+            std::memcpy(dst + (elements_to_copy[ax] * (j + i * bcast_dim)),
+                        src + (elements_to_copy[ax] * i),
+                        elements_to_copy[ax] * sizeof(IType));
+#pragma GCC diagnostic pop
+          }
+        }
+      }
+      // when first of broadcastable axis is broadcasted,
+      // run same algorithm for next brodcast axis with 'new' input
+      // this is why loops are iterating from the end
+      src = dst;
+    }
+  }
+}
+
 /**
  * When CPU context is used the no. of kernel launches are equal to

Review Comment:
   When kernel implementation is used, than yes - optimized implementation utilizes memcpy w/o running kernel



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1640,7 +1728,7 @@ void LpNormCompute(const nnvm::NodeAttrs& attrs,
   }
 #else
   const std::string& red = param.ord == 1 ? "red::sum{}" : "red::nrm2{}";
-  const std::string& op  = param.ord == 1 ? "abs" : "identity";
+  const std::string& op = param.ord == 1 ? "abs" : "identity";

Review Comment:
   Yes, idk why it was formatted



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r861500749


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];

Review Comment:
   Because dst_shape is already broadcasted shape. 
   e.g. shapes: (6, 1, 4, 1, 2) broadcasted to (6, 5, 4, 3, 2) - first braodcasting axis elements_to_copy is equal to 2 and in for the second one it's equal to 24 (4*3*2)
   
   in preaxis_dims I am calculating how many different dimensions are in src tensor: for the first broadcasted axis it's 24 (6*1*4) and for second axis it's 6   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bartekkuncer commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r860678925


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {
+      if (i < dst_shape.ndim()) {
+        in_shape[i]  = src_shape[i];
+        out_shape[i] = src_shape[i];
+      } else {
+        in_shape[i]  = 1;

Review Comment:
   What is in_shape's purpose?



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {
+      if (i < dst_shape.ndim()) {
+        in_shape[i]  = src_shape[i];
+        out_shape[i] = src_shape[i];
+      } else {
+        in_shape[i]  = 1;
+        out_shape[i] = 1;
+      }
+    }
+
+    if (dst_shape.ndim() == 2) {

Review Comment:
   Why are you using just a '2' as parameter but for the other one you declare a variable? Maybe instead of branching the kernel call use if only for determining ndim e.g. `const int ndim = dst_shape.ndim() == 2 ? 2 : MXNET_SPECIAL_MAX_NDIM;` ?



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {

Review Comment:
   maybe stick to one notation -> `i++`



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;

Review Comment:
   You can simplify this one:
   ```suggestion
       preaxis_dims[ax] = src_shape[0];
       for (int i =1; i < axis; i++) {
         preaxis_dims[ax] *= src_shape[i];
       }
   ```
   You can do similar thing above as well.



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger

Review Comment:
   ```suggestion
     // there is no need to check further axis' elements to copy as it for sure will be larger
   ```



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis's elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> in_shape;
+    mshadow::Shape<MXNET_SPECIAL_MAX_NDIM> out_shape;
+    for (int i = 0; i < MXNET_SPECIAL_MAX_NDIM; ++i) {
+      if (i < dst_shape.ndim()) {
+        in_shape[i]  = src_shape[i];
+        out_shape[i] = src_shape[i];

Review Comment:
   Why src_shape not dst_shape?



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,94 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+    preaxis_dims[ax] = 1;
+    for (int i = axis - 1; i >= 0; i--) {
+      preaxis_dims[ax] *= src_shape[i];

Review Comment:
   Why one is calculated on src_shape and the other on dst_shape?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1111033875

   @mxnet-bot run ci [unix-cpu, centos-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1144578524

   Jenkins CI successfully triggered : [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1105403178

   @mxnet-bot run ci [centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] anko-intel commented on a diff in pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
anko-intel commented on code in PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#discussion_r860672535


##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1640,7 +1728,7 @@ void LpNormCompute(const nnvm::NodeAttrs& attrs,
   }
 #else
   const std::string& red = param.ord == 1 ? "red::sum{}" : "red::nrm2{}";
-  const std::string& op  = param.ord == 1 ? "abs" : "identity";
+  const std::string& op = param.ord == 1 ? "abs" : "identity";

Review Comment:
   Did you format the code with clang-formatter ? It looks strange that it breaks previous good alignment.



##########
src/operator/tensor/broadcast_reduce_op.h:
##########
@@ -1354,6 +1354,79 @@ struct direct_copy {
   }
 };
 
+template <typename IType, typename OType>
+void BroadcastCPU(const OpContext& ctx,
+                  const std::vector<TBlob>& inputs,
+                  const std::vector<OpReqType>& req,
+                  const std::vector<TBlob>& outputs,
+                  const mxnet::TShape& src_shape,
+                  const mxnet::TShape& dst_shape,
+                  ShapeAndStride aux_data) {
+  using namespace mshadow;
+  using namespace mshadow::expr;
+  using namespace mxnet_op;
+  constexpr size_t ELEMENTS_THRESHOLD = 256;
+  Stream<cpu>* s                      = ctx.get_stream<cpu>();
+
+  std::vector<size_t> elements_to_copy(aux_data.num_broadcast_axes);
+  std::vector<size_t> preaxis_dims(aux_data.num_broadcast_axes);
+  for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+    index_t axis = aux_data.axes[ax];
+
+    elements_to_copy[ax] = 1;
+    for (int i = axis + 1; i < dst_shape.ndim(); i++) {
+      elements_to_copy[ax] *= dst_shape[i];
+    }
+
+    preaxis_dims[ax] = src_shape[0];
+    for (int i = 1; i < axis; i++) {
+      preaxis_dims[ax] *= src_shape[i];
+    }
+  }
+
+  // there is no need to check further axis' elements to copy as it for sure will be larger
+  if (elements_to_copy[0] < ELEMENTS_THRESHOLD || !std::is_same<IType, OType>::value) {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    OType* dst = static_cast<OType*>(outputs[0].dptr_);
+
+    const int ndim = dst_shape.ndim() == 2 ? 2 : MXNET_SPECIAL_MAX_NDIM;
+    Kernel<broadcast_kernel_cpu<mshadow_op::identity>, cpu>::Launch(
+        s, src_shape.Size(), src, dst, aux_data, req[0], ndim);
+
+  } else {
+    IType* src = static_cast<IType*>(inputs[0].dptr_);
+    IType* dst = static_cast<IType*>(outputs[0].dptr_);
+    // broadcast axis independently with result reusage
+    const int omp_threads = mxnet::engine::OpenMP::Get()->GetRecommendedOMPThreadCount();
+    for (int ax = 0; ax < aux_data.num_broadcast_axes; ax++) {
+      index_t axis     = aux_data.axes[ax];
+      size_t bcast_dim = dst_shape[axis];
+
+#pragma omp parallel num_threads(omp_threads)
+      {
+        // start from the end to avoid overwriting values when src == dst
+        for (int i = preaxis_dims[ax] - 1; i >= 0; i--) {
+#pragma omp for
+          for (int j = bcast_dim - 1; j >= 0; j--) {
+#pragma GCC diagnostic push
+#if __GNUC__ >= 8
+#pragma GCC diagnostic ignored "-Wclass-memaccess"
+#endif
+            std::memcpy(dst + (elements_to_copy[ax] * (j + i * bcast_dim)),
+                        src + (elements_to_copy[ax] * i),
+                        elements_to_copy[ax] * sizeof(IType));
+#pragma GCC diagnostic pop
+          }
+        }
+      }
+      // when first of broadcastable axis is broadcasted,
+      // run same algorithm for next brodcast axis with 'new' input
+      // this is why loops are iterating from the end
+      src = dst;
+    }
+  }
+}
+
 /**
  * When CPU context is used the no. of kernel launches are equal to

Review Comment:
   is this comment still valid ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1109828244

   Jenkins CI successfully triggered : [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1144578444

   @mxnet-bot run ci [unix-cpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-mxnet] bgawrych commented on pull request #21004: broadcast_like CPU optimization

Posted by GitBox <gi...@apache.org>.
bgawrych commented on PR #21004:
URL: https://github.com/apache/incubator-mxnet/pull/21004#issuecomment-1104975435

   @mxnet-bot run ci [website, unix-gpu, centos-gpu]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org