You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/07/25 13:31:02 UTC
[GitHub] [tvm] valmat07 opened a new pull request, #12173: [OpenCL] Fix profiling hang for OpenCL device
valmat07 opened a new pull request, #12173:
URL: https://github.com/apache/tvm/pull/12173
Profiling for OpenCL runs was hanging because nested timer runs were not correctly handled. This is because each call to the `Timer::Start` function on OpenCL device causes the OpenCL event queue to be cleared.
The hang was due to the fact that the minimum run time was not reached for profiling, if it is set greater than zero. Since the queue is cleared every `Start` call, the time of only one run is returned, not `number`.
It happens in this cycle:
https://github.com/apache/tvm/blob/75ec1cffa9f160dd3165fedbf4408731ebfa797a/src/runtime/profiling.cc#L876-L891
Here `pf.CallPacked(args, &temp)` call function which contains one more call `Timer::Start` and `Timer::End`.
Also profiling was hanging for `__nop` functions like `reshape` due to there are no OpenCL calls for this node. So that OpenCL timer returns 0, which is less than the minimum time value.
The `__nop` functions are now skipped for profiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #12173: [OpenCL] Fix profiling hang for OpenCL device
Posted by GitBox <gi...@apache.org>.
echuraev commented on PR #12173:
URL: https://github.com/apache/tvm/pull/12173#issuecomment-1196623920
@masahi could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on pull request #12173: [OpenCL] Fix profiling hang for OpenCL device
Posted by GitBox <gi...@apache.org>.
echuraev commented on PR #12173:
URL: https://github.com/apache/tvm/pull/12173#issuecomment-1195009913
@valmat07 Thank you for your reply! Yes, I see, it looks like it should work. In this case, could you please add a test which will illustrate the problem which you have fixed in this PR? You can add test here: https://github.com/apache/tvm/tree/main/tests/cpp-runtime/opencl
I think that test can be simple as this one:
https://github.com/apache/tvm/blob/9963b59ffa489db61358dedd35a2453a5ca666b9/tests/cpp/profiling_test.cc#L28-L39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] valmat07 commented on pull request #12173: [OpenCL] Fix profiling hang for OpenCL device
Posted by GitBox <gi...@apache.org>.
valmat07 commented on PR #12173:
URL: https://github.com/apache/tvm/pull/12173#issuecomment-1194267819
@echuraev I have looked deeper into your question and i guess everything should work since every `Start` function call will create a new object and at the end of the first iteration the destructor will be called.
Please look:
https://github.com/apache/tvm/blob/75ec1cffa9f160dd3165fedbf4408731ebfa797a/src/runtime/profiling.cc#L92-L103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] echuraev commented on a diff in pull request #12173: [OpenCL] Fix profiling hang for OpenCL device
Posted by GitBox <gi...@apache.org>.
echuraev commented on code in PR #12173:
URL: https://github.com/apache/tvm/pull/12173#discussion_r930617226
##########
tests/cpp-runtime/opencl/opencl_timer_test.cc:
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <gtest/gtest.h>
+#include <tvm/runtime/profiling.h>
+
+#include "../src/runtime/opencl/opencl_common.h"
+
+using namespace tvm::runtime;
+using namespace tvm::runtime::cl;
+
+#define BUFF_SIZE 1024
+#define NUM_REPEAT 10
+
+TEST(OpenCLTimerNode, nested_timers) {
+
+ OpenCLWorkspace* workspace = OpenCLWorkspace::Global();
+ OpenCLThreadEntry* thr = workspace->GetThreadEntry();
+ cl_command_queue queue = workspace->GetQueue(thr->device);
+
+ int err;
+ cl_int* tmp_buf = new cl_int[BUFF_SIZE];
+ int64_t nested_time_sum = 0;
+
+ Timer init_timer = Timer::Start(thr->device);
+ for (int i = 0; i < NUM_REPEAT; ++i) {
+ Timer nested_timer = Timer::Start(thr->device);
+ //create some events
+ cl_event ev = clCreateUserEvent(workspace->context, &err);
+ OPENCL_CHECK_ERROR(err);
+ cl_mem cl_buf = clCreateBuffer(workspace->context, CL_MEM_READ_ONLY, BUFF_SIZE * sizeof(cl_int), NULL, &err);
+ OPENCL_CHECK_ERROR(err);
+ OPENCL_CALL(clEnqueueWriteBuffer(queue, cl_buf, false, 0, BUFF_SIZE * sizeof(cl_int), tmp_buf, 0, NULL, &ev));
+ OPENCL_CALL(clReleaseMemObject(cl_buf));
+ workspace->events[thr->device.device_id].push_back(ev);
+ nested_timer->Stop();
+ nested_time_sum += nested_timer->SyncAndGetElapsedNanos();
+ }
+ init_timer->Stop();
+
+ free(tmp_buf);
Review Comment:
You have allocated this object with `new`, so you must use `delete` instead of `free`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #12173: [OpenCL] Fix profiling hang for OpenCL device
Posted by GitBox <gi...@apache.org>.
masahi merged PR #12173:
URL: https://github.com/apache/tvm/pull/12173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org