You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/02/09 23:42:05 UTC
[GitHub] [incubator-mxnet] Zha0q1 opened a new pull request #19876: [v1.x] Tool to help update/restore onnx support
Zha0q1 opened a new pull request #19876:
URL: https://github.com/apache/incubator-mxnet/pull/19876
As we frequently improve our ONNX support, users might want to get timely access to the latest changes. However, MXNet releases might not happen as frequent as the users would want and they might have other compatibility concerns to update the entire mxnet module. To help with this, this pr adds a simple tool to update(replace) and restore the `mxnet/contrib/onnx` folder to the latest version on github. For example, a user that has mxnet 1.7 installed can use this script to get the latest onnx support without updating mxnet to a later version. This script helps make our onnx support more accessible and users can report bugs back and expect them fixed more easily.
The script will automatically find the currently installed mxnet. The original files will be backuped to `mxnet/contrib/onnx/backup`. `--branch` option controls which mxnet branch to checkout and if `--restore` option is specified then files in `backup` will be restored.
To update to changes in v1.x
```
python update_onnx.py
```
To update to changes in a specific branch
```
python update_onnx.py --branch <branch>
```
To restore the original files
```
python update_onnx.py --restore
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19876: [v1.x] Tool to help update/restore onnx support
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19876:
URL: https://github.com/apache/incubator-mxnet/pull/19876#discussion_r576395830
##########
File path: tools/onnx/update_onnx.py
##########
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet
+import os
+import logging
+import argparse
+
+def update_onnx(branch='v1.x'):
+ # Detect MXNet
+ print('Detected MXNet version: %s' % mxnet.__version__)
+ mx_path = os.path.abspath(mxnet.__file__)
+ mx_path = mx_path[:mx_path.rfind('/')]
+ onnx_path = mx_path + '/contrib/onnx/'
+ if os.path.isdir(onnx_path):
+ print('Found ONNX path: %s' % onnx_path)
+ else:
+ logging.error('ONNX path not found. %s does not exist' % onnx_path)
+
+ # Backup the current onnx dir
+ backup_path = onnx_path + 'backup'
+ os.system('mkdir %s' % backup_path)
+ os.system('mv -v %s/* %s' % (onnx_path, backup_path))
+
+ # Clone the latest repo and copy the onnx dir
+ clone_path = './mxnet_repo_tmp'
+ os.system('mkdir %s' % clone_path)
+ cwd = os.getcwd()
+ os.chdir(clone_path)
+ os.system('git clone https://github.com/apache/incubator-mxnet mxnet')
+ os.chdir('./mxnet')
+ os.system('git checkout %s' % branch)
+ os.system('cp -r python/mxnet/contrib/onnx/* %s/' % onnx_path)
+ os.chdir(cwd)
+ os.system('rm -rf %s' %clone_path)
+ print('Done')
+
+
+def restore_onnx():
+ # Detect MXNet
+ print('Detected MXNet version: %s' % mxnet.__version__)
+ mx_path = os.path.abspath(mxnet.__file__)
+ mx_path = mx_path[:mx_path.rfind('/')]
+ onnx_path = mx_path + '/contrib/onnx'
+ backup_path = onnx_path + '/backup'
+ if os.path.isdir(backup_path):
+ print('Found ONNX path: %s' % onnx_path)
+ print('Found ONNX backup path: %s' % backup_path)
+ else:
+ logging.error('ONNX backup path not found. %s does not exist' % backup_path)
+
+ # Restore backup
+ os.chdir(onnx_path)
Review comment:
For this to work on windows, could you just check if the backup path exists and move it back then? The find/cp/rm commands won't work under windows.
##########
File path: tools/onnx/update_onnx.py
##########
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import mxnet
+import os
+import logging
+import argparse
+
+def update_onnx(branch='v1.x'):
+ # Detect MXNet
+ print('Detected MXNet version: %s' % mxnet.__version__)
+ mx_path = os.path.abspath(mxnet.__file__)
+ mx_path = mx_path[:mx_path.rfind('/')]
+ onnx_path = mx_path + '/contrib/onnx/'
+ if os.path.isdir(onnx_path):
+ print('Found ONNX path: %s' % onnx_path)
+ else:
+ logging.error('ONNX path not found. %s does not exist' % onnx_path)
+
+ # Backup the current onnx dir
+ backup_path = onnx_path + 'backup'
+ os.system('mkdir %s' % backup_path)
+ os.system('mv -v %s/* %s' % (onnx_path, backup_path))
+
+ # Clone the latest repo and copy the onnx dir
+ clone_path = './mxnet_repo_tmp'
+ os.system('mkdir %s' % clone_path)
+ cwd = os.getcwd()
+ os.chdir(clone_path)
+ os.system('git clone https://github.com/apache/incubator-mxnet mxnet')
+ os.chdir('./mxnet')
+ os.system('git checkout %s' % branch)
+ os.system('cp -r python/mxnet/contrib/onnx/* %s/' % onnx_path)
+ os.chdir(cwd)
+ os.system('rm -rf %s' %clone_path)
+ print('Done')
+
+
+def restore_onnx():
+ # Detect MXNet
+ print('Detected MXNet version: %s' % mxnet.__version__)
+ mx_path = os.path.abspath(mxnet.__file__)
+ mx_path = mx_path[:mx_path.rfind('/')]
Review comment:
You might want to use os.path.join(mx_path, "..", "contrib", "onnx"), so this will work in windows as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19876: [v1.x] Tool to help update/restore onnx support
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19876:
URL: https://github.com/apache/incubator-mxnet/pull/19876#issuecomment-776320624
Hey @Zha0q1 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [windows-cpu, website, clang, centos-cpu, windows-gpu, centos-gpu, edge, unix-gpu, unix-cpu, miscellaneous, sanity]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org