You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/07/09 06:07:12 UTC
[GitHub] [incubator-tvm] fernchen opened a new pull request #6023: [CI][Caffe Frontend] add caffe environment
fernchen opened a new pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023
Recently，we try to introduce Caffe frontend to TVM, please to ref to https://github.com/apache/incubator-tvm/pull/6000 , but original CI environment dose not support for Caffe. In order to pass CI test, we add Caffe environment including Caffe dependent packages and pycaffe for ssd-version, it comes from https://github.com/weiliu89/caffe
@zhiics @tqchen @FrozenGene
Could you please have a look on this PR? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452601208
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
I agree with you. In fact, we already maintain one caffe proto file internally, which includes various layers from different caffe distribution. If add a new layer, we will add layer definiton in caffe.proto firstly and then generate caffe_pb2.py which will be used by caffe frontend. But, now the problem is that BVLC can not include all the layers that supported by our Caffe frontend, if we add BVLC distribution in CI, the tests case will fail.
For the Caffe frontend pr, I think there are two steps:
1. CI pr: we can use SSD distribution temporarily， it convers as many layers as possible;
2. Caffe Frontend pr: we can offer caffe.proto too if neccessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457811327
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
Ok，i will add more comments on it later. By the way, do you know why this pr cant not pass the ci tests? The following information shows that the it pass the buiding windows-latest but fail to build macOS-latest. When i click the "Details", it just display the "This check failed", and no more other info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452591374
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
Yes, our caffe frontend will support for some layers and networks that has not been implemeted in BVLC, such as ProrBox in SSDMobilenet, and this custom distribution can support all tests in https://github.com/apache/incubator-tvm/pull/6000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-659944951
Gental ping @tqchen @siju-samuel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen merged pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen merged pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-661029354
Hi @FrozenGene @siju-samuel @tqchen :
I have changed the caffe deps into BVLC distribuition, now we support the official caffe version for TVM firstly.
May you please review them? If there is no problem, I hope to merge as soon as possible in order to pass the CI test of caff front-end codes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-660408499
> > 1. We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;
> > 2. We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.
>
> I vote for second option. BVLC caffe models will have more users compared to forked versions.
> Another concern is lot of dependancies are getting added for running the ci with custom distribution, i think bvlc have support for precompiled caffe installation.
Yes, bvlc distribution has supported for precompiled caffe installation, and It's really convenient to use.
If other people agree, we can support the official version first, and then gradually support other distribution. We can even maintain our own Caffe branch to support other layers, of course it is another topic. @tqchen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457802827
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
please add a more detailed comment about this case then. thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-656436451
May you help me review this pr? @siju-samuel @Huyuwei @hlu1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-665749065
ci-cpu is now updated to the latest state
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457790092
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
When installing pre-compiled caffe, it first installs the tzdata dependency, and then you need select the time zone manually. In other words, it is neccessary to install precompiled caffe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457816815
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
I have already added more comments on it, please review it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-658560667
Based on disscution above, there are two kinds of ways to support caffe frontend in TVM:
1. We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;
2. We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.
We are very welcome to participate in the discussion！https://discuss.tvm.ai/t/introduce-new-frontend-for-caffe/6918
@tqchen @siju-samuel @zhiics @Huyuwei @hlu1 @kazum @srkreddy1238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-665125691
Thanks @fernchen This PR is now merged. Note that we still need to update the ci-cpu image, will rely to this thread once that is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452597410
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
we should have one discussion about it. Because Caffe proto could be customized, we can not support every distribution. The default version we should support is the official distribution of course. But I also know SSD is one important usage. How about we maintain one Caffe proto like MXNet (https://github.com/apache/incubator-mxnet/tree/v1.7.x/tools/caffe_converter) and generate caffe_pb2.py? The Caffe proto should be based on BVLC official version but if we add customization we could discuss. Maybe this is one trade off to support important usage which is not included in the BVLC version. @tqchen @siju-samuel do you have better suggestion about it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452602027
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
Then if we all agree we should maintain our TVM Caffe proto, in our CI, the caffe_pb will be generated by this Caffe proto (like we could support SSD now). Then we could pass the test in your pr.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457772084
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
Can you comment about the time zone setup and whether it is necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452601615
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
If we add BVLC distribution, we could seperate our pr and avoid the ops BVLC distribution doesn't support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r457813539
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# avoid manually selecting the time zone
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
Review comment:
please re-trigger it again. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r461209556
##########
File path: docker/Dockerfile.ci_gpu
##########
@@ -83,6 +83,9 @@ RUN bash /install/ubuntu_install_dgl.sh
COPY install/ubuntu_install_vulkan.sh /install/ubuntu_install_vulkan.sh
RUN bash /install/ubuntu_install_vulkan.sh
+COPY install/ubuntu_install_caffe.sh /install/ubuntu_install_caffe.sh
Review comment:
Can we just install the caffe on the CPU and run ci tests through the CPU side? This would reduce our pressure in terms of depending on the GPU resources
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,34 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# The precompiled caffe dependents on tzdata.
+# While installing tzdata in docker, we need set the time zone manually,
+# which will cause the container to hang during installation.
+# So in order to avoid manually selecting the time zone, set as following:
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+
+apt install caffe-cpu -y
+
+pip install --upgrade scikit-image
Review comment:
Need pip3 instead of pip to introduce it to the python3 env.
##########
File path: docker/Dockerfile.ci_gpu
##########
@@ -83,6 +83,9 @@ RUN bash /install/ubuntu_install_dgl.sh
COPY install/ubuntu_install_vulkan.sh /install/ubuntu_install_vulkan.sh
RUN bash /install/ubuntu_install_vulkan.sh
+COPY install/ubuntu_install_caffe.sh /install/ubuntu_install_caffe.sh
Review comment:
We can add the tests to here https://github.com/apache/incubator-tvm/blob/master/tests/scripts/task_python_frontend_cpu.sh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452587675
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
Do we have special reason we use this custom distribution of Caffe instead of BVLC?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] siju-samuel commented on pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
siju-samuel commented on pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#issuecomment-660085200
> 1. We add caffe frontend based on SSD distribution, then this ci pr is good enough to test;
> 2. We add caffe frontend based on our maintaining tvm caffe, then I will support the layers of BVLC distribution firstly.
I vote for second option. BVLC caffe models will have more users compared to forked versions.
Another concern is lot of dependancies are getting added for running the ci with custom distribution, i think bvlc have support for precompiled caffe installation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] fernchen commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
fernchen commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r461307199
##########
File path: docker/Dockerfile.ci_gpu
##########
@@ -83,6 +83,9 @@ RUN bash /install/ubuntu_install_dgl.sh
COPY install/ubuntu_install_vulkan.sh /install/ubuntu_install_vulkan.sh
RUN bash /install/ubuntu_install_vulkan.sh
+COPY install/ubuntu_install_caffe.sh /install/ubuntu_install_caffe.sh
Review comment:
Good, if we just install the caffe on the CPU, it will reduce a lot of work.
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,34 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+apt-get update --fix-missing
+
+# The precompiled caffe dependents on tzdata.
+# While installing tzdata in docker, we need set the time zone manually,
+# which will cause the container to hang during installation.
+# So in order to avoid manually selecting the time zone, set as following:
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+
+apt install caffe-cpu -y
+
+pip install --upgrade scikit-image
Review comment:
I have updated the codes, please review them, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request #6023: [CI][Caffe Frontend] add caffe environment
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #6023:
URL: https://github.com/apache/incubator-tvm/pull/6023#discussion_r452601615
##########
File path: docker/install/ubuntu_install_caffe.sh
##########
@@ -0,0 +1,93 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+set -u
+set -o pipefail
+
+# Prerequisite
+apt-get update --fix-missing
+
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y tzdata
+ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
+dpkg-reconfigure --frontend noninteractive tzdata
+
+apt-get install libprotobuf-dev -y
+apt-get install libleveldb-dev -y
+apt-get install libsnappy-dev -y
+apt-get install libopencv-dev -y
+apt-get install libhdf5-serial-dev -y
+apt-get install protobuf-compiler -y
+apt-get install libgflags-dev -y
+apt-get install libgoogle-glog-dev -y
+apt-get install liblmdb-dev -y
+apt-get install libatlas-base-dev -y
+apt-get install --no-install-recommends libboost-all-dev -y
+apt-get install gfortran -y
+
+cd /
+mkdir caffe
+cd caffe
+wget https://github.com/weiliu89/caffe/archive/ssd.zip -O ssd.zip
Review comment:
If we add BVLC distribution, we could separate our pr and avoid the ops BVLC distribution doesn't support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org