You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/01 16:30:18 UTC
[GitHub] [ozone] sodonnel opened a new pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
sodonnel opened a new pull request #3030:
URL: https://github.com/apache/ozone/pull/3030
## What changes were proposed in this pull request?
Add a new EC environment for acceptance tests and some basic tests to ensure EC writes and reads work correctly.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6231
## How was this patch tested?
New acceptance tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r797897560
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
+
+Test key Creation
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
+ Should Be Equal As Integers ${result} 0
Review comment:
`Execute` is the same as `Run and Return RC` + checking that RC is 0. So this (and following similar items) can be simplified to:
```suggestion
Execute ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
```
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
+
+Test key Creation
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}2mb /tmp/2mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}3mb /tmp/3mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}100mb /tmp/100mb
+ Should Be Equal As Integers ${result} 0
+
+ # Now Get the files and check they match
+ ${result} = Run and Return RC ozone sh key get /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/${prefix}1mb
+ Should Be Equal As Integers ${result} 0
+ Compare files /tmp/${prefix}1mb /tmp/1mb
Review comment:
We have a custom keyword `Key Should Match Local File` for this purpose.
```suggestion
Key Should Match Local File /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
```
The following "import" is needed for the keyword:
```
Resource ../ozone-lib/shell.robot
```
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
Review comment:
Nit: Since it also creates a couple of files, I think we can call this something more generic, like `Prepare for tests` (and adjust `Suite Setup` accordingly).
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
Review comment:
Can you please create a keyword for this group of checks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#issuecomment-1029923823
I think I have addressed the main comments, but I still need to move the test suite into an existing environment. If what we have now looks good, I will continue to do that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r800552464
##########
File path: hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
##########
@@ -61,3 +61,21 @@ Create Key
${output} = Execute ozone sh key put ${key} ${file}
Should not contain ${output} Failed
Log Uploaded ${file} to ${key}
+
+Verify Bucket EC Replication Config
+ [arguments] ${volume} ${encoding} ${data} ${parity} ${chunksize}
+ ${result} = Execute ozone sh bucket list ${volume} | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
Review comment:
I did not mean to carry $prefix into this part of the code. I missed that then copying and pasting I think. Your solution is better so I have committed them and will let the CI run and hopefully it passes again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799378044
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
Review comment:
Looks good. I think we can defer adding the keyword arguments to the next patch, unless you think it would be simpler now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
adoroszlai merged pull request #3030:
URL: https://github.com/apache/ozone/pull/3030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799365707
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
+
+Test key Creation
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
+ Should Be Equal As Integers ${result} 0
Review comment:
Thanks for the info on this. I have made the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799398090
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
Review comment:
I think I have something working now. I will push a couple of commits shortly so you can see how they look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#issuecomment-1027808592
@adoroszlai This PR is for the EC branch, but it is a new acceptance test suite for EC. Only basic tests for now, but we plan to build on them. Could you give this a review to ensure everything looks OK from a Docker and Robot Test framework point of view? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799368977
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
Review comment:
How do you think this would work? We made need to check for RS-3-2 or RS-6-3, or a different value from 1048576 in some future tests.
Something like:
```
Verify Key EC Replication Config
[arguments] ${encoding} ${data} ${parity} ${chunkSize}
...
```
Would probably work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r800058691
##########
File path: hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
##########
@@ -61,3 +61,21 @@ Create Key
${output} = Execute ozone sh key put ${key} ${file}
Should not contain ${output} Failed
Log Uploaded ${file} to ${key}
+
+Verify Bucket EC Replication Config
+ [arguments] ${volume} ${encoding} ${data} ${parity} ${chunksize}
+ ${result} = Execute ozone sh bucket list ${volume} | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
Review comment:
I think directly passing bucket is more reusable, instead of relying on suite variable and naming convention.
```suggestion
[arguments] ${bucket} ${encoding} ${data} ${parity} ${chunksize}
${result} = Execute ozone sh bucket info ${bucket} | jq -r '.replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
```
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Resource ../ozone-lib/shell.robot
+Suite Setup Prepare For Tests
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Prepare For Tests
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ Verify Bucket EC Replication Config /${prefix}vol1 RS 3 2 1048576
Review comment:
```suggestion
Verify Bucket EC Replication Config /${prefix}vol1/${prefix}ec RS 3 2 1048576
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799365932
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
+ ${random} = Generate Random String 5 [NUMBERS]
+ Set Suite Variable ${prefix} ${random}
+ Execute dd if=/dev/urandom of=/tmp/1mb bs=1048576 count=1
+ Execute dd if=/dev/urandom of=/tmp/2mb bs=1048576 count=2
+ Execute dd if=/dev/urandom of=/tmp/3mb bs=1048576 count=3
+ Execute dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
+
+*** Test Cases ***
+
+Test Bucket Creation
+ ${result} = Execute ozone sh volume create /${prefix}vol1
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket create /${prefix}vol1/${prefix}ratis
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ratis")) | .replicationConfig.replicationType'
+ Should contain ${result} RATIS
+ ${result} = Execute ozone sh bucket create --replication rs-3-2-1024k --type EC /${prefix}vol1/${prefix}ec
+ Should not contain ${result} Failed
+ ${result} = Execute ozone sh bucket list /${prefix}vol1 | jq -r '.[] | select(.name | contains("${prefix}ec")) | .replicationConfig.replicationType, .replicationConfig.codec, .replicationConfig.data, .replicationConfig.parity, .replicationConfig.ecChunkSize'
+ Should Match Regexp ${result} ^(?m)EC$
+ Should Match Regexp ${result} ^(?m)RS$
+ Should Match Regexp ${result} ^(?m)3$
+ Should Match Regexp ${result} ^(?m)2$
+ Should Match Regexp ${result} ^(?m)1048576$
+
+Test key Creation
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/1mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}2mb /tmp/2mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}3mb /tmp/3mb
+ Should Be Equal As Integers ${result} 0
+ ${result} = Run and Return RC ozone sh key put /${prefix}vol1/${prefix}ec/${prefix}100mb /tmp/100mb
+ Should Be Equal As Integers ${result} 0
+
+ # Now Get the files and check they match
+ ${result} = Run and Return RC ozone sh key get /${prefix}vol1/${prefix}ec/${prefix}1mb /tmp/${prefix}1mb
+ Should Be Equal As Integers ${result} 0
+ Compare files /tmp/${prefix}1mb /tmp/1mb
Review comment:
Nice - that makes these checks a lot easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #3030: HDDS-6231. EC: Create EC acceptance test environment and some basic tests
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #3030:
URL: https://github.com/apache/ozone/pull/3030#discussion_r799366076
##########
File path: hadoop-ozone/dist/src/main/smoketest/ec/basic.robot
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test EC shell commands
+Library OperatingSystem
+Resource ../commonlib.robot
+Suite Setup Generate prefix
+
+*** Variables ***
+${SCM} scm
+
+*** Keywords ***
+
+Generate prefix
Review comment:
OK - makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org