You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sentry.apache.org by Lei Xu <le...@cloudera.com> on 2017/06/02 22:20:40 UTC
Review Request 59767: SENTRY-1731. Enable
TestHDFSIntegrationEnd2End.testEnd2End
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/
-----------------------------------------------------------
Review request for sentry and Alexander Kolbasov.
Bugs: SENTRY-1731
https://issues.apache.org/jira/browse/SENTRY-1731
Repository: sentry
Description
-------
Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
Diffs
-----
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
Diff: https://reviews.apache.org/r/59767/diff/1/
Testing
-------
mvn test -Dtest=TestHDFSintegrationEnd2End.
Thanks,
Lei Xu
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by Na Li <li...@cloudera.com>.
> On June 4, 2017, 7:38 p.m., Alexander Kolbasov wrote:
> > The fix is fine, but there are test failures - are they related?
The failure is at org.apache.sentry.tests.e2e.dbprovider.TestDbPrivilegesAtColumnScope, not org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationEnd2End.
could one test class affects another test class? Or it causes more delay in another test class's execution. We saw test cases within a test class affects each other, such as add notification event in meta store that causes exception in next testing case.
I feel we have a general issue of flaky test cases due to timing. When configuing from hive, it takes time for Sentry to get the change, and then it takes time for HDFS to get the change. If the thread that gets update is delayed because other threads in other testing cases are running, then the test will fail. We need to add support to get the notification ID associated with the change, and verify after Sentry gets that notification ID. And the changeID associated with the change at Sentry, and verify when HDFS gets this changeID. In this way, we can avoid timing caused flaky test cases.
One issue is if the code to get update has bug, the test case may run forever. So we need to have a timeout that is much larger than the polling interval.
- Na
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/#review176884
-----------------------------------------------------------
On June 3, 2017, 12:35 a.m., Lei Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59767/
> -----------------------------------------------------------
>
> (Updated June 3, 2017, 12:35 a.m.)
>
>
> Review request for sentry and Alexander Kolbasov.
>
>
> Bugs: SENTRY-1713
> https://issues.apache.org/jira/browse/SENTRY-1713
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
>
>
> Diffs
> -----
>
> sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
>
>
> Diff: https://reviews.apache.org/r/59767/diff/1/
>
>
> Testing
> -------
>
> mvn test -Dtest=TestHDFSintegrationEnd2End.
>
>
> Thanks,
>
> Lei Xu
>
>
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by Lei Xu <le...@cloudera.com>.
> On June 4, 2017, 12:38 p.m., Alexander Kolbasov wrote:
> > The fix is fine, but there are test failures - are they related?
>
> Na Li wrote:
> The failure is at org.apache.sentry.tests.e2e.dbprovider.TestDbPrivilegesAtColumnScope, not org.apache.sentry.tests.e2e.hdfs.TestHDFSIntegrationEnd2End.
>
> could one test class affects another test class? Or it causes more delay in another test class's execution. We saw test cases within a test class affects each other, such as add notification event in meta store that causes exception in next testing case.
>
> I feel we have a general issue of flaky test cases due to timing. When configuing from hive, it takes time for Sentry to get the change, and then it takes time for HDFS to get the change. If the thread that gets update is delayed because other threads in other testing cases are running, then the test will fail. We need to add support to get the notification ID associated with the change, and verify after Sentry gets that notification ID. And the changeID associated with the change at Sentry, and verify when HDFS gets this changeID. In this way, we can avoid timing caused flaky test cases.
>
> One issue is if the code to get update has bug, the test case may run forever. So we need to have a timeout that is much larger than the polling interval.
This patch does not impact the product code and TestdbPrivilegesAtColumeScope, so they should not be related. Btw, {{TestHDFSIntegrationBase}} has setUp/tearDown to clean up the resource for each run.
- Lei
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/#review176884
-----------------------------------------------------------
On June 2, 2017, 5:35 p.m., Lei Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59767/
> -----------------------------------------------------------
>
> (Updated June 2, 2017, 5:35 p.m.)
>
>
> Review request for sentry and Alexander Kolbasov.
>
>
> Bugs: SENTRY-1713
> https://issues.apache.org/jira/browse/SENTRY-1713
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
>
>
> Diffs
> -----
>
> sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
>
>
> Diff: https://reviews.apache.org/r/59767/diff/1/
>
>
> Testing
> -------
>
> mvn test -Dtest=TestHDFSintegrationEnd2End.
>
>
> Thanks,
>
> Lei Xu
>
>
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by Alexander Kolbasov <ak...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/#review176884
-----------------------------------------------------------
The fix is fine, but there are test failures - are they related?
- Alexander Kolbasov
On June 3, 2017, 12:35 a.m., Lei Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59767/
> -----------------------------------------------------------
>
> (Updated June 3, 2017, 12:35 a.m.)
>
>
> Review request for sentry and Alexander Kolbasov.
>
>
> Bugs: SENTRY-1713
> https://issues.apache.org/jira/browse/SENTRY-1713
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
>
>
> Diffs
> -----
>
> sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
>
>
> Diff: https://reviews.apache.org/r/59767/diff/1/
>
>
> Testing
> -------
>
> mvn test -Dtest=TestHDFSintegrationEnd2End.
>
>
> Thanks,
>
> Lei Xu
>
>
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by kalyan kumar kalvagadda <kk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/#review176916
-----------------------------------------------------------
Ship it!
Ship It!
- kalyan kumar kalvagadda
On June 3, 2017, 12:35 a.m., Lei Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59767/
> -----------------------------------------------------------
>
> (Updated June 3, 2017, 12:35 a.m.)
>
>
> Review request for sentry and Alexander Kolbasov.
>
>
> Bugs: SENTRY-1713
> https://issues.apache.org/jira/browse/SENTRY-1713
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
>
>
> Diffs
> -----
>
> sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
>
>
> Diff: https://reviews.apache.org/r/59767/diff/1/
>
>
> Testing
> -------
>
> mvn test -Dtest=TestHDFSintegrationEnd2End.
>
>
> Thanks,
>
> Lei Xu
>
>
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by Na Li <li...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/#review176880
-----------------------------------------------------------
Ship it!
Ship It!
- Na Li
On June 3, 2017, 12:35 a.m., Lei Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59767/
> -----------------------------------------------------------
>
> (Updated June 3, 2017, 12:35 a.m.)
>
>
> Review request for sentry and Alexander Kolbasov.
>
>
> Bugs: SENTRY-1713
> https://issues.apache.org/jira/browse/SENTRY-1713
>
>
> Repository: sentry
>
>
> Description
> -------
>
> Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
>
>
> Diffs
> -----
>
> sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
>
>
> Diff: https://reviews.apache.org/r/59767/diff/1/
>
>
> Testing
> -------
>
> mvn test -Dtest=TestHDFSintegrationEnd2End.
>
>
> Thanks,
>
> Lei Xu
>
>
Re: Review Request 59767: SENTRY-1713. Enable
TestHDFSIntegrationEnd2End.testEnd2End
Posted by Lei Xu <le...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59767/
-----------------------------------------------------------
(Updated June 2, 2017, 5:35 p.m.)
Review request for sentry and Alexander Kolbasov.
Summary (updated)
-----------------
SENTRY-1713. Enable TestHDFSIntegrationEnd2End.testEnd2End
Bugs: SENTRY-1713
https://issues.apache.org/jira/browse/SENTRY-1713
Repository: sentry
Description
-------
Re-enable ```TestHDFSIntegrationEnd2End.testEnd2End```
Diffs
-----
sentry-tests/sentry-tests-hive/src/test/java/org/apache/sentry/tests/e2e/hdfs/TestHDFSIntegrationEnd2End.java 1ed92eae
Diff: https://reviews.apache.org/r/59767/diff/1/
Testing
-------
mvn test -Dtest=TestHDFSintegrationEnd2End.
Thanks,
Lei Xu