You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Amos Bird <am...@gmail.com> on 2016/09/17 04:06:13 UTC
test errors in local env
Hi there,
I followed the wiki
https://cwiki.apache.org/confluence/display/IMPALA/How+to+load+and+run+Impala+tests
carefully but still have some problems in my local env.
1. I need to manually execute "hdfs dfs -mkdir /test-warehouse/emptytable"
to get rid of some fe test error.
2. I have authz-policy.ini in HDFS, but I still get authorization errors.
TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time elapsed: 0.333 sec <<< FAILURE!
java.lang.AssertionError: got error:
User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
expected:
User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2220)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2203)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2197)
at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(AuthorizationTest.java:512)
TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time elapsed: 0.324 sec <<< FAILURE!
java.lang.AssertionError: got error:
User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
expected:
User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2220)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2203)
at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(AuthorizationTest.java:2197)
at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(AuthorizationTest.java:512)
Results :
Failed tests:
AuthorizationTest.TestSelect:512->AuthzError:2197->AuthzError:2203->AuthzError:2220 got error:
User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
expected:
User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
AuthorizationTest.TestSelect:512->AuthzError:2197->AuthzError:2203->AuthzError:2220 got error:
User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
expected:
User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
3. For end-to-end tests, I encountered two kinds of errors
a) connection refused.
SET sync_ddl=False;
-- executing against localhost:21000
DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
___________________ ERROR at setup of TestLoadData.test_load[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] ___________________
[gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/bin/../infra/python/env/bin/python
metadata/test_load.py:77: in setup_method
"{0}/{1}/100101.txt".format(STAGING_PATH, i))
util/hdfs_util.py:122: in copy
data = self.read_file(src)
../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:183: in read_file
response = requests.get(uri, allow_redirects=True)
../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in get
return request('get', url, params=params, **kwargs)
../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in request
response = session.request(method=method, url=url, **kwargs)
../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:465: in request
resp = self.send(prep, **send_kwargs)
../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:594: in send
history = [resp for resp in gen] if allow_redirects else []
../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:196: in resolve_redirects
**adapter_kwargs
../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:573: in send
r = adapter.send(request, **kwargs)
../infra/python/env/lib/python2.6/site-packages/requests/adapters.py:415: in send
raise ConnectionError(err, request=request)
E ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
b) stats not match
[gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/bin/../infra/python/env/bin/python
metadata/test_metadata_query_statements.py:67: in test_show_stats
self.run_test_case('QueryTest/show-stats', vector, "functional")
common/impala_test_suite.py:342: in run_test_case
self.__verify_results_and_errors(vector, test_section, result, use_db)
common/impala_test_suite.py:234: in __verify_results_and_errors
replace_filenames_with_placeholder)
common/test_result_verifier.py:398: in verify_raw_results
VERIFIER_MAP[verifier](expected, actual)
common/test_result_verifier.py:231: in verify_query_result_is_equal
assert expected_results == actual_results
...
-- executing against localhost:21000
show column stats alltypes_clone;
MainThread: Comparing QueryTestResults (expected vs actual):
'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',736,-1,8,8
'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
'smallint_col','SMALLINT',10,-1,2,2 == 'smallint_col','SMALLINT',10,-1,2,2
'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-1,-1
'timestamp_col','TIMESTAMP',7554,-1,16,16 != 'timestamp_col','TIMESTAMP',7552,-1,16,16
'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,1,1
'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help is
much appreciated!
Best regards,
Amos
Re: test errors in local env
Posted by Amos Bird <am...@gmail.com>.
Java 7 solves this problem. Thanks Lars!
Lars Volker writes:
> Hi Amos,
>
> apologies for the delay? Is this issue still open? If so, can you try to
> run the tests with Java 7?
>
> Thanks, Lars
>
> On Thu, Sep 22, 2016 at 6:38 PM, Amos Bird <am...@gmail.com> wrote:
>
>> Thanks Lars,
>>
>> My Java version is
>>
>> $ $JAVA_HOME/bin/java -version
>> java version "1.8.0_65"
>> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
>> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
>>
>> > Hi Amos,
>> >
>> > I tried to see if IMPALA-3643
>> > <https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible
>> for
>> > me. In the process I also ran into the "NoClassDefFoundError" error you
>> saw
>> > and found out, that this happens if my local hadoop services
>> > (./testdata/bin/run-all.sh) isn't running.
>> >
>> > Once I have the local hadoop services running I'm able to repro the issue
>> > in IMPALA-3643. The only thing I remember doing differently from the
>> > recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
>> > LinkedHashSet
>> > <https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html>
>> don't
>> > read like they'd change the ordering, however it doesn't say that the
>> > ordering has to be stable *between different versions of Java*. I will
>> try
>> > to confirm that this issue does not repro with Java 7 when I have time.
>> Can
>> > you check the Java version you're running. Mine is this:
>> >
>> > $ $JAVA_HOME/bin/java -version
>> > java version "1.8.0_101"
>> > Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
>> > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
>> >
>> > Alex, could a difference in the implementation between Java 7 and 8 be a
>> > reasonable explanation for this?
>> >
>> > Cheers, Lars
>> >
>> > On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <am...@gmail.com> wrote:
>> >
>> >>
>> >> Thank you for helping me out :D
>> >>
>> >> > Thanks for keeping this discussion going.
>> >> >
>> >> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com>
>> wrote:
>> >> >
>> >> >>
>> >> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com>
>> >> wrote:
>> >> >> >
>> >> >> >>
>> >> >> >> Hi there,
>> >> >> >>
>> >> >> >> I followed the wiki
>> >> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> >> >> to+load+and+run+Impala+tests
>> >> >> >> carefully but still have some problems in my local env.
>> >> >> >>
>> >> >> >> 1. I need to manually execute "hdfs dfs -mkdir
>> >> >> /test-warehouse/emptytable"
>> >> >> >> to get rid of some fe test error.
>> >> >> >>
>> >> >> >>
>> >> >> > Ideally, you should not have to do this. Could you tell me what
>> errors
>> >> >> you
>> >> >> > encountered? Sounds like there may be a test or data loading bug we
>> >> >> should
>> >> >> > fix.
>> >> >>
>> >> >> The error is :
>> >> >>
>> >> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
>> >> >> elapsed: 0.033 sec <<< FAILURE!
>> >> >> java.lang.AssertionError: got error:
>> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> does
>> >> >> not exist.
>> >> >> expected:
>> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> >> >> contains no visible files.
>> >> >> at org.junit.Assert.fail(Assert.java:88)
>> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> >> FrontendTestBase.java:312)
>> >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> >> FrontendTestBase.java:292)
>> >> >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> >> >> AnalyzeStmtsTest.java:2860)
>> >> >>
>> >> >>
>> >> > Do you have a table functional.emptytable? If yes, then what location
>> is
>> >> > reported in "show create table"?
>> >> Query: show create table functional.emptytable
>> >> +-------------------------------------------------------------+
>> >> | result |
>> >> +-------------------------------------------------------------+
>> >> | CREATE EXTERNAL TABLE functional.emptytable ( |
>> >> | field STRING |
>> >> | ) |
>> >> | PARTITIONED BY ( |
>> >> | f2 INT |
>> >> | ) |
>> >> | STORED AS TEXTFILE |
>> >> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
>> >> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') |
>> >> +-------------------------------------------------------------+
>> >> Fetched 1 row(s) in 5.51s
>> >>
>> >> > Does the directory exist in HDFS?
>> >> No.
>> >>
>> >> >
>> >> > You could try to manually reload the table and see if the directory is
>> >> > created:
>> >> > bin/load-data.py -f -w functional-query --table_names=emptytable
>> >> > --table_formats=text/none
>> >> After executing this command the directory appears.
>> >>
>> >> >
>> >> >>>
>> >> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
>> >> >> errors.
>> >> >> >>
>> >> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)
>> Time
>> >> >> >> elapsed: 0.333 sec <<< FAILURE!
>> >> >> >> java.lang.AssertionError: got error:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> default.nodb
>> >> >> >> expected:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> nodb.alltypes
>> >> >> >> at org.junit.Assert.fail(Assert.java:88)
>> >> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2220)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2203)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2197)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> >> >> AuthorizationTest.java:512)
>> >> >> >>
>> >> >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)
>> Time
>> >> >> >> elapsed: 0.324 sec <<< FAILURE!
>> >> >> >> java.lang.AssertionError: got error:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> default.nodb
>> >> >> >> expected:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> nodb.alltypes
>> >> >> >> at org.junit.Assert.fail(Assert.java:88)
>> >> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2220)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2203)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> >> AuthorizationTest.java:2197)
>> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> >> >> AuthorizationTest.java:512)
>> >> >> >>
>> >> >> >>
>> >> >> >> Results :
>> >> >> >>
>> >> >> >> Failed tests:
>> >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> >> >> AuthzError:2203->AuthzError:2220 got error:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> default.nodb
>> >> >> >> expected:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> nodb.alltypes
>> >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> >> >> AuthzError:2203->AuthzError:2220 got error:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> default.nodb
>> >> >> >> expected:
>> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> >> nodb.alltypes
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> > Strange. In this test, we register two authorization requests, and
>> it
>> >> >> seems
>> >> >> > like those are not checked in the expected order. However, that
>> should
>> >> >> not
>> >> >> > be possible because we store them in a LinkedHashSet.
>> >> >> > Could you dig into this a little further to see if you can figure
>> out
>> >> why
>> >> >> > the order is wrong?
>> >> >> >
>> >> >> > This is where we register the authorization requests:
>> >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> >> >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
>> >> >> >
>> >> >> > This is where we check the authorization requests:
>> >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> >> >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>> >> >> >
>> >> >> >
>> >> >>
>> >> >> I tried directly executing "select 1 from nodb.alltypes" in
>> >> >> impala-shell, leading to this error:
>> >> >> ERROR: AnalysisException: Could not resolve table reference:
>> >> >> 'nodb.alltypes'
>> >> >>
>> >> >> How can I reproduce the authorization tests in impala-shell so I can
>> >> >> debug it?
>> >> >>
>> >> >>
>> >> >>
>> >> > FYI, this is actually a known issue and may have something to do with
>> the
>> >> > JRE version you are running: https://issues.cloudera.org/
>> >> browse/IMPALA-3643
>> >> > As far as I can tell the bug should be "impossible" because we use a
>> >> > LinkedHashSet, but maybe certain JREs do not properly honor the
>> >> guarantees.
>> >> >
>> >> > The AuthorizationTests in particular require a non-trivial setup, so
>> I'd
>> >> > not recommend trying to debug via the Impala shell.
>> >> >
>> >> > I'd recommend debugging in one of these ways:
>> >> > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from
>> the
>> >> FE
>> >> > directory. Attach debugger and break in TestSelect().
>> >> > - Run the JUnit test from an IDE such as Eclipse and then debug the
>> test.
>> >> > I'm afraid there is no easy way to just run that single query in our
>> >> > current test setup. You will need to run the whole suite, but you can
>> >> break
>> >> > TestSelect() or hack the code in various places to set useful
>> >> breakpoints.
>> >> >
>> >> > Hope that helps.
>> >>
>> >> I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest
>> >> fails with ClassNoDefine.
>> >> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008
>> sec
>> >> <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest
>> >> initializationError(com.cloudera.impala.analysis.AuthorizationTest)
>> Time
>> >> elapsed: 0.006 sec <<< ERROR!
>> >> java.lang.NoClassDefFoundError: Could not initialize class
>> >> com.cloudera.impala.analysis.AuthorizationTest
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> >> NativeMethodAccessorImpl.java:62)
>> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >> DelegatingMethodAccessorImpl.java:43)
>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>> >> FrameworkMethod.java:50)
>> >> at org.junit.internal.runners.model.ReflectiveCallable.run(
>> >> ReflectiveCallable.java:12)
>> >> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
>> >> FrameworkMethod.java:47)
>> >> at org.junit.runners.Parameterized.allParameters(
>> Parameterized.java:280)
>> >> at org.junit.runners.Parameterized.<init>(Parameterized.java:248)
>> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>> >> NativeConstructorAccessorImpl.java:62)
>> >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>> >> DelegatingConstructorAccessorImpl.java:45)
>> >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> >> at org.junit.internal.builders.AnnotatedBuilder.buildRunner(
>> >> AnnotatedBuilder.java:104)
>> >> at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(
>> >> AnnotatedBuilder.java:86)
>> >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
>> >> RunnerBuilder.java:59)
>> >> at org.junit.internal.builders.AllDefaultPossibilitiesBuilder
>> >> .runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
>> >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
>> >> RunnerBuilder.java:59)
>> >> at org.junit.internal.requests.ClassRequest.getRunner(
>> >> ClassRequest.java:33)
>> >> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
>> >> JUnit4Provider.java:283)
>> >> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(
>> >> JUnit4Provider.java:173)
>> >> at org.apache.maven.surefire.junit4.JUnit4Provider.
>> >> executeTestSet(JUnit4Provider.java:153)
>> >> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
>> >> JUnit4Provider.java:128)
>> >> at org.apache.maven.surefire.booter.ForkedBooter.
>> >> invokeProviderInSameClassLoader(ForkedBooter.java:203)
>> >> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(
>> >> ForkedBooter.java:155)
>> >> at org.apache.maven.surefire.booter.ForkedBooter.main(
>> >> ForkedBooter.java:103)
>> >>
>> >> >
>> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> 3. For end-to-end tests, I encountered two kinds of errors
>> >> >> >>
>> >> >> >> a) connection refused.
>> >> >> >>
>> >> >> >> SET sync_ddl=False;
>> >> >> >> -- executing against localhost:21000
>> >> >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>> >> >> >>
>> >> >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
>> >> >> option:
>> >> >> >> {'disable_codegen': False, 'abort_on_error': 1,
>> >> 'exec_single_node_rows_
>> >> >> threshold':
>> >> >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
>> >> >> >> ___________________
>> >> >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> >> >> bin/../infra/python/env/bin/python
>> >> >> >> metadata/test_load.py:77: in setup_method
>> >> >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
>> >> >> >> util/hdfs_util.py:122: in copy
>> >> >> >> data = self.read_file(src)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> pywebhdfs/webhdfs.py:183:
>> >> >> >> in read_file
>> >> >> >> response = requests.get(uri, allow_redirects=True)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/api.py:69:
>> >> in
>> >> >> get
>> >> >> >> return request('get', url, params=params, **kwargs)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/api.py:50:
>> >> in
>> >> >> >> request
>> >> >> >> response = session.request(method=method, url=url, **kwargs)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> requests/sessions.py:465:
>> >> >> >> in request
>> >> >> >> resp = self.send(prep, **send_kwargs)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> requests/sessions.py:594:
>> >> >> >> in send
>> >> >> >> history = [resp for resp in gen] if allow_redirects else []
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> requests/sessions.py:196:
>> >> >> >> in resolve_redirects
>> >> >> >> **adapter_kwargs
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> requests/sessions.py:573:
>> >> >> >> in send
>> >> >> >> r = adapter.send(request, **kwargs)
>> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> >> requests/adapters.py:415:
>> >> >> >> in send
>> >> >> >> raise ConnectionError(err, request=request)
>> >> >> >> E ConnectionError: ('Connection aborted.', error(111,
>> 'Connection
>> >> >> >> refused'))
>> >> >> >>
>> >> >> >>
>> >> >> > The connection refused issue is very bizarre. One thing that I
>> >> noticed is
>> >> >> > that your Python does not seem to match what we use (Python 2.7.3).
>> >> >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if
>> you
>> >> get
>> >> >> > the expected version into infra/python/env/local/bin?
>> >> >> >
>> >> >> > Alternatively, maybe there's a problem with your /etc/hosts? You
>> can
>> >> try
>> >> >> > searching online for WebHdfs and /etc/hosts
>> >> >> >
>> >> >>
>> >> >> well, I find this 'find_py26.py' file under deps. Is it normal?
>> >> >>
>> >> >
>> >> > Yes, that's normal. That file looks Python 2.6 on your system but
>> should
>> >> > not be relevant for running tests because we use the Python from our
>> >> > virtualenv and not the one on your system.
>> >> >
>> >> > What's your output when you run "impala-python --version". You should
>> get
>> >> > 'Python 2.7.3".
>> >>
>> >> [amos@t450s tests]$ impala-python --version
>> >> Python 2.6.6
>> >>
>> >> > Also, what's the Python version on your system? Our virtualenv will
>> use
>> >> > Python 2.6 if your system has a Python < 2.6.
>> >> > You could try to upgrade your system Python and then
>> >> > re-run infra/python/bootstrap_virtualenv.py
>> >>
>> >> My system has python 2.6.6.
>> >>
>> >> >
>> >> > Still, theoretically Python 2.6 in the virtual env should work. I
>> think
>> >> > it's more likely you are having a connection problem due to a
>> >> misconfigured
>> >> > /etc/hosts.
>> >> >
>> >> > Are you running the test from a shell that has bin/impala-config.sh
>> and
>> >> > bin/set-classpath.sh sourced?
>> >> Yes.
>> >>
>> >> >
>> >> > To further debug this you could try to specify your namenode address
>> when
>> >> > running the test to see whether it is somehow picking up a wrong
>> address:
>> >> > cd tests
>> >> > ./run-tests.py metadata/test_load.py --namenode_http_address=
>> >> localhost:50070
>> >> >
>> >> > And see if that works.
>> >>
>> >> Unfortunately, no.
>> >>
>> >> >
>> >> >
>> >> >
>> >> >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
>> >> >> download_requirements find_py26.py pip_download.py
>> requirements.txt
>> >> >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
>> >> >> requirements
>> >> >> #!/bin/bash
>> >> >>
>> >> >> # Licensed to the Apache Software Foundation (ASF) under one
>> >> >> # or more contributor license agreements. See the NOTICE file
>> >> >> # distributed with this work for additional information
>> >> >> # regarding copyright ownership. The ASF licenses this file
>> >> >> # to you under the Apache License, Version 2.0 (the
>> >> >> # "License"); you may not use this file except in compliance
>> >> >> # with the License. You may obtain a copy of the License at
>> >> >> #
>> >> >> # http://www.apache.org/licenses/LICENSE-2.0
>> >> >> #
>> >> >> # Unless required by applicable law or agreed to in writing,
>> >> >> # software distributed under the License is distributed on an
>> >> >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> >> >> # KIND, either express or implied. See the License for the
>> >> >> # specific language governing permissions and limitations
>> >> >> # under the License.
>> >> >>
>> >> >> set -euo pipefail
>> >> >>
>> >> >> DIR="$(dirname "$0")"
>> >> >>
>> >> >> pushd "$DIR"
>> >> >> PY26="$(./find_py26.py)"
>> >> >> # Directly download packages listed in requirements.txt, but don't
>> >> install
>> >> >> them.
>> >> >> "$PY26" pip_download.py
>> >> >> # For virtualenv, other scripts rely on the .tar.gz package (not a
>> .whl
>> >> >> package).
>> >> >> "$PY26" pip_download.py virtualenv 13.1.0
>> >> >> # kudu-python is downloaded separately because pip install attempts
>> to
>> >> >> execute a
>> >> >> # setup.py subcommand for kudu-python that can fail even if the
>> download
>> >> >> succeeds.
>> >> >> "$PY26" pip_download.py kudu-python 0.1.1
>> >> >> popd
>> >> >>
>> >> >>
>> >> >>
>> >> >> > b) stats not match
>> >> >> >>
>> >> >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> >> >> bin/../infra/python/env/bin/python
>> >> >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
>> >> >> >> self.run_test_case('QueryTest/show-stats', vector,
>> "functional")
>> >> >> >> common/impala_test_suite.py:342: in run_test_case
>> >> >> >> self.__verify_results_and_errors(vector, test_section,
>> result,
>> >> >> use_db)
>> >> >> >> common/impala_test_suite.py:234: in __verify_results_and_errors
>> >> >> >> replace_filenames_with_placeholder)
>> >> >> >> common/test_result_verifier.py:398: in verify_raw_results
>> >> >> >> VERIFIER_MAP[verifier](expected, actual)
>> >> >> >> common/test_result_verifier.py:231: in
>> verify_query_result_is_equal
>> >> >> >> assert expected_results == actual_results
>> >> >> >>
>> >> >> >> ...
>> >> >> >>
>> >> >> >> -- executing against localhost:21000
>> >> >> >> show column stats alltypes_clone;
>> >> >> >>
>> >> >> >> MainThread: Comparing QueryTestResults (expected vs actual):
>> >> >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,
>> 8
>> >> >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
>> >> >> >> 'date_string_col','STRING',736,-1,8,8 ==
>> 'date_string_col','STRING',
>> >> >> >> 736,-1,8,8
>> >> >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,
>> 8
>> >> >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
>> >> >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
>> >> >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
>> >> >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
>> >> >> >> 'smallint_col','SMALLINT',10,-1,2,2 ==
>> >> 'smallint_col','SMALLINT',10,-
>> >> >> 1,2,2
>> >> >> >> 'string_col','STRING',10,-1,-1,-1 ==
>> 'string_col','STRING',10,-1,-
>> >> 1,-1
>> >> >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
>> >> >> 'timestamp_col','TIMESTAMP',
>> >> >> >> 7552,-1,16,16
>> >> >> >> 'tinyint_col','TINYINT',10,-1,1,1 ==
>> 'tinyint_col','TINYINT',10,-1,
>> >> 1,1
>> >> >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
>> >> >> >>
>> >> >> >>
>> >> >> >> Very strange. Can you do a compute stats on functional.alltypes
>> and
>> >> >> > confirm that the NDV for timestamp_col are 7552 in your setup?
>> >> >>
>> >> >> Yes.
>> >> >>
>> >> >
>> >> > I'll need to ask around for help. I have no idea why this is
>> happening.
>> >>
>> >> Thanks :)
>> >>
>> >> >
>> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any
>> help
>> >> is
>> >> >> >> much appreciated!
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >>
>> >> >> >> Best regards,
>> >> >> >> Amos
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>
Re: test errors in local env
Posted by Lars Volker <lv...@cloudera.com>.
Hi Amos,
apologies for the delay? Is this issue still open? If so, can you try to
run the tests with Java 7?
Thanks, Lars
On Thu, Sep 22, 2016 at 6:38 PM, Amos Bird <am...@gmail.com> wrote:
> Thanks Lars,
>
> My Java version is
>
> $ $JAVA_HOME/bin/java -version
> java version "1.8.0_65"
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
>
> > Hi Amos,
> >
> > I tried to see if IMPALA-3643
> > <https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible
> for
> > me. In the process I also ran into the "NoClassDefFoundError" error you
> saw
> > and found out, that this happens if my local hadoop services
> > (./testdata/bin/run-all.sh) isn't running.
> >
> > Once I have the local hadoop services running I'm able to repro the issue
> > in IMPALA-3643. The only thing I remember doing differently from the
> > recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
> > LinkedHashSet
> > <https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html>
> don't
> > read like they'd change the ordering, however it doesn't say that the
> > ordering has to be stable *between different versions of Java*. I will
> try
> > to confirm that this issue does not repro with Java 7 when I have time.
> Can
> > you check the Java version you're running. Mine is this:
> >
> > $ $JAVA_HOME/bin/java -version
> > java version "1.8.0_101"
> > Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
> >
> > Alex, could a difference in the implementation between Java 7 and 8 be a
> > reasonable explanation for this?
> >
> > Cheers, Lars
> >
> > On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <am...@gmail.com> wrote:
> >
> >>
> >> Thank you for helping me out :D
> >>
> >> > Thanks for keeping this discussion going.
> >> >
> >> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com>
> wrote:
> >> >
> >> >>
> >> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com>
> >> wrote:
> >> >> >
> >> >> >>
> >> >> >> Hi there,
> >> >> >>
> >> >> >> I followed the wiki
> >> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> >> >> to+load+and+run+Impala+tests
> >> >> >> carefully but still have some problems in my local env.
> >> >> >>
> >> >> >> 1. I need to manually execute "hdfs dfs -mkdir
> >> >> /test-warehouse/emptytable"
> >> >> >> to get rid of some fe test error.
> >> >> >>
> >> >> >>
> >> >> > Ideally, you should not have to do this. Could you tell me what
> errors
> >> >> you
> >> >> > encountered? Sounds like there may be a test or data loading bug we
> >> >> should
> >> >> > fix.
> >> >>
> >> >> The error is :
> >> >>
> >> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
> >> >> elapsed: 0.033 sec <<< FAILURE!
> >> >> java.lang.AssertionError: got error:
> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> does
> >> >> not exist.
> >> >> expected:
> >> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> >> >> contains no visible files.
> >> >> at org.junit.Assert.fail(Assert.java:88)
> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> >> FrontendTestBase.java:312)
> >> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> >> FrontendTestBase.java:292)
> >> >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> >> >> AnalyzeStmtsTest.java:2860)
> >> >>
> >> >>
> >> > Do you have a table functional.emptytable? If yes, then what location
> is
> >> > reported in "show create table"?
> >> Query: show create table functional.emptytable
> >> +-------------------------------------------------------------+
> >> | result |
> >> +-------------------------------------------------------------+
> >> | CREATE EXTERNAL TABLE functional.emptytable ( |
> >> | field STRING |
> >> | ) |
> >> | PARTITIONED BY ( |
> >> | f2 INT |
> >> | ) |
> >> | STORED AS TEXTFILE |
> >> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
> >> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') |
> >> +-------------------------------------------------------------+
> >> Fetched 1 row(s) in 5.51s
> >>
> >> > Does the directory exist in HDFS?
> >> No.
> >>
> >> >
> >> > You could try to manually reload the table and see if the directory is
> >> > created:
> >> > bin/load-data.py -f -w functional-query --table_names=emptytable
> >> > --table_formats=text/none
> >> After executing this command the directory appears.
> >>
> >> >
> >> >>>
> >> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
> >> >> errors.
> >> >> >>
> >> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest)
> Time
> >> >> >> elapsed: 0.333 sec <<< FAILURE!
> >> >> >> java.lang.AssertionError: got error:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> default.nodb
> >> >> >> expected:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> nodb.alltypes
> >> >> >> at org.junit.Assert.fail(Assert.java:88)
> >> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2220)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2203)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2197)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> >> >> AuthorizationTest.java:512)
> >> >> >>
> >> >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest)
> Time
> >> >> >> elapsed: 0.324 sec <<< FAILURE!
> >> >> >> java.lang.AssertionError: got error:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> default.nodb
> >> >> >> expected:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> nodb.alltypes
> >> >> >> at org.junit.Assert.fail(Assert.java:88)
> >> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2220)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2203)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> >> AuthorizationTest.java:2197)
> >> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> >> >> AuthorizationTest.java:512)
> >> >> >>
> >> >> >>
> >> >> >> Results :
> >> >> >>
> >> >> >> Failed tests:
> >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> >> >> AuthzError:2203->AuthzError:2220 got error:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> default.nodb
> >> >> >> expected:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> nodb.alltypes
> >> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> >> >> AuthzError:2203->AuthzError:2220 got error:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> default.nodb
> >> >> >> expected:
> >> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> >> nodb.alltypes
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> > Strange. In this test, we register two authorization requests, and
> it
> >> >> seems
> >> >> > like those are not checked in the expected order. However, that
> should
> >> >> not
> >> >> > be possible because we store them in a LinkedHashSet.
> >> >> > Could you dig into this a little further to see if you can figure
> out
> >> why
> >> >> > the order is wrong?
> >> >> >
> >> >> > This is where we register the authorization requests:
> >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> >> >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
> >> >> >
> >> >> > This is where we check the authorization requests:
> >> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> >> >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
> >> >> >
> >> >> >
> >> >>
> >> >> I tried directly executing "select 1 from nodb.alltypes" in
> >> >> impala-shell, leading to this error:
> >> >> ERROR: AnalysisException: Could not resolve table reference:
> >> >> 'nodb.alltypes'
> >> >>
> >> >> How can I reproduce the authorization tests in impala-shell so I can
> >> >> debug it?
> >> >>
> >> >>
> >> >>
> >> > FYI, this is actually a known issue and may have something to do with
> the
> >> > JRE version you are running: https://issues.cloudera.org/
> >> browse/IMPALA-3643
> >> > As far as I can tell the bug should be "impossible" because we use a
> >> > LinkedHashSet, but maybe certain JREs do not properly honor the
> >> guarantees.
> >> >
> >> > The AuthorizationTests in particular require a non-trivial setup, so
> I'd
> >> > not recommend trying to debug via the Impala shell.
> >> >
> >> > I'd recommend debugging in one of these ways:
> >> > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from
> the
> >> FE
> >> > directory. Attach debugger and break in TestSelect().
> >> > - Run the JUnit test from an IDE such as Eclipse and then debug the
> test.
> >> > I'm afraid there is no easy way to just run that single query in our
> >> > current test setup. You will need to run the whole suite, but you can
> >> break
> >> > TestSelect() or hack the code in various places to set useful
> >> breakpoints.
> >> >
> >> > Hope that helps.
> >>
> >> I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest
> >> fails with ClassNoDefine.
> >> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008
> sec
> >> <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest
> >> initializationError(com.cloudera.impala.analysis.AuthorizationTest)
> Time
> >> elapsed: 0.006 sec <<< ERROR!
> >> java.lang.NoClassDefFoundError: Could not initialize class
> >> com.cloudera.impala.analysis.AuthorizationTest
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke(
> >> NativeMethodAccessorImpl.java:62)
> >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >> DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> >> FrameworkMethod.java:50)
> >> at org.junit.internal.runners.model.ReflectiveCallable.run(
> >> ReflectiveCallable.java:12)
> >> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> >> FrameworkMethod.java:47)
> >> at org.junit.runners.Parameterized.allParameters(
> Parameterized.java:280)
> >> at org.junit.runners.Parameterized.<init>(Parameterized.java:248)
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> >> NativeConstructorAccessorImpl.java:62)
> >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> >> DelegatingConstructorAccessorImpl.java:45)
> >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> >> at org.junit.internal.builders.AnnotatedBuilder.buildRunner(
> >> AnnotatedBuilder.java:104)
> >> at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(
> >> AnnotatedBuilder.java:86)
> >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
> >> RunnerBuilder.java:59)
> >> at org.junit.internal.builders.AllDefaultPossibilitiesBuilder
> >> .runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> >> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
> >> RunnerBuilder.java:59)
> >> at org.junit.internal.requests.ClassRequest.getRunner(
> >> ClassRequest.java:33)
> >> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
> >> JUnit4Provider.java:283)
> >> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(
> >> JUnit4Provider.java:173)
> >> at org.apache.maven.surefire.junit4.JUnit4Provider.
> >> executeTestSet(JUnit4Provider.java:153)
> >> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
> >> JUnit4Provider.java:128)
> >> at org.apache.maven.surefire.booter.ForkedBooter.
> >> invokeProviderInSameClassLoader(ForkedBooter.java:203)
> >> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(
> >> ForkedBooter.java:155)
> >> at org.apache.maven.surefire.booter.ForkedBooter.main(
> >> ForkedBooter.java:103)
> >>
> >> >
> >> >>
> >> >> >>
> >> >> >>
> >> >> >> 3. For end-to-end tests, I encountered two kinds of errors
> >> >> >>
> >> >> >> a) connection refused.
> >> >> >>
> >> >> >> SET sync_ddl=False;
> >> >> >> -- executing against localhost:21000
> >> >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
> >> >> >>
> >> >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
> >> >> option:
> >> >> >> {'disable_codegen': False, 'abort_on_error': 1,
> >> 'exec_single_node_rows_
> >> >> threshold':
> >> >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
> >> >> >> ___________________
> >> >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> >> >> bin/../infra/python/env/bin/python
> >> >> >> metadata/test_load.py:77: in setup_method
> >> >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
> >> >> >> util/hdfs_util.py:122: in copy
> >> >> >> data = self.read_file(src)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> pywebhdfs/webhdfs.py:183:
> >> >> >> in read_file
> >> >> >> response = requests.get(uri, allow_redirects=True)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/api.py:69:
> >> in
> >> >> get
> >> >> >> return request('get', url, params=params, **kwargs)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/api.py:50:
> >> in
> >> >> >> request
> >> >> >> response = session.request(method=method, url=url, **kwargs)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> requests/sessions.py:465:
> >> >> >> in request
> >> >> >> resp = self.send(prep, **send_kwargs)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> requests/sessions.py:594:
> >> >> >> in send
> >> >> >> history = [resp for resp in gen] if allow_redirects else []
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> requests/sessions.py:196:
> >> >> >> in resolve_redirects
> >> >> >> **adapter_kwargs
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> requests/sessions.py:573:
> >> >> >> in send
> >> >> >> r = adapter.send(request, **kwargs)
> >> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> >> requests/adapters.py:415:
> >> >> >> in send
> >> >> >> raise ConnectionError(err, request=request)
> >> >> >> E ConnectionError: ('Connection aborted.', error(111,
> 'Connection
> >> >> >> refused'))
> >> >> >>
> >> >> >>
> >> >> > The connection refused issue is very bizarre. One thing that I
> >> noticed is
> >> >> > that your Python does not seem to match what we use (Python 2.7.3).
> >> >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if
> you
> >> get
> >> >> > the expected version into infra/python/env/local/bin?
> >> >> >
> >> >> > Alternatively, maybe there's a problem with your /etc/hosts? You
> can
> >> try
> >> >> > searching online for WebHdfs and /etc/hosts
> >> >> >
> >> >>
> >> >> well, I find this 'find_py26.py' file under deps. Is it normal?
> >> >>
> >> >
> >> > Yes, that's normal. That file looks Python 2.6 on your system but
> should
> >> > not be relevant for running tests because we use the Python from our
> >> > virtualenv and not the one on your system.
> >> >
> >> > What's your output when you run "impala-python --version". You should
> get
> >> > 'Python 2.7.3".
> >>
> >> [amos@t450s tests]$ impala-python --version
> >> Python 2.6.6
> >>
> >> > Also, what's the Python version on your system? Our virtualenv will
> use
> >> > Python 2.6 if your system has a Python < 2.6.
> >> > You could try to upgrade your system Python and then
> >> > re-run infra/python/bootstrap_virtualenv.py
> >>
> >> My system has python 2.6.6.
> >>
> >> >
> >> > Still, theoretically Python 2.6 in the virtual env should work. I
> think
> >> > it's more likely you are having a connection problem due to a
> >> misconfigured
> >> > /etc/hosts.
> >> >
> >> > Are you running the test from a shell that has bin/impala-config.sh
> and
> >> > bin/set-classpath.sh sourced?
> >> Yes.
> >>
> >> >
> >> > To further debug this you could try to specify your namenode address
> when
> >> > running the test to see whether it is somehow picking up a wrong
> address:
> >> > cd tests
> >> > ./run-tests.py metadata/test_load.py --namenode_http_address=
> >> localhost:50070
> >> >
> >> > And see if that works.
> >>
> >> Unfortunately, no.
> >>
> >> >
> >> >
> >> >
> >> >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
> >> >> download_requirements find_py26.py pip_download.py
> requirements.txt
> >> >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
> >> >> requirements
> >> >> #!/bin/bash
> >> >>
> >> >> # Licensed to the Apache Software Foundation (ASF) under one
> >> >> # or more contributor license agreements. See the NOTICE file
> >> >> # distributed with this work for additional information
> >> >> # regarding copyright ownership. The ASF licenses this file
> >> >> # to you under the Apache License, Version 2.0 (the
> >> >> # "License"); you may not use this file except in compliance
> >> >> # with the License. You may obtain a copy of the License at
> >> >> #
> >> >> # http://www.apache.org/licenses/LICENSE-2.0
> >> >> #
> >> >> # Unless required by applicable law or agreed to in writing,
> >> >> # software distributed under the License is distributed on an
> >> >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> >> >> # KIND, either express or implied. See the License for the
> >> >> # specific language governing permissions and limitations
> >> >> # under the License.
> >> >>
> >> >> set -euo pipefail
> >> >>
> >> >> DIR="$(dirname "$0")"
> >> >>
> >> >> pushd "$DIR"
> >> >> PY26="$(./find_py26.py)"
> >> >> # Directly download packages listed in requirements.txt, but don't
> >> install
> >> >> them.
> >> >> "$PY26" pip_download.py
> >> >> # For virtualenv, other scripts rely on the .tar.gz package (not a
> .whl
> >> >> package).
> >> >> "$PY26" pip_download.py virtualenv 13.1.0
> >> >> # kudu-python is downloaded separately because pip install attempts
> to
> >> >> execute a
> >> >> # setup.py subcommand for kudu-python that can fail even if the
> download
> >> >> succeeds.
> >> >> "$PY26" pip_download.py kudu-python 0.1.1
> >> >> popd
> >> >>
> >> >>
> >> >>
> >> >> > b) stats not match
> >> >> >>
> >> >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> >> >> bin/../infra/python/env/bin/python
> >> >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
> >> >> >> self.run_test_case('QueryTest/show-stats', vector,
> "functional")
> >> >> >> common/impala_test_suite.py:342: in run_test_case
> >> >> >> self.__verify_results_and_errors(vector, test_section,
> result,
> >> >> use_db)
> >> >> >> common/impala_test_suite.py:234: in __verify_results_and_errors
> >> >> >> replace_filenames_with_placeholder)
> >> >> >> common/test_result_verifier.py:398: in verify_raw_results
> >> >> >> VERIFIER_MAP[verifier](expected, actual)
> >> >> >> common/test_result_verifier.py:231: in
> verify_query_result_is_equal
> >> >> >> assert expected_results == actual_results
> >> >> >>
> >> >> >> ...
> >> >> >>
> >> >> >> -- executing against localhost:21000
> >> >> >> show column stats alltypes_clone;
> >> >> >>
> >> >> >> MainThread: Comparing QueryTestResults (expected vs actual):
> >> >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,
> 8
> >> >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
> >> >> >> 'date_string_col','STRING',736,-1,8,8 ==
> 'date_string_col','STRING',
> >> >> >> 736,-1,8,8
> >> >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,
> 8
> >> >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
> >> >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
> >> >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
> >> >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
> >> >> >> 'smallint_col','SMALLINT',10,-1,2,2 ==
> >> 'smallint_col','SMALLINT',10,-
> >> >> 1,2,2
> >> >> >> 'string_col','STRING',10,-1,-1,-1 ==
> 'string_col','STRING',10,-1,-
> >> 1,-1
> >> >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
> >> >> 'timestamp_col','TIMESTAMP',
> >> >> >> 7552,-1,16,16
> >> >> >> 'tinyint_col','TINYINT',10,-1,1,1 ==
> 'tinyint_col','TINYINT',10,-1,
> >> 1,1
> >> >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
> >> >> >>
> >> >> >>
> >> >> >> Very strange. Can you do a compute stats on functional.alltypes
> and
> >> >> > confirm that the NDV for timestamp_col are 7552 in your setup?
> >> >>
> >> >> Yes.
> >> >>
> >> >
> >> > I'll need to ask around for help. I have no idea why this is
> happening.
> >>
> >> Thanks :)
> >>
> >> >
> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any
> help
> >> is
> >> >> >> much appreciated!
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >>
> >> >> >> Best regards,
> >> >> >> Amos
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>
Re: test errors in local env
Posted by Amos Bird <am...@gmail.com>.
Thanks Lars,
My Java version is
$ $JAVA_HOME/bin/java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
> Hi Amos,
>
> I tried to see if IMPALA-3643
> <https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible for
> me. In the process I also ran into the "NoClassDefFoundError" error you saw
> and found out, that this happens if my local hadoop services
> (./testdata/bin/run-all.sh) isn't running.
>
> Once I have the local hadoop services running I'm able to repro the issue
> in IMPALA-3643. The only thing I remember doing differently from the
> recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
> LinkedHashSet
> <https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html> don't
> read like they'd change the ordering, however it doesn't say that the
> ordering has to be stable *between different versions of Java*. I will try
> to confirm that this issue does not repro with Java 7 when I have time. Can
> you check the Java version you're running. Mine is this:
>
> $ $JAVA_HOME/bin/java -version
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
>
> Alex, could a difference in the implementation between Java 7 and 8 be a
> reasonable explanation for this?
>
> Cheers, Lars
>
> On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <am...@gmail.com> wrote:
>
>>
>> Thank you for helping me out :D
>>
>> > Thanks for keeping this discussion going.
>> >
>> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com> wrote:
>> >
>> >>
>> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com>
>> wrote:
>> >> >
>> >> >>
>> >> >> Hi there,
>> >> >>
>> >> >> I followed the wiki
>> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> >> to+load+and+run+Impala+tests
>> >> >> carefully but still have some problems in my local env.
>> >> >>
>> >> >> 1. I need to manually execute "hdfs dfs -mkdir
>> >> /test-warehouse/emptytable"
>> >> >> to get rid of some fe test error.
>> >> >>
>> >> >>
>> >> > Ideally, you should not have to do this. Could you tell me what errors
>> >> you
>> >> > encountered? Sounds like there may be a test or data loading bug we
>> >> should
>> >> > fix.
>> >>
>> >> The error is :
>> >>
>> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
>> >> elapsed: 0.033 sec <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
>> >> not exist.
>> >> expected:
>> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> >> contains no visible files.
>> >> at org.junit.Assert.fail(Assert.java:88)
>> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> FrontendTestBase.java:312)
>> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> >> FrontendTestBase.java:292)
>> >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> >> AnalyzeStmtsTest.java:2860)
>> >>
>> >>
>> > Do you have a table functional.emptytable? If yes, then what location is
>> > reported in "show create table"?
>> Query: show create table functional.emptytable
>> +-------------------------------------------------------------+
>> | result |
>> +-------------------------------------------------------------+
>> | CREATE EXTERNAL TABLE functional.emptytable ( |
>> | field STRING |
>> | ) |
>> | PARTITIONED BY ( |
>> | f2 INT |
>> | ) |
>> | STORED AS TEXTFILE |
>> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
>> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') |
>> +-------------------------------------------------------------+
>> Fetched 1 row(s) in 5.51s
>>
>> > Does the directory exist in HDFS?
>> No.
>>
>> >
>> > You could try to manually reload the table and see if the directory is
>> > created:
>> > bin/load-data.py -f -w functional-query --table_names=emptytable
>> > --table_formats=text/none
>> After executing this command the directory appears.
>>
>> >
>> >>>
>> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
>> >> errors.
>> >> >>
>> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
>> >> >> elapsed: 0.333 sec <<< FAILURE!
>> >> >> java.lang.AssertionError: got error:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> default.nodb
>> >> >> expected:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> nodb.alltypes
>> >> >> at org.junit.Assert.fail(Assert.java:88)
>> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2220)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2203)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2197)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> >> AuthorizationTest.java:512)
>> >> >>
>> >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
>> >> >> elapsed: 0.324 sec <<< FAILURE!
>> >> >> java.lang.AssertionError: got error:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> default.nodb
>> >> >> expected:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> nodb.alltypes
>> >> >> at org.junit.Assert.fail(Assert.java:88)
>> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2220)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2203)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> >> AuthorizationTest.java:2197)
>> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> >> AuthorizationTest.java:512)
>> >> >>
>> >> >>
>> >> >> Results :
>> >> >>
>> >> >> Failed tests:
>> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> >> AuthzError:2203->AuthzError:2220 got error:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> default.nodb
>> >> >> expected:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> nodb.alltypes
>> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> >> AuthzError:2203->AuthzError:2220 got error:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> default.nodb
>> >> >> expected:
>> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> >> nodb.alltypes
>> >> >>
>> >> >>
>> >> >>
>> >> > Strange. In this test, we register two authorization requests, and it
>> >> seems
>> >> > like those are not checked in the expected order. However, that should
>> >> not
>> >> > be possible because we store them in a LinkedHashSet.
>> >> > Could you dig into this a little further to see if you can figure out
>> why
>> >> > the order is wrong?
>> >> >
>> >> > This is where we register the authorization requests:
>> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
>> >> >
>> >> > This is where we check the authorization requests:
>> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>> >> >
>> >> >
>> >>
>> >> I tried directly executing "select 1 from nodb.alltypes" in
>> >> impala-shell, leading to this error:
>> >> ERROR: AnalysisException: Could not resolve table reference:
>> >> 'nodb.alltypes'
>> >>
>> >> How can I reproduce the authorization tests in impala-shell so I can
>> >> debug it?
>> >>
>> >>
>> >>
>> > FYI, this is actually a known issue and may have something to do with the
>> > JRE version you are running: https://issues.cloudera.org/
>> browse/IMPALA-3643
>> > As far as I can tell the bug should be "impossible" because we use a
>> > LinkedHashSet, but maybe certain JREs do not properly honor the
>> guarantees.
>> >
>> > The AuthorizationTests in particular require a non-trivial setup, so I'd
>> > not recommend trying to debug via the Impala shell.
>> >
>> > I'd recommend debugging in one of these ways:
>> > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from the
>> FE
>> > directory. Attach debugger and break in TestSelect().
>> > - Run the JUnit test from an IDE such as Eclipse and then debug the test.
>> > I'm afraid there is no easy way to just run that single query in our
>> > current test setup. You will need to run the whole suite, but you can
>> break
>> > TestSelect() or hack the code in various places to set useful
>> breakpoints.
>> >
>> > Hope that helps.
>>
>> I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest
>> fails with ClassNoDefine.
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec
>> <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest
>> initializationError(com.cloudera.impala.analysis.AuthorizationTest) Time
>> elapsed: 0.006 sec <<< ERROR!
>> java.lang.NoClassDefFoundError: Could not initialize class
>> com.cloudera.impala.analysis.AuthorizationTest
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>> FrameworkMethod.java:50)
>> at org.junit.internal.runners.model.ReflectiveCallable.run(
>> ReflectiveCallable.java:12)
>> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
>> FrameworkMethod.java:47)
>> at org.junit.runners.Parameterized.allParameters(Parameterized.java:280)
>> at org.junit.runners.Parameterized.<init>(Parameterized.java:248)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>> NativeConstructorAccessorImpl.java:62)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>> DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> at org.junit.internal.builders.AnnotatedBuilder.buildRunner(
>> AnnotatedBuilder.java:104)
>> at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(
>> AnnotatedBuilder.java:86)
>> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
>> RunnerBuilder.java:59)
>> at org.junit.internal.builders.AllDefaultPossibilitiesBuilder
>> .runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
>> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
>> RunnerBuilder.java:59)
>> at org.junit.internal.requests.ClassRequest.getRunner(
>> ClassRequest.java:33)
>> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
>> JUnit4Provider.java:283)
>> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(
>> JUnit4Provider.java:173)
>> at org.apache.maven.surefire.junit4.JUnit4Provider.
>> executeTestSet(JUnit4Provider.java:153)
>> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
>> JUnit4Provider.java:128)
>> at org.apache.maven.surefire.booter.ForkedBooter.
>> invokeProviderInSameClassLoader(ForkedBooter.java:203)
>> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(
>> ForkedBooter.java:155)
>> at org.apache.maven.surefire.booter.ForkedBooter.main(
>> ForkedBooter.java:103)
>>
>> >
>> >>
>> >> >>
>> >> >>
>> >> >> 3. For end-to-end tests, I encountered two kinds of errors
>> >> >>
>> >> >> a) connection refused.
>> >> >>
>> >> >> SET sync_ddl=False;
>> >> >> -- executing against localhost:21000
>> >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>> >> >>
>> >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
>> >> option:
>> >> >> {'disable_codegen': False, 'abort_on_error': 1,
>> 'exec_single_node_rows_
>> >> threshold':
>> >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
>> >> >> ___________________
>> >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> >> bin/../infra/python/env/bin/python
>> >> >> metadata/test_load.py:77: in setup_method
>> >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
>> >> >> util/hdfs_util.py:122: in copy
>> >> >> data = self.read_file(src)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> pywebhdfs/webhdfs.py:183:
>> >> >> in read_file
>> >> >> response = requests.get(uri, allow_redirects=True)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69:
>> in
>> >> get
>> >> >> return request('get', url, params=params, **kwargs)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50:
>> in
>> >> >> request
>> >> >> response = session.request(method=method, url=url, **kwargs)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> requests/sessions.py:465:
>> >> >> in request
>> >> >> resp = self.send(prep, **send_kwargs)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> requests/sessions.py:594:
>> >> >> in send
>> >> >> history = [resp for resp in gen] if allow_redirects else []
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> requests/sessions.py:196:
>> >> >> in resolve_redirects
>> >> >> **adapter_kwargs
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> requests/sessions.py:573:
>> >> >> in send
>> >> >> r = adapter.send(request, **kwargs)
>> >> >> ../infra/python/env/lib/python2.6/site-packages/
>> >> requests/adapters.py:415:
>> >> >> in send
>> >> >> raise ConnectionError(err, request=request)
>> >> >> E ConnectionError: ('Connection aborted.', error(111, 'Connection
>> >> >> refused'))
>> >> >>
>> >> >>
>> >> > The connection refused issue is very bizarre. One thing that I
>> noticed is
>> >> > that your Python does not seem to match what we use (Python 2.7.3).
>> >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if you
>> get
>> >> > the expected version into infra/python/env/local/bin?
>> >> >
>> >> > Alternatively, maybe there's a problem with your /etc/hosts? You can
>> try
>> >> > searching online for WebHdfs and /etc/hosts
>> >> >
>> >>
>> >> well, I find this 'find_py26.py' file under deps. Is it normal?
>> >>
>> >
>> > Yes, that's normal. That file looks Python 2.6 on your system but should
>> > not be relevant for running tests because we use the Python from our
>> > virtualenv and not the one on your system.
>> >
>> > What's your output when you run "impala-python --version". You should get
>> > 'Python 2.7.3".
>>
>> [amos@t450s tests]$ impala-python --version
>> Python 2.6.6
>>
>> > Also, what's the Python version on your system? Our virtualenv will use
>> > Python 2.6 if your system has a Python < 2.6.
>> > You could try to upgrade your system Python and then
>> > re-run infra/python/bootstrap_virtualenv.py
>>
>> My system has python 2.6.6.
>>
>> >
>> > Still, theoretically Python 2.6 in the virtual env should work. I think
>> > it's more likely you are having a connection problem due to a
>> misconfigured
>> > /etc/hosts.
>> >
>> > Are you running the test from a shell that has bin/impala-config.sh and
>> > bin/set-classpath.sh sourced?
>> Yes.
>>
>> >
>> > To further debug this you could try to specify your namenode address when
>> > running the test to see whether it is somehow picking up a wrong address:
>> > cd tests
>> > ./run-tests.py metadata/test_load.py --namenode_http_address=
>> localhost:50070
>> >
>> > And see if that works.
>>
>> Unfortunately, no.
>>
>> >
>> >
>> >
>> >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
>> >> download_requirements find_py26.py pip_download.py requirements.txt
>> >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
>> >> requirements
>> >> #!/bin/bash
>> >>
>> >> # Licensed to the Apache Software Foundation (ASF) under one
>> >> # or more contributor license agreements. See the NOTICE file
>> >> # distributed with this work for additional information
>> >> # regarding copyright ownership. The ASF licenses this file
>> >> # to you under the Apache License, Version 2.0 (the
>> >> # "License"); you may not use this file except in compliance
>> >> # with the License. You may obtain a copy of the License at
>> >> #
>> >> # http://www.apache.org/licenses/LICENSE-2.0
>> >> #
>> >> # Unless required by applicable law or agreed to in writing,
>> >> # software distributed under the License is distributed on an
>> >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> >> # KIND, either express or implied. See the License for the
>> >> # specific language governing permissions and limitations
>> >> # under the License.
>> >>
>> >> set -euo pipefail
>> >>
>> >> DIR="$(dirname "$0")"
>> >>
>> >> pushd "$DIR"
>> >> PY26="$(./find_py26.py)"
>> >> # Directly download packages listed in requirements.txt, but don't
>> install
>> >> them.
>> >> "$PY26" pip_download.py
>> >> # For virtualenv, other scripts rely on the .tar.gz package (not a .whl
>> >> package).
>> >> "$PY26" pip_download.py virtualenv 13.1.0
>> >> # kudu-python is downloaded separately because pip install attempts to
>> >> execute a
>> >> # setup.py subcommand for kudu-python that can fail even if the download
>> >> succeeds.
>> >> "$PY26" pip_download.py kudu-python 0.1.1
>> >> popd
>> >>
>> >>
>> >>
>> >> > b) stats not match
>> >> >>
>> >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> >> bin/../infra/python/env/bin/python
>> >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
>> >> >> self.run_test_case('QueryTest/show-stats', vector, "functional")
>> >> >> common/impala_test_suite.py:342: in run_test_case
>> >> >> self.__verify_results_and_errors(vector, test_section, result,
>> >> use_db)
>> >> >> common/impala_test_suite.py:234: in __verify_results_and_errors
>> >> >> replace_filenames_with_placeholder)
>> >> >> common/test_result_verifier.py:398: in verify_raw_results
>> >> >> VERIFIER_MAP[verifier](expected, actual)
>> >> >> common/test_result_verifier.py:231: in verify_query_result_is_equal
>> >> >> assert expected_results == actual_results
>> >> >>
>> >> >> ...
>> >> >>
>> >> >> -- executing against localhost:21000
>> >> >> show column stats alltypes_clone;
>> >> >>
>> >> >> MainThread: Comparing QueryTestResults (expected vs actual):
>> >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
>> >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
>> >> >> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
>> >> >> 736,-1,8,8
>> >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
>> >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
>> >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
>> >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
>> >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
>> >> >> 'smallint_col','SMALLINT',10,-1,2,2 ==
>> 'smallint_col','SMALLINT',10,-
>> >> 1,2,2
>> >> >> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-
>> 1,-1
>> >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
>> >> 'timestamp_col','TIMESTAMP',
>> >> >> 7552,-1,16,16
>> >> >> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,
>> 1,1
>> >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
>> >> >>
>> >> >>
>> >> >> Very strange. Can you do a compute stats on functional.alltypes and
>> >> > confirm that the NDV for timestamp_col are 7552 in your setup?
>> >>
>> >> Yes.
>> >>
>> >
>> > I'll need to ask around for help. I have no idea why this is happening.
>>
>> Thanks :)
>>
>> >
>> >>
>> >> >
>> >> >
>> >> >
>> >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help
>> is
>> >> >> much appreciated!
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >> Best regards,
>> >> >> Amos
>> >> >>
>> >>
>> >>
>>
>>
Re: test errors in local env
Posted by Lars Volker <lv...@cloudera.com>.
Hi Amos,
I tried to see if IMPALA-3643
<https://issues.cloudera.org/browse/IMPALA-3643> is still reproducible for
me. In the process I also ran into the "NoClassDefFoundError" error you saw
and found out, that this happens if my local hadoop services
(./testdata/bin/run-all.sh) isn't running.
Once I have the local hadoop services running I'm able to repro the issue
in IMPALA-3643. The only thing I remember doing differently from the
recommended ways of dev-machine setup is using Java 8. The Java 8 docs on
LinkedHashSet
<https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html> don't
read like they'd change the ordering, however it doesn't say that the
ordering has to be stable *between different versions of Java*. I will try
to confirm that this issue does not repro with Java 7 when I have time. Can
you check the Java version you're running. Mine is this:
$ $JAVA_HOME/bin/java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
Alex, could a difference in the implementation between Java 7 and 8 be a
reasonable explanation for this?
Cheers, Lars
On Thu, Sep 22, 2016 at 12:52 AM, Amos Bird <am...@gmail.com> wrote:
>
> Thank you for helping me out :D
>
> > Thanks for keeping this discussion going.
> >
> > On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com> wrote:
> >
> >>
> >> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com>
> wrote:
> >> >
> >> >>
> >> >> Hi there,
> >> >>
> >> >> I followed the wiki
> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> >> to+load+and+run+Impala+tests
> >> >> carefully but still have some problems in my local env.
> >> >>
> >> >> 1. I need to manually execute "hdfs dfs -mkdir
> >> /test-warehouse/emptytable"
> >> >> to get rid of some fe test error.
> >> >>
> >> >>
> >> > Ideally, you should not have to do this. Could you tell me what errors
> >> you
> >> > encountered? Sounds like there may be a test or data loading bug we
> >> should
> >> > fix.
> >>
> >> The error is :
> >>
> >> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
> >> elapsed: 0.033 sec <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
> >> not exist.
> >> expected:
> >> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> >> contains no visible files.
> >> at org.junit.Assert.fail(Assert.java:88)
> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> FrontendTestBase.java:312)
> >> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> >> FrontendTestBase.java:292)
> >> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> >> AnalyzeStmtsTest.java:2860)
> >>
> >>
> > Do you have a table functional.emptytable? If yes, then what location is
> > reported in "show create table"?
> Query: show create table functional.emptytable
> +-------------------------------------------------------------+
> | result |
> +-------------------------------------------------------------+
> | CREATE EXTERNAL TABLE functional.emptytable ( |
> | field STRING |
> | ) |
> | PARTITIONED BY ( |
> | f2 INT |
> | ) |
> | STORED AS TEXTFILE |
> | LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
> | TBLPROPERTIES ('transient_lastDdlTime'='1464782625') |
> +-------------------------------------------------------------+
> Fetched 1 row(s) in 5.51s
>
> > Does the directory exist in HDFS?
> No.
>
> >
> > You could try to manually reload the table and see if the directory is
> > created:
> > bin/load-data.py -f -w functional-query --table_names=emptytable
> > --table_formats=text/none
> After executing this command the directory appears.
>
> >
> >>>
> >> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
> >> errors.
> >> >>
> >> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
> >> >> elapsed: 0.333 sec <<< FAILURE!
> >> >> java.lang.AssertionError: got error:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> default.nodb
> >> >> expected:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> nodb.alltypes
> >> >> at org.junit.Assert.fail(Assert.java:88)
> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2220)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2203)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2197)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> >> AuthorizationTest.java:512)
> >> >>
> >> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
> >> >> elapsed: 0.324 sec <<< FAILURE!
> >> >> java.lang.AssertionError: got error:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> default.nodb
> >> >> expected:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> nodb.alltypes
> >> >> at org.junit.Assert.fail(Assert.java:88)
> >> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2220)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2203)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> >> AuthorizationTest.java:2197)
> >> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> >> AuthorizationTest.java:512)
> >> >>
> >> >>
> >> >> Results :
> >> >>
> >> >> Failed tests:
> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> >> AuthzError:2203->AuthzError:2220 got error:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> default.nodb
> >> >> expected:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> nodb.alltypes
> >> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> >> AuthzError:2203->AuthzError:2220 got error:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> default.nodb
> >> >> expected:
> >> >> User 'amos' does not have privileges to execute 'SELECT' on:
> >> nodb.alltypes
> >> >>
> >> >>
> >> >>
> >> > Strange. In this test, we register two authorization requests, and it
> >> seems
> >> > like those are not checked in the expected order. However, that should
> >> not
> >> > be possible because we store them in a LinkedHashSet.
> >> > Could you dig into this a little further to see if you can figure out
> why
> >> > the order is wrong?
> >> >
> >> > This is where we register the authorization requests:
> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> >> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
> >> >
> >> > This is where we check the authorization requests:
> >> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> >> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
> >> >
> >> >
> >>
> >> I tried directly executing "select 1 from nodb.alltypes" in
> >> impala-shell, leading to this error:
> >> ERROR: AnalysisException: Could not resolve table reference:
> >> 'nodb.alltypes'
> >>
> >> How can I reproduce the authorization tests in impala-shell so I can
> >> debug it?
> >>
> >>
> >>
> > FYI, this is actually a known issue and may have something to do with the
> > JRE version you are running: https://issues.cloudera.org/
> browse/IMPALA-3643
> > As far as I can tell the bug should be "impossible" because we use a
> > LinkedHashSet, but maybe certain JREs do not properly honor the
> guarantees.
> >
> > The AuthorizationTests in particular require a non-trivial setup, so I'd
> > not recommend trying to debug via the Impala shell.
> >
> > I'd recommend debugging in one of these ways:
> > - Run the test manually via "mvn test -Dtest=AuthorizationTest" from the
> FE
> > directory. Attach debugger and break in TestSelect().
> > - Run the JUnit test from an IDE such as Eclipse and then debug the test.
> > I'm afraid there is no easy way to just run that single query in our
> > current test setup. You will need to run the whole suite, but you can
> break
> > TestSelect() or hack the code in various places to set useful
> breakpoints.
> >
> > Hope that helps.
>
> I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest
> fails with ClassNoDefine.
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec
> <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest
> initializationError(com.cloudera.impala.analysis.AuthorizationTest) Time
> elapsed: 0.006 sec <<< ERROR!
> java.lang.NoClassDefFoundError: Could not initialize class
> com.cloudera.impala.analysis.AuthorizationTest
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(
> ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(
> FrameworkMethod.java:47)
> at org.junit.runners.Parameterized.allParameters(Parameterized.java:280)
> at org.junit.runners.Parameterized.<init>(Parameterized.java:248)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.junit.internal.builders.AnnotatedBuilder.buildRunner(
> AnnotatedBuilder.java:104)
> at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(
> AnnotatedBuilder.java:86)
> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
> RunnerBuilder.java:59)
> at org.junit.internal.builders.AllDefaultPossibilitiesBuilder
> .runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(
> RunnerBuilder.java:59)
> at org.junit.internal.requests.ClassRequest.getRunner(
> ClassRequest.java:33)
> at org.apache.maven.surefire.junit4.JUnit4Provider.execute(
> JUnit4Provider.java:283)
> at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(
> JUnit4Provider.java:173)
> at org.apache.maven.surefire.junit4.JUnit4Provider.
> executeTestSet(JUnit4Provider.java:153)
> at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(
> JUnit4Provider.java:128)
> at org.apache.maven.surefire.booter.ForkedBooter.
> invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(
> ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(
> ForkedBooter.java:103)
>
> >
> >>
> >> >>
> >> >>
> >> >> 3. For end-to-end tests, I encountered two kinds of errors
> >> >>
> >> >> a) connection refused.
> >> >>
> >> >> SET sync_ddl=False;
> >> >> -- executing against localhost:21000
> >> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
> >> >>
> >> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
> >> option:
> >> >> {'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_
> >> threshold':
> >> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
> >> >> ___________________
> >> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> >> bin/../infra/python/env/bin/python
> >> >> metadata/test_load.py:77: in setup_method
> >> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
> >> >> util/hdfs_util.py:122: in copy
> >> >> data = self.read_file(src)
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> pywebhdfs/webhdfs.py:183:
> >> >> in read_file
> >> >> response = requests.get(uri, allow_redirects=True)
> >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69:
> in
> >> get
> >> >> return request('get', url, params=params, **kwargs)
> >> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50:
> in
> >> >> request
> >> >> response = session.request(method=method, url=url, **kwargs)
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> requests/sessions.py:465:
> >> >> in request
> >> >> resp = self.send(prep, **send_kwargs)
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> requests/sessions.py:594:
> >> >> in send
> >> >> history = [resp for resp in gen] if allow_redirects else []
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> requests/sessions.py:196:
> >> >> in resolve_redirects
> >> >> **adapter_kwargs
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> requests/sessions.py:573:
> >> >> in send
> >> >> r = adapter.send(request, **kwargs)
> >> >> ../infra/python/env/lib/python2.6/site-packages/
> >> requests/adapters.py:415:
> >> >> in send
> >> >> raise ConnectionError(err, request=request)
> >> >> E ConnectionError: ('Connection aborted.', error(111, 'Connection
> >> >> refused'))
> >> >>
> >> >>
> >> > The connection refused issue is very bizarre. One thing that I
> noticed is
> >> > that your Python does not seem to match what we use (Python 2.7.3).
> >> > Could you re-run infra/python/bootstrap_virtualenv.py and see if you
> get
> >> > the expected version into infra/python/env/local/bin?
> >> >
> >> > Alternatively, maybe there's a problem with your /etc/hosts? You can
> try
> >> > searching online for WebHdfs and /etc/hosts
> >> >
> >>
> >> well, I find this 'find_py26.py' file under deps. Is it normal?
> >>
> >
> > Yes, that's normal. That file looks Python 2.6 on your system but should
> > not be relevant for running tests because we use the Python from our
> > virtualenv and not the one on your system.
> >
> > What's your output when you run "impala-python --version". You should get
> > 'Python 2.7.3".
>
> [amos@t450s tests]$ impala-python --version
> Python 2.6.6
>
> > Also, what's the Python version on your system? Our virtualenv will use
> > Python 2.6 if your system has a Python < 2.6.
> > You could try to upgrade your system Python and then
> > re-run infra/python/bootstrap_virtualenv.py
>
> My system has python 2.6.6.
>
> >
> > Still, theoretically Python 2.6 in the virtual env should work. I think
> > it's more likely you are having a connection problem due to a
> misconfigured
> > /etc/hosts.
> >
> > Are you running the test from a shell that has bin/impala-config.sh and
> > bin/set-classpath.sh sourced?
> Yes.
>
> >
> > To further debug this you could try to specify your namenode address when
> > running the test to see whether it is somehow picking up a wrong address:
> > cd tests
> > ./run-tests.py metadata/test_load.py --namenode_http_address=
> localhost:50070
> >
> > And see if that works.
>
> Unfortunately, no.
>
> >
> >
> >
> >> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
> >> download_requirements find_py26.py pip_download.py requirements.txt
> >> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
> >> requirements
> >> #!/bin/bash
> >>
> >> # Licensed to the Apache Software Foundation (ASF) under one
> >> # or more contributor license agreements. See the NOTICE file
> >> # distributed with this work for additional information
> >> # regarding copyright ownership. The ASF licenses this file
> >> # to you under the Apache License, Version 2.0 (the
> >> # "License"); you may not use this file except in compliance
> >> # with the License. You may obtain a copy of the License at
> >> #
> >> # http://www.apache.org/licenses/LICENSE-2.0
> >> #
> >> # Unless required by applicable law or agreed to in writing,
> >> # software distributed under the License is distributed on an
> >> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> >> # KIND, either express or implied. See the License for the
> >> # specific language governing permissions and limitations
> >> # under the License.
> >>
> >> set -euo pipefail
> >>
> >> DIR="$(dirname "$0")"
> >>
> >> pushd "$DIR"
> >> PY26="$(./find_py26.py)"
> >> # Directly download packages listed in requirements.txt, but don't
> install
> >> them.
> >> "$PY26" pip_download.py
> >> # For virtualenv, other scripts rely on the .tar.gz package (not a .whl
> >> package).
> >> "$PY26" pip_download.py virtualenv 13.1.0
> >> # kudu-python is downloaded separately because pip install attempts to
> >> execute a
> >> # setup.py subcommand for kudu-python that can fail even if the download
> >> succeeds.
> >> "$PY26" pip_download.py kudu-python 0.1.1
> >> popd
> >>
> >>
> >>
> >> > b) stats not match
> >> >>
> >> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> >> bin/../infra/python/env/bin/python
> >> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
> >> >> self.run_test_case('QueryTest/show-stats', vector, "functional")
> >> >> common/impala_test_suite.py:342: in run_test_case
> >> >> self.__verify_results_and_errors(vector, test_section, result,
> >> use_db)
> >> >> common/impala_test_suite.py:234: in __verify_results_and_errors
> >> >> replace_filenames_with_placeholder)
> >> >> common/test_result_verifier.py:398: in verify_raw_results
> >> >> VERIFIER_MAP[verifier](expected, actual)
> >> >> common/test_result_verifier.py:231: in verify_query_result_is_equal
> >> >> assert expected_results == actual_results
> >> >>
> >> >> ...
> >> >>
> >> >> -- executing against localhost:21000
> >> >> show column stats alltypes_clone;
> >> >>
> >> >> MainThread: Comparing QueryTestResults (expected vs actual):
> >> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
> >> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
> >> >> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
> >> >> 736,-1,8,8
> >> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
> >> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
> >> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
> >> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
> >> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
> >> >> 'smallint_col','SMALLINT',10,-1,2,2 ==
> 'smallint_col','SMALLINT',10,-
> >> 1,2,2
> >> >> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-
> 1,-1
> >> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
> >> 'timestamp_col','TIMESTAMP',
> >> >> 7552,-1,16,16
> >> >> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,
> 1,1
> >> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
> >> >>
> >> >>
> >> >> Very strange. Can you do a compute stats on functional.alltypes and
> >> > confirm that the NDV for timestamp_col are 7552 in your setup?
> >>
> >> Yes.
> >>
> >
> > I'll need to ask around for help. I have no idea why this is happening.
>
> Thanks :)
>
> >
> >>
> >> >
> >> >
> >> >
> >> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help
> is
> >> >> much appreciated!
> >> >
> >> >
> >> >
> >> >
> >> >>
> >> >> Best regards,
> >> >> Amos
> >> >>
> >>
> >>
>
>
Re: test errors in local env
Posted by Amos Bird <am...@gmail.com>.
Thank you for helping me out :D
> Thanks for keeping this discussion going.
>
> On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com> wrote:
>
>>
>> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com> wrote:
>> >
>> >>
>> >> Hi there,
>> >>
>> >> I followed the wiki
>> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> >> to+load+and+run+Impala+tests
>> >> carefully but still have some problems in my local env.
>> >>
>> >> 1. I need to manually execute "hdfs dfs -mkdir
>> /test-warehouse/emptytable"
>> >> to get rid of some fe test error.
>> >>
>> >>
>> > Ideally, you should not have to do this. Could you tell me what errors
>> you
>> > encountered? Sounds like there may be a test or data loading bug we
>> should
>> > fix.
>>
>> The error is :
>>
>> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
>> elapsed: 0.033 sec <<< FAILURE!
>> java.lang.AssertionError: got error:
>> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
>> not exist.
>> expected:
>> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
>> contains no visible files.
>> at org.junit.Assert.fail(Assert.java:88)
>> at org.junit.Assert.assertTrue(Assert.java:41)
>> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> FrontendTestBase.java:312)
>> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
>> FrontendTestBase.java:292)
>> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
>> AnalyzeStmtsTest.java:2860)
>>
>>
> Do you have a table functional.emptytable? If yes, then what location is
> reported in "show create table"?
Query: show create table functional.emptytable
+-------------------------------------------------------------+
| result |
+-------------------------------------------------------------+
| CREATE EXTERNAL TABLE functional.emptytable ( |
| field STRING |
| ) |
| PARTITIONED BY ( |
| f2 INT |
| ) |
| STORED AS TEXTFILE |
| LOCATION 'hdfs://localhost:20500/test-warehouse/emptytable' |
| TBLPROPERTIES ('transient_lastDdlTime'='1464782625') |
+-------------------------------------------------------------+
Fetched 1 row(s) in 5.51s
> Does the directory exist in HDFS?
No.
>
> You could try to manually reload the table and see if the directory is
> created:
> bin/load-data.py -f -w functional-query --table_names=emptytable
> --table_formats=text/none
After executing this command the directory appears.
>
>>>
>> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
>> errors.
>> >>
>> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
>> >> elapsed: 0.333 sec <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >> at org.junit.Assert.fail(Assert.java:88)
>> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2220)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2203)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2197)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> AuthorizationTest.java:512)
>> >>
>> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
>> >> elapsed: 0.324 sec <<< FAILURE!
>> >> java.lang.AssertionError: got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >> at org.junit.Assert.fail(Assert.java:88)
>> >> at org.junit.Assert.assertTrue(Assert.java:41)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2220)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2203)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> >> AuthorizationTest.java:2197)
>> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> >> AuthorizationTest.java:512)
>> >>
>> >>
>> >> Results :
>> >>
>> >> Failed tests:
>> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> AuthzError:2203->AuthzError:2220 got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> >> AuthzError:2203->AuthzError:2220 got error:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> default.nodb
>> >> expected:
>> >> User 'amos' does not have privileges to execute 'SELECT' on:
>> nodb.alltypes
>> >>
>> >>
>> >>
>> > Strange. In this test, we register two authorization requests, and it
>> seems
>> > like those are not checked in the expected order. However, that should
>> not
>> > be possible because we store them in a LinkedHashSet.
>> > Could you dig into this a little further to see if you can figure out why
>> > the order is wrong?
>> >
>> > This is where we register the authorization requests:
>> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
>> >
>> > This is where we check the authorization requests:
>> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
>> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>> >
>> >
>>
>> I tried directly executing "select 1 from nodb.alltypes" in
>> impala-shell, leading to this error:
>> ERROR: AnalysisException: Could not resolve table reference:
>> 'nodb.alltypes'
>>
>> How can I reproduce the authorization tests in impala-shell so I can
>> debug it?
>>
>>
>>
> FYI, this is actually a known issue and may have something to do with the
> JRE version you are running: https://issues.cloudera.org/browse/IMPALA-3643
> As far as I can tell the bug should be "impossible" because we use a
> LinkedHashSet, but maybe certain JREs do not properly honor the guarantees.
>
> The AuthorizationTests in particular require a non-trivial setup, so I'd
> not recommend trying to debug via the Impala shell.
>
> I'd recommend debugging in one of these ways:
> - Run the test manually via "mvn test -Dtest=AuthorizationTest" from the FE
> directory. Attach debugger and break in TestSelect().
> - Run the JUnit test from an IDE such as Eclipse and then debug the test.
> I'm afraid there is no easy way to just run that single query in our
> current test setup. You will need to run the whole suite, but you can break
> TestSelect() or hack the code in various places to set useful breakpoints.
>
> Hope that helps.
I tried jdk1.8.0_102. Compilation works just fine but AuthorizationTest
fails with ClassNoDefine.
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec <<< FAILURE! - in com.cloudera.impala.analysis.AuthorizationTest
initializationError(com.cloudera.impala.analysis.AuthorizationTest) Time elapsed: 0.006 sec <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class com.cloudera.impala.analysis.AuthorizationTest
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.runners.Parameterized.allParameters(Parameterized.java:280)
at org.junit.runners.Parameterized.<init>(Parameterized.java:248)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
>>
>> >>
>> >>
>> >> 3. For end-to-end tests, I encountered two kinds of errors
>> >>
>> >> a) connection refused.
>> >>
>> >> SET sync_ddl=False;
>> >> -- executing against localhost:21000
>> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>> >>
>> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
>> option:
>> >> {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_
>> threshold':
>> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
>> >> ___________________
>> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> bin/../infra/python/env/bin/python
>> >> metadata/test_load.py:77: in setup_method
>> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
>> >> util/hdfs_util.py:122: in copy
>> >> data = self.read_file(src)
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> pywebhdfs/webhdfs.py:183:
>> >> in read_file
>> >> response = requests.get(uri, allow_redirects=True)
>> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in
>> get
>> >> return request('get', url, params=params, **kwargs)
>> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in
>> >> request
>> >> response = session.request(method=method, url=url, **kwargs)
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/sessions.py:465:
>> >> in request
>> >> resp = self.send(prep, **send_kwargs)
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/sessions.py:594:
>> >> in send
>> >> history = [resp for resp in gen] if allow_redirects else []
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/sessions.py:196:
>> >> in resolve_redirects
>> >> **adapter_kwargs
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/sessions.py:573:
>> >> in send
>> >> r = adapter.send(request, **kwargs)
>> >> ../infra/python/env/lib/python2.6/site-packages/
>> requests/adapters.py:415:
>> >> in send
>> >> raise ConnectionError(err, request=request)
>> >> E ConnectionError: ('Connection aborted.', error(111, 'Connection
>> >> refused'))
>> >>
>> >>
>> > The connection refused issue is very bizarre. One thing that I noticed is
>> > that your Python does not seem to match what we use (Python 2.7.3).
>> > Could you re-run infra/python/bootstrap_virtualenv.py and see if you get
>> > the expected version into infra/python/env/local/bin?
>> >
>> > Alternatively, maybe there's a problem with your /etc/hosts? You can try
>> > searching online for WebHdfs and /etc/hosts
>> >
>>
>> well, I find this 'find_py26.py' file under deps. Is it normal?
>>
>
> Yes, that's normal. That file looks Python 2.6 on your system but should
> not be relevant for running tests because we use the Python from our
> virtualenv and not the one on your system.
>
> What's your output when you run "impala-python --version". You should get
> 'Python 2.7.3".
[amos@t450s tests]$ impala-python --version
Python 2.6.6
> Also, what's the Python version on your system? Our virtualenv will use
> Python 2.6 if your system has a Python < 2.6.
> You could try to upgrade your system Python and then
> re-run infra/python/bootstrap_virtualenv.py
My system has python 2.6.6.
>
> Still, theoretically Python 2.6 in the virtual env should work. I think
> it's more likely you are having a connection problem due to a misconfigured
> /etc/hosts.
>
> Are you running the test from a shell that has bin/impala-config.sh and
> bin/set-classpath.sh sourced?
Yes.
>
> To further debug this you could try to specify your namenode address when
> running the test to see whether it is somehow picking up a wrong address:
> cd tests
> ./run-tests.py metadata/test_load.py --namenode_http_address=localhost:50070
>
> And see if that works.
Unfortunately, no.
>
>
>
>> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
>> download_requirements find_py26.py pip_download.py requirements.txt
>> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
>> requirements
>> #!/bin/bash
>>
>> # Licensed to the Apache Software Foundation (ASF) under one
>> # or more contributor license agreements. See the NOTICE file
>> # distributed with this work for additional information
>> # regarding copyright ownership. The ASF licenses this file
>> # to you under the Apache License, Version 2.0 (the
>> # "License"); you may not use this file except in compliance
>> # with the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing,
>> # software distributed under the License is distributed on an
>> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> # KIND, either express or implied. See the License for the
>> # specific language governing permissions and limitations
>> # under the License.
>>
>> set -euo pipefail
>>
>> DIR="$(dirname "$0")"
>>
>> pushd "$DIR"
>> PY26="$(./find_py26.py)"
>> # Directly download packages listed in requirements.txt, but don't install
>> them.
>> "$PY26" pip_download.py
>> # For virtualenv, other scripts rely on the .tar.gz package (not a .whl
>> package).
>> "$PY26" pip_download.py virtualenv 13.1.0
>> # kudu-python is downloaded separately because pip install attempts to
>> execute a
>> # setup.py subcommand for kudu-python that can fail even if the download
>> succeeds.
>> "$PY26" pip_download.py kudu-python 0.1.1
>> popd
>>
>>
>>
>> > b) stats not match
>> >>
>> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> >> bin/../infra/python/env/bin/python
>> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
>> >> self.run_test_case('QueryTest/show-stats', vector, "functional")
>> >> common/impala_test_suite.py:342: in run_test_case
>> >> self.__verify_results_and_errors(vector, test_section, result,
>> use_db)
>> >> common/impala_test_suite.py:234: in __verify_results_and_errors
>> >> replace_filenames_with_placeholder)
>> >> common/test_result_verifier.py:398: in verify_raw_results
>> >> VERIFIER_MAP[verifier](expected, actual)
>> >> common/test_result_verifier.py:231: in verify_query_result_is_equal
>> >> assert expected_results == actual_results
>> >>
>> >> ...
>> >>
>> >> -- executing against localhost:21000
>> >> show column stats alltypes_clone;
>> >>
>> >> MainThread: Comparing QueryTestResults (expected vs actual):
>> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
>> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
>> >> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
>> >> 736,-1,8,8
>> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
>> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
>> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
>> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
>> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
>> >> 'smallint_col','SMALLINT',10,-1,2,2 == 'smallint_col','SMALLINT',10,-
>> 1,2,2
>> >> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-1,-1
>> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
>> 'timestamp_col','TIMESTAMP',
>> >> 7552,-1,16,16
>> >> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,1,1
>> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
>> >>
>> >>
>> >> Very strange. Can you do a compute stats on functional.alltypes and
>> > confirm that the NDV for timestamp_col are 7552 in your setup?
>>
>> Yes.
>>
>
> I'll need to ask around for help. I have no idea why this is happening.
Thanks :)
>
>>
>> >
>> >
>> >
>> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help is
>> >> much appreciated!
>> >
>> >
>> >
>> >
>> >>
>> >> Best regards,
>> >> Amos
>> >>
>>
>>
Re: test errors in local env
Posted by Alex Behm <al...@cloudera.com>.
Thanks for keeping this discussion going.
On Sun, Sep 18, 2016 at 8:13 AM, Amos Bird <am...@gmail.com> wrote:
>
> > On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com> wrote:
> >
> >>
> >> Hi there,
> >>
> >> I followed the wiki
> >> https://cwiki.apache.org/confluence/display/IMPALA/How+
> >> to+load+and+run+Impala+tests
> >> carefully but still have some problems in my local env.
> >>
> >> 1. I need to manually execute "hdfs dfs -mkdir
> /test-warehouse/emptytable"
> >> to get rid of some fe test error.
> >>
> >>
> > Ideally, you should not have to do this. Could you tell me what errors
> you
> > encountered? Sounds like there may be a test or data loading bug we
> should
> > fix.
>
> The error is :
>
> TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time
> elapsed: 0.033 sec <<< FAILURE!
> java.lang.AssertionError: got error:
> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does
> not exist.
> expected:
> INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable'
> contains no visible files.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> FrontendTestBase.java:312)
> at com.cloudera.impala.common.FrontendTestBase.AnalysisError(
> FrontendTestBase.java:292)
> at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(
> AnalyzeStmtsTest.java:2860)
>
>
Do you have a table functional.emptytable? If yes, then what location is
reported in "show create table"?
Does the directory exist in HDFS?
You could try to manually reload the table and see if the directory is
created:
bin/load-data.py -f -w functional-query --table_names=emptytable
--table_formats=text/none
>>
> >> 2. I have authz-policy.ini in HDFS, but I still get authorization
> errors.
> >>
> >> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
> >> elapsed: 0.333 sec <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >> at org.junit.Assert.fail(Assert.java:88)
> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2220)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2203)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2197)
> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> AuthorizationTest.java:512)
> >>
> >> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
> >> elapsed: 0.324 sec <<< FAILURE!
> >> java.lang.AssertionError: got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >> at org.junit.Assert.fail(Assert.java:88)
> >> at org.junit.Assert.assertTrue(Assert.java:41)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2220)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2203)
> >> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> >> AuthorizationTest.java:2197)
> >> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> >> AuthorizationTest.java:512)
> >>
> >>
> >> Results :
> >>
> >> Failed tests:
> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> AuthzError:2203->AuthzError:2220 got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >> AuthorizationTest.TestSelect:512->AuthzError:2197->
> >> AuthzError:2203->AuthzError:2220 got error:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> default.nodb
> >> expected:
> >> User 'amos' does not have privileges to execute 'SELECT' on:
> nodb.alltypes
> >>
> >>
> >>
> > Strange. In this test, we register two authorization requests, and it
> seems
> > like those are not checked in the expected order. However, that should
> not
> > be possible because we store them in a LinkedHashSet.
> > Could you dig into this a little further to see if you can figure out why
> > the order is wrong?
> >
> > This is where we register the authorization requests:
> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> main/java/com/cloudera/impala/analysis/Analyzer.java#L544
> >
> > This is where we check the authorization requests:
> > https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/
> main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
> >
> >
>
> I tried directly executing "select 1 from nodb.alltypes" in
> impala-shell, leading to this error:
> ERROR: AnalysisException: Could not resolve table reference:
> 'nodb.alltypes'
>
> How can I reproduce the authorization tests in impala-shell so I can
> debug it?
>
>
>
FYI, this is actually a known issue and may have something to do with the
JRE version you are running: https://issues.cloudera.org/browse/IMPALA-3643
As far as I can tell the bug should be "impossible" because we use a
LinkedHashSet, but maybe certain JREs do not properly honor the guarantees.
The AuthorizationTests in particular require a non-trivial setup, so I'd
not recommend trying to debug via the Impala shell.
I'd recommend debugging in one of these ways:
- Run the test manually via "mvn test -Dtest=AuthorizationTest" from the FE
directory. Attach debugger and break in TestSelect().
- Run the JUnit test from an IDE such as Eclipse and then debug the test.
I'm afraid there is no easy way to just run that single query in our
current test setup. You will need to run the whole suite, but you can break
TestSelect() or hack the code in various places to set useful breakpoints.
Hope that helps.
>
> >>
> >>
> >> 3. For end-to-end tests, I encountered two kinds of errors
> >>
> >> a) connection refused.
> >>
> >> SET sync_ddl=False;
> >> -- executing against localhost:21000
> >> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
> >>
> >> ___________________ ERROR at setup of TestLoadData.test_load[exec_
> option:
> >> {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_
> threshold':
> >> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
> >> ___________________
> >> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> bin/../infra/python/env/bin/python
> >> metadata/test_load.py:77: in setup_method
> >> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
> >> util/hdfs_util.py:122: in copy
> >> data = self.read_file(src)
> >> ../infra/python/env/lib/python2.6/site-packages/
> pywebhdfs/webhdfs.py:183:
> >> in read_file
> >> response = requests.get(uri, allow_redirects=True)
> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in
> get
> >> return request('get', url, params=params, **kwargs)
> >> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in
> >> request
> >> response = session.request(method=method, url=url, **kwargs)
> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/sessions.py:465:
> >> in request
> >> resp = self.send(prep, **send_kwargs)
> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/sessions.py:594:
> >> in send
> >> history = [resp for resp in gen] if allow_redirects else []
> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/sessions.py:196:
> >> in resolve_redirects
> >> **adapter_kwargs
> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/sessions.py:573:
> >> in send
> >> r = adapter.send(request, **kwargs)
> >> ../infra/python/env/lib/python2.6/site-packages/
> requests/adapters.py:415:
> >> in send
> >> raise ConnectionError(err, request=request)
> >> E ConnectionError: ('Connection aborted.', error(111, 'Connection
> >> refused'))
> >>
> >>
> > The connection refused issue is very bizarre. One thing that I noticed is
> > that your Python does not seem to match what we use (Python 2.7.3).
> > Could you re-run infra/python/bootstrap_virtualenv.py and see if you get
> > the expected version into infra/python/env/local/bin?
> >
> > Alternatively, maybe there's a problem with your /etc/hosts? You can try
> > searching online for WebHdfs and /etc/hosts
> >
>
> well, I find this 'find_py26.py' file under deps. Is it normal?
>
Yes, that's normal. That file looks Python 2.6 on your system but should
not be relevant for running tests because we use the Python from our
virtualenv and not the one on your system.
What's your output when you run "impala-python --version". You should get
'Python 2.7.3".
Also, what's the Python version on your system? Our virtualenv will use
Python 2.6 if your system has a Python < 2.6.
You could try to upgrade your system Python and then
re-run infra/python/bootstrap_virtualenv.py
Still, theoretically Python 2.6 in the virtual env should work. I think
it's more likely you are having a connection problem due to a misconfigured
/etc/hosts.
Are you running the test from a shell that has bin/impala-config.sh and
bin/set-classpath.sh sourced?
To further debug this you could try to specify your namenode address when
running the test to see whether it is somehow picking up a wrong address:
cd tests
./run-tests.py metadata/test_load.py --namenode_http_address=localhost:50070
And see if that works.
> [amos@nobida143 incubator-impala]$ ls infra/python/deps/
> download_requirements find_py26.py pip_download.py requirements.txt
> [amos@nobida143 incubator-impala]$ cat infra/python/deps/download_
> requirements
> #!/bin/bash
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing,
> # software distributed under the License is distributed on an
> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> # KIND, either express or implied. See the License for the
> # specific language governing permissions and limitations
> # under the License.
>
> set -euo pipefail
>
> DIR="$(dirname "$0")"
>
> pushd "$DIR"
> PY26="$(./find_py26.py)"
> # Directly download packages listed in requirements.txt, but don't install
> them.
> "$PY26" pip_download.py
> # For virtualenv, other scripts rely on the .tar.gz package (not a .whl
> package).
> "$PY26" pip_download.py virtualenv 13.1.0
> # kudu-python is downloaded separately because pip install attempts to
> execute a
> # setup.py subcommand for kudu-python that can fail even if the download
> succeeds.
> "$PY26" pip_download.py kudu-python 0.1.1
> popd
>
>
>
> > b) stats not match
> >>
> >> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> >> bin/../infra/python/env/bin/python
> >> metadata/test_metadata_query_statements.py:67: in test_show_stats
> >> self.run_test_case('QueryTest/show-stats', vector, "functional")
> >> common/impala_test_suite.py:342: in run_test_case
> >> self.__verify_results_and_errors(vector, test_section, result,
> use_db)
> >> common/impala_test_suite.py:234: in __verify_results_and_errors
> >> replace_filenames_with_placeholder)
> >> common/test_result_verifier.py:398: in verify_raw_results
> >> VERIFIER_MAP[verifier](expected, actual)
> >> common/test_result_verifier.py:231: in verify_query_result_is_equal
> >> assert expected_results == actual_results
> >>
> >> ...
> >>
> >> -- executing against localhost:21000
> >> show column stats alltypes_clone;
> >>
> >> MainThread: Comparing QueryTestResults (expected vs actual):
> >> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
> >> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
> >> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
> >> 736,-1,8,8
> >> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
> >> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
> >> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
> >> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
> >> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
> >> 'smallint_col','SMALLINT',10,-1,2,2 == 'smallint_col','SMALLINT',10,-
> 1,2,2
> >> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-1,-1
> >> 'timestamp_col','TIMESTAMP',7554,-1,16,16 !=
> 'timestamp_col','TIMESTAMP',
> >> 7552,-1,16,16
> >> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,1,1
> >> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
> >>
> >>
> >> Very strange. Can you do a compute stats on functional.alltypes and
> > confirm that the NDV for timestamp_col are 7552 in your setup?
>
> Yes.
>
I'll need to ask around for help. I have no idea why this is happening.
>
> >
> >
> >
> >> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help is
> >> much appreciated!
> >
> >
> >
> >
> >>
> >> Best regards,
> >> Amos
> >>
>
>
Re: test errors in local env
Posted by Amos Bird <am...@gmail.com>.
> On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com> wrote:
>
>>
>> Hi there,
>>
>> I followed the wiki
>> https://cwiki.apache.org/confluence/display/IMPALA/How+
>> to+load+and+run+Impala+tests
>> carefully but still have some problems in my local env.
>>
>> 1. I need to manually execute "hdfs dfs -mkdir /test-warehouse/emptytable"
>> to get rid of some fe test error.
>>
>>
> Ideally, you should not have to do this. Could you tell me what errors you
> encountered? Sounds like there may be a test or data loading bug we should
> fix.
The error is :
TestLoadData(com.cloudera.impala.analysis.AnalyzeStmtsTest) Time elapsed: 0.033 sec <<< FAILURE!
java.lang.AssertionError: got error:
INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' does not exist.
expected:
INPATH location 'hdfs://localhost:20500/test-warehouse/emptytable' contains no visible files.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at com.cloudera.impala.common.FrontendTestBase.AnalysisError(FrontendTestBase.java:312)
at com.cloudera.impala.common.FrontendTestBase.AnalysisError(FrontendTestBase.java:292)
at com.cloudera.impala.analysis.AnalyzeStmtsTest.TestLoadData(AnalyzeStmtsTest.java:2860)
>>
>> 2. I have authz-policy.ini in HDFS, but I still get authorization errors.
>>
>> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
>> elapsed: 0.333 sec <<< FAILURE!
>> java.lang.AssertionError: got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>> at org.junit.Assert.fail(Assert.java:88)
>> at org.junit.Assert.assertTrue(Assert.java:41)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2220)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2203)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2197)
>> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> AuthorizationTest.java:512)
>>
>> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
>> elapsed: 0.324 sec <<< FAILURE!
>> java.lang.AssertionError: got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>> at org.junit.Assert.fail(Assert.java:88)
>> at org.junit.Assert.assertTrue(Assert.java:41)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2220)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2203)
>> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
>> AuthorizationTest.java:2197)
>> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
>> AuthorizationTest.java:512)
>>
>>
>> Results :
>>
>> Failed tests:
>> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> AuthzError:2203->AuthzError:2220 got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>> AuthorizationTest.TestSelect:512->AuthzError:2197->
>> AuthzError:2203->AuthzError:2220 got error:
>> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
>> expected:
>> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>>
>>
>>
> Strange. In this test, we register two authorization requests, and it seems
> like those are not checked in the expected order. However, that should not
> be possible because we store them in a LinkedHashSet.
> Could you dig into this a little further to see if you can figure out why
> the order is wrong?
>
> This is where we register the authorization requests:
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java#L544
>
> This is where we check the authorization requests:
> https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>
>
I tried directly executing "select 1 from nodb.alltypes" in
impala-shell, leading to this error:
ERROR: AnalysisException: Could not resolve table reference: 'nodb.alltypes'
How can I reproduce the authorization tests in impala-shell so I can
debug it?
>
>>
>>
>> 3. For end-to-end tests, I encountered two kinds of errors
>>
>> a) connection refused.
>>
>> SET sync_ddl=False;
>> -- executing against localhost:21000
>> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>>
>> ___________________ ERROR at setup of TestLoadData.test_load[exec_option:
>> {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold':
>> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
>> ___________________
>> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> bin/../infra/python/env/bin/python
>> metadata/test_load.py:77: in setup_method
>> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
>> util/hdfs_util.py:122: in copy
>> data = self.read_file(src)
>> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:183:
>> in read_file
>> response = requests.get(uri, allow_redirects=True)
>> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in get
>> return request('get', url, params=params, **kwargs)
>> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in
>> request
>> response = session.request(method=method, url=url, **kwargs)
>> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:465:
>> in request
>> resp = self.send(prep, **send_kwargs)
>> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:594:
>> in send
>> history = [resp for resp in gen] if allow_redirects else []
>> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:196:
>> in resolve_redirects
>> **adapter_kwargs
>> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:573:
>> in send
>> r = adapter.send(request, **kwargs)
>> ../infra/python/env/lib/python2.6/site-packages/requests/adapters.py:415:
>> in send
>> raise ConnectionError(err, request=request)
>> E ConnectionError: ('Connection aborted.', error(111, 'Connection
>> refused'))
>>
>>
> The connection refused issue is very bizarre. One thing that I noticed is
> that your Python does not seem to match what we use (Python 2.7.3).
> Could you re-run infra/python/bootstrap_virtualenv.py and see if you get
> the expected version into infra/python/env/local/bin?
>
> Alternatively, maybe there's a problem with your /etc/hosts? You can try
> searching online for WebHdfs and /etc/hosts
>
well, I find this 'find_py26.py' file under deps. Is it normal?
[amos@nobida143 incubator-impala]$ ls infra/python/deps/
download_requirements find_py26.py pip_download.py requirements.txt
[amos@nobida143 incubator-impala]$ cat infra/python/deps/download_requirements
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
set -euo pipefail
DIR="$(dirname "$0")"
pushd "$DIR"
PY26="$(./find_py26.py)"
# Directly download packages listed in requirements.txt, but don't install them.
"$PY26" pip_download.py
# For virtualenv, other scripts rely on the .tar.gz package (not a .whl package).
"$PY26" pip_download.py virtualenv 13.1.0
# kudu-python is downloaded separately because pip install attempts to execute a
# setup.py subcommand for kudu-python that can fail even if the download succeeds.
"$PY26" pip_download.py kudu-python 0.1.1
popd
> b) stats not match
>>
>> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
>> bin/../infra/python/env/bin/python
>> metadata/test_metadata_query_statements.py:67: in test_show_stats
>> self.run_test_case('QueryTest/show-stats', vector, "functional")
>> common/impala_test_suite.py:342: in run_test_case
>> self.__verify_results_and_errors(vector, test_section, result, use_db)
>> common/impala_test_suite.py:234: in __verify_results_and_errors
>> replace_filenames_with_placeholder)
>> common/test_result_verifier.py:398: in verify_raw_results
>> VERIFIER_MAP[verifier](expected, actual)
>> common/test_result_verifier.py:231: in verify_query_result_is_equal
>> assert expected_results == actual_results
>>
>> ...
>>
>> -- executing against localhost:21000
>> show column stats alltypes_clone;
>>
>> MainThread: Comparing QueryTestResults (expected vs actual):
>> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
>> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
>> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
>> 736,-1,8,8
>> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
>> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
>> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
>> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
>> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
>> 'smallint_col','SMALLINT',10,-1,2,2 == 'smallint_col','SMALLINT',10,-1,2,2
>> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-1,-1
>> 'timestamp_col','TIMESTAMP',7554,-1,16,16 != 'timestamp_col','TIMESTAMP',
>> 7552,-1,16,16
>> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,1,1
>> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
>>
>>
>> Very strange. Can you do a compute stats on functional.alltypes and
> confirm that the NDV for timestamp_col are 7552 in your setup?
Yes.
>
>
>
>> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help is
>> much appreciated!
>
>
>
>
>>
>> Best regards,
>> Amos
>>
Re: test errors in local env
Posted by Alex Behm <al...@cloudera.com>.
Sorry this is so difficult. We are still in the process of learning and
improving the experience for external contributors. Your help is much
appreciated.
Some responses below.
On Fri, Sep 16, 2016 at 9:06 PM, Amos Bird <am...@gmail.com> wrote:
>
> Hi there,
>
> I followed the wiki
> https://cwiki.apache.org/confluence/display/IMPALA/How+
> to+load+and+run+Impala+tests
> carefully but still have some problems in my local env.
>
> 1. I need to manually execute "hdfs dfs -mkdir /test-warehouse/emptytable"
> to get rid of some fe test error.
>
>
Ideally, you should not have to do this. Could you tell me what errors you
encountered? Sounds like there may be a test or data loading bug we should
fix.
>
> 2. I have authz-policy.ini in HDFS, but I still get authorization errors.
>
> TestSelect[0](com.cloudera.impala.analysis.AuthorizationTest) Time
> elapsed: 0.333 sec <<< FAILURE!
> java.lang.AssertionError: got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2220)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2203)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2197)
> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> AuthorizationTest.java:512)
>
> TestSelect[1](com.cloudera.impala.analysis.AuthorizationTest) Time
> elapsed: 0.324 sec <<< FAILURE!
> java.lang.AssertionError: got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2220)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2203)
> at com.cloudera.impala.analysis.AuthorizationTest.AuthzError(
> AuthorizationTest.java:2197)
> at com.cloudera.impala.analysis.AuthorizationTest.TestSelect(
> AuthorizationTest.java:512)
>
>
> Results :
>
> Failed tests:
> AuthorizationTest.TestSelect:512->AuthzError:2197->
> AuthzError:2203->AuthzError:2220 got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
> AuthorizationTest.TestSelect:512->AuthzError:2197->
> AuthzError:2203->AuthzError:2220 got error:
> User 'amos' does not have privileges to execute 'SELECT' on: default.nodb
> expected:
> User 'amos' does not have privileges to execute 'SELECT' on: nodb.alltypes
>
>
>
Strange. In this test, we register two authorization requests, and it seems
like those are not checked in the expected order. However, that should not
be possible because we store them in a LinkedHashSet.
Could you dig into this a little further to see if you can figure out why
the order is wrong?
This is where we register the authorization requests:
https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java#L544
This is where we check the authorization requests:
https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java#L391
>
>
> 3. For end-to-end tests, I encountered two kinds of errors
>
> a) connection refused.
>
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE `test_drop_cleans_hdfs_dirs_fdfd4f8` CASCADE;
>
> ___________________ ERROR at setup of TestLoadData.test_load[exec_option:
> {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold':
> 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none]
> ___________________
> [gw5] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> bin/../infra/python/env/bin/python
> metadata/test_load.py:77: in setup_method
> "{0}/{1}/100101.txt".format(STAGING_PATH, i))
> util/hdfs_util.py:122: in copy
> data = self.read_file(src)
> ../infra/python/env/lib/python2.6/site-packages/pywebhdfs/webhdfs.py:183:
> in read_file
> response = requests.get(uri, allow_redirects=True)
> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:69: in get
> return request('get', url, params=params, **kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/api.py:50: in
> request
> response = session.request(method=method, url=url, **kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:465:
> in request
> resp = self.send(prep, **send_kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:594:
> in send
> history = [resp for resp in gen] if allow_redirects else []
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:196:
> in resolve_redirects
> **adapter_kwargs
> ../infra/python/env/lib/python2.6/site-packages/requests/sessions.py:573:
> in send
> r = adapter.send(request, **kwargs)
> ../infra/python/env/lib/python2.6/site-packages/requests/adapters.py:415:
> in send
> raise ConnectionError(err, request=request)
> E ConnectionError: ('Connection aborted.', error(111, 'Connection
> refused'))
>
>
The connection refused issue is very bizarre. One thing that I noticed is
that your Python does not seem to match what we use (Python 2.7.3).
Could you re-run infra/python/bootstrap_virtualenv.py and see if you get
the expected version into infra/python/env/local/bin?
Alternatively, maybe there's a problem with your /etc/hosts? You can try
searching online for WebHdfs and /etc/hosts
b) stats not match
>
> [gw4] linux2 -- Python 2.6.6 /home/amos/incubator-impala/
> bin/../infra/python/env/bin/python
> metadata/test_metadata_query_statements.py:67: in test_show_stats
> self.run_test_case('QueryTest/show-stats', vector, "functional")
> common/impala_test_suite.py:342: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:234: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:398: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:231: in verify_query_result_is_equal
> assert expected_results == actual_results
>
> ...
>
> -- executing against localhost:21000
> show column stats alltypes_clone;
>
> MainThread: Comparing QueryTestResults (expected vs actual):
> 'bigint_col','BIGINT',10,-1,8,8 == 'bigint_col','BIGINT',10,-1,8,8
> 'bool_col','BOOLEAN',2,-1,1,1 == 'bool_col','BOOLEAN',2,-1,1,1
> 'date_string_col','STRING',736,-1,8,8 == 'date_string_col','STRING',
> 736,-1,8,8
> 'double_col','DOUBLE',-1,-1,8,8 == 'double_col','DOUBLE',-1,-1,8,8
> 'float_col','FLOAT',10,-1,4,4 == 'float_col','FLOAT',10,-1,4,4
> 'id','INT',7505,-1,4,4 == 'id','INT',7505,-1,4,4
> 'int_col','INT',-1,-1,4,4 == 'int_col','INT',-1,-1,4,4
> 'month','INT',12,0,4,4 == 'month','INT',12,0,4,4
> 'smallint_col','SMALLINT',10,-1,2,2 == 'smallint_col','SMALLINT',10,-1,2,2
> 'string_col','STRING',10,-1,-1,-1 == 'string_col','STRING',10,-1,-1,-1
> 'timestamp_col','TIMESTAMP',7554,-1,16,16 != 'timestamp_col','TIMESTAMP',
> 7552,-1,16,16
> 'tinyint_col','TINYINT',10,-1,1,1 == 'tinyint_col','TINYINT',10,-1,1,1
> 'year','INT',2,0,4,4 == 'year','INT',2,0,4,4
>
>
> Very strange. Can you do a compute stats on functional.alltypes and
confirm that the NDV for timestamp_col are 7552 in your setup?
> I'm using CentOS 6.8 final. I have no idea what goes wrong. Any help is
> much appreciated!
>
> Best regards,
> Amos
>