You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kylin.apache.org by dong wang <el...@gmail.com> on 2015/03/04 07:42:22 UTC

fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

when check the log shown in web GUI of this step:

2015-03-03 22:39:39.495 - State of Hadoop job:
job_1425448727678_0004:ACCEPTED - UNDEFINED
2015-03-03 22:39:49.518 - State of Hadoop job:
job_1425448727678_0004:FINISHED - FAILED
no counters for job job_1425448727678_0004


with the same cube, it can be built successfully with kylin-0.6* before

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

so sorry, before, I always checked
./job/target/kylin-job-0.7.1-SNAPSHOT.jar, therefore, no *.SF, *.RSA, *.DSA
files, however, there are files named "NB-JWS.SF" and "NB-JWS.DSA" in
 META-INF directory for kylin-job-0.7.1-SNAPSHOT-job.jar file, 2 ways to
resolve the issue:
1, on linux, use zip command to delete the *.SF, *.DSA, *.RSA files shown
below, then run command "jarsigner -verify kylin-job-0.7.1-SNAPSHOT.jar" to
check whether the delete operations take effect, if you get information
like "jar is unsigned. (signatures missing or not parsable)" instead of
error information, which means the *.jar is OK now.
zip -d kylin-job-0.7.1-SNAPSHOT.jar META-INF/*.SF
zip -d kylin-job-0.7.1-SNAPSHOT.jar META-INF/*.DSA
zip -d kylin-job-0.7.1-SNAPSHOT.jar META-INF/*.RSA

2, fix the issue from the root cause, I changed the job/pom.xml again based
on the fix of issue-621, and submit a pull request:
https://github.com/KylinOLAP/Kylin/pull/450  on github, it works
successfully with cube building test, so, please review the pull request~

2015-03-25 17:56 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> All shade configuration lies in job/pom.xml
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/25/15, 5:45 PM, "dong wang" <el...@gmail.com> wrote:
>
> >after building, I checked the following 2 files:
> >jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT.jar
> >jar is unsigned. (signatures missing or not parsable)
> >
> >jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
> >jarsigner: java.lang.SecurityException: Invalid signature file digest for
> >Manifest main attributes
> >
> >and I also checked ./lib/kylin-job-0.7.1-SNAPSHOT.jar inside tomcat
> >directory of my build from source codes, it returns: jarsigner:
> >java.lang.SecurityException: Invalid signature file digest for Manifest
> >main attributes
> >
> >while official tgz file returns "jar is unsigned. (signatures missing or
> >not parsable)" for ./lib/kylin-job-0.7.1-SNAPSHOT.jar
> >
> >thus, there should be something wrong with
> >"./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar" only,  then, where can I
> >find the related configuration for
> >./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?
> >
> >2015-03-25 17:38 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
> >
> >> kylin-job-0.7.1-SNAPSHOT.jar is used as a dependency for KYLIN
> >>
> >> It only includes class in the job module
> >>
> >> kylin-job-0.7.1-SNAPSHOT-job.jar is the jar file which will be used by
> >>MR
> >> job
> >>
> >> It contains class in the job module and its dependency
> >>
> >> We use maven-shade-plugin to generate "kylin-job-0.7.1-SNAPSHOT-job.jar²
> >> You don¹t have to worry about the building process,
> >> just run mvm clean package -DskipTests
> >> Maven will do the rest of the work.
> >>
> >> Best Regard
> >> Zhou QianHao
> >>
> >>
> >>
> >>
> >>
> >> On 3/25/15, 5:31 PM, "dong wang" <el...@gmail.com> wrote:
> >>
> >> >Hi,Qianhao, no files such as *.SF, *.DSA, *.RSA files because of the
> >>fix
> >> >of
> >> >issue-621, I have another question, what's the differece between
> >> >./job/target/kylin-job-0.7.1-SNAPSHOT.jar and
> >> >./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar? and how to configure to
> >> >build ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?
> >>
> >>
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

All shade configuration lies in job/pom.xml

Best Regard
Zhou QianHao





On 3/25/15, 5:45 PM, "dong wang" <el...@gmail.com> wrote:

>after building, I checked the following 2 files:
>jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT.jar
>jar is unsigned. (signatures missing or not parsable)
>
>jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
>jarsigner: java.lang.SecurityException: Invalid signature file digest for
>Manifest main attributes
>
>and I also checked ./lib/kylin-job-0.7.1-SNAPSHOT.jar inside tomcat
>directory of my build from source codes, it returns: jarsigner:
>java.lang.SecurityException: Invalid signature file digest for Manifest
>main attributes
>
>while official tgz file returns "jar is unsigned. (signatures missing or
>not parsable)" for ./lib/kylin-job-0.7.1-SNAPSHOT.jar
>
>thus, there should be something wrong with
>"./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar" only,  then, where can I
>find the related configuration for
>./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?
>
>2015-03-25 17:38 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>
>> kylin-job-0.7.1-SNAPSHOT.jar is used as a dependency for KYLIN
>>
>> It only includes class in the job module
>>
>> kylin-job-0.7.1-SNAPSHOT-job.jar is the jar file which will be used by
>>MR
>> job
>>
>> It contains class in the job module and its dependency
>>
>> We use maven-shade-plugin to generate "kylin-job-0.7.1-SNAPSHOT-job.jar²
>> You don¹t have to worry about the building process,
>> just run mvm clean package -DskipTests
>> Maven will do the rest of the work.
>>
>> Best Regard
>> Zhou QianHao
>>
>>
>>
>>
>>
>> On 3/25/15, 5:31 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >Hi,Qianhao, no files such as *.SF, *.DSA, *.RSA files because of the
>>fix
>> >of
>> >issue-621, I have another question, what's the differece between
>> >./job/target/kylin-job-0.7.1-SNAPSHOT.jar and
>> >./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar? and how to configure to
>> >build ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?
>>
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

after building, I checked the following 2 files:
jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT.jar
jar is unsigned. (signatures missing or not parsable)

jarsigner -verify ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
jarsigner: java.lang.SecurityException: Invalid signature file digest for
Manifest main attributes

and I also checked ./lib/kylin-job-0.7.1-SNAPSHOT.jar inside tomcat
directory of my build from source codes, it returns: jarsigner:
java.lang.SecurityException: Invalid signature file digest for Manifest
main attributes

while official tgz file returns "jar is unsigned. (signatures missing or
not parsable)" for ./lib/kylin-job-0.7.1-SNAPSHOT.jar

thus, there should be something wrong with
"./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar" only,  then, where can I
find the related configuration for
./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?

2015-03-25 17:38 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> kylin-job-0.7.1-SNAPSHOT.jar is used as a dependency for KYLIN
>
> It only includes class in the job module
>
> kylin-job-0.7.1-SNAPSHOT-job.jar is the jar file which will be used by MR
> job
>
> It contains class in the job module and its dependency
>
> We use maven-shade-plugin to generate "kylin-job-0.7.1-SNAPSHOT-job.jar²
> You don¹t have to worry about the building process,
> just run mvm clean package -DskipTests
> Maven will do the rest of the work.
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/25/15, 5:31 PM, "dong wang" <el...@gmail.com> wrote:
>
> >Hi,Qianhao, no files such as *.SF, *.DSA, *.RSA files because of the fix
> >of
> >issue-621, I have another question, what's the differece between
> >./job/target/kylin-job-0.7.1-SNAPSHOT.jar and
> >./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar? and how to configure to
> >build ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

kylin-job-0.7.1-SNAPSHOT.jar is used as a dependency for KYLIN

It only includes class in the job module

kylin-job-0.7.1-SNAPSHOT-job.jar is the jar file which will be used by MR
job

It contains class in the job module and its dependency

We use maven-shade-plugin to generate "kylin-job-0.7.1-SNAPSHOT-job.jar²
You don¹t have to worry about the building process,
just run mvm clean package -DskipTests
Maven will do the rest of the work.

Best Regard
Zhou QianHao

On 3/25/15, 5:31 PM, "dong wang" <el...@gmail.com> wrote:

>Hi,Qianhao, no files such as *.SF, *.DSA, *.RSA files because of the fix
>of
>issue-621, I have another question, what's the differece between
>./job/target/kylin-job-0.7.1-SNAPSHOT.jar and
>./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar? and how to configure to
>build ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

Hi,Qianhao, no files such as *.SF, *.DSA, *.RSA files because of the fix of
issue-621, I have another question, what's the differece between
./job/target/kylin-job-0.7.1-SNAPSHOT.jar and
./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar? and how to configure to
build ./job/target/kylin-job-0.7.1-SNAPSHOT-job.jar?

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

It should be related with the build environment.
@dong can you please check if there is any *.SF *.DSA *.RSA file in the
kylin-job-0.7.1-SNAPSHOT.jar



Best Regard
Zhou QianHao





On 3/25/15, 4:03 PM, "dong wang" <el...@gmail.com> wrote:

>according to:
>https://docs.oracle.com/javase/tutorial/deployment/jar/verify.html
>
>and when running: jarsigner -verify ./lib/kylin-job-0.7.1-SNAPSHOT.jar, it
>says
>jarsigner: java.lang.SecurityException: Invalid signature file digest for
>Manifest main attributes
>
>thus, there should be still something wrong with this jar when building
>process
>
>2015-03-25 15:52 GMT+08:00 dong wang <el...@gmail.com>:
>
>> note that the issue only happens when building the source code from
>> github, it works OK if directly downloading the tgz file from official
>>site~
>>
>> 2015-03-25 15:35 GMT+08:00 dong wang <el...@gmail.com>:
>>
>>> Hi all, today, I checked out the latest code form
>>> https://github.com/KylinOLAP/Kylin/tree/staging, it can be built
>>> successfully, however, when starting building the cube, the above error
>>> occurs again~ is there anyone who meets with the same problem?
>>>
>>> 2015-03-05 10:32 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>>>
>>>> I have created a ticket:
>>>> https://issues.apache.org/jira/browse/KYLIN-621
>>>>
>>>>
>>>> Best Regard
>>>> Zhou QianHao
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:
>>>>
>>>> >currently, the 2nd issue is also gone, it is caused by that I just
>>>> >installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of
>>>>ALL
>>>> >the
>>>> >nodes in the cluster(actually, when kylin-0.6x, it is not necessary
>>>>for
>>>> >user to install the "hive-hcatalog" in ANY node with CDH 5.3.1
>>>>before),
>>>> >therefore, in the cluster's DATA nodes, it says "Error:
>>>> >java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>>>> >org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
>>>> >
>>>> >2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
>>>> >
>>>> >> 1,  " Job init failed : java.lang.SecurityException: Invalid
>>>>signature
>>>> >> file digest for Manifest main attributes" is GONE after updating
>>>>the
>>>> >> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned
>>>>above
>>>> >> under "job" directory for the job.jar
>>>> >>
>>>> >> 2, to the 2nd error mentioned above, as checked the job log, it
>>>>shows:
>>>> >> Error: java.lang.RuntimeException:
>>>>java.lang.ClassNotFoundException:
>>>> >>Class
>>>> >> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>>>> >>
>>>> >>
>>>> >> however, in tomcat/logs/kylin.log as below, we have actually added
>>>>the
>>>> >> hcatalog lib for the job and
>>>> >>
>>>>
>>>> 
>>>>>>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.
>>>>>>3.2.
>>>> >>jar
>>>> >> indeeded exists.
>>>> >>
>>>> >>
>>>> >> [pool-7-thread-4]:[2015-03-04
>>>> >>
>>>>
>>>> 
>>>>>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.set
>>>>>>JobC
>>>> >>lasspath(AbstractHadoopJob.java:146)]
>>>> >> - append job jar:
>>>> >>
>>>>
>>>> 
>>>>>>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPS
>>>>>>HOT.
>>>> >>jar
>>>> >> [pool-7-thread-4]:[2015-03-04
>>>> >>
>>>>
>>>> 
>>>>>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.set
>>>>>>JobC
>>>> >>lasspath(AbstractHadoopJob.java:152)]
>>>> >> - append kylin.hive.dependency:
>>>> >>
>>>>
>>>> 
>>>>>>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/h
>>>>>>cata
>>>> >>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
>>>> >> to mapreduce.application.classpath
>>>> >>
>>>>
>>>>
>>>
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

according to:
https://docs.oracle.com/javase/tutorial/deployment/jar/verify.html

and when running: jarsigner -verify ./lib/kylin-job-0.7.1-SNAPSHOT.jar, it
says
jarsigner: java.lang.SecurityException: Invalid signature file digest for
Manifest main attributes

thus, there should be still something wrong with this jar when building
process

2015-03-25 15:52 GMT+08:00 dong wang <el...@gmail.com>:

> note that the issue only happens when building the source code from
> github, it works OK if directly downloading the tgz file from official site~
>
> 2015-03-25 15:35 GMT+08:00 dong wang <el...@gmail.com>:
>
>> Hi all, today, I checked out the latest code form
>> https://github.com/KylinOLAP/Kylin/tree/staging, it can be built
>> successfully, however, when starting building the cube, the above error
>> occurs again~ is there anyone who meets with the same problem?
>>
>> 2015-03-05 10:32 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>>
>>> I have created a ticket:
>>> https://issues.apache.org/jira/browse/KYLIN-621
>>>
>>>
>>> Best Regard
>>> Zhou QianHao
>>>
>>>
>>>
>>>
>>>
>>> On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:
>>>
>>> >currently, the 2nd issue is also gone, it is caused by that I just
>>> >installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL
>>> >the
>>> >nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
>>> >user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
>>> >therefore, in the cluster's DATA nodes, it says "Error:
>>> >java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>>> >org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
>>> >
>>> >2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
>>> >
>>> >> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
>>> >> file digest for Manifest main attributes" is GONE after updating the
>>> >> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
>>> >> under "job" directory for the job.jar
>>> >>
>>> >> 2, to the 2nd error mentioned above, as checked the job log, it shows:
>>> >> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> >>Class
>>> >> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>>> >>
>>> >>
>>> >> however, in tomcat/logs/kylin.log as below, we have actually added the
>>> >> hcatalog lib for the job and
>>> >>
>>>
>>> >>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.
>>> >>jar
>>> >> indeeded exists.
>>> >>
>>> >>
>>> >> [pool-7-thread-4]:[2015-03-04
>>> >>
>>>
>>> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>> >>lasspath(AbstractHadoopJob.java:146)]
>>> >> - append job jar:
>>> >>
>>>
>>> >>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.
>>> >>jar
>>> >> [pool-7-thread-4]:[2015-03-04
>>> >>
>>>
>>> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>> >>lasspath(AbstractHadoopJob.java:152)]
>>> >> - append kylin.hive.dependency:
>>> >>
>>>
>>> >>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcata
>>> >>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
>>> >> to mapreduce.application.classpath
>>> >>
>>>
>>>
>>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

note that the issue only happens when building the source code from github,
it works OK if directly downloading the tgz file from official site~

2015-03-25 15:35 GMT+08:00 dong wang <el...@gmail.com>:

> Hi all, today, I checked out the latest code form
> https://github.com/KylinOLAP/Kylin/tree/staging, it can be built
> successfully, however, when starting building the cube, the above error
> occurs again~ is there anyone who meets with the same problem?
>
> 2015-03-05 10:32 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>
>> I have created a ticket:
>> https://issues.apache.org/jira/browse/KYLIN-621
>>
>>
>> Best Regard
>> Zhou QianHao
>>
>>
>>
>>
>>
>> On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >currently, the 2nd issue is also gone, it is caused by that I just
>> >installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL
>> >the
>> >nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
>> >user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
>> >therefore, in the cluster's DATA nodes, it says "Error:
>> >java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>> >org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
>> >
>> >2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
>> >
>> >> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
>> >> file digest for Manifest main attributes" is GONE after updating the
>> >> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
>> >> under "job" directory for the job.jar
>> >>
>> >> 2, to the 2nd error mentioned above, as checked the job log, it shows:
>> >> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> >>Class
>> >> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>> >>
>> >>
>> >> however, in tomcat/logs/kylin.log as below, we have actually added the
>> >> hcatalog lib for the job and
>> >>
>>
>> >>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.
>> >>jar
>> >> indeeded exists.
>> >>
>> >>
>> >> [pool-7-thread-4]:[2015-03-04
>> >>
>>
>> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>> >>lasspath(AbstractHadoopJob.java:146)]
>> >> - append job jar:
>> >>
>>
>> >>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.
>> >>jar
>> >> [pool-7-thread-4]:[2015-03-04
>> >>
>>
>> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>> >>lasspath(AbstractHadoopJob.java:152)]
>> >> - append kylin.hive.dependency:
>> >>
>>
>> >>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcata
>> >>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
>> >> to mapreduce.application.classpath
>> >>
>>
>>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

Hi all, today, I checked out the latest code form
https://github.com/KylinOLAP/Kylin/tree/staging, it can be built
successfully, however, when starting building the cube, the above error
occurs again~ is there anyone who meets with the same problem?

2015-03-05 10:32 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> I have created a ticket:
> https://issues.apache.org/jira/browse/KYLIN-621
>
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:
>
> >currently, the 2nd issue is also gone, it is caused by that I just
> >installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL
> >the
> >nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
> >user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
> >therefore, in the cluster's DATA nodes, it says "Error:
> >java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> >org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
> >
> >2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
> >
> >> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
> >> file digest for Manifest main attributes" is GONE after updating the
> >> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
> >> under "job" directory for the job.jar
> >>
> >> 2, to the 2nd error mentioned above, as checked the job log, it shows:
> >> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> >>Class
> >> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> >>
> >>
> >> however, in tomcat/logs/kylin.log as below, we have actually added the
> >> hcatalog lib for the job and
> >>
> >>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.
> >>jar
> >> indeeded exists.
> >>
> >>
> >> [pool-7-thread-4]:[2015-03-04
> >>
> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
> >>lasspath(AbstractHadoopJob.java:146)]
> >> - append job jar:
> >>
> >>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.
> >>jar
> >> [pool-7-thread-4]:[2015-03-04
> >>
> >>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
> >>lasspath(AbstractHadoopJob.java:152)]
> >> - append kylin.hive.dependency:
> >>
> >>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcata
> >>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
> >> to mapreduce.application.classpath
> >>
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

I have created a ticket:
https://issues.apache.org/jira/browse/KYLIN-621


Best Regard
Zhou QianHao





On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:

>currently, the 2nd issue is also gone, it is caused by that I just
>installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL
>the
>nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
>user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
>therefore, in the cluster's DATA nodes, it says "Error:
>java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
>
>2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
>
>> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
>> file digest for Manifest main attributes" is GONE after updating the
>> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
>> under "job" directory for the job.jar
>>
>> 2, to the 2nd error mentioned above, as checked the job log, it shows:
>> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>Class
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>>
>>
>> however, in tomcat/logs/kylin.log as below, we have actually added the
>> hcatalog lib for the job and
>> 
>>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.
>>jar
>> indeeded exists.
>>
>>
>> [pool-7-thread-4]:[2015-03-04
>> 
>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>lasspath(AbstractHadoopJob.java:146)]
>> - append job jar:
>> 
>>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.
>>jar
>> [pool-7-thread-4]:[2015-03-04
>> 
>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>lasspath(AbstractHadoopJob.java:152)]
>> - append kylin.hive.dependency:
>> 
>>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcata
>>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
>> to mapreduce.application.classpath
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Shi, Shaofeng" <sh...@ebay.com>.

Good catch; Yes the dependency on Hcatalog was introduced since 0.7;
Having all nodes has the same map reduce class path is a basic requirement;

On 3/4/15, 11:30 PM, "dong wang" <el...@gmail.com> wrote:

>currently, the 2nd issue is also gone, it is caused by that I just
>installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL
>the
>nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
>user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
>therefore, in the cluster's DATA nodes, it says "Error:
>java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".
>
>2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:
>
>> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
>> file digest for Manifest main attributes" is GONE after updating the
>> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
>> under "job" directory for the job.jar
>>
>> 2, to the 2nd error mentioned above, as checked the job log, it shows:
>> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>Class
>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>>
>>
>> however, in tomcat/logs/kylin.log as below, we have actually added the
>> hcatalog lib for the job and
>> 
>>/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.
>>jar
>> indeeded exists.
>>
>>
>> [pool-7-thread-4]:[2015-03-04
>> 
>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>lasspath(AbstractHadoopJob.java:146)]
>> - append job jar:
>> 
>>/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.
>>jar
>> [pool-7-thread-4]:[2015-03-04
>> 
>>22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobC
>>lasspath(AbstractHadoopJob.java:152)]
>> - append kylin.hive.dependency:
>> 
>>/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcata
>>log/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
>> to mapreduce.application.classpath
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

currently, the 2nd issue is also gone, it is caused by that I just
installed "hive-hcatalog" in MASTER node when kylin-0.71 instead of ALL the
nodes in the cluster(actually, when kylin-0.6x, it is not necessary for
user to install the "hive-hcatalog" in ANY node with CDH 5.3.1 before),
therefore, in the cluster's DATA nodes, it says "Error:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found".

2015-03-04 22:29 GMT+08:00 dong wang <el...@gmail.com>:

> 1,  " Job init failed : java.lang.SecurityException: Invalid signature
> file digest for Manifest main attributes" is GONE after updating the
> pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned above
> under "job" directory for the job.jar
>
> 2, to the 2nd error mentioned above, as checked the job log, it shows:
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
>
>
> however, in tomcat/logs/kylin.log as below, we have actually added the
> hcatalog lib for the job and
> /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
> indeeded exists.
>
>
> [pool-7-thread-4]:[2015-03-04
> 22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:146)]
> - append job jar:
> /home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.jar
> [pool-7-thread-4]:[2015-03-04
> 22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:152)]
> - append kylin.hive.dependency:
> /usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
> to mapreduce.application.classpath
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

1,  " Job init failed : java.lang.SecurityException: Invalid signature file
digest for Manifest main attributes" is GONE after updating the pom.xml
with adding "<exclude>META-INF/*.SF</exclude>" mentioned above under "job"
directory for the job.jar

2, to the 2nd error mentioned above, as checked the job log, it shows:
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found


however, in tomcat/logs/kylin.log as below, we have actually added the
hcatalog lib for the job and
/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
indeeded exists.


[pool-7-thread-4]:[2015-03-04
22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:146)]
- append job jar:
/home/user/workspace/kylin-0.7.1-SNAPSHOT/./lib/kylin-job-0.7.1-SNAPSHOT.jar
[pool-7-thread-4]:[2015-03-04
22:08:15,357][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:152)]
- append kylin.hive.dependency:
/usr/lib/hive/conf:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.2.jar
to mapreduce.application.classpath

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

as the previous reply, currently after changing the pom for job.jar as
mentioned, the error changes as below:
Task failed task_1425461099463_0009_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

another thing is that the same error occurs when I use the tgz file from
official build.

2015-03-04 20:40 GMT+08:00 dong wang <el...@gmail.com>:

> CDH 5.3.1 + java version "1.7.0_75"
>
> 2015-03-04 20:34 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>
>> Hi dong
>>    Can you please tell us what exactly is your java version of your
>> sandbox.
>>    According to
>> http://stackoverflow.com/questions/8302022/maven-shade-jar-throw-exception
>>    The root cause maybe the java version
>>
>> Best Regard
>> Zhou QianHao
>>
>>
>>
>>
>>
>> On 3/4/15, 7:08 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >I built the jar from the source, as checked, only, there is a file named
>> >*.SF under job/target/kylin-job-0.7.1-SNAPSHOT-job.jar,  then I tried to
>> >update the pom.xml with adding "<exclude>META-INF/*.SF</exclude>"
>> >mentioned
>> >above under "job" directory for the job.jar, then build again, however,
>> >currently, there is stranger problem with few job info:
>> >
>> >Task failed task_1425461099463_0005_m_000000
>> >Job failed as tasks failed. failedMaps:1 failedReduces:0
>> >
>> >
>> >
>> >2015-03-04 18:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>> >
>> >> @Dong
>> >>    Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz?
>> >>
>> >>    If from source code, you will see
>> >>    job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
>> >>    storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar
>> >>
>> >>    If from tgz
>> >>    lib/kylin-job-0.7.1-SNAPSHOT.jar
>> >>    lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar
>> >>
>> >>    Please check if there are *.SF *.RSA *.DSA files in these two jars.
>> >>If
>> >> they do exist, as a workaround, you can remove these files from the
>> jar.
>> >>    Please let us know if it works, and then we will fix it.
>> >>
>> >> Best Regard
>> >> Zhou QianHao
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On 3/4/15, 5:48 PM, "dong wang" <el...@gmail.com> wrote:
>> >>
>> >> >I'm not sure whether the problem is similar to:
>> >> >http://www.cnblogs.com/fuxinci/p/3356087.html
>> >> >
>> >> >2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >
>> >> >> As checked, it says:
>> >> >>
>> >> >> Job init failed : java.lang.SecurityException: Invalid signature
>> file
>> >> >> digest for Manifest main attributes
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifi
>> >>>>er
>> >> >>.java:286)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.j
>> >>>>av
>> >> >>a:239)
>> >> >> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
>> >> >> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
>> >> >> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
>> >> >> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
>> >> >> at
>> >>
>>
>> >>>>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
>> >> >> at sun.misc.Resource.cachedInputStream(Resource.java:77)
>> >> >> at sun.misc.Resource.getByteBuffer(Resource.java:160)
>> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
>> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> >> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> >> >> at java.lang.Class.forName0(Native Method)
>> >> >> at java.lang.Class.forName(Class.java:191)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.
>> >>>>ja
>> >> >>va:1305)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(Jo
>> >>>>bI
>> >> >>mpl.java:1247)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl
>> >>>>.j
>> >> >>ava:139)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
>> >>>>si
>> >> >>tion(JobImpl.java:1451)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
>> >>>>si
>> >> >>tion(JobImpl.java:1388)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.do
>> >>>>Tr
>> >> >>ansition(StateMachineFactory.java:385)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMach
>> >>>>in
>> >> >>eFactory.java:302)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachin
>> >>>>eF
>> >> >>actory.java:46)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.d
>> >>>>oT
>> >> >>ransition(StateMachineFactory.java:448)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
>> >>>>:9
>> >> >>96)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
>> >>>>:1
>> >> >>38)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handl
>> >>>>e(
>> >> >>MRAppMaster.java:1272)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster
>> >>>>.j
>> >> >>ava:1045)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.service.AbstractService.start(AbstractService.java:19
>> >>>>3)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1
>> >>>>47
>> >> >>8)
>> >> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> >> at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> >>>>on
>> >> >>.java:1642)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MR
>> >>>>Ap
>> >> >>pMaster.java:1474)
>> >> >> at
>> >> >>
>> >>
>>
>> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:14
>> >>>>07
>> >> >>)
>> >> >>
>> >> >> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>> >> >>
>> >> >>> @Dong
>> >> >>>     The map-reduce job may encounter some error, can you please
>> look
>> >> >>>into
>> >> >>> the job detail.
>> >> >>>     Just FYI, you can use http://$job history
>> >> >>> server$/jobhistory/job/job_1425448727678_0003/ to see what is
>> really
>> >> >>>going
>> >> >>> on.
>> >> >>>
>> >> >>> Best Regard
>> >> >>> Zhou QianHao
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
>> >> >>>
>> >> >>> >more error info:
>> >> >>> >
>> >> >>> >
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
>> >>>>>>he
>> >> >>>>ckS
>> >> >>> >tatus(HadoopStatusChecker.java:57)]
>> >> >>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getR
>> >>>>>>es
>> >> >>>>pon
>> >> >>> >seBody(HttpMethodBase.java:682)]
>> >> >>> >- Going to buffer response body of large or unknown size. Using
>> >> >>> >getResponseBodyAsStream instead is recommended.
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>> >>>>>>14:19:51,047][DEBUG][
>> org.apache.kylin.job.tools.HadoopStatusGetter.ge
>> >>>>>>tH
>> >> >>>>ttp
>> >> >>> >Response(HadoopStatusGetter.java:90)]
>> >> >>> >- Job job_1425448727678_0003 get status check result.
>> >> >>> >
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
>> >>>>>>he
>> >> >>>>ckS
>> >> >>> >tatus(HadoopStatusChecker.java:57)]
>> >> >>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
>> >> >>> >mapred.ClientServiceDelegate: Application state is completed.
>> >> >>> >FinalApplicationStatus=FAILED. Redirecting to job history server
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updat
>> >>>>>>eJ
>> >> >>>>obC
>> >> >>> >ounter(HadoopCmdOutput.java:89)]
>> >> >>> >- no counters for job job_1425448727678_0003
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>> >>>>>>e.
>> >> >>>>put
>> >> >>> >Resource(ResourceStore.java:171)]
>> >> >>> >- Saving resource
>> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >> >>> >(Store kylin_metadata@hbase)
>> >> >>> >[pool-7-thread-3]:[2015-03-04
>> >> >>>
>> >> >>>
>> >>
>>
>> >>>>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.up
>> >>>>>>da
>> >> >>>>teJ
>> >> >>> >obOutput(ExecutableManager.java:222)]
>> >> >>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to
>> >>ERROR
>> >> >>> >
>> >> >>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
>> >> >>> >
>> >> >>> >> when check the log shown in web GUI of this step:
>> >> >>> >>
>> >> >>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
>> >> >>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
>> >> >>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
>> >> >>> >> job_1425448727678_0004:FINISHED - FAILED
>> >> >>> >> no counters for job job_1425448727678_0004
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> with the same cube, it can be built successfully with kylin-0.6*
>> >> >>>before
>> >> >>> >>
>> >> >>>
>> >> >>>
>> >> >>
>> >>
>> >>
>>
>>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

CDH 5.3.1 + java version "1.7.0_75"

2015-03-04 20:34 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> Hi dong
>    Can you please tell us what exactly is your java version of your
> sandbox.
>    According to
> http://stackoverflow.com/questions/8302022/maven-shade-jar-throw-exception
>    The root cause maybe the java version
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/4/15, 7:08 PM, "dong wang" <el...@gmail.com> wrote:
>
> >I built the jar from the source, as checked, only, there is a file named
> >*.SF under job/target/kylin-job-0.7.1-SNAPSHOT-job.jar,  then I tried to
> >update the pom.xml with adding "<exclude>META-INF/*.SF</exclude>"
> >mentioned
> >above under "job" directory for the job.jar, then build again, however,
> >currently, there is stranger problem with few job info:
> >
> >Task failed task_1425461099463_0005_m_000000
> >Job failed as tasks failed. failedMaps:1 failedReduces:0
> >
> >
> >
> >2015-03-04 18:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
> >
> >> @Dong
> >>    Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz?
> >>
> >>    If from source code, you will see
> >>    job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
> >>    storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar
> >>
> >>    If from tgz
> >>    lib/kylin-job-0.7.1-SNAPSHOT.jar
> >>    lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar
> >>
> >>    Please check if there are *.SF *.RSA *.DSA files in these two jars.
> >>If
> >> they do exist, as a workaround, you can remove these files from the jar.
> >>    Please let us know if it works, and then we will fix it.
> >>
> >> Best Regard
> >> Zhou QianHao
> >>
> >>
> >>
> >>
> >>
> >> On 3/4/15, 5:48 PM, "dong wang" <el...@gmail.com> wrote:
> >>
> >> >I'm not sure whether the problem is similar to:
> >> >http://www.cnblogs.com/fuxinci/p/3356087.html
> >> >
> >> >2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:
> >> >
> >> >> As checked, it says:
> >> >>
> >> >> Job init failed : java.lang.SecurityException: Invalid signature file
> >> >> digest for Manifest main attributes
> >> >> at
> >> >>
> >>
> >>>>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifi
> >>>>er
> >> >>.java:286)
> >> >> at
> >> >>
> >>
> >>>>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.j
> >>>>av
> >> >>a:239)
> >> >> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
> >> >> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
> >> >> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
> >> >> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
> >> >> at
> >>
> >>>>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> >> >> at sun.misc.Resource.cachedInputStream(Resource.java:77)
> >> >> at sun.misc.Resource.getByteBuffer(Resource.java:160)
> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >> >> at java.security.AccessController.doPrivileged(Native Method)
> >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> >> >> at java.lang.Class.forName0(Native Method)
> >> >> at java.lang.Class.forName(Class.java:191)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.
> >>>>ja
> >> >>va:1305)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(Jo
> >>>>bI
> >> >>mpl.java:1247)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl
> >>>>.j
> >> >>ava:139)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
> >>>>si
> >> >>tion(JobImpl.java:1451)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
> >>>>si
> >> >>tion(JobImpl.java:1388)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.do
> >>>>Tr
> >> >>ansition(StateMachineFactory.java:385)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMach
> >>>>in
> >> >>eFactory.java:302)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachin
> >>>>eF
> >> >>actory.java:46)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.d
> >>>>oT
> >> >>ransition(StateMachineFactory.java:448)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
> >>>>:9
> >> >>96)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
> >>>>:1
> >> >>38)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handl
> >>>>e(
> >> >>MRAppMaster.java:1272)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster
> >>>>.j
> >> >>ava:1045)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.service.AbstractService.start(AbstractService.java:19
> >>>>3)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1
> >>>>47
> >> >>8)
> >> >> at java.security.AccessController.doPrivileged(Native Method)
> >> >> at javax.security.auth.Subject.doAs(Subject.java:415)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>>>on
> >> >>.java:1642)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MR
> >>>>Ap
> >> >>pMaster.java:1474)
> >> >> at
> >> >>
> >>
> >>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:14
> >>>>07
> >> >>)
> >> >>
> >> >> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
> >> >>
> >> >>> @Dong
> >> >>>     The map-reduce job may encounter some error, can you please look
> >> >>>into
> >> >>> the job detail.
> >> >>>     Just FYI, you can use http://$job history
> >> >>> server$/jobhistory/job/job_1425448727678_0003/ to see what is really
> >> >>>going
> >> >>> on.
> >> >>>
> >> >>> Best Regard
> >> >>> Zhou QianHao
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
> >> >>>
> >> >>> >more error info:
> >> >>> >
> >> >>> >
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
> >>>>>>he
> >> >>>>ckS
> >> >>> >tatus(HadoopStatusChecker.java:57)]
> >> >>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getR
> >>>>>>es
> >> >>>>pon
> >> >>> >seBody(HttpMethodBase.java:682)]
> >> >>> >- Going to buffer response body of large or unknown size. Using
> >> >>> >getResponseBodyAsStream instead is recommended.
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,047][DEBUG][
> org.apache.kylin.job.tools.HadoopStatusGetter.ge
> >>>>>>tH
> >> >>>>ttp
> >> >>> >Response(HadoopStatusGetter.java:90)]
> >> >>> >- Job job_1425448727678_0003 get status check result.
> >> >>> >
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
> >>>>>>he
> >> >>>>ckS
> >> >>> >tatus(HadoopStatusChecker.java:57)]
> >> >>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
> >> >>> >mapred.ClientServiceDelegate: Application state is completed.
> >> >>> >FinalApplicationStatus=FAILED. Redirecting to job history server
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updat
> >>>>>>eJ
> >> >>>>obC
> >> >>> >ounter(HadoopCmdOutput.java:89)]
> >> >>> >- no counters for job job_1425448727678_0003
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStor
> >>>>>>e.
> >> >>>>put
> >> >>> >Resource(ResourceStore.java:171)]
> >> >>> >- Saving resource
> >> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >> >>> >(Store kylin_metadata@hbase)
> >> >>> >[pool-7-thread-3]:[2015-03-04
> >> >>>
> >> >>>
> >>
> >>>>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.up
> >>>>>>da
> >> >>>>teJ
> >> >>> >obOutput(ExecutableManager.java:222)]
> >> >>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to
> >>ERROR
> >> >>> >
> >> >>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
> >> >>> >
> >> >>> >> when check the log shown in web GUI of this step:
> >> >>> >>
> >> >>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
> >> >>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
> >> >>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
> >> >>> >> job_1425448727678_0004:FINISHED - FAILED
> >> >>> >> no counters for job job_1425448727678_0004
> >> >>> >>
> >> >>> >>
> >> >>> >> with the same cube, it can be built successfully with kylin-0.6*
> >> >>>before
> >> >>> >>
> >> >>>
> >> >>>
> >> >>
> >>
> >>
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

Hi dong
   Can you please tell us what exactly is your java version of your
sandbox.
   According to 
http://stackoverflow.com/questions/8302022/maven-shade-jar-throw-exception
   The root cause maybe the java version

Best Regard
Zhou QianHao





On 3/4/15, 7:08 PM, "dong wang" <el...@gmail.com> wrote:

>I built the jar from the source, as checked, only, there is a file named
>*.SF under job/target/kylin-job-0.7.1-SNAPSHOT-job.jar,  then I tried to
>update the pom.xml with adding "<exclude>META-INF/*.SF</exclude>"
>mentioned
>above under "job" directory for the job.jar, then build again, however,
>currently, there is stranger problem with few job info:
>
>Task failed task_1425461099463_0005_m_000000
>Job failed as tasks failed. failedMaps:1 failedReduces:0
>
>
>
>2015-03-04 18:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>
>> @Dong
>>    Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz?
>>
>>    If from source code, you will see
>>    job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
>>    storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar
>>
>>    If from tgz
>>    lib/kylin-job-0.7.1-SNAPSHOT.jar
>>    lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar
>>
>>    Please check if there are *.SF *.RSA *.DSA files in these two jars.
>>If
>> they do exist, as a workaround, you can remove these files from the jar.
>>    Please let us know if it works, and then we will fix it.
>>
>> Best Regard
>> Zhou QianHao
>>
>>
>>
>>
>>
>> On 3/4/15, 5:48 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >I'm not sure whether the problem is similar to:
>> >http://www.cnblogs.com/fuxinci/p/3356087.html
>> >
>> >2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:
>> >
>> >> As checked, it says:
>> >>
>> >> Job init failed : java.lang.SecurityException: Invalid signature file
>> >> digest for Manifest main attributes
>> >> at
>> >>
>> 
>>>>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifi
>>>>er
>> >>.java:286)
>> >> at
>> >>
>> 
>>>>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.j
>>>>av
>> >>a:239)
>> >> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
>> >> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
>> >> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
>> >> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
>> >> at
>> 
>>>>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
>> >> at sun.misc.Resource.cachedInputStream(Resource.java:77)
>> >> at sun.misc.Resource.getByteBuffer(Resource.java:160)
>> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
>> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> >> at java.lang.Class.forName0(Native Method)
>> >> at java.lang.Class.forName(Class.java:191)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.
>>>>ja
>> >>va:1305)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(Jo
>>>>bI
>> >>mpl.java:1247)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl
>>>>.j
>> >>ava:139)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
>>>>si
>> >>tion(JobImpl.java:1451)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.tran
>>>>si
>> >>tion(JobImpl.java:1388)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.do
>>>>Tr
>> >>ansition(StateMachineFactory.java:385)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMach
>>>>in
>> >>eFactory.java:302)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachin
>>>>eF
>> >>actory.java:46)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.d
>>>>oT
>> >>ransition(StateMachineFactory.java:448)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
>>>>:9
>> >>96)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java
>>>>:1
>> >>38)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handl
>>>>e(
>> >>MRAppMaster.java:1272)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster
>>>>.j
>> >>ava:1045)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.service.AbstractService.start(AbstractService.java:19
>>>>3)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1
>>>>47
>> >>8)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>>>>on
>> >>.java:1642)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MR
>>>>Ap
>> >>pMaster.java:1474)
>> >> at
>> >>
>> 
>>>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:14
>>>>07
>> >>)
>> >>
>> >> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>> >>
>> >>> @Dong
>> >>>     The map-reduce job may encounter some error, can you please look
>> >>>into
>> >>> the job detail.
>> >>>     Just FYI, you can use http://$job history
>> >>> server$/jobhistory/job/job_1425448727678_0003/ to see what is really
>> >>>going
>> >>> on.
>> >>>
>> >>> Best Regard
>> >>> Zhou QianHao
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
>> >>>
>> >>> >more error info:
>> >>> >
>> >>> >
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
>>>>>>he
>> >>>>ckS
>> >>> >tatus(HadoopStatusChecker.java:57)]
>> >>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getR
>>>>>>es
>> >>>>pon
>> >>> >seBody(HttpMethodBase.java:682)]
>> >>> >- Going to buffer response body of large or unknown size. Using
>> >>> >getResponseBodyAsStream instead is recommended.
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.ge
>>>>>>tH
>> >>>>ttp
>> >>> >Response(HadoopStatusGetter.java:90)]
>> >>> >- Job job_1425448727678_0003 get status check result.
>> >>> >
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.c
>>>>>>he
>> >>>>ckS
>> >>> >tatus(HadoopStatusChecker.java:57)]
>> >>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
>> >>> >mapred.ClientServiceDelegate: Application state is completed.
>> >>> >FinalApplicationStatus=FAILED. Redirecting to job history server
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updat
>>>>>>eJ
>> >>>>obC
>> >>> >ounter(HadoopCmdOutput.java:89)]
>> >>> >- no counters for job job_1425448727678_0003
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStor
>>>>>>e.
>> >>>>put
>> >>> >Resource(ResourceStore.java:171)]
>> >>> >- Saving resource
>> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >>> >(Store kylin_metadata@hbase)
>> >>> >[pool-7-thread-3]:[2015-03-04
>> >>>
>> >>>
>> 
>>>>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.up
>>>>>>da
>> >>>>teJ
>> >>> >obOutput(ExecutableManager.java:222)]
>> >>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to
>>ERROR
>> >>> >
>> >>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
>> >>> >
>> >>> >> when check the log shown in web GUI of this step:
>> >>> >>
>> >>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
>> >>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
>> >>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
>> >>> >> job_1425448727678_0004:FINISHED - FAILED
>> >>> >> no counters for job job_1425448727678_0004
>> >>> >>
>> >>> >>
>> >>> >> with the same cube, it can be built successfully with kylin-0.6*
>> >>>before
>> >>> >>
>> >>>
>> >>>
>> >>
>>
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

I also tried with tgz from official binary build, it says the same error as
above

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

I built the jar from the source, as checked, only, there is a file named
*.SF under job/target/kylin-job-0.7.1-SNAPSHOT-job.jar,  then I tried to
update the pom.xml with adding "<exclude>META-INF/*.SF</exclude>" mentioned
above under "job" directory for the job.jar, then build again, however,
currently, there is stranger problem with few job info:

Task failed task_1425461099463_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0



2015-03-04 18:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> @Dong
>    Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz?
>
>    If from source code, you will see
>    job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
>    storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar
>
>    If from tgz
>    lib/kylin-job-0.7.1-SNAPSHOT.jar
>    lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar
>
>    Please check if there are *.SF *.RSA *.DSA files in these two jars. If
> they do exist, as a workaround, you can remove these files from the jar.
>    Please let us know if it works, and then we will fix it.
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/4/15, 5:48 PM, "dong wang" <el...@gmail.com> wrote:
>
> >I'm not sure whether the problem is similar to:
> >http://www.cnblogs.com/fuxinci/p/3356087.html
> >
> >2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:
> >
> >> As checked, it says:
> >>
> >> Job init failed : java.lang.SecurityException: Invalid signature file
> >> digest for Manifest main attributes
> >> at
> >>
> >>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier
> >>.java:286)
> >> at
> >>
> >>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.jav
> >>a:239)
> >> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
> >> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
> >> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
> >> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
> >> at
> >>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> >> at sun.misc.Resource.cachedInputStream(Resource.java:77)
> >> at sun.misc.Resource.getByteBuffer(Resource.java:160)
> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:191)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.ja
> >>va:1305)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(JobI
> >>mpl.java:1247)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl.j
> >>ava:139)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transi
> >>tion(JobImpl.java:1451)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transi
> >>tion(JobImpl.java:1388)
> >> at
> >>
> >>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTr
> >>ansition(StateMachineFactory.java:385)
> >> at
> >>
> >>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachin
> >>eFactory.java:302)
> >> at
> >>
> >>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineF
> >>actory.java:46)
> >> at
> >>
> >>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doT
> >>ransition(StateMachineFactory.java:448)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:9
> >>96)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1
> >>38)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(
> >>MRAppMaster.java:1272)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.j
> >>ava:1045)
> >> at
> >>
> >>org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:147
> >>8)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:415)
> >> at
> >>
> >>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
> >>.java:1642)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp
> >>pMaster.java:1474)
> >> at
> >>
> >>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407
> >>)
> >>
> >> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
> >>
> >>> @Dong
> >>>     The map-reduce job may encounter some error, can you please look
> >>>into
> >>> the job detail.
> >>>     Just FYI, you can use http://$job history
> >>> server$/jobhistory/job/job_1425448727678_0003/ to see what is really
> >>>going
> >>> on.
> >>>
> >>> Best Regard
> >>> Zhou QianHao
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
> >>>
> >>> >more error info:
> >>> >
> >>> >
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.che
> >>>>ckS
> >>> >tatus(HadoopStatusChecker.java:57)]
> >>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getRes
> >>>>pon
> >>> >seBody(HttpMethodBase.java:682)]
> >>> >- Going to buffer response body of large or unknown size. Using
> >>> >getResponseBodyAsStream instead is recommended.
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getH
> >>>>ttp
> >>> >Response(HadoopStatusGetter.java:90)]
> >>> >- Job job_1425448727678_0003 get status check result.
> >>> >
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.che
> >>>>ckS
> >>> >tatus(HadoopStatusChecker.java:57)]
> >>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
> >>> >mapred.ClientServiceDelegate: Application state is completed.
> >>> >FinalApplicationStatus=FAILED. Redirecting to job history server
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJ
> >>>>obC
> >>> >ounter(HadoopCmdOutput.java:89)]
> >>> >- no counters for job job_1425448727678_0003
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
> >>>>put
> >>> >Resource(ResourceStore.java:171)]
> >>> >- Saving resource
> >>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >>> >(Store kylin_metadata@hbase)
> >>> >[pool-7-thread-3]:[2015-03-04
> >>>
> >>>
> >>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.upda
> >>>>teJ
> >>> >obOutput(ExecutableManager.java:222)]
> >>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR
> >>> >
> >>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
> >>> >
> >>> >> when check the log shown in web GUI of this step:
> >>> >>
> >>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
> >>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
> >>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
> >>> >> job_1425448727678_0004:FINISHED - FAILED
> >>> >> no counters for job job_1425448727678_0004
> >>> >>
> >>> >>
> >>> >> with the same cube, it can be built successfully with kylin-0.6*
> >>>before
> >>> >>
> >>>
> >>>
> >>
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

@Dong
   Are you running kylin from source code or kylin-0.7.1-SNAPSHOT.tgz?
   
   If from source code, you will see
   job/target/kylin-job-0.7.1-SNAPSHOT-job.jar
   storage/target/kylin-storage-0.7.1-SNAPSHOT-coprocessor.jar

   If from tgz
   lib/kylin-job-0.7.1-SNAPSHOT.jar
   lib/kylin-coprocessor-0.7.1-SNAPSHOT.jar
   
   Please check if there are *.SF *.RSA *.DSA files in these two jars. If
they do exist, as a workaround, you can remove these files from the jar.
   Please let us know if it works, and then we will fix it.

Best Regard
Zhou QianHao





On 3/4/15, 5:48 PM, "dong wang" <el...@gmail.com> wrote:

>I'm not sure whether the problem is similar to:
>http://www.cnblogs.com/fuxinci/p/3356087.html
>
>2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:
>
>> As checked, it says:
>>
>> Job init failed : java.lang.SecurityException: Invalid signature file
>> digest for Manifest main attributes
>> at
>> 
>>sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier
>>.java:286)
>> at
>> 
>>sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.jav
>>a:239)
>> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
>> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
>> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
>> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
>> at 
>>sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
>> at sun.misc.Resource.cachedInputStream(Resource.java:77)
>> at sun.misc.Resource.getByteBuffer(Resource.java:160)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:191)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.ja
>>va:1305)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(JobI
>>mpl.java:1247)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl.j
>>ava:139)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transi
>>tion(JobImpl.java:1451)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transi
>>tion(JobImpl.java:1388)
>> at
>> 
>>org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTr
>>ansition(StateMachineFactory.java:385)
>> at
>> 
>>org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachin
>>eFactory.java:302)
>> at
>> 
>>org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineF
>>actory.java:46)
>> at
>> 
>>org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doT
>>ransition(StateMachineFactory.java:448)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:9
>>96)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1
>>38)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(
>>MRAppMaster.java:1272)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.j
>>ava:1045)
>> at
>> 
>>org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:147
>>8)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
>>.java:1642)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp
>>pMaster.java:1474)
>> at
>> 
>>org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407
>>)
>>
>> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>>
>>> @Dong
>>>     The map-reduce job may encounter some error, can you please look
>>>into
>>> the job detail.
>>>     Just FYI, you can use http://$job history
>>> server$/jobhistory/job/job_1425448727678_0003/ to see what is really
>>>going
>>> on.
>>>
>>> Best Regard
>>> Zhou QianHao
>>>
>>>
>>>
>>>
>>>
>>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
>>>
>>> >more error info:
>>> >
>>> >
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.che
>>>>ckS
>>> >tatus(HadoopStatusChecker.java:57)]
>>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getRes
>>>>pon
>>> >seBody(HttpMethodBase.java:682)]
>>> >- Going to buffer response body of large or unknown size. Using
>>> >getResponseBodyAsStream instead is recommended.
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getH
>>>>ttp
>>> >Response(HadoopStatusGetter.java:90)]
>>> >- Job job_1425448727678_0003 get status check result.
>>> >
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.che
>>>>ckS
>>> >tatus(HadoopStatusChecker.java:57)]
>>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
>>> >mapred.ClientServiceDelegate: Application state is completed.
>>> >FinalApplicationStatus=FAILED. Redirecting to job history server
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJ
>>>>obC
>>> >ounter(HadoopCmdOutput.java:89)]
>>> >- no counters for job job_1425448727678_0003
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.
>>>>put
>>> >Resource(ResourceStore.java:171)]
>>> >- Saving resource
>>>/execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>>> >(Store kylin_metadata@hbase)
>>> >[pool-7-thread-3]:[2015-03-04
>>>
>>> 
>>>>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.upda
>>>>teJ
>>> >obOutput(ExecutableManager.java:222)]
>>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR
>>> >
>>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
>>> >
>>> >> when check the log shown in web GUI of this step:
>>> >>
>>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
>>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
>>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
>>> >> job_1425448727678_0004:FINISHED - FAILED
>>> >> no counters for job job_1425448727678_0004
>>> >>
>>> >>
>>> >> with the same cube, it can be built successfully with kylin-0.6*
>>>before
>>> >>
>>>
>>>
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

I'm not sure whether the problem is similar to:
http://www.cnblogs.com/fuxinci/p/3356087.html

2015-03-04 17:38 GMT+08:00 dong wang <el...@gmail.com>:

> As checked, it says:
>
> Job init failed : java.lang.SecurityException: Invalid signature file
> digest for Manifest main attributes
> at
> sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
> at
> sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
> at java.util.jar.JarVerifier.update(JarVerifier.java:228)
> at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
> at java.util.jar.JarFile.getInputStream(JarFile.java:415)
> at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> at sun.misc.Resource.cachedInputStream(Resource.java:77)
> at sun.misc.Resource.getByteBuffer(Resource.java:160)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.java:1305)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(JobImpl.java:1247)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1451)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
>
> 2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:
>
>> @Dong
>>     The map-reduce job may encounter some error, can you please look into
>> the job detail.
>>     Just FYI, you can use http://$job history
>> server$/jobhistory/job/job_1425448727678_0003/ to see what is really going
>> on.
>>
>> Best Regard
>> Zhou QianHao
>>
>>
>>
>>
>>
>> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
>>
>> >more error info:
>> >
>> >
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
>> >tatus(HadoopStatusChecker.java:57)]
>> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getRespon
>> >seBody(HttpMethodBase.java:682)]
>> >- Going to buffer response body of large or unknown size. Using
>> >getResponseBodyAsStream instead is recommended.
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttp
>> >Response(HadoopStatusGetter.java:90)]
>> >- Job job_1425448727678_0003 get status check result.
>> >
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
>> >tatus(HadoopStatusChecker.java:57)]
>> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
>> >mapred.ClientServiceDelegate: Application state is completed.
>> >FinalApplicationStatus=FAILED. Redirecting to job history server
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJobC
>> >ounter(HadoopCmdOutput.java:89)]
>> >- no counters for job job_1425448727678_0003
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>> >Resource(ResourceStore.java:171)]
>> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>> >(Store kylin_metadata@hbase)
>> >[pool-7-thread-3]:[2015-03-04
>>
>> >14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJ
>> >obOutput(ExecutableManager.java:222)]
>> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR
>> >
>> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
>> >
>> >> when check the log shown in web GUI of this step:
>> >>
>> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
>> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
>> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
>> >> job_1425448727678_0004:FINISHED - FAILED
>> >> no counters for job job_1425448727678_0004
>> >>
>> >>
>> >> with the same cube, it can be built successfully with kylin-0.6* before
>> >>
>>
>>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

As checked, it says:

Job init failed : java.lang.SecurityException: Invalid signature file
digest for Manifest main attributes
at
sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
at
sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
at java.util.jar.JarFile.getInputStream(JarFile.java:415)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
at sun.misc.Resource.cachedInputStream(Resource.java:77)
at sun.misc.Resource.getByteBuffer(Resource.java:160)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.isChainJob(JobImpl.java:1305)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.makeUberDecision(JobImpl.java:1247)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.access$3800(JobImpl.java:139)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1451)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InitTransition.transition(JobImpl.java:1388)
at
org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:996)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:138)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1272)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1045)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)

2015-03-04 17:03 GMT+08:00 Zhou, Qianhao <qi...@ebay.com>:

> @Dong
>     The map-reduce job may encounter some error, can you please look into
> the job detail.
>     Just FYI, you can use http://$job history
> server$/jobhistory/job/job_1425448727678_0003/ to see what is really going
> on.
>
> Best Regard
> Zhou QianHao
>
>
>
>
>
> On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:
>
> >more error info:
> >
> >
> >[pool-7-thread-3]:[2015-03-04
> >14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
> >tatus(HadoopStatusChecker.java:57)]
> >- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
> >[pool-7-thread-3]:[2015-03-04
> >14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getRespon
> >seBody(HttpMethodBase.java:682)]
> >- Going to buffer response body of large or unknown size. Using
> >getResponseBodyAsStream instead is recommended.
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttp
> >Response(HadoopStatusGetter.java:90)]
> >- Job job_1425448727678_0003 get status check result.
> >
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
> >tatus(HadoopStatusChecker.java:57)]
> >- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
> >mapred.ClientServiceDelegate: Application state is completed.
> >FinalApplicationStatus=FAILED. Redirecting to job history server
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJobC
> >ounter(HadoopCmdOutput.java:89)]
> >- no counters for job job_1425448727678_0003
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
> >Resource(ResourceStore.java:171)]
> >- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
> >(Store kylin_metadata@hbase)
> >[pool-7-thread-3]:[2015-03-04
> >14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJ
> >obOutput(ExecutableManager.java:222)]
> >- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR
> >
> >2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
> >
> >> when check the log shown in web GUI of this step:
> >>
> >> 2015-03-03 22:39:39.495 - State of Hadoop job:
> >> job_1425448727678_0004:ACCEPTED - UNDEFINED
> >> 2015-03-03 22:39:49.518 - State of Hadoop job:
> >> job_1425448727678_0004:FINISHED - FAILED
> >> no counters for job job_1425448727678_0004
> >>
> >>
> >> with the same cube, it can be built successfully with kylin-0.6* before
> >>
>
>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by "Zhou, Qianhao" <qi...@ebay.com>.

@Dong
    The map-reduce job may encounter some error, can you please look into
the job detail.
    Just FYI, you can use http://$job history
server$/jobhistory/job/job_1425448727678_0003/ to see what is really going
on.

Best Regard
Zhou QianHao





On 3/4/15, 3:21 PM, "dong wang" <el...@gmail.com> wrote:

>more error info:
>
>
>[pool-7-thread-3]:[2015-03-04
>14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
>tatus(HadoopStatusChecker.java:57)]
>- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
>[pool-7-thread-3]:[2015-03-04
>14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>[pool-7-thread-3]:[2015-03-04
>14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getRespon
>seBody(HttpMethodBase.java:682)]
>- Going to buffer response body of large or unknown size. Using
>getResponseBodyAsStream instead is recommended.
>[pool-7-thread-3]:[2015-03-04
>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttp
>Response(HadoopStatusGetter.java:90)]
>- Job job_1425448727678_0003 get status check result.
>
>[pool-7-thread-3]:[2015-03-04
>14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkS
>tatus(HadoopStatusChecker.java:57)]
>- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
>[pool-7-thread-3]:[2015-03-04
>14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>[pool-7-thread-3]:[2015-03-04
>14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
>mapred.ClientServiceDelegate: Application state is completed.
>FinalApplicationStatus=FAILED. Redirecting to job history server
>[pool-7-thread-3]:[2015-03-04
>14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJobC
>ounter(HadoopCmdOutput.java:89)]
>- no counters for job job_1425448727678_0003
>[pool-7-thread-3]:[2015-03-04
>14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>[pool-7-thread-3]:[2015-03-04
>14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>[pool-7-thread-3]:[2015-03-04
>14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.put
>Resource(ResourceStore.java:171)]
>- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
>(Store kylin_metadata@hbase)
>[pool-7-thread-3]:[2015-03-04
>14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJ
>obOutput(ExecutableManager.java:222)]
>- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR
>
>2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:
>
>> when check the log shown in web GUI of this step:
>>
>> 2015-03-03 22:39:39.495 - State of Hadoop job:
>> job_1425448727678_0004:ACCEPTED - UNDEFINED
>> 2015-03-03 22:39:49.518 - State of Hadoop job:
>> job_1425448727678_0004:FINISHED - FAILED
>> no counters for job job_1425448727678_0004
>>
>>
>> with the same cube, it can be built successfully with kylin-0.6* before
>>

Re: fail when Step Name: Extract Fact Table Distinct Columns with latest inverted-index branch

Posted by dong wang <el...@gmail.com>.

more error info:


[pool-7-thread-3]:[2015-03-04
14:19:41,023][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:57)]
- State of Hadoop job: job_1425448727678_0003:ACCEPTED-UNDEFINED
[pool-7-thread-3]:[2015-03-04
14:19:41,030][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
[pool-7-thread-3]:[2015-03-04
14:19:51,046][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)]
- Going to buffer response body of large or unknown size. Using
getResponseBodyAsStream instead is recommended.
[pool-7-thread-3]:[2015-03-04
14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:90)]
- Job job_1425448727678_0003 get status check result.

[pool-7-thread-3]:[2015-03-04
14:19:51,047][DEBUG][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:57)]
- State of Hadoop job: job_1425448727678_0003:FINISHED-FAILED
[pool-7-thread-3]:[2015-03-04
14:19:51,058][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
[pool-7-thread-3]:[2015-03-04
14:19:51,068][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
2015-03-04 14:19:51,074 INFO  [pool-7-thread-3]
mapred.ClientServiceDelegate: Application state is completed.
FinalApplicationStatus=FAILED. Redirecting to job history server
[pool-7-thread-3]:[2015-03-04
14:19:51,135][WARN][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:89)]
- no counters for job job_1425448727678_0003
[pool-7-thread-3]:[2015-03-04
14:19:51,141][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
[pool-7-thread-3]:[2015-03-04
14:19:51,149][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
[pool-7-thread-3]:[2015-03-04
14:19:51,162][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/99cbf62c-d0d4-4427-b075-e848cd8d83e9-01
(Store kylin_metadata@hbase)
[pool-7-thread-3]:[2015-03-04
14:19:51,165][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:99cbf62c-d0d4-4427-b075-e848cd8d83e9-01 from RUNNING to ERROR

2015-03-04 14:42 GMT+08:00 dong wang <el...@gmail.com>:

> when check the log shown in web GUI of this step:
>
> 2015-03-03 22:39:39.495 - State of Hadoop job:
> job_1425448727678_0004:ACCEPTED - UNDEFINED
> 2015-03-03 22:39:49.518 - State of Hadoop job:
> job_1425448727678_0004:FINISHED - FAILED
> no counters for job job_1425448727678_0004
>
>
> with the same cube, it can be built successfully with kylin-0.6* before
>