You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Raviteja Lokineni <ra...@gmail.com> on 2016/08/04 17:53:33 UTC
MAHOUT-1876 - Lucene compatibility
Hi Devs,
Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
Problem statement: Mahout should be compatible with the latest lucene
version. I was trying to solve a text clustering problem and I stumbled
upon the error that the lucene version in use is not supported. That's when
I raised this issue and one of the guys suggested me to try fixing it and I
started doing so.
Proposed solution:
1. Change lucene version in the POM file and fix all the compilation
failures
2. Fix any failing tests due to this change
Current progress:
1. Fixed all the compilation issues
2. Comparing the test failures of before-fix v/s after-fix
I might need some help in test failures, I happen to see that the same
tests are failing in both before-fix and after-fix.
Let me know if any comments.
Thanks,
--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade
E: raviteja.lokineni@gmail.com
[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>
Re: MAHOUT-1876 - Lucene compatibility
Posted by Raviteja Lokineni <ra...@gmail.com>.
Are there any pre-requisites to run tests on local machines?
Most of the MR module tests failed with:
java.lang.NullPointerException: null
at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:845EFDB07D41873E]:0)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
RawLocalFileSystem.java:633)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
JobSubmissionFiles.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.mahout.cf.taste.hadoop.preparation.PreparePreferenceMatrixJob.
run(PreparePreferenceMatrixJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
run(RecommenderJob.java:168)
at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
testCompleteJobWithFiltering(RecommenderJobTest.java:881)
On Thu, Aug 4, 2016 at 6:03 PM, Andrew Palumbo <ap...@outlook.com> wrote:
> Actually, Travis CI is not set up to test all mahout modules. Could you
> please test again on your local machine, and report any errors.
>
> Andy
>
>
> -------- Original message --------
> From: Raviteja Lokineni <ra...@gmail.com>
> Date: 08/04/2016 4:01 PM (GMT-05:00)
> To: mahout <de...@mahout.apache.org>
> Subject: Re: MAHOUT-1876 - Lucene compatibility
>
> All the tests are successful even with my code change and upgrade to
> Lucene. (The test failure that I reported above was on my local machine, I
> guess you can ignore that.)
>
> So is this pull request good to merge?
>
> https://github.com/apache/mahout/pull/247
>
> https://travis-ci.org/apache/mahout/builds/149840408
>
> On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
> > Most of the MR tests are failing with:
> >
> > java.lang.NullPointerException: null
> > at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> > 845EFDB07D41873E]:0)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> > at org.apache.hadoop.util.Shell.run(Shell.java:418)
> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> > Shell.java:650)
> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> > at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> > RawLocalFileSystem.java:633)
> > at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> > RawLocalFileSystem.java:421)
> > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(
> FilterFileSystem.java:281)
> > at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> > JobSubmissionFiles.java:125)
> > at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> > JobSubmitter.java:348)
> > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1556)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> > at org.apache.mahout.cf.taste.hadoop.preparation.
> > PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> > run(RecommenderJob.java:168)
> > at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> > testCompleteJobWithFiltering(RecommenderJobTest.java:881)
> >
> > Do I need to have hadoop installed on my local machine?
> >
> > On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> > raviteja.lokineni@gmail.com> wrote:
> >
> >> Hi Devs,
> >>
> >> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
> >>
> >> Problem statement: Mahout should be compatible with the latest lucene
> >> version. I was trying to solve a text clustering problem and I stumbled
> >> upon the error that the lucene version in use is not supported. That's
> when
> >> I raised this issue and one of the guys suggested me to try fixing it
> and I
> >> started doing so.
> >>
> >> Proposed solution:
> >>
> >> 1. Change lucene version in the POM file and fix all the compilation
> >> failures
> >> 2. Fix any failing tests due to this change
> >>
> >> Current progress:
> >>
> >> 1. Fixed all the compilation issues
> >> 2. Comparing the test failures of before-fix v/s after-fix
> >>
> >> I might need some help in test failures, I happen to see that the same
> >> tests are failing in both before-fix and after-fix.
> >>
> >> Let me know if any comments.
> >>
> >> Thanks,
> >> --
> >> *Raviteja Lokineni* | Business Intelligence Developer
> >> TD Ameritrade
> >>
> >> E: raviteja.lokineni@gmail.com
> >>
> >> [image: View Raviteja Lokineni's profile on LinkedIn]
> >> <http://in.linkedin.com/in/ravitejalokineni>
> >>
> >>
> >
> >
> > --
> > *Raviteja Lokineni* | Business Intelligence Developer
> > TD Ameritrade
> >
> > E: raviteja.lokineni@gmail.com
> >
> > [image: View Raviteja Lokineni's profile on LinkedIn]
> > <http://in.linkedin.com/in/ravitejalokineni>
> >
> >
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade
E: raviteja.lokineni@gmail.com
[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>
RE: MAHOUT-1876 - Lucene compatibility
Posted by Andrew Palumbo <ap...@outlook.com>.
Actually, Travis CI is not set up to test all mahout modules. Could you please test again on your local machine, and report any errors.
Andy
-------- Original message --------
From: Raviteja Lokineni <ra...@gmail.com>
Date: 08/04/2016 4:01 PM (GMT-05:00)
To: mahout <de...@mahout.apache.org>
Subject: Re: MAHOUT-1876 - Lucene compatibility
All the tests are successful even with my code change and upgrade to
Lucene. (The test failure that I reported above was on my local machine, I
guess you can ignore that.)
So is this pull request good to merge?
https://github.com/apache/mahout/pull/247
https://travis-ci.org/apache/mahout/builds/149840408
On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:
> Most of the MR tests are failing with:
>
> java.lang.NullPointerException: null
> at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> 845EFDB07D41873E]:0)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:633)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> RawLocalFileSystem.java:421)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
> at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> JobSubmissionFiles.java:125)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.mahout.cf.taste.hadoop.preparation.
> PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> run(RecommenderJob.java:168)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> testCompleteJobWithFiltering(RecommenderJobTest.java:881)
>
> Do I need to have hadoop installed on my local machine?
>
> On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
>> Hi Devs,
>>
>> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>>
>> Problem statement: Mahout should be compatible with the latest lucene
>> version. I was trying to solve a text clustering problem and I stumbled
>> upon the error that the lucene version in use is not supported. That's when
>> I raised this issue and one of the guys suggested me to try fixing it and I
>> started doing so.
>>
>> Proposed solution:
>>
>> 1. Change lucene version in the POM file and fix all the compilation
>> failures
>> 2. Fix any failing tests due to this change
>>
>> Current progress:
>>
>> 1. Fixed all the compilation issues
>> 2. Comparing the test failures of before-fix v/s after-fix
>>
>> I might need some help in test failures, I happen to see that the same
>> tests are failing in both before-fix and after-fix.
>>
>> Let me know if any comments.
>>
>> Thanks,
>> --
>> *Raviteja Lokineni* | Business Intelligence Developer
>> TD Ameritrade
>>
>> E: raviteja.lokineni@gmail.com
>>
>> [image: View Raviteja Lokineni's profile on LinkedIn]
>> <http://in.linkedin.com/in/ravitejalokineni>
>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>
--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade
E: raviteja.lokineni@gmail.com
[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>
Re: MAHOUT-1876 - Lucene compatibility
Posted by Raviteja Lokineni <ra...@gmail.com>.
All the tests are successful even with my code change and upgrade to
Lucene. (The test failure that I reported above was on my local machine, I
guess you can ignore that.)
So is this pull request good to merge?
https://github.com/apache/mahout/pull/247
https://travis-ci.org/apache/mahout/builds/149840408
On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:
> Most of the MR tests are failing with:
>
> java.lang.NullPointerException: null
> at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> 845EFDB07D41873E]:0)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:633)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> RawLocalFileSystem.java:421)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
> at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> JobSubmissionFiles.java:125)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.mahout.cf.taste.hadoop.preparation.
> PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> run(RecommenderJob.java:168)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> testCompleteJobWithFiltering(RecommenderJobTest.java:881)
>
> Do I need to have hadoop installed on my local machine?
>
> On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
>> Hi Devs,
>>
>> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>>
>> Problem statement: Mahout should be compatible with the latest lucene
>> version. I was trying to solve a text clustering problem and I stumbled
>> upon the error that the lucene version in use is not supported. That's when
>> I raised this issue and one of the guys suggested me to try fixing it and I
>> started doing so.
>>
>> Proposed solution:
>>
>> 1. Change lucene version in the POM file and fix all the compilation
>> failures
>> 2. Fix any failing tests due to this change
>>
>> Current progress:
>>
>> 1. Fixed all the compilation issues
>> 2. Comparing the test failures of before-fix v/s after-fix
>>
>> I might need some help in test failures, I happen to see that the same
>> tests are failing in both before-fix and after-fix.
>>
>> Let me know if any comments.
>>
>> Thanks,
>> --
>> *Raviteja Lokineni* | Business Intelligence Developer
>> TD Ameritrade
>>
>> E: raviteja.lokineni@gmail.com
>>
>> [image: View Raviteja Lokineni's profile on LinkedIn]
>> <http://in.linkedin.com/in/ravitejalokineni>
>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>
--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade
E: raviteja.lokineni@gmail.com
[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>
Re: MAHOUT-1876 - Lucene compatibility
Posted by Raviteja Lokineni <ra...@gmail.com>.
Most of the MR tests are failing with:
java.lang.NullPointerException: null
at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:845EFDB07D41873E]:0)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at
org.apache.mahout.cf.taste.hadoop.preparation.PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.run(RecommenderJob.java:168)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.testCompleteJobWithFiltering(RecommenderJobTest.java:881)
Do I need to have hadoop installed on my local machine?
On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:
> Hi Devs,
>
> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>
> Problem statement: Mahout should be compatible with the latest lucene
> version. I was trying to solve a text clustering problem and I stumbled
> upon the error that the lucene version in use is not supported. That's when
> I raised this issue and one of the guys suggested me to try fixing it and I
> started doing so.
>
> Proposed solution:
>
> 1. Change lucene version in the POM file and fix all the compilation
> failures
> 2. Fix any failing tests due to this change
>
> Current progress:
>
> 1. Fixed all the compilation issues
> 2. Comparing the test failures of before-fix v/s after-fix
>
> I might need some help in test failures, I happen to see that the same
> tests are failing in both before-fix and after-fix.
>
> Let me know if any comments.
>
> Thanks,
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>
--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade
E: raviteja.lokineni@gmail.com
[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>