You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Raviteja Lokineni <ra...@gmail.com> on 2016/08/04 17:53:33 UTC

MAHOUT-1876 - Lucene compatibility

Hi Devs,

Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876

Problem statement: Mahout should be compatible with the latest lucene
version. I was trying to solve a text clustering problem and I stumbled
upon the error that the lucene version in use is not supported. That's when
I raised this issue and one of the guys suggested me to try fixing it and I
started doing so.

Proposed solution:

   1. Change lucene version in the POM file and fix all the compilation
   failures
   2. Fix any failing tests due to this change

Current progress:

   1. Fixed all the compilation issues
   2. Comparing the test failures of before-fix v/s after-fix

I might need some help in test failures, I happen to see that the same
tests are failing in both before-fix and after-fix.

Let me know if any comments.

Thanks,
-- 
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade

E: raviteja.lokineni@gmail.com

[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>

Re: MAHOUT-1876 - Lucene compatibility

Posted by Raviteja Lokineni <ra...@gmail.com>.
Are there any pre-requisites to run tests on local machines?

Most of the MR module tests failed with:

java.lang.NullPointerException: null
at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:845EFDB07D41873E]:0)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
RawLocalFileSystem.java:633)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
JobSubmissionFiles.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.mahout.cf.taste.hadoop.preparation.PreparePreferenceMatrixJob.
run(PreparePreferenceMatrixJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
run(RecommenderJob.java:168)
at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
testCompleteJobWithFiltering(RecommenderJobTest.java:881)

On Thu, Aug 4, 2016 at 6:03 PM, Andrew Palumbo <ap...@outlook.com> wrote:

> Actually, Travis CI is not set up to test all mahout modules.  Could you
> please test again on your local machine, and report any errors.
>
> Andy
>
>
> -------- Original message --------
> From: Raviteja Lokineni <ra...@gmail.com>
> Date: 08/04/2016 4:01 PM (GMT-05:00)
> To: mahout <de...@mahout.apache.org>
> Subject: Re: MAHOUT-1876 - Lucene compatibility
>
> All the tests are successful even with my code change and upgrade to
> Lucene. (The test failure that I reported above was on my local machine, I
> guess you can ignore that.)
>
> So is this pull request good to merge?
>
> https://github.com/apache/mahout/pull/247
>
> https://travis-ci.org/apache/mahout/builds/149840408
>
> On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
> > Most of the MR tests are failing with:
> >
> > java.lang.NullPointerException: null
> > at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> > 845EFDB07D41873E]:0)
> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> > at org.apache.hadoop.util.Shell.run(Shell.java:418)
> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> > Shell.java:650)
> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> > at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> > at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> > RawLocalFileSystem.java:633)
> > at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> > RawLocalFileSystem.java:421)
> > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(
> FilterFileSystem.java:281)
> > at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> > JobSubmissionFiles.java:125)
> > at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> > JobSubmitter.java:348)
> > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1556)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> > at org.apache.mahout.cf.taste.hadoop.preparation.
> > PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> > run(RecommenderJob.java:168)
> > at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> > testCompleteJobWithFiltering(RecommenderJobTest.java:881)
> >
> > Do I need to have hadoop installed on my local machine?
> >
> > On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> > raviteja.lokineni@gmail.com> wrote:
> >
> >> Hi Devs,
> >>
> >> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
> >>
> >> Problem statement: Mahout should be compatible with the latest lucene
> >> version. I was trying to solve a text clustering problem and I stumbled
> >> upon the error that the lucene version in use is not supported. That's
> when
> >> I raised this issue and one of the guys suggested me to try fixing it
> and I
> >> started doing so.
> >>
> >> Proposed solution:
> >>
> >>    1. Change lucene version in the POM file and fix all the compilation
> >>    failures
> >>    2. Fix any failing tests due to this change
> >>
> >> Current progress:
> >>
> >>    1. Fixed all the compilation issues
> >>    2. Comparing the test failures of before-fix v/s after-fix
> >>
> >> I might need some help in test failures, I happen to see that the same
> >> tests are failing in both before-fix and after-fix.
> >>
> >> Let me know if any comments.
> >>
> >> Thanks,
> >> --
> >> *Raviteja Lokineni* | Business Intelligence Developer
> >> TD Ameritrade
> >>
> >> E: raviteja.lokineni@gmail.com
> >>
> >> [image: View Raviteja Lokineni's profile on LinkedIn]
> >> <http://in.linkedin.com/in/ravitejalokineni>
> >>
> >>
> >
> >
> > --
> > *Raviteja Lokineni* | Business Intelligence Developer
> > TD Ameritrade
> >
> > E: raviteja.lokineni@gmail.com
> >
> > [image: View Raviteja Lokineni's profile on LinkedIn]
> > <http://in.linkedin.com/in/ravitejalokineni>
> >
> >
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>



-- 
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade

E: raviteja.lokineni@gmail.com

[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>

RE: MAHOUT-1876 - Lucene compatibility

Posted by Andrew Palumbo <ap...@outlook.com>.
Actually, Travis CI is not set up to test all mahout modules.  Could you please test again on your local machine, and report any errors.

Andy


-------- Original message --------
From: Raviteja Lokineni <ra...@gmail.com>
Date: 08/04/2016 4:01 PM (GMT-05:00)
To: mahout <de...@mahout.apache.org>
Subject: Re: MAHOUT-1876 - Lucene compatibility

All the tests are successful even with my code change and upgrade to
Lucene. (The test failure that I reported above was on my local machine, I
guess you can ignore that.)

So is this pull request good to merge?

https://github.com/apache/mahout/pull/247

https://travis-ci.org/apache/mahout/builds/149840408

On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:

> Most of the MR tests are failing with:
>
> java.lang.NullPointerException: null
> at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> 845EFDB07D41873E]:0)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:633)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> RawLocalFileSystem.java:421)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
> at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> JobSubmissionFiles.java:125)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.mahout.cf.taste.hadoop.preparation.
> PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> run(RecommenderJob.java:168)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> testCompleteJobWithFiltering(RecommenderJobTest.java:881)
>
> Do I need to have hadoop installed on my local machine?
>
> On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
>> Hi Devs,
>>
>> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>>
>> Problem statement: Mahout should be compatible with the latest lucene
>> version. I was trying to solve a text clustering problem and I stumbled
>> upon the error that the lucene version in use is not supported. That's when
>> I raised this issue and one of the guys suggested me to try fixing it and I
>> started doing so.
>>
>> Proposed solution:
>>
>>    1. Change lucene version in the POM file and fix all the compilation
>>    failures
>>    2. Fix any failing tests due to this change
>>
>> Current progress:
>>
>>    1. Fixed all the compilation issues
>>    2. Comparing the test failures of before-fix v/s after-fix
>>
>> I might need some help in test failures, I happen to see that the same
>> tests are failing in both before-fix and after-fix.
>>
>> Let me know if any comments.
>>
>> Thanks,
>> --
>> *Raviteja Lokineni* | Business Intelligence Developer
>> TD Ameritrade
>>
>> E: raviteja.lokineni@gmail.com
>>
>> [image: View Raviteja Lokineni's profile on LinkedIn]
>> <http://in.linkedin.com/in/ravitejalokineni>
>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>


--
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade

E: raviteja.lokineni@gmail.com

[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>

Re: MAHOUT-1876 - Lucene compatibility

Posted by Raviteja Lokineni <ra...@gmail.com>.
All the tests are successful even with my code change and upgrade to
Lucene. (The test failure that I reported above was on my local machine, I
guess you can ignore that.)

So is this pull request good to merge?

https://github.com/apache/mahout/pull/247

https://travis-ci.org/apache/mahout/builds/149840408

On Thu, Aug 4, 2016 at 2:05 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:

> Most of the MR tests are failing with:
>
> java.lang.NullPointerException: null
> at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:
> 845EFDB07D41873E]:0)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(
> Shell.java:650)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(
> RawLocalFileSystem.java:633)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(
> RawLocalFileSystem.java:421)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
> at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(
> JobSubmissionFiles.java:125)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.mahout.cf.taste.hadoop.preparation.
> PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.
> run(RecommenderJob.java:168)
> at org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.
> testCompleteJobWithFiltering(RecommenderJobTest.java:881)
>
> Do I need to have hadoop installed on my local machine?
>
> On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
> raviteja.lokineni@gmail.com> wrote:
>
>> Hi Devs,
>>
>> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>>
>> Problem statement: Mahout should be compatible with the latest lucene
>> version. I was trying to solve a text clustering problem and I stumbled
>> upon the error that the lucene version in use is not supported. That's when
>> I raised this issue and one of the guys suggested me to try fixing it and I
>> started doing so.
>>
>> Proposed solution:
>>
>>    1. Change lucene version in the POM file and fix all the compilation
>>    failures
>>    2. Fix any failing tests due to this change
>>
>> Current progress:
>>
>>    1. Fixed all the compilation issues
>>    2. Comparing the test failures of before-fix v/s after-fix
>>
>> I might need some help in test failures, I happen to see that the same
>> tests are failing in both before-fix and after-fix.
>>
>> Let me know if any comments.
>>
>> Thanks,
>> --
>> *Raviteja Lokineni* | Business Intelligence Developer
>> TD Ameritrade
>>
>> E: raviteja.lokineni@gmail.com
>>
>> [image: View Raviteja Lokineni's profile on LinkedIn]
>> <http://in.linkedin.com/in/ravitejalokineni>
>>
>>
>
>
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>


-- 
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade

E: raviteja.lokineni@gmail.com

[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>

Re: MAHOUT-1876 - Lucene compatibility

Posted by Raviteja Lokineni <ra...@gmail.com>.
Most of the MR tests are failing with:

java.lang.NullPointerException: null
at __randomizedtesting.SeedInfo.seed([C8C254E3D9D80CC9:845EFDB07D41873E]:0)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at
org.apache.mahout.cf.taste.hadoop.preparation.PreparePreferenceMatrixJob.run(PreparePreferenceMatrixJob.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.run(RecommenderJob.java:168)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJobTest.testCompleteJobWithFiltering(RecommenderJobTest.java:881)

Do I need to have hadoop installed on my local machine?

On Thu, Aug 4, 2016 at 1:53 PM, Raviteja Lokineni <
raviteja.lokineni@gmail.com> wrote:

> Hi Devs,
>
> Issue link: https://issues.apache.org/jira/browse/MAHOUT-1876
>
> Problem statement: Mahout should be compatible with the latest lucene
> version. I was trying to solve a text clustering problem and I stumbled
> upon the error that the lucene version in use is not supported. That's when
> I raised this issue and one of the guys suggested me to try fixing it and I
> started doing so.
>
> Proposed solution:
>
>    1. Change lucene version in the POM file and fix all the compilation
>    failures
>    2. Fix any failing tests due to this change
>
> Current progress:
>
>    1. Fixed all the compilation issues
>    2. Comparing the test failures of before-fix v/s after-fix
>
> I might need some help in test failures, I happen to see that the same
> tests are failing in both before-fix and after-fix.
>
> Let me know if any comments.
>
> Thanks,
> --
> *Raviteja Lokineni* | Business Intelligence Developer
> TD Ameritrade
>
> E: raviteja.lokineni@gmail.com
>
> [image: View Raviteja Lokineni's profile on LinkedIn]
> <http://in.linkedin.com/in/ravitejalokineni>
>
>


-- 
*Raviteja Lokineni* | Business Intelligence Developer
TD Ameritrade

E: raviteja.lokineni@gmail.com

[image: View Raviteja Lokineni's profile on LinkedIn]
<http://in.linkedin.com/in/ravitejalokineni>