You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Irindu Nugawela <nb...@gmail.com> on 2018/04/28 03:03:13 UTC
Fwd: GSOC 2018: MongoDB Output Connector
---------- Forwarded message ----------
From: Irindu Nugawela <nb...@gmail.com>
Date: 28 April 2018 at 07:42
Subject: Re: GSOC 2018: MongoDB Output Connector
To: Piergiorgio Lucidi <pi...@apache.org>
Cc: Karl Wright <da...@gmail.com>
Hi Piergiorgio,
I have noticed that you are using mongo-java-driver
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.11.3</version>
</dependency>
for Gridfs connector but the latest stable is
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.6.3</version>
</dependency>
there are some API changes between the two which one do you want me to use.
At the moment I cannot use the version 3.6.3 inside the manifoldCF
framework even though I add it as a dependency for my project I encounter
some problems with certain API calls which I suspect because version 2.11.3
jar is already there. Please guide me with next steps, shall I stick to
version 2.11.3?
On 26 April 2018 at 16:27, Piergiorgio Lucidi <pi...@apache.org>
wrote:
> Just shared on Twitter:
> https://twitter.com/ApacheManifold/status/989457998715277312
>
> ^__^
>
> 2018-04-26 11:39 GMT+02:00 Irindu Nugawela <nb...@gmail.com>:
>
>> Hi Piergiorgio,
>> Thank you very much. yes, you may publish about my involvement in our
>> project and please let me know if I can help you in any way possible. Yes,
>> the twitter profile you have listed belongs to me.
>>
>> On 26 April 2018 at 12:23, Piergiorgio Lucidi <pi...@apache.org>
>> wrote:
>>
>>> Hi Irindu,
>>>
>>> no problem, take your time.
>>>
>>> In the meanwhile I'm wondering if you allow me to publish a post about
>>> your involvement in the project on our Twitter profile:
>>> https://twitter.com/apachemanifold
>>>
>>> I noticed that you have a Twitter profile, could you please to confirm
>>> that your profile is the following?
>>> https://twitter.com/IrinduPera
>>>
>>> Please let me know and if you give me your permission I'll write a post.
>>> Thank you and take care.
>>>
>>> Cheers,
>>> PJ
>>>
>>> 2018-04-26 3:33 GMT+02:00 Irindu Nugawela <nb...@gmail.com>:
>>>
>>>> Hi Piergiorgio,
>>>>
>>>> Thank you very much for giving me this valuable opportunity, I am going
>>>> to do my best with it. Secondly, I am very sorry that I was not very active
>>>> in the last few days that I have been suffering from a viral fever. I am
>>>> recovering now and will work extra hours in the upcoming weeks to get back
>>>> on track. Please help me with the next steps :-)
>>>>
>>>> On 16 April 2018 at 13:52, Piergiorgio Lucidi <pi...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Irindu,
>>>>>
>>>>> very nice and great job!
>>>>>
>>>>> ^__^
>>>>>
>>>>> Cheers,
>>>>> PJ
>>>>>
>>>>>
>>>>> 2018-04-16 8:00 GMT+02:00 Irindu Nugawela <nb...@gmail.com>:
>>>>>
>>>>>> Hi Piergiorgio ,
>>>>>> I have come up with the implementation for edit configuration and
>>>>>> view configuration with the crawler-ui. It's a small step in the long
>>>>>> journey.
>>>>>> Apache ManifoldCF_ View Output Connection Statu...
>>>>>> <https://drive.google.com/file/d/1X3q32ml40yzJ1moTjoXew2K1g5wNk0Gs/view?usp=drive_web>
>>>>>>
>>>>>>
>>>>>>
>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>>>>>> www.avast.com
>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>>>>> <#m_621489084084143594_m_-3875022077604017546_m_2871376415022697920_m_-1171787217413983024_m_1069976683875003658_m_645939314292940779_m_2231955365225803757_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>>>
>>>>>> On 15 March 2018 at 00:45, Irindu Nugawela <nb...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>> I was able to successfully build ManifoldCF with IntelliJ Idea as
>>>>>>> well.
>>>>>>>
>>>>>>> I have gone through MongoDB - Java
>>>>>>> <https://www.tutorialspoint.com/mongodb/mongodb_java.htm> and
>>>>>>> identified the MongoDB parameters as follows
>>>>>>>
>>>>>>>
>>>>>>> 1. MongoCredential by a useraname and a password
>>>>>>> 2. MongoDatabase
>>>>>>> 3. collection
>>>>>>>
>>>>>>> I am planning to have a single username and a password for a Manifold-MongoDB connection
>>>>>>>
>>>>>>> I have identified the operations that we are going to support as
>>>>>>>
>>>>>>> 1. Inserting documents
>>>>>>> 2. Querying documents
>>>>>>> 3. Updating documents
>>>>>>> 4. Deleting documents
>>>>>>> 5. Indexing documents based on a particular field
>>>>>>>
>>>>>>> I have also studied the document format we are going to store
>>>>>>> available here
>>>>>>> <https://github.com/apache/manifoldcf/blob/trunk/framework/agents/src/main/java/org/apache/manifoldcf/agents/interfaces/RepositoryDocument.java> which
>>>>>>> MongoDB accepts in BSON format,
>>>>>>>
>>>>>>> I have identified the following fields
>>>>>>>
>>>>>>> 1. binaryFieldData
>>>>>>> 2. binaryLength
>>>>>>> 3. fileName
>>>>>>> 4. contentMimeType
>>>>>>> 5. createdDate
>>>>>>> 6. modifiedDate
>>>>>>> 7. indexingDate
>>>>>>> 8. originalSize
>>>>>>>
>>>>>>> of a Repository Document that I am going to convert to BSON format
>>>>>>> to write to MongoDB
>>>>>>>
>>>>>>> Please tell me am I on the correct path, and correct me if I am wrong
>>>>>>>
>>>>>>> On 27 February 2018 at 15:42, Piergiorgio Lucidi <
>>>>>>> piergiorgio@apache.org> wrote:
>>>>>>>
>>>>>>>> Hi Irindu,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2018-02-25 13:53 GMT+01:00 Irindu Nugawela <nb...@gmail.com>:
>>>>>>>>
>>>>>>>> > Hi All,
>>>>>>>> > I am currently to working on the $subject. I have some issues
>>>>>>>> that I need
>>>>>>>> > to clarify.
>>>>>>>> > First of all, why did we choose Mongo DB as an output target? My
>>>>>>>> first
>>>>>>>> > thought of the project was that it was about writing a repository
>>>>>>>> connector
>>>>>>>> > for MongoDB because MongoDB is a database programme. ( I am aware
>>>>>>>> of its
>>>>>>>> > NoSQL
>>>>>>>> > architecture and its indexing capabilities ). But it is not a
>>>>>>>> search engine
>>>>>>>> > per se.
>>>>>>>> >
>>>>>>>>
>>>>>>>> We want to extend the scenario of ManifoldCF and we would like to
>>>>>>>> add
>>>>>>>> content migration capability.
>>>>>>>> Imagine to have a tool that helps you not only for the indexing
>>>>>>>> process but
>>>>>>>> also in migrating contents.
>>>>>>>>
>>>>>>>> In this way you could migrate contents from a content repository to
>>>>>>>> another
>>>>>>>> one, for example you could migrate contents from Alfresco to
>>>>>>>> MongoDB.
>>>>>>>>
>>>>>>>>
>>>>>>>> >
>>>>>>>> > Then what are the expected operations other than document
>>>>>>>> addition and
>>>>>>>> > deletion? (what activities should be supported?)
>>>>>>>> >
>>>>>>>>
>>>>>>>> The excepted operation are the same but related to migrate contents
>>>>>>>> and not
>>>>>>>> related only to manage the indexing process.
>>>>>>>>
>>>>>>>>
>>>>>>>> >
>>>>>>>> > What capabilities of MongoDB you have already identified that you
>>>>>>>> expect to
>>>>>>>> > be useful for us. (what capabilities of MongoDB you have
>>>>>>>> identified as
>>>>>>>> > useful That you've decided to write an Output connector for it.)
>>>>>>>> >
>>>>>>>>
>>>>>>>> Currently we only have a repository connector for GridFS that allow
>>>>>>>> us to
>>>>>>>> read binaries from MongoDB.
>>>>>>>> We also want to store contents against a MongoDB instance, in this
>>>>>>>> way we
>>>>>>>> can migrate contents on MongoDB from any repository with metadata
>>>>>>>> and
>>>>>>>> binaries.
>>>>>>>>
>>>>>>>> I think it could be very cool :)
>>>>>>>>
>>>>>>>> In the enterprise context is very critical to migrate contents /
>>>>>>>> documents
>>>>>>>> from different kind of repositories and typically it is the first
>>>>>>>> step for
>>>>>>>> a digital transformation project where a company is trying to
>>>>>>>> orchestrate
>>>>>>>> different silos in a smart way.
>>>>>>>>
>>>>>>>> Please let me know if you need more informations.
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> PJ
>>>>>>>>
>>>>>>>>
>>>>>>>> >
>>>>>>>> > I have been through the ManifoldCF Architecture with DaddyWri
>>>>>>>> > <https://github.com/DaddyWri>/manifoldcfinaction Chapter1.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Thanks and Regards,
>>>>>>>> > Irindu Nugawela,
>>>>>>>> > Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>>>>> > Faculty of Engineering University of Peradeniya
>>>>>>>> >
>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_
>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t
>>>>>>>> erm=icon>
>>>>>>>> > Virus-free.
>>>>>>>> > www.avast.com
>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_
>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t
>>>>>>>> erm=link>
>>>>>>>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Piergiorgio Lucidi
>>>>>>>> https://www.open4dev.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Irindu Nugawela,
>>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>>>> Faculty of Engineering University of Peradeniya
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Irindu Nugawela,
>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>>> Faculty of Engineering University of Peradeniya
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Piergiorgio Lucidi
>>>>> https://www.open4dev.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Irindu Nugawela,
>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>> Faculty of Engineering University of Peradeniya
>>>>
>>>
>>>
>>>
>>> --
>>> Piergiorgio Lucidi
>>> https://www.open4dev.com
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Irindu Nugawela,
>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>> Faculty of Engineering University of Peradeniya
>>
>
>
>
> --
> Piergiorgio Lucidi
> https://www.open4dev.com
>
--
Thanks and Regards,
Irindu Nugawela,
Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
Faculty of Engineering University of Peradeniya
--
Thanks and Regards,
Irindu Nugawela,
Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
Faculty of Engineering University of Peradeniya