You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Shameera Rathnayaka <sh...@gmail.com> on 2016/04/04 05:14:34 UTC
Re: Job Monitor POC
Hi Siddharth,
I tested your poc and it is working. Your next move should be to intregrate
this with airavata, i saw you have created a subtask for this. Let's work
on that according to your availability. You can do this part by part,
create your own working branch and work on it, let's do code review once
you finish each part. By the way nice work, seems you have understood the
problem correctly.
Thanks,
Shameera.
On Sat, Mar 19, 2016 at 12:37 AM Siddharth Jain <ja...@umail.iu.edu> wrote:
> Hello all,
>
> I have completed creating a stand-alone implementation/POC of job monitor in
> the monitoring module of the codebase.
>
>
>
> I did fork from a dev branch and code, but I was unable to commit. I tried
> creating patch, but the patch did not aggregate all the changes, so I went
> ahead and just created a copy of the module (I think it should be as good
> as a patch) for now and attached it to the JIRA issues:
>
> https://issues.apache.org/jira/browse/AIRAVATA-1912
>
> https://issues.apache.org/jira/browse/AIRAVATA-1914
>
> https://issues.apache.org/jira/browse/AIRAVATA-1915
>
>
>
> This code addresses the sub-task:
>
> · AIRAVATA-1914
>
> · AIRAVATA-1915
>
> I have also flagged the above two sub-tasks as patch.
>
>
>
> Briefly this is what stand-alone implementation does right now:
>
> · Creates a fanout exchange with durable queues (will persist after
> server restarts) and auto-recovery option enabled; declares and binds all
> the consumer queues
>
> · Fetch e-mails from a dummy email address
>
> · Converts the *non-serializable* javax Message objects into custom
> *serializable* MessageExtract objects, which consists of only the
> necessary information, viz. “from”, “to”, “subject” and “content” fields,
> which are being used in the existing codebase
>
> · Serialize the messages
>
> · Send the message to the exchange, which in turn will deliver the
> message to all the queues bound to it
>
> · Two consumers threads fetch the messages from their respective
> queues
>
> · Deserialize the message and convert it into javax Message object,
> so that it can be used with the existing code
>
> · Print the received message content
>
>
>
> I have tried to keep all the components object-oriented. Following is the
> explanation of the components,
>
> Core Components:
>
> · RabbitMQEmailPublisher: RabbitMQ publisher for e-mail messages
>
> · EmailReceiver: Represents a consumer, which spawns a thread and
> keeps on receiving messages and process them. Multiple EmailReceiver can be
> instantiated and they will automatically run in different threads.
>
> · GmailSMTPMailBox: represents gmail mail box
>
> · EmailConsumer: This has the implementation of call back for
> received messages.
>
> · MessageExtract: Represents the extract from e-mail, which is
> serializable
>
> Simulation Components:
>
> · FetchPublish: Fetches emails with GmailSMTPMailBox and publishes
> to message bus with RabbitMQEmailPublisher
>
> · Simulator (also the main class to run):
>
> o Instantiates two consumers (EmailReceiver) in separate threads
>
> o Fetches email and publishes it with FetchPublish
>
> o Shuts down the consumers after a minute
>
>
>
> Following is the TO-DO list:
>
> · Use the actual email server configs
>
> o Currently I am using my own dummy account for testing, did not want
> to accidentally mess up the production mail-boxes
>
> · Use the codebase broker config
>
> o Currently I am using a local broker url and config
>
> · Fetch the actual email text/body
>
> o Currently I am using a hard-coded string for email text, as somehow
> when I extract the content, I always get multi-part content. The code you
> guys already have in the repository seems to be written with an assumption
> that the received message content will be text only, I tried creating an
> email like that, but it always has multi-part component. So once I have
> integrated the code with the existing monitor, I can get back to fetching
> the actual content
>
> · Open the inbox in read_write mode
>
> o Currently I am opening the inbox in read mode, so that the messages
> are not marked as read and so I don’t have to repeatedly create new dummy
> messages
>
> · Probably implement a RabbitMQ Factory singleton
>
> · Disable message(from message queue) auto-ack and only ack after
> processing has been completed, this might help making the workflow more
> robust
>
> · Use the codebase properties file for configuration of broker and
> email service, right now creating properties on run time
>
> · Create test-cases
>
> I looked into integrating this into the live work-flow, but following
> things kind of need time:
>
> · The message processing work-flow will have to be inverted, it’s
> kind of different than we assumed. Currently the messages are fetched
> explicitly and then processing is done. The way I have implemented the
> consumers is, the queue messages will be delivered asynchronously, so we
> don’t have to poll for it. So all the processing part will have to be wired
> with call-back function.
>
> · Setup PGA and its dependencies on local for testing
>
> · Understand in which part of the life-cycle the consumer(or
> equivalent of that) threads are to be invocated and what specific shutdown
> interrupts are they to follow and respond to
>
> · Understand properties files conventions
>
> Hence the aforementioned list with the TO-DO list will be part of
> milestone 1.
>
>
>
> Best regards,
>
> Siddharth Jain
>
--
Shameera Rathnayaka
Re: Job Monitor POC
Posted by Siddharth Jain <ja...@umail.iu.edu>.
Hi Shameera,
Thank you for testing it :)
As much as I would like to get started with integrating the POC this month,
I cannot, because I do not have the bandwidth to work on it. This being the
last month of the semester, there are too many things going on, especially
with the subjects I have taken. Apart from the regular assignments and
classes, I have two (might become three) programming projects to develop
and deliver; three end-term exams to write in this month. My last exam is
on May 6th. I can start working on it from May 7th on wards.
Best regards,
Siddharth Jain
On Sun, Apr 3, 2016 at 11:14 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Siddharth,
>
> I tested your poc and it is working. Your next move should be to
> intregrate this with airavata, i saw you have created a subtask for this.
> Let's work on that according to your availability. You can do this part by
> part, create your own working branch and work on it, let's do code review
> once you finish each part. By the way nice work, seems you have understood
> the problem correctly.
>
> Thanks,
> Shameera.
>
>
> On Sat, Mar 19, 2016 at 12:37 AM Siddharth Jain <ja...@umail.iu.edu>
> wrote:
>
>> Hello all,
>>
>> I have completed creating a stand-alone implementation/POC of job monitor in
>> the monitoring module of the codebase.
>>
>>
>>
>> I did fork from a dev branch and code, but I was unable to commit. I
>> tried creating patch, but the patch did not aggregate all the changes, so I
>> went ahead and just created a copy of the module (I think it should be as
>> good as a patch) for now and attached it to the JIRA issues:
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1912
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1914
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1915
>>
>>
>>
>> This code addresses the sub-task:
>>
>> · AIRAVATA-1914
>>
>> · AIRAVATA-1915
>>
>> I have also flagged the above two sub-tasks as patch.
>>
>>
>>
>> Briefly this is what stand-alone implementation does right now:
>>
>> · Creates a fanout exchange with durable queues (will persist
>> after server restarts) and auto-recovery option enabled; declares and binds
>> all the consumer queues
>>
>> · Fetch e-mails from a dummy email address
>>
>> · Converts the *non-serializable* javax Message objects into
>> custom *serializable* MessageExtract objects, which consists of only the
>> necessary information, viz. “from”, “to”, “subject” and “content” fields,
>> which are being used in the existing codebase
>>
>> · Serialize the messages
>>
>> · Send the message to the exchange, which in turn will deliver the
>> message to all the queues bound to it
>>
>> · Two consumers threads fetch the messages from their respective
>> queues
>>
>> · Deserialize the message and convert it into javax Message
>> object, so that it can be used with the existing code
>>
>> · Print the received message content
>>
>>
>>
>> I have tried to keep all the components object-oriented. Following is the
>> explanation of the components,
>>
>> Core Components:
>>
>> · RabbitMQEmailPublisher: RabbitMQ publisher for e-mail messages
>>
>> · EmailReceiver: Represents a consumer, which spawns a thread and
>> keeps on receiving messages and process them. Multiple EmailReceiver can be
>> instantiated and they will automatically run in different threads.
>>
>> · GmailSMTPMailBox: represents gmail mail box
>>
>> · EmailConsumer: This has the implementation of call back for
>> received messages.
>>
>> · MessageExtract: Represents the extract from e-mail, which is
>> serializable
>>
>> Simulation Components:
>>
>> · FetchPublish: Fetches emails with GmailSMTPMailBox and publishes
>> to message bus with RabbitMQEmailPublisher
>>
>> · Simulator (also the main class to run):
>>
>> o Instantiates two consumers (EmailReceiver) in separate threads
>>
>> o Fetches email and publishes it with FetchPublish
>>
>> o Shuts down the consumers after a minute
>>
>>
>>
>> Following is the TO-DO list:
>>
>> · Use the actual email server configs
>>
>> o Currently I am using my own dummy account for testing, did not want
>> to accidentally mess up the production mail-boxes
>>
>> · Use the codebase broker config
>>
>> o Currently I am using a local broker url and config
>>
>> · Fetch the actual email text/body
>>
>> o Currently I am using a hard-coded string for email text, as somehow
>> when I extract the content, I always get multi-part content. The code you
>> guys already have in the repository seems to be written with an assumption
>> that the received message content will be text only, I tried creating
>> an email like that, but it always has multi-part component. So once I have
>> integrated the code with the existing monitor, I can get back to fetching
>> the actual content
>>
>> · Open the inbox in read_write mode
>>
>> o Currently I am opening the inbox in read mode, so that the messages
>> are not marked as read and so I don’t have to repeatedly create new dummy
>> messages
>>
>> · Probably implement a RabbitMQ Factory singleton
>>
>> · Disable message(from message queue) auto-ack and only ack after
>> processing has been completed, this might help making the workflow more
>> robust
>>
>> · Use the codebase properties file for configuration of broker and
>> email service, right now creating properties on run time
>>
>> · Create test-cases
>>
>> I looked into integrating this into the live work-flow, but following
>> things kind of need time:
>>
>> · The message processing work-flow will have to be inverted, it’s
>> kind of different than we assumed. Currently the messages are fetched
>> explicitly and then processing is done. The way I have implemented the
>> consumers is, the queue messages will be delivered asynchronously, so we
>> don’t have to poll for it. So all the processing part will have to be wired
>> with call-back function.
>>
>> · Setup PGA and its dependencies on local for testing
>>
>> · Understand in which part of the life-cycle the consumer(or
>> equivalent of that) threads are to be invocated and what specific shutdown
>> interrupts are they to follow and respond to
>>
>> · Understand properties files conventions
>>
>> Hence the aforementioned list with the TO-DO list will be part of
>> milestone 1.
>>
>>
>>
>> Best regards,
>>
>> Siddharth Jain
>>
> --
> Shameera Rathnayaka
>