You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by Calvin Kirs <ki...@apache.org> on 2021/06/16 03:19:40 UTC
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Hi Rumesh,
Do you have any problems that I can help you with?
(BTW, the official coding period has started, so let's discuss it on the
mailing list)
Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
> Hi Rumesh,
> You need to pay extra attention to several points.
>
> Scalability: If we have increased the number of ways to configure
> parameters, such as system startup parameters, configuration files,
> configuration center settings. How quickly you can expand is a point you
> should focus on.
>
> Another point, the valid range of parameters, such as the A parameter,
> belongs to the JVM startup parameters, the user-startup and did not set,
> then even when the configuration file to append this parameter, this
> parameter will not take effect.
>
> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>
>> Hi Calvin,
>>
>> Thank you for accepting my proposal. I am working setting up
>> DolphinScheduler and playing with the code base. Once I am comfortable with
>> design I will start a public thread to discuss the GSoC work.
>>
>> I am looking forward to working with you two.
>>
>> Regards
>> Rumesh
>>
>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org> wrote:
>>
>>> hi, Rumesh
>>>
>>> I believe you will have a comprehensive understanding of
>>> DolphinScheduler next.
>>>
>>> If you encounter any problems in the process of getting familiar with
>>> the project, you can talk to me or Kevin, Kevin a very enthusiastic and
>>> professional mentor, who was also the Mentor of DolphinScheduler during the
>>> Apache incubator.
>>>
>>> Your proposal looks good, but before you develop the code, it is better
>>> to make the design public and discuss it together, it will make the whole
>>> work more efficient.
>>>
>>> No matter what problems you encounter, please communicate with us in
>>> time. We'll be happy to help you.
>>> We prefer open communication, such as mailing lists [1], or Github
>>> issues [2]. Of course, you can also choose the official Slack [3] group (
>>> important things must be reflected in the email list).
>>>
>>>
>>> [1]:
>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>> [3]:
>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>>
>>> --
>>> Best wishes！
>>> Calvin Kirs
>>>
>>
>
> --
> Best wishes！
> CalvinKirs
>
--
Best wishes！
CalvinKirs
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Rumesh Perera <nr...@gmail.com>.
Hi Calvin,
Thank you for the pointers. I will update the thread with both design work
and code work I have done so far.
Regards
Rumesh
On Wed, Jul 7, 2021 at 3:25 PM Calvin Kirs <ki...@apache.org> wrote:
> Hi Rumesh,
>
> Please don't forget the first evaluations on the 13th, it doesn't require
> you to submit code immediately, I'd rather see your design, flowcharts,
> class diagrams, etc. Coding is the last step to be done, until then it
> doesn't major.
>
> Also, if you run into any problems please contact the community or Kevin
> and me immediately.
>
> It is best for us to maintain effective communication once a week, which
> can ensure that we are always on the right path.
>
> Rumesh Perera <nr...@gmail.com> 于2021年7月2日周五 下午8:23写道：
>
>> Hi Calvin,
>>
>> Thank you for the follow up, I will submit work in progress PR for this.
>> Then we will take it from there.
>>
>> Regards
>> Rumesh
>>
>> On Thu, Jul 1, 2021 at 11:58 AM Calvin Kirs <ki...@apache.org> wrote:
>>
>>> Hi Rumesh:
>>>
>>> The first evaluations will take place on July 13,
>>> I would like to know what is your progress so far and if you have any
>>> problems that I can help you with?
>>>
>>> Lidong Dai <da...@gmail.com> 于2021年6月21日周一 下午6:34写道：
>>>
>>>> hi Rumesh,
>>>>
>>>> Please subscribe our mailing list first, or this mailing list can't show
>>>> your mail directly
>>>> please referer the instructions:
>>>>
>>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>>> , it is very easy!
>>>>
>>>>
>>>> Best Regards
>>>> ---------------
>>>> DolphinScheduler PMC
>>>> Lidong Dai
>>>> lidongdai@apache.org
>>>> ---------------
>>>>
>>>>
>>>> On Mon, Jun 21, 2021 at 11:25 AM Calvin Kirs <ki...@apache.org> wrote:
>>>>
>>>> > Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
>>>> >
>>>> > > Hi Mentors/all,
>>>> > >
>>>> > > This is to initiate the design discussion currently I am working
>>>> on. This
>>>> > > project is about introducing parameter injection from java JVM
>>>> arguments
>>>> > to
>>>> > > DolphinScheduler server runtime, so that one can override a
>>>> particular
>>>> > > parameter which is externalized to a configuration file. It was
>>>> suggested
>>>> > > that implementation should be extensible so that even if we
>>>> introduce
>>>> > > another way of parameter injection in future Eg:- configuration
>>>> center,
>>>> > > we should be able to extend our implementation to accommodate new
>>>> methods
>>>> > > of parameter injection.
>>>> > >
>>>> > > I would like to highlight following design points and request your
>>>> > > feedback,
>>>> > >
>>>> > > 1. Priority of parameter assignment if the same parameter is
>>>> supplied
>>>> > with
>>>> > > different methods like JVM, configuration file - Should we always
>>>> treat
>>>> > the
>>>> > > JVM startup parameter to have the highest priority, then
>>>> configuration
>>>> > file
>>>> > > and other methods and so on. Should these priority levels remain
>>>> > consistent
>>>> > > across all the parameters?
>>>> > >
>>>> >
>>>> > Yes, we need to follow this priority override rule. When jvm starts
>>>> with
>>>> > parameter A set, then this time even if the configuration file has
>>>> > parameter A set it has no effect. The value of parameter A is
>>>> determined by
>>>> > what the JVM sets.
>>>> >
>>>> >
>>>> > > 2. Validation of parameter - If a parameter validation is failed (
>>>> Eg:-
>>>> > > valid data type or valid range ) at JVM startup, Should we just
>>>> log/print
>>>> > > to the console and exit the Java runtime or default value will be
>>>> > assigned
>>>> > > from the configuration file. or from code and continue the Server
>>>> > process?
>>>> > >
>>>> >
>>>> > In my opinion exit is the best option.
>>>> >
>>>> > > 3. Are there any parameters we should treat as mandatory JVM
>>>> parameters?
>>>> > > If not supplied we basically exit the Java runtime printing to the
>>>> > console
>>>> > > that parameter is mandatory to be supplied at startup.
>>>> > >
>>>> > > I don't think we need to, as a public approach, we don't need to
>>>> care
>>>> > about specific businesses.
>>>> >
>>>> > > @Calvin Kirs <ki...@apache.org> Can you please provide example for
>>>> the
>>>> > > point you added - Another point, the valid range of parameters,
>>>> such as
>>>> > the
>>>> > > A parameter, belongs to the JVM startup parameters, the
>>>> user-startup and
>>>> > > did not set, then even when the configuration file to append this
>>>> > > parameter, this parameter will not take effect.
>>>> > >
>>>> > > Regards
>>>> > > Rumesh
>>>> > >
>>>> > > On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com>
>>>> > wrote:
>>>> > >
>>>> > >> Hi Calvin,
>>>> > >>
>>>> > >> Thank you for the pointers provided. I will make an update to the
>>>> > >> public thread with all the details of the design I am working on
>>>> right
>>>> > now.
>>>> > >>
>>>> > >> Regards
>>>> > >> Rumesh
>>>> > >>
>>>> > >> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org>
>>>> wrote:
>>>> > >>
>>>> > >>> Hi Rumesh,
>>>> > >>> Do you have any problems that I can help you with?
>>>> > >>> (BTW, the official coding period has started, so let's discuss it
>>>> on
>>>> > the
>>>> > >>> mailing list)
>>>> > >>>
>>>> > >>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>>>> > >>>
>>>> > >>>> Hi Rumesh,
>>>> > >>>> You need to pay extra attention to several points.
>>>> > >>>>
>>>> > >>>> Scalability: If we have increased the number of ways to configure
>>>> > >>>> parameters, such as system startup parameters, configuration
>>>> files,
>>>> > >>>> configuration center settings. How quickly you can expand is a
>>>> point
>>>> > you
>>>> > >>>> should focus on.
>>>> > >>>>
>>>> > >>>> Another point, the valid range of parameters, such as the A
>>>> parameter,
>>>> > >>>> belongs to the JVM startup parameters, the user-startup and did
>>>> not
>>>> > set,
>>>> > >>>> then even when the configuration file to append this parameter,
>>>> this
>>>> > >>>> parameter will not take effect.
>>>> > >>>>
>>>> > >>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>>>> > >>>>
>>>> > >>>>> Hi Calvin,
>>>> > >>>>>
>>>> > >>>>> Thank you for accepting my proposal. I am working setting up
>>>> > >>>>> DolphinScheduler and playing with the code base. Once I am
>>>> > comfortable with
>>>> > >>>>> design I will start a public thread to discuss the GSoC work.
>>>> > >>>>>
>>>> > >>>>> I am looking forward to working with you two.
>>>> > >>>>>
>>>> > >>>>> Regards
>>>> > >>>>> Rumesh
>>>> > >>>>>
>>>> > >>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org>
>>>> > wrote:
>>>> > >>>>>
>>>> > >>>>>> hi, Rumesh
>>>> > >>>>>>
>>>> > >>>>>> I believe you will have a comprehensive understanding of
>>>> > >>>>>> DolphinScheduler next.
>>>> > >>>>>>
>>>> > >>>>>> If you encounter any problems in the process of getting
>>>> familiar
>>>> > with
>>>> > >>>>>> the project, you can talk to me or Kevin, Kevin a very
>>>> enthusiastic
>>>> > and
>>>> > >>>>>> professional mentor, who was also the Mentor of
>>>> DolphinScheduler
>>>> > during the
>>>> > >>>>>> Apache incubator.
>>>> > >>>>>>
>>>> > >>>>>> Your proposal looks good, but before you develop the code, it
>>>> is
>>>> > >>>>>> better to make the design public and discuss it together, it
>>>> will
>>>> > make the
>>>> > >>>>>> whole work more efficient.
>>>> > >>>>>>
>>>> > >>>>>> No matter what problems you encounter, please communicate with
>>>> us in
>>>> > >>>>>> time. We'll be happy to help you.
>>>> > >>>>>> We prefer open communication, such as mailing lists [1], or
>>>> Github
>>>> > >>>>>> issues [2]. Of course, you can also choose the official Slack
>>>> [3]
>>>> > group (
>>>> > >>>>>> important things must be reflected in the email list).
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>> [1]:
>>>> > >>>>>>
>>>> >
>>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>>> > >>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>>> > >>>>>> [3]:
>>>> > >>>>>>
>>>> >
>>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>>> > >>>>>>
>>>> > >>>>>> --
>>>> > >>>>>> Best wishes！
>>>> > >>>>>> Calvin Kirs
>>>> > >>>>>>
>>>> > >>>>>
>>>> > >>>>
>>>> > >>>> --
>>>> > >>>> Best wishes！
>>>> > >>>> CalvinKirs
>>>> > >>>>
>>>> > >>>
>>>> > >>>
>>>> > >>> --
>>>> > >>> Best wishes！
>>>> > >>> CalvinKirs
>>>> > >>>
>>>> > >>
>>>> >
>>>> > --
>>>> > Best wishes！
>>>> > CalvinKirs
>>>> >
>>>>
>>>
>>>
>>> --
>>> Best wishes！
>>> CalvinKirs
>>>
>>
>
> --
> Best wishes！
> CalvinKirs
>
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Calvin Kirs <ki...@apache.org>.
Hi Rumesh,
Please don't forget the first evaluations on the 13th, it doesn't require
you to submit code immediately, I'd rather see your design, flowcharts,
class diagrams, etc. Coding is the last step to be done, until then it
doesn't major.
Also, if you run into any problems please contact the community or Kevin
and me immediately.
It is best for us to maintain effective communication once a week, which
can ensure that we are always on the right path.
Rumesh Perera <nr...@gmail.com> 于2021年7月2日周五 下午8:23写道：
> Hi Calvin,
>
> Thank you for the follow up, I will submit work in progress PR for this.
> Then we will take it from there.
>
> Regards
> Rumesh
>
> On Thu, Jul 1, 2021 at 11:58 AM Calvin Kirs <ki...@apache.org> wrote:
>
>> Hi Rumesh:
>>
>> The first evaluations will take place on July 13,
>> I would like to know what is your progress so far and if you have any
>> problems that I can help you with?
>>
>> Lidong Dai <da...@gmail.com> 于2021年6月21日周一 下午6:34写道：
>>
>>> hi Rumesh,
>>>
>>> Please subscribe our mailing list first, or this mailing list can't show
>>> your mail directly
>>> please referer the instructions:
>>>
>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>> , it is very easy!
>>>
>>>
>>> Best Regards
>>> ---------------
>>> DolphinScheduler PMC
>>> Lidong Dai
>>> lidongdai@apache.org
>>> ---------------
>>>
>>>
>>> On Mon, Jun 21, 2021 at 11:25 AM Calvin Kirs <ki...@apache.org> wrote:
>>>
>>> > Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
>>> >
>>> > > Hi Mentors/all,
>>> > >
>>> > > This is to initiate the design discussion currently I am working on.
>>> This
>>> > > project is about introducing parameter injection from java JVM
>>> arguments
>>> > to
>>> > > DolphinScheduler server runtime, so that one can override a
>>> particular
>>> > > parameter which is externalized to a configuration file. It was
>>> suggested
>>> > > that implementation should be extensible so that even if we introduce
>>> > > another way of parameter injection in future Eg:- configuration
>>> center,
>>> > > we should be able to extend our implementation to accommodate new
>>> methods
>>> > > of parameter injection.
>>> > >
>>> > > I would like to highlight following design points and request your
>>> > > feedback,
>>> > >
>>> > > 1. Priority of parameter assignment if the same parameter is supplied
>>> > with
>>> > > different methods like JVM, configuration file - Should we always
>>> treat
>>> > the
>>> > > JVM startup parameter to have the highest priority, then
>>> configuration
>>> > file
>>> > > and other methods and so on. Should these priority levels remain
>>> > consistent
>>> > > across all the parameters?
>>> > >
>>> >
>>> > Yes, we need to follow this priority override rule. When jvm starts
>>> with
>>> > parameter A set, then this time even if the configuration file has
>>> > parameter A set it has no effect. The value of parameter A is
>>> determined by
>>> > what the JVM sets.
>>> >
>>> >
>>> > > 2. Validation of parameter - If a parameter validation is failed (
>>> Eg:-
>>> > > valid data type or valid range ) at JVM startup, Should we just
>>> log/print
>>> > > to the console and exit the Java runtime or default value will be
>>> > assigned
>>> > > from the configuration file. or from code and continue the Server
>>> > process?
>>> > >
>>> >
>>> > In my opinion exit is the best option.
>>> >
>>> > > 3. Are there any parameters we should treat as mandatory JVM
>>> parameters?
>>> > > If not supplied we basically exit the Java runtime printing to the
>>> > console
>>> > > that parameter is mandatory to be supplied at startup.
>>> > >
>>> > > I don't think we need to, as a public approach, we don't need to care
>>> > about specific businesses.
>>> >
>>> > > @Calvin Kirs <ki...@apache.org> Can you please provide example for
>>> the
>>> > > point you added - Another point, the valid range of parameters, such
>>> as
>>> > the
>>> > > A parameter, belongs to the JVM startup parameters, the user-startup
>>> and
>>> > > did not set, then even when the configuration file to append this
>>> > > parameter, this parameter will not take effect.
>>> > >
>>> > > Regards
>>> > > Rumesh
>>> > >
>>> > > On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com>
>>> > wrote:
>>> > >
>>> > >> Hi Calvin,
>>> > >>
>>> > >> Thank you for the pointers provided. I will make an update to the
>>> > >> public thread with all the details of the design I am working on
>>> right
>>> > now.
>>> > >>
>>> > >> Regards
>>> > >> Rumesh
>>> > >>
>>> > >> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org>
>>> wrote:
>>> > >>
>>> > >>> Hi Rumesh,
>>> > >>> Do you have any problems that I can help you with?
>>> > >>> (BTW, the official coding period has started, so let's discuss it
>>> on
>>> > the
>>> > >>> mailing list)
>>> > >>>
>>> > >>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>>> > >>>
>>> > >>>> Hi Rumesh,
>>> > >>>> You need to pay extra attention to several points.
>>> > >>>>
>>> > >>>> Scalability: If we have increased the number of ways to configure
>>> > >>>> parameters, such as system startup parameters, configuration
>>> files,
>>> > >>>> configuration center settings. How quickly you can expand is a
>>> point
>>> > you
>>> > >>>> should focus on.
>>> > >>>>
>>> > >>>> Another point, the valid range of parameters, such as the A
>>> parameter,
>>> > >>>> belongs to the JVM startup parameters, the user-startup and did
>>> not
>>> > set,
>>> > >>>> then even when the configuration file to append this parameter,
>>> this
>>> > >>>> parameter will not take effect.
>>> > >>>>
>>> > >>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>>> > >>>>
>>> > >>>>> Hi Calvin,
>>> > >>>>>
>>> > >>>>> Thank you for accepting my proposal. I am working setting up
>>> > >>>>> DolphinScheduler and playing with the code base. Once I am
>>> > comfortable with
>>> > >>>>> design I will start a public thread to discuss the GSoC work.
>>> > >>>>>
>>> > >>>>> I am looking forward to working with you two.
>>> > >>>>>
>>> > >>>>> Regards
>>> > >>>>> Rumesh
>>> > >>>>>
>>> > >>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org>
>>> > wrote:
>>> > >>>>>
>>> > >>>>>> hi, Rumesh
>>> > >>>>>>
>>> > >>>>>> I believe you will have a comprehensive understanding of
>>> > >>>>>> DolphinScheduler next.
>>> > >>>>>>
>>> > >>>>>> If you encounter any problems in the process of getting familiar
>>> > with
>>> > >>>>>> the project, you can talk to me or Kevin, Kevin a very
>>> enthusiastic
>>> > and
>>> > >>>>>> professional mentor, who was also the Mentor of DolphinScheduler
>>> > during the
>>> > >>>>>> Apache incubator.
>>> > >>>>>>
>>> > >>>>>> Your proposal looks good, but before you develop the code, it is
>>> > >>>>>> better to make the design public and discuss it together, it
>>> will
>>> > make the
>>> > >>>>>> whole work more efficient.
>>> > >>>>>>
>>> > >>>>>> No matter what problems you encounter, please communicate with
>>> us in
>>> > >>>>>> time. We'll be happy to help you.
>>> > >>>>>> We prefer open communication, such as mailing lists [1], or
>>> Github
>>> > >>>>>> issues [2]. Of course, you can also choose the official Slack
>>> [3]
>>> > group (
>>> > >>>>>> important things must be reflected in the email list).
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>>> [1]:
>>> > >>>>>>
>>> >
>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>> > >>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>> > >>>>>> [3]:
>>> > >>>>>>
>>> >
>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>> > >>>>>>
>>> > >>>>>> --
>>> > >>>>>> Best wishes！
>>> > >>>>>> Calvin Kirs
>>> > >>>>>>
>>> > >>>>>
>>> > >>>>
>>> > >>>> --
>>> > >>>> Best wishes！
>>> > >>>> CalvinKirs
>>> > >>>>
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Best wishes！
>>> > >>> CalvinKirs
>>> > >>>
>>> > >>
>>> >
>>> > --
>>> > Best wishes！
>>> > CalvinKirs
>>> >
>>>
>>
>>
>> --
>> Best wishes！
>> CalvinKirs
>>
>
--
Best wishes！
CalvinKirs
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Rumesh Perera <nr...@gmail.com>.
Hi Calvin,
Thank you for the follow up, I will submit work in progress PR for this.
Then we will take it from there.
Regards
Rumesh
On Thu, Jul 1, 2021 at 11:58 AM Calvin Kirs <ki...@apache.org> wrote:
> Hi Rumesh:
>
> The first evaluations will take place on July 13,
> I would like to know what is your progress so far and if you have any
> problems that I can help you with?
>
> Lidong Dai <da...@gmail.com> 于2021年6月21日周一 下午6:34写道：
>
>> hi Rumesh,
>>
>> Please subscribe our mailing list first, or this mailing list can't show
>> your mail directly
>> please referer the instructions:
>>
>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>> , it is very easy!
>>
>>
>> Best Regards
>> ---------------
>> DolphinScheduler PMC
>> Lidong Dai
>> lidongdai@apache.org
>> ---------------
>>
>>
>> On Mon, Jun 21, 2021 at 11:25 AM Calvin Kirs <ki...@apache.org> wrote:
>>
>> > Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
>> >
>> > > Hi Mentors/all,
>> > >
>> > > This is to initiate the design discussion currently I am working on.
>> This
>> > > project is about introducing parameter injection from java JVM
>> arguments
>> > to
>> > > DolphinScheduler server runtime, so that one can override a
>> particular
>> > > parameter which is externalized to a configuration file. It was
>> suggested
>> > > that implementation should be extensible so that even if we introduce
>> > > another way of parameter injection in future Eg:- configuration
>> center,
>> > > we should be able to extend our implementation to accommodate new
>> methods
>> > > of parameter injection.
>> > >
>> > > I would like to highlight following design points and request your
>> > > feedback,
>> > >
>> > > 1. Priority of parameter assignment if the same parameter is supplied
>> > with
>> > > different methods like JVM, configuration file - Should we always
>> treat
>> > the
>> > > JVM startup parameter to have the highest priority, then configuration
>> > file
>> > > and other methods and so on. Should these priority levels remain
>> > consistent
>> > > across all the parameters?
>> > >
>> >
>> > Yes, we need to follow this priority override rule. When jvm starts with
>> > parameter A set, then this time even if the configuration file has
>> > parameter A set it has no effect. The value of parameter A is
>> determined by
>> > what the JVM sets.
>> >
>> >
>> > > 2. Validation of parameter - If a parameter validation is failed (
>> Eg:-
>> > > valid data type or valid range ) at JVM startup, Should we just
>> log/print
>> > > to the console and exit the Java runtime or default value will be
>> > assigned
>> > > from the configuration file. or from code and continue the Server
>> > process?
>> > >
>> >
>> > In my opinion exit is the best option.
>> >
>> > > 3. Are there any parameters we should treat as mandatory JVM
>> parameters?
>> > > If not supplied we basically exit the Java runtime printing to the
>> > console
>> > > that parameter is mandatory to be supplied at startup.
>> > >
>> > > I don't think we need to, as a public approach, we don't need to care
>> > about specific businesses.
>> >
>> > > @Calvin Kirs <ki...@apache.org> Can you please provide example for the
>> > > point you added - Another point, the valid range of parameters, such
>> as
>> > the
>> > > A parameter, belongs to the JVM startup parameters, the user-startup
>> and
>> > > did not set, then even when the configuration file to append this
>> > > parameter, this parameter will not take effect.
>> > >
>> > > Regards
>> > > Rumesh
>> > >
>> > > On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com>
>> > wrote:
>> > >
>> > >> Hi Calvin,
>> > >>
>> > >> Thank you for the pointers provided. I will make an update to the
>> > >> public thread with all the details of the design I am working on
>> right
>> > now.
>> > >>
>> > >> Regards
>> > >> Rumesh
>> > >>
>> > >> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
>> > >>
>> > >>> Hi Rumesh,
>> > >>> Do you have any problems that I can help you with?
>> > >>> (BTW, the official coding period has started, so let's discuss it on
>> > the
>> > >>> mailing list)
>> > >>>
>> > >>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>> > >>>
>> > >>>> Hi Rumesh,
>> > >>>> You need to pay extra attention to several points.
>> > >>>>
>> > >>>> Scalability: If we have increased the number of ways to configure
>> > >>>> parameters, such as system startup parameters, configuration files,
>> > >>>> configuration center settings. How quickly you can expand is a
>> point
>> > you
>> > >>>> should focus on.
>> > >>>>
>> > >>>> Another point, the valid range of parameters, such as the A
>> parameter,
>> > >>>> belongs to the JVM startup parameters, the user-startup and did not
>> > set,
>> > >>>> then even when the configuration file to append this parameter,
>> this
>> > >>>> parameter will not take effect.
>> > >>>>
>> > >>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>> > >>>>
>> > >>>>> Hi Calvin,
>> > >>>>>
>> > >>>>> Thank you for accepting my proposal. I am working setting up
>> > >>>>> DolphinScheduler and playing with the code base. Once I am
>> > comfortable with
>> > >>>>> design I will start a public thread to discuss the GSoC work.
>> > >>>>>
>> > >>>>> I am looking forward to working with you two.
>> > >>>>>
>> > >>>>> Regards
>> > >>>>> Rumesh
>> > >>>>>
>> > >>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org>
>> > wrote:
>> > >>>>>
>> > >>>>>> hi, Rumesh
>> > >>>>>>
>> > >>>>>> I believe you will have a comprehensive understanding of
>> > >>>>>> DolphinScheduler next.
>> > >>>>>>
>> > >>>>>> If you encounter any problems in the process of getting familiar
>> > with
>> > >>>>>> the project, you can talk to me or Kevin, Kevin a very
>> enthusiastic
>> > and
>> > >>>>>> professional mentor, who was also the Mentor of DolphinScheduler
>> > during the
>> > >>>>>> Apache incubator.
>> > >>>>>>
>> > >>>>>> Your proposal looks good, but before you develop the code, it is
>> > >>>>>> better to make the design public and discuss it together, it will
>> > make the
>> > >>>>>> whole work more efficient.
>> > >>>>>>
>> > >>>>>> No matter what problems you encounter, please communicate with
>> us in
>> > >>>>>> time. We'll be happy to help you.
>> > >>>>>> We prefer open communication, such as mailing lists [1], or
>> Github
>> > >>>>>> issues [2]. Of course, you can also choose the official Slack [3]
>> > group (
>> > >>>>>> important things must be reflected in the email list).
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> [1]:
>> > >>>>>>
>> >
>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>> > >>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>> > >>>>>> [3]:
>> > >>>>>>
>> >
>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Best wishes！
>> > >>>>>> Calvin Kirs
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>> --
>> > >>>> Best wishes！
>> > >>>> CalvinKirs
>> > >>>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Best wishes！
>> > >>> CalvinKirs
>> > >>>
>> > >>
>> >
>> > --
>> > Best wishes！
>> > CalvinKirs
>> >
>>
>
>
> --
> Best wishes！
> CalvinKirs
>
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Calvin Kirs <ki...@apache.org>.
Hi Rumesh:
The first evaluations will take place on July 13,
I would like to know what is your progress so far and if you have any
problems that I can help you with?
Lidong Dai <da...@gmail.com> 于2021年6月21日周一 下午6:34写道：
> hi Rumesh,
>
> Please subscribe our mailing list first, or this mailing list can't show
> your mail directly
> please referer the instructions:
>
> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
> , it is very easy!
>
>
> Best Regards
> ---------------
> DolphinScheduler PMC
> Lidong Dai
> lidongdai@apache.org
> ---------------
>
>
> On Mon, Jun 21, 2021 at 11:25 AM Calvin Kirs <ki...@apache.org> wrote:
>
> > Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
> >
> > > Hi Mentors/all,
> > >
> > > This is to initiate the design discussion currently I am working on.
> This
> > > project is about introducing parameter injection from java JVM
> arguments
> > to
> > > DolphinScheduler server runtime, so that one can override a particular
> > > parameter which is externalized to a configuration file. It was
> suggested
> > > that implementation should be extensible so that even if we introduce
> > > another way of parameter injection in future Eg:- configuration center,
> > > we should be able to extend our implementation to accommodate new
> methods
> > > of parameter injection.
> > >
> > > I would like to highlight following design points and request your
> > > feedback,
> > >
> > > 1. Priority of parameter assignment if the same parameter is supplied
> > with
> > > different methods like JVM, configuration file - Should we always treat
> > the
> > > JVM startup parameter to have the highest priority, then configuration
> > file
> > > and other methods and so on. Should these priority levels remain
> > consistent
> > > across all the parameters?
> > >
> >
> > Yes, we need to follow this priority override rule. When jvm starts with
> > parameter A set, then this time even if the configuration file has
> > parameter A set it has no effect. The value of parameter A is determined
> by
> > what the JVM sets.
> >
> >
> > > 2. Validation of parameter - If a parameter validation is failed ( Eg:-
> > > valid data type or valid range ) at JVM startup, Should we just
> log/print
> > > to the console and exit the Java runtime or default value will be
> > assigned
> > > from the configuration file. or from code and continue the Server
> > process?
> > >
> >
> > In my opinion exit is the best option.
> >
> > > 3. Are there any parameters we should treat as mandatory JVM
> parameters?
> > > If not supplied we basically exit the Java runtime printing to the
> > console
> > > that parameter is mandatory to be supplied at startup.
> > >
> > > I don't think we need to, as a public approach, we don't need to care
> > about specific businesses.
> >
> > > @Calvin Kirs <ki...@apache.org> Can you please provide example for the
> > > point you added - Another point, the valid range of parameters, such as
> > the
> > > A parameter, belongs to the JVM startup parameters, the user-startup
> and
> > > did not set, then even when the configuration file to append this
> > > parameter, this parameter will not take effect.
> > >
> > > Regards
> > > Rumesh
> > >
> > > On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com>
> > wrote:
> > >
> > >> Hi Calvin,
> > >>
> > >> Thank you for the pointers provided. I will make an update to the
> > >> public thread with all the details of the design I am working on right
> > now.
> > >>
> > >> Regards
> > >> Rumesh
> > >>
> > >> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
> > >>
> > >>> Hi Rumesh,
> > >>> Do you have any problems that I can help you with?
> > >>> (BTW, the official coding period has started, so let's discuss it on
> > the
> > >>> mailing list)
> > >>>
> > >>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
> > >>>
> > >>>> Hi Rumesh,
> > >>>> You need to pay extra attention to several points.
> > >>>>
> > >>>> Scalability: If we have increased the number of ways to configure
> > >>>> parameters, such as system startup parameters, configuration files,
> > >>>> configuration center settings. How quickly you can expand is a point
> > you
> > >>>> should focus on.
> > >>>>
> > >>>> Another point, the valid range of parameters, such as the A
> parameter,
> > >>>> belongs to the JVM startup parameters, the user-startup and did not
> > set,
> > >>>> then even when the configuration file to append this parameter, this
> > >>>> parameter will not take effect.
> > >>>>
> > >>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
> > >>>>
> > >>>>> Hi Calvin,
> > >>>>>
> > >>>>> Thank you for accepting my proposal. I am working setting up
> > >>>>> DolphinScheduler and playing with the code base. Once I am
> > comfortable with
> > >>>>> design I will start a public thread to discuss the GSoC work.
> > >>>>>
> > >>>>> I am looking forward to working with you two.
> > >>>>>
> > >>>>> Regards
> > >>>>> Rumesh
> > >>>>>
> > >>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org>
> > wrote:
> > >>>>>
> > >>>>>> hi, Rumesh
> > >>>>>>
> > >>>>>> I believe you will have a comprehensive understanding of
> > >>>>>> DolphinScheduler next.
> > >>>>>>
> > >>>>>> If you encounter any problems in the process of getting familiar
> > with
> > >>>>>> the project, you can talk to me or Kevin, Kevin a very
> enthusiastic
> > and
> > >>>>>> professional mentor, who was also the Mentor of DolphinScheduler
> > during the
> > >>>>>> Apache incubator.
> > >>>>>>
> > >>>>>> Your proposal looks good, but before you develop the code, it is
> > >>>>>> better to make the design public and discuss it together, it will
> > make the
> > >>>>>> whole work more efficient.
> > >>>>>>
> > >>>>>> No matter what problems you encounter, please communicate with us
> in
> > >>>>>> time. We'll be happy to help you.
> > >>>>>> We prefer open communication, such as mailing lists [1], or Github
> > >>>>>> issues [2]. Of course, you can also choose the official Slack [3]
> > group (
> > >>>>>> important things must be reflected in the email list).
> > >>>>>>
> > >>>>>>
> > >>>>>> [1]:
> > >>>>>>
> >
> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
> > >>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
> > >>>>>> [3]:
> > >>>>>>
> >
> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best wishes！
> > >>>>>> Calvin Kirs
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Best wishes！
> > >>>> CalvinKirs
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Best wishes！
> > >>> CalvinKirs
> > >>>
> > >>
> >
> > --
> > Best wishes！
> > CalvinKirs
> >
>
--
Best wishes！
CalvinKirs
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Lidong Dai <da...@gmail.com>.
hi Rumesh,
Please subscribe our mailing list first, or this mailing list can't show
your mail directly
please referer the instructions:
https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
, it is very easy!
Best Regards
---------------
DolphinScheduler PMC
Lidong Dai
lidongdai@apache.org
---------------
On Mon, Jun 21, 2021 at 11:25 AM Calvin Kirs <ki...@apache.org> wrote:
> Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
>
> > Hi Mentors/all,
> >
> > This is to initiate the design discussion currently I am working on. This
> > project is about introducing parameter injection from java JVM arguments
> to
> > DolphinScheduler server runtime, so that one can override a particular
> > parameter which is externalized to a configuration file. It was suggested
> > that implementation should be extensible so that even if we introduce
> > another way of parameter injection in future Eg:- configuration center,
> > we should be able to extend our implementation to accommodate new methods
> > of parameter injection.
> >
> > I would like to highlight following design points and request your
> > feedback,
> >
> > 1. Priority of parameter assignment if the same parameter is supplied
> with
> > different methods like JVM, configuration file - Should we always treat
> the
> > JVM startup parameter to have the highest priority, then configuration
> file
> > and other methods and so on. Should these priority levels remain
> consistent
> > across all the parameters?
> >
>
> Yes, we need to follow this priority override rule. When jvm starts with
> parameter A set, then this time even if the configuration file has
> parameter A set it has no effect. The value of parameter A is determined by
> what the JVM sets.
>
>
> > 2. Validation of parameter - If a parameter validation is failed ( Eg:-
> > valid data type or valid range ) at JVM startup, Should we just log/print
> > to the console and exit the Java runtime or default value will be
> assigned
> > from the configuration file. or from code and continue the Server
> process?
> >
>
> In my opinion exit is the best option.
>
> > 3. Are there any parameters we should treat as mandatory JVM parameters?
> > If not supplied we basically exit the Java runtime printing to the
> console
> > that parameter is mandatory to be supplied at startup.
> >
> > I don't think we need to, as a public approach, we don't need to care
> about specific businesses.
>
> > @Calvin Kirs <ki...@apache.org> Can you please provide example for the
> > point you added - Another point, the valid range of parameters, such as
> the
> > A parameter, belongs to the JVM startup parameters, the user-startup and
> > did not set, then even when the configuration file to append this
> > parameter, this parameter will not take effect.
> >
> > Regards
> > Rumesh
> >
> > On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com>
> wrote:
> >
> >> Hi Calvin,
> >>
> >> Thank you for the pointers provided. I will make an update to the
> >> public thread with all the details of the design I am working on right
> now.
> >>
> >> Regards
> >> Rumesh
> >>
> >> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
> >>
> >>> Hi Rumesh,
> >>> Do you have any problems that I can help you with?
> >>> (BTW, the official coding period has started, so let's discuss it on
> the
> >>> mailing list)
> >>>
> >>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
> >>>
> >>>> Hi Rumesh,
> >>>> You need to pay extra attention to several points.
> >>>>
> >>>> Scalability: If we have increased the number of ways to configure
> >>>> parameters, such as system startup parameters, configuration files,
> >>>> configuration center settings. How quickly you can expand is a point
> you
> >>>> should focus on.
> >>>>
> >>>> Another point, the valid range of parameters, such as the A parameter,
> >>>> belongs to the JVM startup parameters, the user-startup and did not
> set,
> >>>> then even when the configuration file to append this parameter, this
> >>>> parameter will not take effect.
> >>>>
> >>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
> >>>>
> >>>>> Hi Calvin,
> >>>>>
> >>>>> Thank you for accepting my proposal. I am working setting up
> >>>>> DolphinScheduler and playing with the code base. Once I am
> comfortable with
> >>>>> design I will start a public thread to discuss the GSoC work.
> >>>>>
> >>>>> I am looking forward to working with you two.
> >>>>>
> >>>>> Regards
> >>>>> Rumesh
> >>>>>
> >>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org>
> wrote:
> >>>>>
> >>>>>> hi, Rumesh
> >>>>>>
> >>>>>> I believe you will have a comprehensive understanding of
> >>>>>> DolphinScheduler next.
> >>>>>>
> >>>>>> If you encounter any problems in the process of getting familiar
> with
> >>>>>> the project, you can talk to me or Kevin, Kevin a very enthusiastic
> and
> >>>>>> professional mentor, who was also the Mentor of DolphinScheduler
> during the
> >>>>>> Apache incubator.
> >>>>>>
> >>>>>> Your proposal looks good, but before you develop the code, it is
> >>>>>> better to make the design public and discuss it together, it will
> make the
> >>>>>> whole work more efficient.
> >>>>>>
> >>>>>> No matter what problems you encounter, please communicate with us in
> >>>>>> time. We'll be happy to help you.
> >>>>>> We prefer open communication, such as mailing lists [1], or Github
> >>>>>> issues [2]. Of course, you can also choose the official Slack [3]
> group (
> >>>>>> important things must be reflected in the email list).
> >>>>>>
> >>>>>>
> >>>>>> [1]:
> >>>>>>
> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
> >>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
> >>>>>> [3]:
> >>>>>>
> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
> >>>>>>
> >>>>>> --
> >>>>>> Best wishes！
> >>>>>> Calvin Kirs
> >>>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Best wishes！
> >>>> CalvinKirs
> >>>>
> >>>
> >>>
> >>> --
> >>> Best wishes！
> >>> CalvinKirs
> >>>
> >>
>
> --
> Best wishes！
> CalvinKirs
>
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Calvin Kirs <ki...@apache.org>.
Rumesh Perera <nr...@gmail.com> 于2021年6月21日周一 上午4:42写道：
> Hi Mentors/all,
>
> This is to initiate the design discussion currently I am working on. This
> project is about introducing parameter injection from java JVM arguments to
> DolphinScheduler server runtime, so that one can override a particular
> parameter which is externalized to a configuration file. It was suggested
> that implementation should be extensible so that even if we introduce
> another way of parameter injection in future Eg:- configuration center,
> we should be able to extend our implementation to accommodate new methods
> of parameter injection.
>
> I would like to highlight following design points and request your
> feedback,
>
> 1. Priority of parameter assignment if the same parameter is supplied with
> different methods like JVM, configuration file - Should we always treat the
> JVM startup parameter to have the highest priority, then configuration file
> and other methods and so on. Should these priority levels remain consistent
> across all the parameters?
>
Yes, we need to follow this priority override rule. When jvm starts with
parameter A set, then this time even if the configuration file has
parameter A set it has no effect. The value of parameter A is determined by
what the JVM sets.
> 2. Validation of parameter - If a parameter validation is failed ( Eg:-
> valid data type or valid range ) at JVM startup, Should we just log/print
> to the console and exit the Java runtime or default value will be assigned
> from the configuration file. or from code and continue the Server process?
>
In my opinion exit is the best option.
> 3. Are there any parameters we should treat as mandatory JVM parameters?
> If not supplied we basically exit the Java runtime printing to the console
> that parameter is mandatory to be supplied at startup.
>
> I don't think we need to, as a public approach, we don't need to care
about specific businesses.
> @Calvin Kirs <ki...@apache.org> Can you please provide example for the
> point you added - Another point, the valid range of parameters, such as the
> A parameter, belongs to the JVM startup parameters, the user-startup and
> did not set, then even when the configuration file to append this
> parameter, this parameter will not take effect.
>
> Regards
> Rumesh
>
> On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com> wrote:
>
>> Hi Calvin,
>>
>> Thank you for the pointers provided. I will make an update to the
>> public thread with all the details of the design I am working on right now.
>>
>> Regards
>> Rumesh
>>
>> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
>>
>>> Hi Rumesh,
>>> Do you have any problems that I can help you with?
>>> (BTW, the official coding period has started, so let's discuss it on the
>>> mailing list)
>>>
>>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>>>
>>>> Hi Rumesh,
>>>> You need to pay extra attention to several points.
>>>>
>>>> Scalability: If we have increased the number of ways to configure
>>>> parameters, such as system startup parameters, configuration files,
>>>> configuration center settings. How quickly you can expand is a point you
>>>> should focus on.
>>>>
>>>> Another point, the valid range of parameters, such as the A parameter,
>>>> belongs to the JVM startup parameters, the user-startup and did not set,
>>>> then even when the configuration file to append this parameter, this
>>>> parameter will not take effect.
>>>>
>>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>>>>
>>>>> Hi Calvin,
>>>>>
>>>>> Thank you for accepting my proposal. I am working setting up
>>>>> DolphinScheduler and playing with the code base. Once I am comfortable with
>>>>> design I will start a public thread to discuss the GSoC work.
>>>>>
>>>>> I am looking forward to working with you two.
>>>>>
>>>>> Regards
>>>>> Rumesh
>>>>>
>>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org> wrote:
>>>>>
>>>>>> hi, Rumesh
>>>>>>
>>>>>> I believe you will have a comprehensive understanding of
>>>>>> DolphinScheduler next.
>>>>>>
>>>>>> If you encounter any problems in the process of getting familiar with
>>>>>> the project, you can talk to me or Kevin, Kevin a very enthusiastic and
>>>>>> professional mentor, who was also the Mentor of DolphinScheduler during the
>>>>>> Apache incubator.
>>>>>>
>>>>>> Your proposal looks good, but before you develop the code, it is
>>>>>> better to make the design public and discuss it together, it will make the
>>>>>> whole work more efficient.
>>>>>>
>>>>>> No matter what problems you encounter, please communicate with us in
>>>>>> time. We'll be happy to help you.
>>>>>> We prefer open communication, such as mailing lists [1], or Github
>>>>>> issues [2]. Of course, you can also choose the official Slack [3] group (
>>>>>> important things must be reflected in the email list).
>>>>>>
>>>>>>
>>>>>> [1]:
>>>>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>>>>> [3]:
>>>>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>>>>>
>>>>>> --
>>>>>> Best wishes！
>>>>>> Calvin Kirs
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Best wishes！
>>>> CalvinKirs
>>>>
>>>
>>>
>>> --
>>> Best wishes！
>>> CalvinKirs
>>>
>>
--
Best wishes！
CalvinKirs
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Rumesh Perera <nr...@gmail.com>.
Hi Mentors/all,
This is to initiate the design discussion currently I am working on. This
project is about introducing parameter injection from java JVM arguments to
DolphinScheduler server runtime, so that one can override a particular
parameter which is externalized to a configuration file. It was suggested
that implementation should be extensible so that even if we introduce
another way of parameter injection in future Eg:- configuration center, we
should be able to extend our implementation to accommodate new methods of
parameter injection.
I would like to highlight following design points and request your feedback,
1. Priority of parameter assignment if the same parameter is supplied with
different methods like JVM, configuration file - Should we always treat the
JVM startup parameter to have the highest priority, then configuration file
and other methods and so on. Should these priority levels remain consistent
across all the parameters?
2. Validation of parameter - If a parameter validation is failed ( Eg:-
valid data type or valid range ) at JVM startup, Should we just log/print
to the console and exit the Java runtime or default value will be assigned
from the configuration file. or from code and continue the Server process?
3. Are there any parameters we should treat as mandatory JVM parameters? If
not supplied we basically exit the Java runtime printing to the console
that parameter is mandatory to be supplied at startup.
@Calvin Kirs <ki...@apache.org> Can you please provide example for the point
you added - Another point, the valid range of parameters, such as the A
parameter, belongs to the JVM startup parameters, the user-startup and did
not set, then even when the configuration file to append this parameter,
this parameter will not take effect.
Regards
Rumesh
On Thu, Jun 17, 2021 at 6:55 AM Rumesh Perera <nr...@gmail.com> wrote:
> Hi Calvin,
>
> Thank you for the pointers provided. I will make an update to the
> public thread with all the details of the design I am working on right now.
>
> Regards
> Rumesh
>
> On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
>
>> Hi Rumesh,
>> Do you have any problems that I can help you with?
>> (BTW, the official coding period has started, so let's discuss it on the
>> mailing list)
>>
>> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>>
>>> Hi Rumesh,
>>> You need to pay extra attention to several points.
>>>
>>> Scalability: If we have increased the number of ways to configure
>>> parameters, such as system startup parameters, configuration files,
>>> configuration center settings. How quickly you can expand is a point you
>>> should focus on.
>>>
>>> Another point, the valid range of parameters, such as the A parameter,
>>> belongs to the JVM startup parameters, the user-startup and did not set,
>>> then even when the configuration file to append this parameter, this
>>> parameter will not take effect.
>>>
>>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>>>
>>>> Hi Calvin,
>>>>
>>>> Thank you for accepting my proposal. I am working setting up
>>>> DolphinScheduler and playing with the code base. Once I am comfortable with
>>>> design I will start a public thread to discuss the GSoC work.
>>>>
>>>> I am looking forward to working with you two.
>>>>
>>>> Regards
>>>> Rumesh
>>>>
>>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org> wrote:
>>>>
>>>>> hi, Rumesh
>>>>>
>>>>> I believe you will have a comprehensive understanding of
>>>>> DolphinScheduler next.
>>>>>
>>>>> If you encounter any problems in the process of getting familiar with
>>>>> the project, you can talk to me or Kevin, Kevin a very enthusiastic and
>>>>> professional mentor, who was also the Mentor of DolphinScheduler during the
>>>>> Apache incubator.
>>>>>
>>>>> Your proposal looks good, but before you develop the code, it is
>>>>> better to make the design public and discuss it together, it will make the
>>>>> whole work more efficient.
>>>>>
>>>>> No matter what problems you encounter, please communicate with us in
>>>>> time. We'll be happy to help you.
>>>>> We prefer open communication, such as mailing lists [1], or Github
>>>>> issues [2]. Of course, you can also choose the official Slack [3] group (
>>>>> important things must be reflected in the email list).
>>>>>
>>>>>
>>>>> [1]:
>>>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>>>> [3]:
>>>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>>>>
>>>>> --
>>>>> Best wishes！
>>>>> Calvin Kirs
>>>>>
>>>>
>>>
>>> --
>>> Best wishes！
>>> CalvinKirs
>>>
>>
>>
>> --
>> Best wishes！
>> CalvinKirs
>>
>
Re: [GSoC]COMDEV-412 Apache DolphinScheduler Parameter coverage
Posted by Rumesh Perera <nr...@gmail.com>.
Hi Calvin,
Thank you for the pointers provided. I will make an update to the
public thread with all the details of the design I am working on right now.
Regards
Rumesh
On Wed, Jun 16, 2021 at 8:50 AM Calvin Kirs <ki...@apache.org> wrote:
> Hi Rumesh,
> Do you have any problems that I can help you with?
> (BTW, the official coding period has started, so let's discuss it on the
> mailing list)
>
> Calvin Kirs <ki...@apache.org> 于2021年6月9日周三 下午4:11写道：
>
>> Hi Rumesh,
>> You need to pay extra attention to several points.
>>
>> Scalability: If we have increased the number of ways to configure
>> parameters, such as system startup parameters, configuration files,
>> configuration center settings. How quickly you can expand is a point you
>> should focus on.
>>
>> Another point, the valid range of parameters, such as the A parameter,
>> belongs to the JVM startup parameters, the user-startup and did not set,
>> then even when the configuration file to append this parameter, this
>> parameter will not take effect.
>>
>> Rumesh Perera <nr...@gmail.com> 于2021年5月27日周四 上午3:40写道：
>>
>>> Hi Calvin,
>>>
>>> Thank you for accepting my proposal. I am working setting up
>>> DolphinScheduler and playing with the code base. Once I am comfortable with
>>> design I will start a public thread to discuss the GSoC work.
>>>
>>> I am looking forward to working with you two.
>>>
>>> Regards
>>> Rumesh
>>>
>>> On Mon, May 24, 2021 at 11:23 PM Calvin Kirs <ki...@apache.org> wrote:
>>>
>>>> hi, Rumesh
>>>>
>>>> I believe you will have a comprehensive understanding of
>>>> DolphinScheduler next.
>>>>
>>>> If you encounter any problems in the process of getting familiar with
>>>> the project, you can talk to me or Kevin, Kevin a very enthusiastic and
>>>> professional mentor, who was also the Mentor of DolphinScheduler during the
>>>> Apache incubator.
>>>>
>>>> Your proposal looks good, but before you develop the code, it is better
>>>> to make the design public and discuss it together, it will make the whole
>>>> work more efficient.
>>>>
>>>> No matter what problems you encounter, please communicate with us in
>>>> time. We'll be happy to help you.
>>>> We prefer open communication, such as mailing lists [1], or Github
>>>> issues [2]. Of course, you can also choose the official Slack [3] group (
>>>> important things must be reflected in the email list).
>>>>
>>>>
>>>> [1]:
>>>> https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
>>>> [2]:https://github.com/apache/dolphinscheduler/issues
>>>> [3]:
>>>> https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
>>>>
>>>> --
>>>> Best wishes！
>>>> Calvin Kirs
>>>>
>>>
>>
>> --
>> Best wishes！
>> CalvinKirs
>>
>
>
> --
> Best wishes！
> CalvinKirs
>