You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Josh Elser <el...@apache.org> on 2016/11/08 17:34:11 UTC
[DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Re-vitalizing some discussion here (updating the subject accordingly)
Thinking ahead with the assumption that the PMC and Francis all want to
bring this code under governance of the Calcite PMC, what does "adoption
of an Avatica client" really entail? My thoughts..
* Perform ASF IP Clearance [1]. Dbl-checking copyright is the most
difficult thing (in most cases, it shouldn't be difficult).
* Create a new repository for the new client
* Define what (if any) released artifacts are (in addition to source-code)
* Ensure code meets licensing requirements of ASF
* Establish build, test, and release steps for the new client
* Establish CI for commits/patches (manual, if not automated?)
My thinking is that we can rely on the backwards compatibility
guarantees of Avatica's wire protocol to deal with the differing release
schedules. Specifically, an older version of the Golang client should
still work with newer versions of the Avatica server -- they should not
require simultaneous releases of all Avatica clients with the de-facto
Java client and Avatica server.
If we can codify these practices, it would make adoption of future
clients less intimidating, letting us focus on the positives of having a
large collection of clients in other languages.
Thoughts?
- Josh
[1] http://incubator.apache.org/ip-clearance/
Julian Hyde wrote:
> Probably not, in the near term at least. The two projects have separate release schedules, artifacts and web sites, which seems to give them enough breathing space for now. Maybe well split source code repositories at some point. Because the code is all managed by Apaches IP governance, any option we choose would be fairly straightforward.
>
> Julian
>
>
>> On May 17, 2016, at 8:11 PM, F21<f2...@gmail.com> wrote:
>>
>> Thanks for opening the issue on JIRA, Julian.
>>
>> Let me know if there's anything I can do to speed up the process. Will Avatica be spun out as its own project?
>>
>> Francis
>>
>> On 18/05/2016 1:06 PM, Julian Hyde wrote:
>>> It sounds as if there is general approval for this. I have logged https://issues.apache.org/jira/browse/CALCITE-1240<https://issues.apache.org/jira/browse/CALCITE-1240> to track.
>>>
>>> Julian
>>>
>>>> On May 17, 2016, at 8:00 PM, Josh Elser<jo...@gmail.com> wrote:
>>>>
>>>> Big +1 from me.
>>>>
>>>> I think if you're amenable to it, Francis, I'm more than willing to help make this a "formal" part of Avatica!
>>>>
>>>> Congrats and great work on what you have done already!
>>>>
>>>> F21 wrote:
>>>>> That would be really great! I think that would help make a lot of
>>>>> Phoenix drivers currently available to support avatica generically. It
>>>>> would also reduce the burden of driver maintainers maintaining a list of
>>>>> errors.
>>>>>
>>>>> On 18/05/2016 3:48 AM, Julian Hyde wrote:
>>>>>> Would it help if we added a function to Avaticas API, so a client
>>>>>> could ask for that map when connecting? It would allow the driver to
>>>>>> work against multiple servers, and in the phoenix-only case it would
>>>>>> mean that you wouldt have to upgrade the client driver if you
>>>>>> upgraded the phoenix server.
>>>>>>
>>>>>> (Were still talking hypotheticals. I would like to hear more of a
>>>>>> consensus from the community before we include this in Avatica.)
>>>>>>
>>>>>> Julian
>>>>>>
>>>>>>
>>>>>>> On May 16, 2016, at 10:57 PM, F21<f2...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hey Julian,
>>>>>>>
>>>>>>> The code should be useful for avatica in general. The only phoenix
>>>>>>> specific bit is the map of Phoenix error codes here:
>>>>>>> https://github.com/Boostport/avatica/blob/master/errors.go#L77
>>>>>>>
>>>>>>> I think other database backends can have their own maps. It might
>>>>>>> also be nice to be able to interrogate the avatica server to see if
>>>>>>> the backend is Phoenix or some other database, and the switch the
>>>>>>> errors map accordingly.
>>>>>>>
>>>>>>> Francis
>>>>>>>
>>>>>>> On 17/05/2016 3:54 PM, Julian Hyde wrote:
>>>>>>>> Excellent! Thanks for doing this.
>>>>>>>>
>>>>>>>> I haven't yet looked at the code to see how much is specific to
>>>>>>>> Phoenix and whether it would work against any Avatica provider. If
>>>>>>>> it is generic, and if you are amenable, there might be a place for
>>>>>>>> it in the Avatica project.
>>>>>>>>
>>>>>>>> What do others think?
>>>>>>>>
>>>>>>>> Julian
>>>>>>>>
>>>>>>>>> On May 16, 2016, at 10:43 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I have just open sourced a golang driver for Phoenix and Avatica.
>>>>>>>>>
>>>>>>>>> The code is licensed using the Apache 2 License and is available
>>>>>>>>> here: https://github.com/Boostport/avatica
>>>>>>>>>
>>>>>>>>> Contributions are very welcomed :)
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Francis
>>>>>>>>>
>
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by Josh Elser <el...@apache.org>.
Julian Hyde wrote:
>> On Nov 8, 2016, at 4:15 PM, F21<f2...@gmail.com> wrote:
>>
>> On 9/11/2016 11:06 AM, Julian Hyde wrote:
>>> A couple more things:
>>> * Integrate the driver and its documentation into Avatica\u2019s web site. I agree with putting source code in a separate repo, but all drivers should be on the same web site.
>> This can be easily done for the readme and usage docs. For the API docs, the godoc tool can generate html files from the source code, however for opensource projects, we tend to defer that to godoc.org. For example, if the project is hosted at github.com/Boostport/avatica, then going to godoc.org/github.com/Boostport/avatica would automatically pull in the source and generate the documents. Would it be acceptable to defer the API docs to godoc.org?
>
> Yes, entirely acceptable.
+1 duh. Documentation is rather important :)
> We must think of the consumers of our software, and deliver it in the way most convenient for them. Which means adhering to the norms of their community.
>
>>> * The driver should know which ranges of protocol versions it can handle, and give a useful error if it encounters a protocol version outside that range.
>> There doesn't appear to be any versioning for the Protobuf protocol (I am assuming that with Josh's work, the goal is to make using old fields compatible and deprecate them). Can you expand on this one?
>
> I don\u2019t know enough about Protobuf to answer that one. But if Protobuf doesn\u2019t know its version, at least an Avatica server does. the Go client is connecting to an Avatica server whose version is too or too recent, a user can reasonably expect a descriptive and useful error message, and I assume that it is technically possible to make that happen. If I\u2019m wrong, can you correct me, Josh.
>
> Julian
>
Right now, we haven't had to build such a thing. Wire protocol
versioning is likely going to be a pre-req when we go to Avatica-2.0 (or
whichever version we decide to start removing things that we've
deprecated). Thus far, clients don't need to do anything different WRT
the version of the server being communicated with; that's the "power" of
protobuf.
Does that help clarify things?
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by Julian Hyde <jh...@apache.org>.
> On Nov 8, 2016, at 4:15 PM, F21 <f2...@gmail.com> wrote:
>
> On 9/11/2016 11:06 AM, Julian Hyde wrote:
>> A couple more things:
>> * Integrate the driver and its documentation into Avatica’s web site. I agree with putting source code in a separate repo, but all drivers should be on the same web site.
> This can be easily done for the readme and usage docs. For the API docs, the godoc tool can generate html files from the source code, however for opensource projects, we tend to defer that to godoc.org. For example, if the project is hosted at github.com/Boostport/avatica, then going to godoc.org/github.com/Boostport/avatica would automatically pull in the source and generate the documents. Would it be acceptable to defer the API docs to godoc.org?
Yes, entirely acceptable.
We must think of the consumers of our software, and deliver it in the way most convenient for them. Which means adhering to the norms of their community.
>> * The driver should know which ranges of protocol versions it can handle, and give a useful error if it encounters a protocol version outside that range.
> There doesn't appear to be any versioning for the Protobuf protocol (I am assuming that with Josh's work, the goal is to make using old fields compatible and deprecate them). Can you expand on this one?
I don’t know enough about Protobuf to answer that one. But if Protobuf doesn’t know its version, at least an Avatica server does. the Go client is connecting to an Avatica server whose version is too or too recent, a user can reasonably expect a descriptive and useful error message, and I assume that it is technically possible to make that happen. If I’m wrong, can you correct me, Josh.
Julian
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by F21 <f2...@gmail.com>.
Hey Julian,
On 9/11/2016 11:06 AM, Julian Hyde wrote:
> A couple more things:
> * Integrate the driver and its documentation into Avatica\u2019s web site. I agree with putting source code in a separate repo, but all drivers should be on the same web site.
This can be easily done for the readme and usage docs. For the API docs,
the godoc tool can generate html files from the source code, however for
opensource projects, we tend to defer that to godoc.org. For example, if
the project is hosted at github.com/Boostport/avatica, then going to
godoc.org/github.com/Boostport/avatica would automatically pull in the
source and generate the documents. Would it be acceptable to defer the
API docs to godoc.org?
> * The driver should know which ranges of protocol versions it can handle, and give a useful error if it encounters a protocol version outside that range.
There doesn't appear to be any versioning for the Protobuf protocol (I
am assuming that with Josh's work, the goal is to make using old fields
compatible and deprecate them). Can you expand on this one?
>
> Other than that, +1.
>
> Julian
>
>
>
>> On Nov 8, 2016, at 2:22 PM, Josh Elser <el...@apache.org> wrote:
>>
>> Great, thanks for the input, Francis. I think the Go driver would be a good "guinea pig" for the process.
>>
>> Let's less this proposed process shake out with input from others and then we can start planning how to move forward specifically with the Golang driver.
>>
>> F21 wrote:
>>> Hey all,
>>>
>>> My thoughts in-line:
>>>
>>> On 9/11/2016 4:34 AM, Josh Elser wrote:
>>>> Re-vitalizing some discussion here (updating the subject accordingly)
>>>>
>>>> Thinking ahead with the assumption that the PMC and Francis all want
>>>> to bring this code under governance of the Calcite PMC, what does
>>>> "adoption of an Avatica client" really entail? My thoughts..
>>>>
>>>> * Perform ASF IP Clearance [1]. Dbl-checking copyright is the most
>>>> difficult thing (in most cases, it shouldn't be difficult).
>>> This shouldn't be a problem. However the project imports some
>>> dependencies that are not licensed under the Apache 2 License. The
>>> dependencies are not stored in the git repository, but are imported by
>>> the user's package manager. Will this be an issue? See dependency list
>>> here: https://github.com/Boostport/avatica/blob/master/vendor/vendor.json
>>>> * Create a new repository for the new client
>>> +1. In general, it is best to have the code in its own repository, so
>>> that it's easier to import.
>>>> * Define what (if any) released artifacts are (in addition to
>>>> source-code)
>>> Go library maintainers do not release any artifacts (in general), and we
>>> are not providing any artifacts, so this should not be a problem.
>>>> * Ensure code meets licensing requirements of ASF
>>> The code is licensed under the Apache 2 License. Is there anything else
>>> that needs to be done?
>> Short-answer: yes :)
>>
>> Having the ASL already applied to the code base makes this process very easy, however.
>>
>>>> * Establish build, test, and release steps for the new client
>>> This should be quite straight forward. Since we do not need to build
>>> anything, we just need to make sure we have tests running on master and
>>> any pull requests. When required, pushing a tag will generate a release.
>>> As Go's package management story is somewhat fragmented, we do need to
>>> provide some instructions on installing the package and pulling in the
>>> dependencies. It currently uses govendor, but I want to switch to glide
>>> soon, as glide is more mature and works better (from my experience).
>>>> * Establish CI for commits/patches (manual, if not automated?)
>>> Automated CI is a must. I currently use Wercker, as it allows me to
>>> easily build in docker containers using the official Go image. I can
>>> then easily update Go's version by switching the image in the
>>> configuration file. I believe Jenkins does support using docker for
>>> builds, so I hope this is something that is possible. For commits and
>>> patches, I would prefer to use Github pull requests, just like Calcite.
>>> We should also get the ASF bot to automatically build PRs and ping the
>>> relevant JIRAs (if any).
>> Cool. Definitely implementation details here. Some leg-work would be required to figure out exactly how we want to approach this. I'm sure we have lots of freedom.
>>
>>>> My thinking is that we can rely on the backwards compatibility
>>>> guarantees of Avatica's wire protocol to deal with the differing
>>>> release schedules. Specifically, an older version of the Golang client
>>>> should still work with newer versions of the Avatica server -- they
>>>> should not require simultaneous releases of all Avatica clients with
>>>> the de-facto Java client and Avatica server.
>>>>
>>> +1. In fact, I believe that most clients have switched to using
>>> protobufs, so backwards compatibility should not be a big problem unless
>>> we accidentally break something.
>> Agreed.
>>
>>>> If we can codify these practices, it would make adoption of future
>>>> clients less intimidating, letting us focus on the positives of having
>>>> a large collection of clients in other languages.
>>>>
>>>> Thoughts?
>>> +1 Having lots of clients would be a huge plus for Calcite and other
>>> projects such as Phoenix, etc. Go 1.8, which is due in a few months will
>>> bring a lot of new features to the database/sql package (see
>>> https://docs.google.com/document/d/1F778e7ZSNiSmbju3jsEWzShcb8lIO4kDyfKDNm4PNd8/edit?usp=sharing).
>>> I haven't had a chance to update the project to include the new
>>> features, but I am really excited :)
>>>> - Josh
>>>>
>>>> [1] http://incubator.apache.org/ip-clearance/
>>>>
>>>> Julian Hyde wrote:
>>>>> Probably not, in the near term at least. The two projects have
>>>>> separate release schedules, artifacts and web sites, which seems to
>>>>> give them enough breathing space for now. Maybe we\u2019ll split source
>>>>> code repositories at some point. Because the code is all managed by
>>>>> Apache\u2019s IP governance, any option we choose would be fairly
>>>>> straightforward.
>>>>>
>>>>> Julian
>>>>>
>>>>>
>>>>>> On May 17, 2016, at 8:11 PM, F21<f2...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks for opening the issue on JIRA, Julian.
>>>>>>
>>>>>> Let me know if there's anything I can do to speed up the process.
>>>>>> Will Avatica be spun out as its own project?
>>>>>>
>>>>>> Francis
>>>>>>
>>>>>> On 18/05/2016 1:06 PM, Julian Hyde wrote:
>>>>>>> It sounds as if there is general approval for this. I have logged
>>>>>>> https://issues.apache.org/jira/browse/CALCITE-1240<https://issues.apache.org/jira/browse/CALCITE-1240>
>>>>>>> to track.
>>>>>>>
>>>>>>> Julian
>>>>>>>
>>>>>>>> On May 17, 2016, at 8:00 PM, Josh Elser<jo...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Big +1 from me.
>>>>>>>>
>>>>>>>> I think if you're amenable to it, Francis, I'm more than willing
>>>>>>>> to help make this a "formal" part of Avatica!
>>>>>>>>
>>>>>>>> Congrats and great work on what you have done already!
>>>>>>>>
>>>>>>>> F21 wrote:
>>>>>>>>> That would be really great! I think that would help make a lot of
>>>>>>>>> Phoenix drivers currently available to support avatica
>>>>>>>>> generically. It
>>>>>>>>> would also reduce the burden of driver maintainers maintaining a
>>>>>>>>> list of
>>>>>>>>> errors.
>>>>>>>>>
>>>>>>>>> On 18/05/2016 3:48 AM, Julian Hyde wrote:
>>>>>>>>>> Would it help if we added a function to Avatica\u2019s API, so a client
>>>>>>>>>> could ask for that map when connecting? It would allow the
>>>>>>>>>> driver to
>>>>>>>>>> work against multiple servers, and in the phoenix-only case it
>>>>>>>>>> would
>>>>>>>>>> mean that you would\u2019t have to upgrade the client driver if you
>>>>>>>>>> upgraded the phoenix server.
>>>>>>>>>>
>>>>>>>>>> (We\u2019re still talking hypotheticals. I would like to hear more of a
>>>>>>>>>> consensus from the community before we include this in Avatica.)
>>>>>>>>>>
>>>>>>>>>> Julian
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 16, 2016, at 10:57 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey Julian,
>>>>>>>>>>>
>>>>>>>>>>> The code should be useful for avatica in general. The only phoenix
>>>>>>>>>>> specific bit is the map of Phoenix error codes here:
>>>>>>>>>>> https://github.com/Boostport/avatica/blob/master/errors.go#L77
>>>>>>>>>>>
>>>>>>>>>>> I think other database backends can have their own maps. It might
>>>>>>>>>>> also be nice to be able to interrogate the avatica server to
>>>>>>>>>>> see if
>>>>>>>>>>> the backend is Phoenix or some other database, and the switch the
>>>>>>>>>>> errors map accordingly.
>>>>>>>>>>>
>>>>>>>>>>> Francis
>>>>>>>>>>>
>>>>>>>>>>> On 17/05/2016 3:54 PM, Julian Hyde wrote:
>>>>>>>>>>>> Excellent! Thanks for doing this.
>>>>>>>>>>>>
>>>>>>>>>>>> I haven't yet looked at the code to see how much is specific to
>>>>>>>>>>>> Phoenix and whether it would work against any Avatica
>>>>>>>>>>>> provider. If
>>>>>>>>>>>> it is generic, and if you are amenable, there might be a place
>>>>>>>>>>>> for
>>>>>>>>>>>> it in the Avatica project.
>>>>>>>>>>>>
>>>>>>>>>>>> What do others think?
>>>>>>>>>>>>
>>>>>>>>>>>> Julian
>>>>>>>>>>>>
>>>>>>>>>>>>> On May 16, 2016, at 10:43 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have just open sourced a golang driver for Phoenix and
>>>>>>>>>>>>> Avatica.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The code is licensed using the Apache 2 License and is available
>>>>>>>>>>>>> here: https://github.com/Boostport/avatica
>>>>>>>>>>>>>
>>>>>>>>>>>>> Contributions are very welcomed :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Francis
>>>>>>>>>>>>>
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by Julian Hyde <jh...@apache.org>.
A couple more things:
* Integrate the driver and its documentation into Avatica’s web site. I agree with putting source code in a separate repo, but all drivers should be on the same web site.
* The driver should know which ranges of protocol versions it can handle, and give a useful error if it encounters a protocol version outside that range.
Other than that, +1.
Julian
> On Nov 8, 2016, at 2:22 PM, Josh Elser <el...@apache.org> wrote:
>
> Great, thanks for the input, Francis. I think the Go driver would be a good "guinea pig" for the process.
>
> Let's less this proposed process shake out with input from others and then we can start planning how to move forward specifically with the Golang driver.
>
> F21 wrote:
>> Hey all,
>>
>> My thoughts in-line:
>>
>> On 9/11/2016 4:34 AM, Josh Elser wrote:
>>> Re-vitalizing some discussion here (updating the subject accordingly)
>>>
>>> Thinking ahead with the assumption that the PMC and Francis all want
>>> to bring this code under governance of the Calcite PMC, what does
>>> "adoption of an Avatica client" really entail? My thoughts..
>>>
>>> * Perform ASF IP Clearance [1]. Dbl-checking copyright is the most
>>> difficult thing (in most cases, it shouldn't be difficult).
>> This shouldn't be a problem. However the project imports some
>> dependencies that are not licensed under the Apache 2 License. The
>> dependencies are not stored in the git repository, but are imported by
>> the user's package manager. Will this be an issue? See dependency list
>> here: https://github.com/Boostport/avatica/blob/master/vendor/vendor.json
>>> * Create a new repository for the new client
>> +1. In general, it is best to have the code in its own repository, so
>> that it's easier to import.
>>> * Define what (if any) released artifacts are (in addition to
>>> source-code)
>> Go library maintainers do not release any artifacts (in general), and we
>> are not providing any artifacts, so this should not be a problem.
>>> * Ensure code meets licensing requirements of ASF
>> The code is licensed under the Apache 2 License. Is there anything else
>> that needs to be done?
>
> Short-answer: yes :)
>
> Having the ASL already applied to the code base makes this process very easy, however.
>
>>> * Establish build, test, and release steps for the new client
>> This should be quite straight forward. Since we do not need to build
>> anything, we just need to make sure we have tests running on master and
>> any pull requests. When required, pushing a tag will generate a release.
>> As Go's package management story is somewhat fragmented, we do need to
>> provide some instructions on installing the package and pulling in the
>> dependencies. It currently uses govendor, but I want to switch to glide
>> soon, as glide is more mature and works better (from my experience).
>>> * Establish CI for commits/patches (manual, if not automated?)
>> Automated CI is a must. I currently use Wercker, as it allows me to
>> easily build in docker containers using the official Go image. I can
>> then easily update Go's version by switching the image in the
>> configuration file. I believe Jenkins does support using docker for
>> builds, so I hope this is something that is possible. For commits and
>> patches, I would prefer to use Github pull requests, just like Calcite.
>> We should also get the ASF bot to automatically build PRs and ping the
>> relevant JIRAs (if any).
>
> Cool. Definitely implementation details here. Some leg-work would be required to figure out exactly how we want to approach this. I'm sure we have lots of freedom.
>
>>>
>>> My thinking is that we can rely on the backwards compatibility
>>> guarantees of Avatica's wire protocol to deal with the differing
>>> release schedules. Specifically, an older version of the Golang client
>>> should still work with newer versions of the Avatica server -- they
>>> should not require simultaneous releases of all Avatica clients with
>>> the de-facto Java client and Avatica server.
>>>
>> +1. In fact, I believe that most clients have switched to using
>> protobufs, so backwards compatibility should not be a big problem unless
>> we accidentally break something.
>
> Agreed.
>
>>> If we can codify these practices, it would make adoption of future
>>> clients less intimidating, letting us focus on the positives of having
>>> a large collection of clients in other languages.
>>>
>>> Thoughts?
>> +1 Having lots of clients would be a huge plus for Calcite and other
>> projects such as Phoenix, etc. Go 1.8, which is due in a few months will
>> bring a lot of new features to the database/sql package (see
>> https://docs.google.com/document/d/1F778e7ZSNiSmbju3jsEWzShcb8lIO4kDyfKDNm4PNd8/edit?usp=sharing).
>> I haven't had a chance to update the project to include the new
>> features, but I am really excited :)
>>>
>>> - Josh
>>>
>>> [1] http://incubator.apache.org/ip-clearance/
>>>
>>> Julian Hyde wrote:
>>>> Probably not, in the near term at least. The two projects have
>>>> separate release schedules, artifacts and web sites, which seems to
>>>> give them enough breathing space for now. Maybe we’ll split source
>>>> code repositories at some point. Because the code is all managed by
>>>> Apache’s IP governance, any option we choose would be fairly
>>>> straightforward.
>>>>
>>>> Julian
>>>>
>>>>
>>>>> On May 17, 2016, at 8:11 PM, F21<f2...@gmail.com> wrote:
>>>>>
>>>>> Thanks for opening the issue on JIRA, Julian.
>>>>>
>>>>> Let me know if there's anything I can do to speed up the process.
>>>>> Will Avatica be spun out as its own project?
>>>>>
>>>>> Francis
>>>>>
>>>>> On 18/05/2016 1:06 PM, Julian Hyde wrote:
>>>>>> It sounds as if there is general approval for this. I have logged
>>>>>> https://issues.apache.org/jira/browse/CALCITE-1240<https://issues.apache.org/jira/browse/CALCITE-1240>
>>>>>> to track.
>>>>>>
>>>>>> Julian
>>>>>>
>>>>>>> On May 17, 2016, at 8:00 PM, Josh Elser<jo...@gmail.com> wrote:
>>>>>>>
>>>>>>> Big +1 from me.
>>>>>>>
>>>>>>> I think if you're amenable to it, Francis, I'm more than willing
>>>>>>> to help make this a "formal" part of Avatica!
>>>>>>>
>>>>>>> Congrats and great work on what you have done already!
>>>>>>>
>>>>>>> F21 wrote:
>>>>>>>> That would be really great! I think that would help make a lot of
>>>>>>>> Phoenix drivers currently available to support avatica
>>>>>>>> generically. It
>>>>>>>> would also reduce the burden of driver maintainers maintaining a
>>>>>>>> list of
>>>>>>>> errors.
>>>>>>>>
>>>>>>>> On 18/05/2016 3:48 AM, Julian Hyde wrote:
>>>>>>>>> Would it help if we added a function to Avatica’s API, so a client
>>>>>>>>> could ask for that map when connecting? It would allow the
>>>>>>>>> driver to
>>>>>>>>> work against multiple servers, and in the phoenix-only case it
>>>>>>>>> would
>>>>>>>>> mean that you would’t have to upgrade the client driver if you
>>>>>>>>> upgraded the phoenix server.
>>>>>>>>>
>>>>>>>>> (We’re still talking hypotheticals. I would like to hear more of a
>>>>>>>>> consensus from the community before we include this in Avatica.)
>>>>>>>>>
>>>>>>>>> Julian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On May 16, 2016, at 10:57 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey Julian,
>>>>>>>>>>
>>>>>>>>>> The code should be useful for avatica in general. The only phoenix
>>>>>>>>>> specific bit is the map of Phoenix error codes here:
>>>>>>>>>> https://github.com/Boostport/avatica/blob/master/errors.go#L77
>>>>>>>>>>
>>>>>>>>>> I think other database backends can have their own maps. It might
>>>>>>>>>> also be nice to be able to interrogate the avatica server to
>>>>>>>>>> see if
>>>>>>>>>> the backend is Phoenix or some other database, and the switch the
>>>>>>>>>> errors map accordingly.
>>>>>>>>>>
>>>>>>>>>> Francis
>>>>>>>>>>
>>>>>>>>>> On 17/05/2016 3:54 PM, Julian Hyde wrote:
>>>>>>>>>>> Excellent! Thanks for doing this.
>>>>>>>>>>>
>>>>>>>>>>> I haven't yet looked at the code to see how much is specific to
>>>>>>>>>>> Phoenix and whether it would work against any Avatica
>>>>>>>>>>> provider. If
>>>>>>>>>>> it is generic, and if you are amenable, there might be a place
>>>>>>>>>>> for
>>>>>>>>>>> it in the Avatica project.
>>>>>>>>>>>
>>>>>>>>>>> What do others think?
>>>>>>>>>>>
>>>>>>>>>>> Julian
>>>>>>>>>>>
>>>>>>>>>>>> On May 16, 2016, at 10:43 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I have just open sourced a golang driver for Phoenix and
>>>>>>>>>>>> Avatica.
>>>>>>>>>>>>
>>>>>>>>>>>> The code is licensed using the Apache 2 License and is available
>>>>>>>>>>>> here: https://github.com/Boostport/avatica
>>>>>>>>>>>>
>>>>>>>>>>>> Contributions are very welcomed :)
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> Francis
>>>>>>>>>>>>
>>>>
>>
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by Josh Elser <el...@apache.org>.
Great, thanks for the input, Francis. I think the Go driver would be a
good "guinea pig" for the process.
Let's less this proposed process shake out with input from others and
then we can start planning how to move forward specifically with the
Golang driver.
F21 wrote:
> Hey all,
>
> My thoughts in-line:
>
> On 9/11/2016 4:34 AM, Josh Elser wrote:
>> Re-vitalizing some discussion here (updating the subject accordingly)
>>
>> Thinking ahead with the assumption that the PMC and Francis all want
>> to bring this code under governance of the Calcite PMC, what does
>> "adoption of an Avatica client" really entail? My thoughts..
>>
>> * Perform ASF IP Clearance [1]. Dbl-checking copyright is the most
>> difficult thing (in most cases, it shouldn't be difficult).
> This shouldn't be a problem. However the project imports some
> dependencies that are not licensed under the Apache 2 License. The
> dependencies are not stored in the git repository, but are imported by
> the user's package manager. Will this be an issue? See dependency list
> here: https://github.com/Boostport/avatica/blob/master/vendor/vendor.json
>> * Create a new repository for the new client
> +1. In general, it is best to have the code in its own repository, so
> that it's easier to import.
>> * Define what (if any) released artifacts are (in addition to
>> source-code)
> Go library maintainers do not release any artifacts (in general), and we
> are not providing any artifacts, so this should not be a problem.
>> * Ensure code meets licensing requirements of ASF
> The code is licensed under the Apache 2 License. Is there anything else
> that needs to be done?
Short-answer: yes :)
Having the ASL already applied to the code base makes this process very
easy, however.
>> * Establish build, test, and release steps for the new client
> This should be quite straight forward. Since we do not need to build
> anything, we just need to make sure we have tests running on master and
> any pull requests. When required, pushing a tag will generate a release.
> As Go's package management story is somewhat fragmented, we do need to
> provide some instructions on installing the package and pulling in the
> dependencies. It currently uses govendor, but I want to switch to glide
> soon, as glide is more mature and works better (from my experience).
>> * Establish CI for commits/patches (manual, if not automated?)
> Automated CI is a must. I currently use Wercker, as it allows me to
> easily build in docker containers using the official Go image. I can
> then easily update Go's version by switching the image in the
> configuration file. I believe Jenkins does support using docker for
> builds, so I hope this is something that is possible. For commits and
> patches, I would prefer to use Github pull requests, just like Calcite.
> We should also get the ASF bot to automatically build PRs and ping the
> relevant JIRAs (if any).
Cool. Definitely implementation details here. Some leg-work would be
required to figure out exactly how we want to approach this. I'm sure we
have lots of freedom.
>>
>> My thinking is that we can rely on the backwards compatibility
>> guarantees of Avatica's wire protocol to deal with the differing
>> release schedules. Specifically, an older version of the Golang client
>> should still work with newer versions of the Avatica server -- they
>> should not require simultaneous releases of all Avatica clients with
>> the de-facto Java client and Avatica server.
>>
> +1. In fact, I believe that most clients have switched to using
> protobufs, so backwards compatibility should not be a big problem unless
> we accidentally break something.
Agreed.
>> If we can codify these practices, it would make adoption of future
>> clients less intimidating, letting us focus on the positives of having
>> a large collection of clients in other languages.
>>
>> Thoughts?
> +1 Having lots of clients would be a huge plus for Calcite and other
> projects such as Phoenix, etc. Go 1.8, which is due in a few months will
> bring a lot of new features to the database/sql package (see
> https://docs.google.com/document/d/1F778e7ZSNiSmbju3jsEWzShcb8lIO4kDyfKDNm4PNd8/edit?usp=sharing).
> I haven't had a chance to update the project to include the new
> features, but I am really excited :)
>>
>> - Josh
>>
>> [1] http://incubator.apache.org/ip-clearance/
>>
>> Julian Hyde wrote:
>>> Probably not, in the near term at least. The two projects have
>>> separate release schedules, artifacts and web sites, which seems to
>>> give them enough breathing space for now. Maybe well split source
>>> code repositories at some point. Because the code is all managed by
>>> Apaches IP governance, any option we choose would be fairly
>>> straightforward.
>>>
>>> Julian
>>>
>>>
>>>> On May 17, 2016, at 8:11 PM, F21<f2...@gmail.com> wrote:
>>>>
>>>> Thanks for opening the issue on JIRA, Julian.
>>>>
>>>> Let me know if there's anything I can do to speed up the process.
>>>> Will Avatica be spun out as its own project?
>>>>
>>>> Francis
>>>>
>>>> On 18/05/2016 1:06 PM, Julian Hyde wrote:
>>>>> It sounds as if there is general approval for this. I have logged
>>>>> https://issues.apache.org/jira/browse/CALCITE-1240<https://issues.apache.org/jira/browse/CALCITE-1240>
>>>>> to track.
>>>>>
>>>>> Julian
>>>>>
>>>>>> On May 17, 2016, at 8:00 PM, Josh Elser<jo...@gmail.com> wrote:
>>>>>>
>>>>>> Big +1 from me.
>>>>>>
>>>>>> I think if you're amenable to it, Francis, I'm more than willing
>>>>>> to help make this a "formal" part of Avatica!
>>>>>>
>>>>>> Congrats and great work on what you have done already!
>>>>>>
>>>>>> F21 wrote:
>>>>>>> That would be really great! I think that would help make a lot of
>>>>>>> Phoenix drivers currently available to support avatica
>>>>>>> generically. It
>>>>>>> would also reduce the burden of driver maintainers maintaining a
>>>>>>> list of
>>>>>>> errors.
>>>>>>>
>>>>>>> On 18/05/2016 3:48 AM, Julian Hyde wrote:
>>>>>>>> Would it help if we added a function to Avaticas API, so a client
>>>>>>>> could ask for that map when connecting? It would allow the
>>>>>>>> driver to
>>>>>>>> work against multiple servers, and in the phoenix-only case it
>>>>>>>> would
>>>>>>>> mean that you wouldt have to upgrade the client driver if you
>>>>>>>> upgraded the phoenix server.
>>>>>>>>
>>>>>>>> (Were still talking hypotheticals. I would like to hear more of a
>>>>>>>> consensus from the community before we include this in Avatica.)
>>>>>>>>
>>>>>>>> Julian
>>>>>>>>
>>>>>>>>
>>>>>>>>> On May 16, 2016, at 10:57 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hey Julian,
>>>>>>>>>
>>>>>>>>> The code should be useful for avatica in general. The only phoenix
>>>>>>>>> specific bit is the map of Phoenix error codes here:
>>>>>>>>> https://github.com/Boostport/avatica/blob/master/errors.go#L77
>>>>>>>>>
>>>>>>>>> I think other database backends can have their own maps. It might
>>>>>>>>> also be nice to be able to interrogate the avatica server to
>>>>>>>>> see if
>>>>>>>>> the backend is Phoenix or some other database, and the switch the
>>>>>>>>> errors map accordingly.
>>>>>>>>>
>>>>>>>>> Francis
>>>>>>>>>
>>>>>>>>> On 17/05/2016 3:54 PM, Julian Hyde wrote:
>>>>>>>>>> Excellent! Thanks for doing this.
>>>>>>>>>>
>>>>>>>>>> I haven't yet looked at the code to see how much is specific to
>>>>>>>>>> Phoenix and whether it would work against any Avatica
>>>>>>>>>> provider. If
>>>>>>>>>> it is generic, and if you are amenable, there might be a place
>>>>>>>>>> for
>>>>>>>>>> it in the Avatica project.
>>>>>>>>>>
>>>>>>>>>> What do others think?
>>>>>>>>>>
>>>>>>>>>> Julian
>>>>>>>>>>
>>>>>>>>>>> On May 16, 2016, at 10:43 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I have just open sourced a golang driver for Phoenix and
>>>>>>>>>>> Avatica.
>>>>>>>>>>>
>>>>>>>>>>> The code is licensed using the Apache 2 License and is available
>>>>>>>>>>> here: https://github.com/Boostport/avatica
>>>>>>>>>>>
>>>>>>>>>>> Contributions are very welcomed :)
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Francis
>>>>>>>>>>>
>>>
>
Re: [DISCUSS] How to merge an Avatica client into the Calcite project
(was Re: Golang driver for Phoenix and Avatica available)
Posted by F21 <f2...@gmail.com>.
Hey all,
My thoughts in-line:
On 9/11/2016 4:34 AM, Josh Elser wrote:
> Re-vitalizing some discussion here (updating the subject accordingly)
>
> Thinking ahead with the assumption that the PMC and Francis all want
> to bring this code under governance of the Calcite PMC, what does
> "adoption of an Avatica client" really entail? My thoughts..
>
> * Perform ASF IP Clearance [1]. Dbl-checking copyright is the most
> difficult thing (in most cases, it shouldn't be difficult).
This shouldn't be a problem. However the project imports some
dependencies that are not licensed under the Apache 2 License. The
dependencies are not stored in the git repository, but are imported by
the user's package manager. Will this be an issue? See dependency list
here: https://github.com/Boostport/avatica/blob/master/vendor/vendor.json
> * Create a new repository for the new client
+1. In general, it is best to have the code in its own repository, so
that it's easier to import.
> * Define what (if any) released artifacts are (in addition to
> source-code)
Go library maintainers do not release any artifacts (in general), and we
are not providing any artifacts, so this should not be a problem.
> * Ensure code meets licensing requirements of ASF
The code is licensed under the Apache 2 License. Is there anything else
that needs to be done?
> * Establish build, test, and release steps for the new client
This should be quite straight forward. Since we do not need to build
anything, we just need to make sure we have tests running on master and
any pull requests. When required, pushing a tag will generate a release.
As Go's package management story is somewhat fragmented, we do need to
provide some instructions on installing the package and pulling in the
dependencies. It currently uses govendor, but I want to switch to glide
soon, as glide is more mature and works better (from my experience).
> * Establish CI for commits/patches (manual, if not automated?)
Automated CI is a must. I currently use Wercker, as it allows me to
easily build in docker containers using the official Go image. I can
then easily update Go's version by switching the image in the
configuration file. I believe Jenkins does support using docker for
builds, so I hope this is something that is possible. For commits and
patches, I would prefer to use Github pull requests, just like Calcite.
We should also get the ASF bot to automatically build PRs and ping the
relevant JIRAs (if any).
>
> My thinking is that we can rely on the backwards compatibility
> guarantees of Avatica's wire protocol to deal with the differing
> release schedules. Specifically, an older version of the Golang client
> should still work with newer versions of the Avatica server -- they
> should not require simultaneous releases of all Avatica clients with
> the de-facto Java client and Avatica server.
>
+1. In fact, I believe that most clients have switched to using
protobufs, so backwards compatibility should not be a big problem unless
we accidentally break something.
> If we can codify these practices, it would make adoption of future
> clients less intimidating, letting us focus on the positives of having
> a large collection of clients in other languages.
>
> Thoughts?
+1 Having lots of clients would be a huge plus for Calcite and other
projects such as Phoenix, etc. Go 1.8, which is due in a few months will
bring a lot of new features to the database/sql package (see
https://docs.google.com/document/d/1F778e7ZSNiSmbju3jsEWzShcb8lIO4kDyfKDNm4PNd8/edit?usp=sharing).
I haven't had a chance to update the project to include the new
features, but I am really excited :)
>
> - Josh
>
> [1] http://incubator.apache.org/ip-clearance/
>
> Julian Hyde wrote:
>> Probably not, in the near term at least. The two projects have
>> separate release schedules, artifacts and web sites, which seems to
>> give them enough breathing space for now. Maybe well split source
>> code repositories at some point. Because the code is all managed by
>> Apaches IP governance, any option we choose would be fairly
>> straightforward.
>>
>> Julian
>>
>>
>>> On May 17, 2016, at 8:11 PM, F21<f2...@gmail.com> wrote:
>>>
>>> Thanks for opening the issue on JIRA, Julian.
>>>
>>> Let me know if there's anything I can do to speed up the process.
>>> Will Avatica be spun out as its own project?
>>>
>>> Francis
>>>
>>> On 18/05/2016 1:06 PM, Julian Hyde wrote:
>>>> It sounds as if there is general approval for this. I have logged
>>>> https://issues.apache.org/jira/browse/CALCITE-1240<https://issues.apache.org/jira/browse/CALCITE-1240>
>>>> to track.
>>>>
>>>> Julian
>>>>
>>>>> On May 17, 2016, at 8:00 PM, Josh Elser<jo...@gmail.com> wrote:
>>>>>
>>>>> Big +1 from me.
>>>>>
>>>>> I think if you're amenable to it, Francis, I'm more than willing
>>>>> to help make this a "formal" part of Avatica!
>>>>>
>>>>> Congrats and great work on what you have done already!
>>>>>
>>>>> F21 wrote:
>>>>>> That would be really great! I think that would help make a lot of
>>>>>> Phoenix drivers currently available to support avatica
>>>>>> generically. It
>>>>>> would also reduce the burden of driver maintainers maintaining a
>>>>>> list of
>>>>>> errors.
>>>>>>
>>>>>> On 18/05/2016 3:48 AM, Julian Hyde wrote:
>>>>>>> Would it help if we added a function to Avaticas API, so a client
>>>>>>> could ask for that map when connecting? It would allow the
>>>>>>> driver to
>>>>>>> work against multiple servers, and in the phoenix-only case it
>>>>>>> would
>>>>>>> mean that you wouldt have to upgrade the client driver if you
>>>>>>> upgraded the phoenix server.
>>>>>>>
>>>>>>> (Were still talking hypotheticals. I would like to hear more of a
>>>>>>> consensus from the community before we include this in Avatica.)
>>>>>>>
>>>>>>> Julian
>>>>>>>
>>>>>>>
>>>>>>>> On May 16, 2016, at 10:57 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hey Julian,
>>>>>>>>
>>>>>>>> The code should be useful for avatica in general. The only phoenix
>>>>>>>> specific bit is the map of Phoenix error codes here:
>>>>>>>> https://github.com/Boostport/avatica/blob/master/errors.go#L77
>>>>>>>>
>>>>>>>> I think other database backends can have their own maps. It might
>>>>>>>> also be nice to be able to interrogate the avatica server to
>>>>>>>> see if
>>>>>>>> the backend is Phoenix or some other database, and the switch the
>>>>>>>> errors map accordingly.
>>>>>>>>
>>>>>>>> Francis
>>>>>>>>
>>>>>>>> On 17/05/2016 3:54 PM, Julian Hyde wrote:
>>>>>>>>> Excellent! Thanks for doing this.
>>>>>>>>>
>>>>>>>>> I haven't yet looked at the code to see how much is specific to
>>>>>>>>> Phoenix and whether it would work against any Avatica
>>>>>>>>> provider. If
>>>>>>>>> it is generic, and if you are amenable, there might be a place
>>>>>>>>> for
>>>>>>>>> it in the Avatica project.
>>>>>>>>>
>>>>>>>>> What do others think?
>>>>>>>>>
>>>>>>>>> Julian
>>>>>>>>>
>>>>>>>>>> On May 16, 2016, at 10:43 PM, F21<f2...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I have just open sourced a golang driver for Phoenix and
>>>>>>>>>> Avatica.
>>>>>>>>>>
>>>>>>>>>> The code is licensed using the Apache 2 License and is available
>>>>>>>>>> here: https://github.com/Boostport/avatica
>>>>>>>>>>
>>>>>>>>>> Contributions are very welcomed :)
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Francis
>>>>>>>>>>
>>