Re: Beam Connector for Reading Data from Delta Lake

[Michael Benenson <mb...@gmail.com>]

I have created PR https://github.com/apache/beam/pull/15434 for JIRA
https://issues.apache.org/jira/browse/BEAM-12824 :  Beam Connector for
Reading Data from Delta Lake

Now build using maven, I do not know how to create build.gradle for
building java & scala code in one project.
Could someone help to create build.gradle?

I put examples for using Delta Lake connector into the new directory
"beam/sdks/java/examples/deltalake", not sure it is the right place for the
examples, could someone suggest another place?

On Tue, Aug 31, 2021 at 8:48 AM Luke Cwik <lc...@google.com> wrote:

> I added you as a contributor to JIRA and assigned BEAM-12824 to you.
>
> Typically you would create your own clone of the repo and make pull
> requests from your clone to the apache/beam repo. See
> https://beam.apache.org/contribute/#contributing-code for more details.
>
> Branches on the main repo are usually reserved for other purposes (e.g.
> release branches).
>
> On Mon, Aug 30, 2021 at 8:56 PM Michael Benenson <mb...@gmail.com>
> wrote:
>
>> Hi, Cham
>> Right, it would be fine to extract common functionality from FileIO.
>>
>> On Mon, Aug 30, 2021 at 1:14 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> +1 to what Kenn said.
>>>
>>> We should probably think about how to not to copy a large amount of code
>>> though. Seems like the following class currently repeats a large amount of
>>> code in FileIO.java ?
>>>
>>> https://github.com/mbenenso/beam-deltalake/blob/main/src/main/java/org/apache/beam/sdk/io/DeltaFileIO.java
>>>
>>> I suspect most of this logic can be shared for code health.
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Mon, Aug 30, 2021 at 1:07 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> By all means, open a pull request to the main Beam repository (for this
>>>> or any other change you would like to propose) with the connector and
>>>> example. We can continue any detailed code-level conversation on the pull
>>>> request. A quick 1-pager (we tend to use Google Docs) can be helpful for
>>>> describing it at a high level so the community generally knows it exists
>>>> and how it works.
>>>>
>>>> Could you also file a JIRA ticket so it is automatically included in
>>>> our release notes once it is released? See
>>>> https://beam.apache.org/community/contact-us/ and let us know your
>>>> JIRA account.
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Aug 30, 2021 at 12:33 PM Michael Benenson <mb...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, folks
>>>>>
>>>>> I have put into the open source Beam connector for reading data from
>>>>> Delta Lake. Please, see https://github.com/mbenenso/beam-deltalake
>>>>> for the connector & https://github.com/mbenenso/beam-deltalake-example
>>>>> for the usage example.
>>>>>
>>>>> Would you like to consider adding this connector into the Beam
>>>>> repository?
>>>>>
>>>>> --
>>>>> Thanks
>>>>>
>>>>> ------------------------------
>>>>> Michael Benenson
>>>>>
>>>>
>>
>> --
>> Thanks
>>
>> ------------------------------
>> Michael Benenson
>>
>

-- 
Thanks

------------------------------
Michael Benenson