You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/10/19 06:20:41 UTC
[GitHub] [iceberg] kbendick opened a new issue #1628: Implement Initial Spark Structured Streaming Source
kbendick opened a new issue #1628:
URL: https://github.com/apache/iceberg/issues/1628
We currently cannot use Spark Structured Streaming for reading from an Iceberg table. This seems like a very common need.
We were previously discussing the issue quite a lot back in February, but discussion on the topic has slowed down quite a lot.
Some of the code that was discussed has been merged in and I'd like to revisit this.
The original PR which has become inactive: https://github.com/apache/iceberg/pull/796
My first PR which fixes some documentation in the MicroBatch builder class and adds some tests to start ensuring that the functionality which has been merged in is working and to start sussing out corner cases. https://github.com/apache/iceberg/pull/1627
I will work off of the conversation that has been in https://github.com/apache/iceberg/pull/796 as well as what I can find in Slack and then I hoped we could revisit issues as they arise. But a decent amount of the previously proposed code has been merged in, so I'd like to take a stab at piecing it together from the discussion that was previously had as well as changes I think that will be needed to support various scenarios, like deletes, different triggers, the global watermark, the per stream watermark that's declared etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jerryshao commented on issue #1628: Implement Initial Spark Structured Streaming Source
Posted by GitBox <gi...@apache.org>.
jerryshao commented on issue #1628:
URL: https://github.com/apache/iceberg/issues/1628#issuecomment-712543551
Hi @kbendick , thanks for your bringing out of this thing. Yes, I'm still working on this. Currently the left comments are mostly style or refactoring issues, so I didn't update the PR actively. I was waiting (for a long while) to see if any committer could check and approve the overall implementations, based on this I could start to change the details and make progress
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shiyi23 commented on issue #1628: Implement Initial Spark Structured Streaming Source
Posted by GitBox <gi...@apache.org>.
shiyi23 commented on issue #1628:
URL: https://github.com/apache/iceberg/issues/1628#issuecomment-745021239
/assign @shiyi23
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #1628: Implement Initial Spark Structured Streaming Source
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #1628:
URL: https://github.com/apache/iceberg/issues/1628#issuecomment-712525508
Actually as I understand it @jerryshao is still somewhat working on this. I'm waiting to hear back but I'm likely to close this ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #1628: Implement Initial Spark Structured Streaming Source
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #1628:
URL: https://github.com/apache/iceberg/issues/1628#issuecomment-712310054
cc @jerryshao as you've been working on and off on the original PR. Wondering where you are in currently working on it or if you'd be looking for assistance if you're currently busy?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #1628: Implement Initial Spark Structured Streaming Source
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #1628:
URL: https://github.com/apache/iceberg/issues/1628#issuecomment-713133254
Thanks for letting me know @jerryshao. For your own update, a lot of things in that PR seem to have been somewhat merged in.
Possibly try forking off of your current branch and merging in master to check the diff. A number of files might go away.
For example, if you check this PR (which I thought I linked to this issue but somehow the link didn't show), you'll see that the `Microbatches` class that's in your PR exists in some form: https://github.com/apache/iceberg/pull/1627/files
I just updated one javadoc comment and then added some tests to verify behavior. So I would think that possibly seeing if you can take your old PR (and the changes that you mentioned if you should like), place them in a new branch based off of the latest master, and see if the overall PR gets smaller.
My guess is that it might still need to be broken up into one or two refactor PRs, but I think the overall size of it will be much smaller. Please let me know if I can be of help in some way and I'll see what I can do (time at work permitting).
Thanks for you reply!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org