You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geaflow.apache.org by "DukeWangYu (via GitHub)" <gi...@apache.org> on 2025/08/20 12:45:06 UTC

[PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

DukeWangYu opened a new pull request, #594:
URL: https://github.com/apache/geaflow/pull/594

   What changes were proposed in this pull request?
   Add streaming source for hive
   
   resolve #592 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org


Re: [PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

Posted by "yaozhongq (via GitHub)" <gi...@apache.org>.
yaozhongq merged PR #594:
URL: https://github.com/apache/geaflow/pull/594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org


Re: [PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

Posted by "DukeWangYu (via GitHub)" <gi...@apache.org>.
DukeWangYu commented on code in PR #594:
URL: https://github.com/apache/geaflow/pull/594#discussion_r2296771876


##########
geaflow/geaflow-dsl/geaflow-dsl-connector/geaflow-dsl-connector-api/src/main/java/org/apache/geaflow/dsl/connector/api/FetchData.java:
##########
@@ -73,6 +73,17 @@ public Offset getNextOffset() {
         return nextOffset;
     }
 
+    public void seek(long seekPos) {
+        long toSkip = seekPos;
+        while (toSkip > 0) {
+            if (!dataIterator.hasNext()) {
+                throw new RuntimeException("skip to many rows " + seekPos);

Review Comment:
   1. More detailed out of bounds error messages have been added to exception
   2. Using RecordReader,skip data is the olay way to seek to target offst. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org


Re: [PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

Posted by "Loognqiang (via GitHub)" <gi...@apache.org>.
Loognqiang commented on code in PR #594:
URL: https://github.com/apache/geaflow/pull/594#discussion_r2293448406


##########
geaflow/geaflow-dsl/geaflow-dsl-connector/geaflow-dsl-connector-hive/src/main/java/org/apache/geaflow/dsl/connector/hive/HiveReader.java:
##########
@@ -69,11 +73,37 @@ public HiveReader(RecordReader<Writable, Writable> recordReader, StructType read
     }
 
     public FetchData<Row> read(long windowSize, String[] partitionValues) {
+        Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
         if (windowSize == Long.MAX_VALUE) {
-            Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
             return FetchData.createBatchFetch(hiveIterator, new HiveOffset(-1L));
         } else {
-            throw new GeaFlowDSLException("Cannot support stream read for hive");
+            long fetchCnt = 0L;
+            List<Row> rows = new ArrayList<>();
+            while (fetchCnt < windowSize) {
+                if (hiveIterator.hasNext()) {
+                    fetchCnt ++;
+                    rows.add(hiveIterator.next());
+                } else {
+                    break;
+                }
+            }
+            fetchOffset += fetchCnt;
+            return FetchData.createStreamFetch(rows, new HiveOffset(fetchOffset), fetchCnt < windowSize);

Review Comment:
   When will the HiveOffset be saved for FO?



##########
geaflow/geaflow-dsl/geaflow-dsl-connector/geaflow-dsl-connector-api/src/main/java/org/apache/geaflow/dsl/connector/api/FetchData.java:
##########
@@ -73,6 +73,17 @@ public Offset getNextOffset() {
         return nextOffset;
     }
 
+    public void seek(long seekPos) {
+        long toSkip = seekPos;
+        while (toSkip > 0) {
+            if (!dataIterator.hasNext()) {
+                throw new RuntimeException("skip to many rows " + seekPos);

Review Comment:
   1. Exception information needs to add "toSkip"
   2. Every seek requires going through this logic, which is very costly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org


Re: [PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

Posted by "DukeWangYu (via GitHub)" <gi...@apache.org>.
DukeWangYu commented on code in PR #594:
URL: https://github.com/apache/geaflow/pull/594#discussion_r2296772536


##########
geaflow/geaflow-dsl/geaflow-dsl-connector/geaflow-dsl-connector-hive/src/main/java/org/apache/geaflow/dsl/connector/hive/HiveReader.java:
##########
@@ -69,11 +73,37 @@ public HiveReader(RecordReader<Writable, Writable> recordReader, StructType read
     }
 
     public FetchData<Row> read(long windowSize, String[] partitionValues) {
+        Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
         if (windowSize == Long.MAX_VALUE) {
-            Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
             return FetchData.createBatchFetch(hiveIterator, new HiveOffset(-1L));
         } else {
-            throw new GeaFlowDSLException("Cannot support stream read for hive");
+            long fetchCnt = 0L;
+            List<Row> rows = new ArrayList<>();
+            while (fetchCnt < windowSize) {
+                if (hiveIterator.hasNext()) {
+                    fetchCnt ++;
+                    rows.add(hiveIterator.next());
+                } else {
+                    break;
+                }
+            }
+            fetchOffset += fetchCnt;
+            return FetchData.createStreamFetch(rows, new HiveOffset(fetchOffset), fetchCnt < windowSize);

Review Comment:
   GeaFlowTableSourceFunction will save the offset after fetch finish. GeaFlowTableSourceFunction will periodically check the split of hive. If there are any changes, fo will appear and reallocate all readers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org


Re: [PR] [ISSUE-592] feat: Add a Streaming Source for Hiveadd stream hive reader [geaflow]

Posted by "DukeWangYu (via GitHub)" <gi...@apache.org>.
DukeWangYu commented on code in PR #594:
URL: https://github.com/apache/geaflow/pull/594#discussion_r2296772536


##########
geaflow/geaflow-dsl/geaflow-dsl-connector/geaflow-dsl-connector-hive/src/main/java/org/apache/geaflow/dsl/connector/hive/HiveReader.java:
##########
@@ -69,11 +73,37 @@ public HiveReader(RecordReader<Writable, Writable> recordReader, StructType read
     }
 
     public FetchData<Row> read(long windowSize, String[] partitionValues) {
+        Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
         if (windowSize == Long.MAX_VALUE) {
-            Iterator<Row> hiveIterator = new HiveIterator(recordReader, deserializer, partitionValues, readSchema);
             return FetchData.createBatchFetch(hiveIterator, new HiveOffset(-1L));
         } else {
-            throw new GeaFlowDSLException("Cannot support stream read for hive");
+            long fetchCnt = 0L;
+            List<Row> rows = new ArrayList<>();
+            while (fetchCnt < windowSize) {
+                if (hiveIterator.hasNext()) {
+                    fetchCnt ++;
+                    rows.add(hiveIterator.next());
+                } else {
+                    break;
+                }
+            }
+            fetchOffset += fetchCnt;
+            return FetchData.createStreamFetch(rows, new HiveOffset(fetchOffset), fetchCnt < windowSize);

Review Comment:
   GeaFlowTableSourceFunction will periodically check the split of paimon. If there are any changes, fo will appear and reallocate all readers



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org