You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jhchee (via GitHub)" <gi...@apache.org> on 2023/04/22 13:43:53 UTC

[GitHub] [hudi] jhchee opened a new issue, #8544: [SUPPORT] Support rate limit when reading Hudi table

jhchee opened a new issue, #8544:
URL: https://github.com/apache/hudi/issues/8544

   **Describe the problem you faced**
   Currently, we are reading on Spark structured streaming to incrementally process Hudi table. One option that we would like to see is rate limiting when reading from Hudi table. This would help much to not overwhelm our downstream consumer.
   
   This was discussed in the past: https://github.com/apache/hudi/issues/3724#issuecomment-986309728
   Similar functionality in the Delta version.
   https://docs.delta.io/latest/delta-streaming.html#limit-input-rate
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.
   2.
   3.
   4.
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version :
   
   * Spark version :
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) :
   
   * Running on Docker? (yes/no) :
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #8544: [SUPPORT] Support rate limit when reading Hudi table

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8544:
URL: https://github.com/apache/hudi/issues/8544#issuecomment-1523559602

   since, this a feature request, we can probably go ahead and close the github issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed issue #8544: [SUPPORT] Support rate limit when reading Hudi table

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #8544: [SUPPORT] Support rate limit when reading Hudi table
URL: https://github.com/apache/hudi/issues/8544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #8544: [SUPPORT] Support rate limit when reading Hudi table

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8544:
URL: https://github.com/apache/hudi/issues/8544#issuecomment-1519301587

   Thanks, this is a useful feature, I have created a JIRA issue: https://issues.apache.org/jira/browse/HUDI-6129


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org