You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:25:49 UTC
[I] DynamoDB/Kinesis Change Capture using Delta Streamer [hudi]
hudi-bot opened a new issue, #14503:
URL: https://github.com/apache/hudi/issues/14503
The goal here is to do CDC from DynamoDB and then have it be ingested into S3 as a Hudi dataset
Few resources:
# DynamoDB Streams [https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html] provides change capture logs in Kinesis.
# Walkthrough [https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.Walkthrough.html] Code [https://github.com/awslabs/dynamodb-streams-kinesis-adapter]
# Spark Streaming has support for reading Kinesis streams [https://spark.apache.org/docs/2.4.4/streaming-kinesis-integration.html] one of the many resources showing how to change the Spark Kinesis example code to consume dynamodb stream [https://medium.com/@ravi72munde/using-spark-streaming-with-dynamodb-d325b9a73c79]
# In DeltaStreamer, we need to add some form of KinesisSource that returns a RDD with new data everytime `fetchNewData` is called [https://github.com/apache/incubator-hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/Source.java] . DeltaStreamer itself does not use Spark Streaming APIs
# Internally, we have Avro, Json, Row sources that extract data in these formats.
Open questions :
# Should this just be a KinesisSource inside Hudi, that needs to be configured differently or do we need two sources: DynamoDBKinesisSource (that does some DynamoDB Stream specific setup/assumptions) and a plain KinesisSource. What's more valuable to do , if we have to pick one.
# For Kafka integration, we just reused the KafkaRDD in Spark Streaming easily and avoided writing a lot of code by hand. Could we pull the same thing off for Kinesis? (probably needs digging through Spark code)
# What's the format of the data for DynamoDB streams?
We should probably flesh these out before going ahead with implementation?
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-310
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-1385
---
## Comments
13/Nov/19 05:11;vinoth;Hi [~vinaypatil18] any updates for us? ;;;
---
26/Jan/21 20:23;shivnarayan;[~vinoth]: Is this still relevant? do we keep it open. ;;;
---
04/Nov/21 01:33;maddy2u;Is this still happening? ;;;
---
06/Nov/21 05:09;vinoth;This has changed hands quite a bit. We still want to take this on. Interested?;;;
---
09/Jan/22 08:57;vinaypatil18;[~vinoth] I remember discussing about this, sry it went to backlog, I am taking this up. ;;;
---
11/Mar/22 15:00;maddy2u;Hi [~vinaypatil18] - Any update on the above? Which release are we targeting this feature?;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DynamoDB/Kinesis Change Capture using Delta Streamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14503: DynamoDB/Kinesis Change Capture using Delta Streamer
URL: https://github.com/apache/hudi/issues/14503
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DynamoDB/Kinesis Change Capture using Delta Streamer [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14503:
URL: https://github.com/apache/hudi/issues/14503#issuecomment-4795175802
This feature request appears to be addressed on master. PR #18224 (commit `4bdcdf9b52b5`, *feat: Create JsonKinesisSource*) introduced a generic `KinesisSource` abstract base and a concrete `JsonKinesisSource` in `hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/`, along with `KinesisSourceConfig` and `KinesisOffsetGen`. This resolves the first open question in the issue (generic KinesisSource vs. DynamoDB-specific source) by choosing the generic path. For DynamoDB CDC, users can enable *Kinesis Data Streams for DynamoDB* (or use the DynamoDB Streams Kinesis Adapter) to land change records as JSON on a Kinesis stream, then ingest with `JsonKinesisSource` via DeltaStreamer. If a DynamoDB-specific source is still desired (e.g., to handle native DDB Stream record envelopes), that could be a separate follow-up — but the core capability requested here exists. Closing seems reasonable; please reopen if there's a specific gap with the current `KinesisSource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org