You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/07/07 19:07:34 UTC
[GitHub] [druid] gianm opened a new pull request #10149: Add availability and consistency docs.
gianm opened a new pull request #10149:
URL: https://github.com/apache/druid/pull/10149
Describes transactional ingestion and atomic replacement. Also, this patch
deletes some bad advice from the javadocs for SegmentTransactionalInsertAction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] surekhasaharan merged pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
surekhasaharan merged pull request #10149:
URL: https://github.com/apache/druid/pull/10149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on a change in pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
gianm commented on a change in pull request #10149:
URL: https://github.com/apache/druid/pull/10149#discussion_r451154818
##########
File path: docs/design/architecture.md
##########
@@ -213,6 +213,75 @@ this will generally start off `true` and then become `false` as the segment is p
published segments. Generally this is a transient state, and segments in this state will soon have their `used` flag
automatically set to false.
+### Availability and consistency
+
+Druid has an architectural separation between ingestion and querying, as described above in
+[Indexing and handoff](#indexing-and-handoff). This means that when understanding Druid's availability and
+consistency properties, we must look at each function separately.
+
+On the **ingestion side**, Druid's primary [ingestion methods](../ingestion/index.md#ingestion-methods) are all
+pull-based and offer transactional guarantees. This means that you are guaranteed that ingestion using these will
+publish in an all-or-nothing manner:
+
+- Supervised "seekable-stream" ingestion methods like [Kafka](../development/extensions-core/kafka-ingestion.md) and
+[Kinesis](../development/extensions-core/kinesis-ingestion.md). With these methods, Druid commits stream offsets to its
+[metadata store](#metadata-storage) alongside segment metadata, in the same transaction. Note that ingestion of data
+that has not yet been published can be rolled back if ingestion tasks fail. In this case, partially-ingested data
+discarded, and Druid will resume ingestion from the last committed set of stream offsets. This ensures exactly-once
Review comment:
Thanks, fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] surekhasaharan commented on a change in pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
surekhasaharan commented on a change in pull request #10149:
URL: https://github.com/apache/druid/pull/10149#discussion_r451118202
##########
File path: docs/design/architecture.md
##########
@@ -213,6 +213,75 @@ this will generally start off `true` and then become `false` as the segment is p
published segments. Generally this is a transient state, and segments in this state will soon have their `used` flag
automatically set to false.
+### Availability and consistency
+
+Druid has an architectural separation between ingestion and querying, as described above in
+[Indexing and handoff](#indexing-and-handoff). This means that when understanding Druid's availability and
+consistency properties, we must look at each function separately.
+
+On the **ingestion side**, Druid's primary [ingestion methods](../ingestion/index.md#ingestion-methods) are all
+pull-based and offer transactional guarantees. This means that you are guaranteed that ingestion using these will
+publish in an all-or-nothing manner:
+
+- Supervised "seekable-stream" ingestion methods like [Kafka](../development/extensions-core/kafka-ingestion.md) and
+[Kinesis](../development/extensions-core/kinesis-ingestion.md). With these methods, Druid commits stream offsets to its
+[metadata store](#metadata-storage) alongside segment metadata, in the same transaction. Note that ingestion of data
+that has not yet been published can be rolled back if ingestion tasks fail. In this case, partially-ingested data
+discarded, and Druid will resume ingestion from the last committed set of stream offsets. This ensures exactly-once
Review comment:
partially-ingested data discarded -> partially-ingested data "is" discarded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
maytasm commented on pull request #10149:
URL: https://github.com/apache/druid/pull/10149#issuecomment-655239969
@gianm @surekhasaharan Travis docs job are now failing due to this change. https://travis-ci.org/github/apache/druid/jobs/705996918
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #10149:
URL: https://github.com/apache/druid/pull/10149#issuecomment-655576987
@surekhasaharan, it looks like the Travis CI build was still in progress when you merged this patch. Please wait for the tests to finish running before merging things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #10149: Add availability and consistency docs.
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #10149:
URL: https://github.com/apache/druid/pull/10149#issuecomment-655246535
@maytasm thank you for the catch. I raised https://github.com/apache/druid/pull/10152.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org