You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/09/28 08:08:13 UTC

[GitHub] [druid] Z08JBECH opened a new issue #11745: Iduplicate data by loading from S3 with Batch

Z08JBECH opened a new issue #11745:
URL: https://github.com/apache/druid/issues/11745


   On our production cluster, we load parse data from s3 bucket with 8 subtasks.
   We have a problem when we load data.
   
   **Sometimes druid create us a segment with 2 partitions, from example : 
   f_stock_mouvement_api_2021-09-26T00:00:00.000Z_2021-09-27T00:00:00.000Z_2021-09-27T13:03:00.661Z
   f_stock_mouvement_api_2021-09-26T00:00:00.000Z_2021-09-27T00:00:00.000Z_2021-09-27T13:03:00.661Z_1
   
   The 2 partitions contains the same data, so we have doublons.**
   
   It seems that the process run in parallel in different subtask. 
   
   we I check the log from all substask i found this in two substasks. 
   -------------------------------
   mycompany",
             "path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0027_part_00.gz"
           }, {
             "bucket" : "mycompany",
             "path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0028_part_00.gz"
           }, {
             "bucket" : "mycompany",
             "path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0029_part_00.gz"
           }, {
             "bucket" : "mycompany",
             "path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0030_part_00.gz"
           }, {
             "bucket" : "mycompany",
             "path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0031_part_00.gz"
           }
   ....
   ---------------------------
   So druid loads some data 2x in the same parallel process with 2 different subtasks. 
   
   
   **My version is 0.21**
   
   Do you know about this problem ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] morrifeldman commented on issue #11745: duplicate data by loading from S3 with Batch

Posted by GitBox <gi...@apache.org>.
morrifeldman commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-930283492


   It sounds similar to what we saw here with data duplication after task failures.  Were there task failures?
   https://github.com/apache/druid/issues/11719


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] morrifeldman commented on issue #11745: duplicate data by loading from S3 with Batch

Posted by GitBox <gi...@apache.org>.
morrifeldman commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-930283492


   It sounds similar to what we saw here with data duplication after task failures.  Were there task failures?
   https://github.com/apache/druid/issues/11719


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] Z08JBECH commented on issue #11745: duplicate data by loading from S3 with Batch

Posted by GitBox <gi...@apache.org>.
Z08JBECH commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-932732175


   > It sounds similar to what we saw here with data duplication after task failures. Were there task failures? #11719
   
   No, task are ok but some data are duplicated.
   I found the problem but i think it is a Druid bug
   
   To solve the problem I have to set the s3 prefix path in order
   Like that :
   cds/f_stock_mouvement_api/year=2021/month=9/day=25
   cds/f_stock_mouvement_api/year=2021/month=9/day=26
   cds/f_stock_mouvement_api/year=2021/month=9/day=27
   cds/f_stock_mouvement_api/year=2021/month=9/day=28
   
   etc
   
   if the path are like that some data are loaded 2 times. 
   cds/f_stock_mouvement_api/year=2021/month=9/day=26
   cds/f_stock_mouvement_api/year=2021/month=9/day=10
   cds/f_stock_mouvement_api/year=2021/month=9/day=13
   cds/f_stock_mouvement_api/year=2021/month=9/day=11
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org