You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/09/28 08:08:13 UTC
[GitHub] [druid] Z08JBECH opened a new issue #11745: Iduplicate data by loading from S3 with Batch
Z08JBECH opened a new issue #11745:
URL: https://github.com/apache/druid/issues/11745
On our production cluster, we load parse data from s3 bucket with 8 subtasks.
We have a problem when we load data.
**Sometimes druid create us a segment with 2 partitions, from example :
f_stock_mouvement_api_2021-09-26T00:00:00.000Z_2021-09-27T00:00:00.000Z_2021-09-27T13:03:00.661Z
f_stock_mouvement_api_2021-09-26T00:00:00.000Z_2021-09-27T00:00:00.000Z_2021-09-27T13:03:00.661Z_1
The 2 partitions contains the same data, so we have doublons.**
It seems that the process run in parallel in different subtask.
we I check the log from all substask i found this in two substasks.
-------------------------------
mycompany",
"path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0027_part_00.gz"
}, {
"bucket" : "mycompany",
"path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0028_part_00.gz"
}, {
"bucket" : "mycompany",
"path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0029_part_00.gz"
}, {
"bucket" : "mycompany",
"path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0030_part_00.gz"
}, {
"bucket" : "mycompany",
"path" : "cds/f_stock_mouvement_api/year=2021/month=9/day=26/0031_part_00.gz"
}
....
---------------------------
So druid loads some data 2x in the same parallel process with 2 different subtasks.
**My version is 0.21**
Do you know about this problem ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] morrifeldman commented on issue #11745: duplicate data by loading from S3 with Batch
Posted by GitBox <gi...@apache.org>.
morrifeldman commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-930283492
It sounds similar to what we saw here with data duplication after task failures. Were there task failures?
https://github.com/apache/druid/issues/11719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] morrifeldman commented on issue #11745: duplicate data by loading from S3 with Batch
Posted by GitBox <gi...@apache.org>.
morrifeldman commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-930283492
It sounds similar to what we saw here with data duplication after task failures. Were there task failures?
https://github.com/apache/druid/issues/11719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Z08JBECH commented on issue #11745: duplicate data by loading from S3 with Batch
Posted by GitBox <gi...@apache.org>.
Z08JBECH commented on issue #11745:
URL: https://github.com/apache/druid/issues/11745#issuecomment-932732175
> It sounds similar to what we saw here with data duplication after task failures. Were there task failures? #11719
No, task are ok but some data are duplicated.
I found the problem but i think it is a Druid bug
To solve the problem I have to set the s3 prefix path in order
Like that :
cds/f_stock_mouvement_api/year=2021/month=9/day=25
cds/f_stock_mouvement_api/year=2021/month=9/day=26
cds/f_stock_mouvement_api/year=2021/month=9/day=27
cds/f_stock_mouvement_api/year=2021/month=9/day=28
etc
if the path are like that some data are loaded 2 times.
cds/f_stock_mouvement_api/year=2021/month=9/day=26
cds/f_stock_mouvement_api/year=2021/month=9/day=10
cds/f_stock_mouvement_api/year=2021/month=9/day=13
cds/f_stock_mouvement_api/year=2021/month=9/day=11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org