You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Dharshana M Krishnamoorthy (Jira)" <ji...@apache.org> on 2021/02/18 07:19:00 UTC
[jira] [Updated] (ATLAS-4164) [Atlas: Spooling] Tables created
after spooling are created before the spooled tables when there is multiple
frequent restart in kafka brokers
[ https://issues.apache.org/jira/browse/ATLAS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dharshana M Krishnamoorthy updated ATLAS-4164:
----------------------------------------------
Description:
Scenario:
* Stop kafka broker
* Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... abc_table_n)
* Make sure the data is spooled
* Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... xyz_table_n)
* Wait for 5 mins for the tables to reflect in atlas
In this case we expect all the abc_table_* to be created before xyz_table_1, meaning all the spooled tables are created before the tables that are created after spooling.
Observation:
createTime of some spooled tables is greater than the create time of the xyz_table_1
Sample data:
createTime for tables that are spooled:
{code:java}
[1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253, 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629, 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947, 1613573521737, 1613573514066, 1613573514555, 1613573515065, 1613573515605]{code}
createTime for tables that are created after spooling:
{code:java}
[1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356, 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659, 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004, 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202] {code}
We expect all spooled tables to have createTime smaller than the tables created after spooling.
But *1613573543253 (Spooled tabled create time) is greater than 1613573540582 (table created after spooling)*
which means, the table created after spooling is created before spooled table
was:
Scenario:
* Stop kafka broker
* Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... abc_table_n)
* Make sure the data is spooled
* Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... xyz_table_n)
* Wait for 5 mins for the tables to reflect in atlas
In this case we expect all the abc_table_* to be created before xyz_table_1, meaning all the spooled tables are created before the tables that are created after spooling.
Observation:
createTime of some spooled tables is greater than the create time of the xyz_table_1
Sample data:
createTime for tables that are spooled:
{code:java}
[1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253, 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629, 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947, 1613573521737, 1613573514066, 1613573514555, 1613573515065, 1613573515605]{code}
createTime for tables that are created after spooling:
{code:java}
[1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356, 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659, 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004, 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202] {code}
*1613573543253 < 1613573540582*
which means, the table created after spooling is created before spooled table
> [Atlas: Spooling] Tables created after spooling are created before the spooled tables when there is multiple frequent restart in kafka brokers
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-4164
> URL: https://issues.apache.org/jira/browse/ATLAS-4164
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Dharshana M Krishnamoorthy
> Assignee: Ashutosh Mestry
> Priority: Major
>
> Scenario:
> * Stop kafka broker
> * Create a few (20) tables save the prefix (abc_table_1, abc_table_2, ... abc_table_n)
> * Make sure the data is spooled
> * Start kafka and create a few more tables (xyz_table_1, xyz_table_2, ... xyz_table_n)
> * Wait for 5 mins for the tables to reflect in atlas
> In this case we expect all the abc_table_* to be created before xyz_table_1, meaning all the spooled tables are created before the tables that are created after spooling.
>
> Observation:
> createTime of some spooled tables is greater than the create time of the xyz_table_1
>
> Sample data:
> createTime for tables that are spooled:
> {code:java}
> [1613573518284, 1613573531470, 1613573531861, 1613573529446, 1613573543253, 1613573525390, 1613573525950, 1613573517796, 1613573518284, 1613573522629, 1613573513524, 1613573524856, 1613573518992, 1613573519477, 1613573519947, 1613573521737, 1613573514066, 1613573514555, 1613573515065, 1613573515605]{code}
> createTime for tables that are created after spooling:
> {code:java}
> [1613573540582, 1613573541300, 1613573551691, 1613573552628, 1613573553356, 1613573555478, 1613573556275, 1613573556940, 1613573557763, 1613573558659, 1613573560673, 1613573561363, 1613573562310, 1613573563096, 1613573564004, 1613573566533, 1613573567602, 1613573568439, 1613573569379, 1613573570202] {code}
> We expect all spooled tables to have createTime smaller than the tables created after spooling.
> But *1613573543253 (Spooled tabled create time) is greater than 1613573540582 (table created after spooling)*
> which means, the table created after spooling is created before spooled table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)