You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Aileen Toleikis <ai...@hpe.com> on 2021/04/20 12:53:18 UTC
Review Request 73286: Initial commit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
Review request for atlas.
Repository: atlas
Description
-------
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs
-----
addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
Diff: https://reviews.apache.org/r/73286/diff/1/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/pom.xml
> > Lines 143 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248948#file2248948line143>
> >
> > Please introduce a property in the root pom and use the reference here.
Handled in diff2: introduced the property in the root pom
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/pom.xml
> > Lines 148 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248948#file2248948line148>
> >
> > Could you please use the ${httpcomponents-httpclient.version} property here from the root pom? Also as I see the DefaultHttpClient has been deprecated since, so it might require some rework in the code too.
Handled in diff2: modified to http version parameter, replaced deprecated DefaultHttpClient with ClosableHttpClient
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/pom.xml
> > Lines 153 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248948#file2248948line153>
> >
> > Please use the ${hive.version} property here.
Handled in diff2: modified to hive version parameter
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
> > Lines 67 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248949#file2248949line67>
> >
> > Please group public and private fields together.
Handled in diff2: regrouped the fields
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
> > Lines 503 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248949#file2248949line504>
> >
> > Can we reuse this HttpClient instead of creating a new one each time?
Handled in diff2: we are reusing the httpclient now
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
> > Lines 38 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248950#file2248950line38>
> >
> > What happens if there are no schema registry running? Does it throw a connection exception or does handle this?
Handled in diff2: added another else for the case of no schema registry found, evth else continues because schema registry is not an requirement
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
> > Lines 51 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248950#file2248950line51>
> >
> > `httpClient` is passed in via an argument which would assume that this is a resource handled by the caller. I think it would be nicer to close this instead in the caller in a `finally` block.
Handled in diff2: closed in finally block
> On April 27, 2021, 3:19 nachm., Viktor Somogyi-Vass wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
> > Lines 62 (patched)
> > <https://reviews.apache.org/r/73286/diff/1/?file=2248950#file2248950line62>
> >
> > Which methods throw IOException and UnsupportedOperationException? Do we have to handle them here?
Handled in diff2: added log-information for the response exception
- Aileen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review222879
-----------------------------------------------------------
On Sept. 3, 2021, 7:42 vorm., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2021, 7:42 vorm.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227
>
>
> Diff: https://reviews.apache.org/r/73286/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Viktor Somogyi-Vass via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review222879
-----------------------------------------------------------
addons/kafka-bridge/pom.xml
Lines 143 (patched)
<https://reviews.apache.org/r/73286/#comment312051>
Please introduce a property in the root pom and use the reference here.
addons/kafka-bridge/pom.xml
Lines 148 (patched)
<https://reviews.apache.org/r/73286/#comment312050>
Could you please use the ${httpcomponents-httpclient.version} property here from the root pom? Also as I see the DefaultHttpClient has been deprecated since, so it might require some rework in the code too.
addons/kafka-bridge/pom.xml
Lines 153 (patched)
<https://reviews.apache.org/r/73286/#comment312052>
Please use the ${hive.version} property here.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 67 (patched)
<https://reviews.apache.org/r/73286/#comment312053>
Please group public and private fields together.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 503 (patched)
<https://reviews.apache.org/r/73286/#comment312056>
Can we reuse this HttpClient instead of creating a new one each time?
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 38 (patched)
<https://reviews.apache.org/r/73286/#comment312059>
What happens if there are no schema registry running? Does it throw a connection exception or does handle this?
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 51 (patched)
<https://reviews.apache.org/r/73286/#comment312057>
`httpClient` is passed in via an argument which would assume that this is a resource handled by the caller. I think it would be nicer to close this instead in the caller in a `finally` block.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 62 (patched)
<https://reviews.apache.org/r/73286/#comment312058>
Which methods throw IOException and UnsupportedOperationException? Do we have to handle them here?
- Viktor Somogyi-Vass
On April 27, 2021, 10:28 a.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated April 27, 2021, 10:28 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
>
>
> Diff: https://reviews.apache.org/r/73286/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
> On April 27, 2021, 3:21 nachm., Viktor Somogyi-Vass wrote:
> > Reviewed your code and made some smaller comments. I think overall this is a very useful feature to add.
Hey Victor,
Thank you so much for reviewing our code!
Thanks for these valid ideas! We will fix them asap and upload a new diff!
- Aileen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review222880
-----------------------------------------------------------
On April 27, 2021, 10:28 vorm., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated April 27, 2021, 10:28 vorm.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
>
>
> Diff: https://reviews.apache.org/r/73286/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
> On April 27, 2021, 3:21 nachm., Viktor Somogyi-Vass wrote:
> > Reviewed your code and made some smaller comments. I think overall this is a very useful feature to add.
>
> Aileen Toleikis wrote:
> Hey Victor,
> Thank you so much for reviewing our code!
> Thanks for these valid ideas! We will fix them asap and upload a new diff!
Hi Viktor,
we finally did it and added your requested changes to the code.
Could you review the code again?
What would be the next step?
Kind regards,
Aileen
- Aileen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review222880
-----------------------------------------------------------
On Sept. 3, 2021, 7:42 vorm., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2021, 7:42 vorm.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227
>
>
> Diff: https://reviews.apache.org/r/73286/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Viktor Somogyi-Vass via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review222880
-----------------------------------------------------------
Reviewed your code and made some smaller comments. I think overall this is a very useful feature to add.
- Viktor Somogyi-Vass
On April 27, 2021, 10:28 a.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated April 27, 2021, 10:28 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
>
>
> Diff: https://reviews.apache.org/r/73286/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
> On Okt. 4, 2021, 3:42 nachm., Ashutosh Mestry wrote:
> > Can you please rebase this against latest master?
Hi Ashutosh!
We rebased our code to the latest master!
And we also added schema evolution to the code, therefore there are more new codelines in the diff.
How do we proceed from here?
Kind regards,
Aileen
- Aileen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
-----------------------------------------------------------
On Okt. 5, 2021, 8:06 vorm., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Okt. 5, 2021, 8:06 vorm.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
>
>
> Diff: https://reviews.apache.org/r/73286/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
> On Oct. 4, 2021, 3:42 p.m., Ashutosh Mestry wrote:
> > Can you please rebase this against latest master?
>
> Aileen Toleikis wrote:
> Hi Ashutosh!
> We rebased our code to the latest master!
> And we also added schema evolution to the code, therefore there are more new codelines in the diff.
> How do we proceed from here?
> Kind regards,
> Aileen
Thanks! I have asked Viktor from our Kafka team to review the changes.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
-----------------------------------------------------------
On Oct. 5, 2021, 8:06 a.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2021, 8:06 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
>
>
> Diff: https://reviews.apache.org/r/73286/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223560
-----------------------------------------------------------
Can you please rebase this against latest master?
- Ashutosh Mestry
On Sept. 3, 2021, 7:42 a.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Sept. 3, 2021, 7:42 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227
>
>
> Diff: https://reviews.apache.org/r/73286/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated Nov. 15, 2021, 4:59 nachm.)
Review request for atlas.
Changes
-------
From: aileen.toleikis@hpe.com
Date: 15.Nov.2021
Subject: Make Kafka Interface aware of Kafka Schema Registry
Recent successful CI build - https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/956/
Repository: atlas
Description
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs (updated)
-----
addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
Diff: https://reviews.apache.org/r/73286/diff/6/
Changes: https://reviews.apache.org/r/73286/diff/5-6/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Sidharth Mishra <si...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223720
-----------------------------------------------------------
Ship it!
Ship It!
- Sidharth Mishra
On Nov. 9, 2021, 1:02 p.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2021, 1:02 p.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
>
>
> Diff: https://reviews.apache.org/r/73286/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated Nov. 9, 2021, 1:02 nachm.)
Review request for atlas.
Changes
-------
SchemaRegistryConnector line 50: inserted try/catch for test
Repository: atlas
Description
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs (updated)
-----
addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
Diff: https://reviews.apache.org/r/73286/diff/5/
Changes: https://reviews.apache.org/r/73286/diff/4-5/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated Okt. 14, 2021, 1:52 nachm.)
Review request for atlas.
Changes
-------
Implemented the change requests from Sarath Subramanian
Repository: atlas
Description
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs (updated)
-----
addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
Diff: https://reviews.apache.org/r/73286/diff/4/
Changes: https://reviews.apache.org/r/73286/diff/3-4/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
> On Okt. 7, 2021, 6:19 nachm., Sarath Subramanian wrote:
> >
Thank you so much! Implemented your changes in the new patch
- Aileen
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223588
-----------------------------------------------------------
On Okt. 14, 2021, 1:52 nachm., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Okt. 14, 2021, 1:52 nachm.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
>
>
> Diff: https://reviews.apache.org/r/73286/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/#review223588
-----------------------------------------------------------
addons/kafka-bridge/pom.xml
Lines 161 (patched)
<https://reviews.apache.org/r/73286/#comment312671>
why hive-exec dependency is needed here?
I see you use JSON libs from hive-exec. Consider removing hive dependency and use json-simple lib.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 63 (patched)
<https://reviews.apache.org/r/73286/#comment312672>
KAFKA_SCHEMA_REGISTRY => KAFKA_SCHEMA_REGISTRY_HOSTNAME
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 102 (patched)
<https://reviews.apache.org/r/73286/#comment312683>
"Entering custom Kafka bridge" => "Kafka bridge"
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 136 (patched)
<https://reviews.apache.org/r/73286/#comment312670>
consider creating a static constant for "KAFKA_SCHEMA_REGISTRY". Something like:
private static final String KAFKA_SCHEMA_REGISTRY_ENV_VARIABLE = "KAFKA_SCHEMA_REGISTRY";
Also, add null check for System.getenv()
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 185 (patched)
<https://reviews.apache.org/r/73286/#comment312684>
change to LOG.error("..., e"). Consider not printing stacktrace here.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 200 (patched)
<https://reviews.apache.org/r/73286/#comment312685>
nit: follow vertical alignment of variable declaration.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 371 (patched)
<https://reviews.apache.org/r/73286/#comment312687>
replace har-coded "atlas.metadata.namespace" with KAFKA_METADATA_NAMESPACE
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 382 (patched)
<https://reviews.apache.org/r/73286/#comment312688>
consider moving the sorting logic: line 382-392 inside createNestedFields() method.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 394 (patched)
<https://reviews.apache.org/r/73286/#comment312689>
why set nested fields in attributes? Shouldn't this be only on the relationshipAttributes?
"attributes" of AtlasEntity should contain only primitive attributes and all relationships (createdFields) should be added only to relationshipAttributes
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 407 (patched)
<https://reviews.apache.org/r/73286/#comment312690>
why hard-code to string? compare with ENUM:
if (field.schema().getType() == Schema.Type.ARRAY)
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 417 (patched)
<https://reviews.apache.org/r/73286/#comment312691>
why hard-code to string? compare with ENUM:
if (field.schema().getType() == Schema.Type.RECORD)
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 418 (patched)
<https://reviews.apache.org/r/73286/#comment312692>
move this 'if' condition inside else-if above line 417
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 476 (patched)
<https://reviews.apache.org/r/73286/#comment312686>
'attributes' is not used other than 476.
atlasClientV2.getEntityByAttribute(typeName, Collections.singletonMap(ATTRIBUTE_QUALIFIED_NAME, qualifiedName));
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 45 (patched)
<https://reviews.apache.org/r/73286/#comment312676>
replace hardcoded 200 to HttpStatus.SC_OK
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 56 (patched)
<https://reviews.apache.org/r/73286/#comment312675>
considering printing list content with separator:
String.join(", ", list);
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 57 (patched)
<https://reviews.apache.org/r/73286/#comment312673>
consider defining own logger for 'SchemaRegistryConnector' instead of using KafkaBridge's logger.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 62 (patched)
<https://reviews.apache.org/r/73286/#comment312681>
consider catching all Exception as:
catch (Exception e) {
.....
}
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 65 (patched)
<https://reviews.apache.org/r/73286/#comment312674>
LOG.error should be added here.
Consider print error stack within logger message and avoid line 63.
LOG.error("......", e)
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 69 (patched)
<https://reviews.apache.org/r/73286/#comment312677>
replace 404 with HttpStatus.SC_NOT_FOUND
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 72 (patched)
<https://reviews.apache.org/r/73286/#comment312678>
"Can not find versions to schema: {} in Kafka" => "No schema versions found for schema: {} in Schema Registry"
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 102 (patched)
<https://reviews.apache.org/r/73286/#comment312679>
create static constant for "schema"
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 104 (patched)
<https://reviews.apache.org/r/73286/#comment312680>
avoid printing stacktrace here and print in log:
LOG.error ("....", e)
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java
Lines 113 (patched)
<https://reviews.apache.org/r/73286/#comment312682>
"Can not" -> "Cannot"
- Sarath Subramanian
On Oct. 5, 2021, 1:06 a.m., Aileen Toleikis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73286/
> -----------------------------------------------------------
>
> (Updated Oct. 5, 2021, 1:06 a.m.)
>
>
> Review request for atlas.
>
>
> Repository: atlas
>
>
> Description
> -------
>
> The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
>
> We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
>
> Changes:
>
> initial commit
>
>
> added schema creation in Atlas
>
>
> added Schema creation, Schema linkage and started to work on versions
>
>
> avro_field creation is working. However, there is no connection to schemas yet
>
>
> smaller refactorings and improved logging
>
>
> added field reference in schemas
>
>
> deleted second README and added gitignore
>
>
> Smaller changes after rebasing on new version
>
>
> moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
>
>
> added testCreateSchema() as test case
>
>
> added testCreateField() and testUpdateField() as test case
>
>
> added testGetSchemas() and testGetSchemaVersions() as test case
>
>
> added testUpdateSchema() as test case
>
>
> removed unnecessary comments
>
>
> Diffs
> -----
>
> addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
> addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
> pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
>
>
> Diff: https://reviews.apache.org/r/73286/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Aileen Toleikis
>
>
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated Okt. 5, 2021, 8:06 vorm.)
Review request for atlas.
Changes
-------
As suggested from Ashutosh Mestry rebased to the latest master!
Added support for schema evolution as well!
Repository: atlas
Description
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs (updated)
-----
addons/kafka-bridge/pom.xml 2ac19fd20e0c0322b0d104641ddd5a2ef89bf9d0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java f9548244364a8f79f346411739348d0d53298c99
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
pom.xml 47768e9fa7d8abcfc81cd148917400ac478a333a
Diff: https://reviews.apache.org/r/73286/diff/3/
Changes: https://reviews.apache.org/r/73286/diff/2-3/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated Sept. 3, 2021, 7:42 vorm.)
Review request for atlas.
Changes
-------
Changes: Updated the code with the suggested changes by Viktor, added support for nested schemas, add try-catch around import in order to make import more resilient
Repository: atlas
Description
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs (updated)
-----
addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
pom.xml 092ba09cee93ca703e29d8ae1afaa20559a06227
Diff: https://reviews.apache.org/r/73286/diff/2/
Changes: https://reviews.apache.org/r/73286/diff/1-2/
Testing
-------
Thanks,
Aileen Toleikis
Re: Review Request 73286: Initial commit
Posted by Aileen Toleikis <ai...@hpe.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73286/
-----------------------------------------------------------
(Updated April 27, 2021, 10:28 vorm.)
Review request for atlas.
Repository: atlas
Description (updated)
-------
The Kafka Community is using Schema Registry more and more heavily but as Atlas is currently unaware of this, this extension helps Atlas make use of the Schemas.
We have tested this extension and we have production environments where Atlas will not be allowed without schema registry access. We have received feedback that this extension would be sufficient to allow production use.
Changes:
initial commit
added schema creation in Atlas
added Schema creation, Schema linkage and started to work on versions
avro_field creation is working. However, there is no connection to schemas yet
smaller refactorings and improved logging
added field reference in schemas
deleted second README and added gitignore
Smaller changes after rebasing on new version
moved HTTP REST calls to a separate class (SchemaRegistryConnector.java)
added testCreateSchema() as test case
added testCreateField() and testUpdateField() as test case
added testGetSchemas() and testGetSchemaVersions() as test case
added testUpdateSchema() as test case
removed unnecessary comments
Diffs
-----
addons/kafka-bridge/pom.xml 7fe97eb39897e32f5e2a4104cc06948e1ec48b27
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java bf74c67454120e3bdeebab0d7cc428d6bba8c021
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/SchemaRegistryConnector.java PRE-CREATION
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/model/KafkaDataTypes.java 0f81b4c37a1d8f84c587a914eb460b290ae36987
addons/kafka-bridge/src/test/java/org/apache/atlas/kafka/bridge/KafkaBridgeTest.java f86ceb58fd250d80f3ae3c71085c8c0adbc9116b
Diff: https://reviews.apache.org/r/73286/diff/1/
Testing
-------
Thanks,
Aileen Toleikis