You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/15 18:58:43 UTC
[GitHub] [hive] belugabehr opened a new pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
belugabehr opened a new pull request #1118:
URL: https://github.com/apache/hive/pull/1118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #1118:
URL: https://github.com/apache/hive/pull/1118#discussion_r448558398
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
+ <field name="cols" persistence-modifier="persistent" table="COLUMNS_V2">
<collection element-type="MFieldSchema"/>
- <foreign-key name="COLUMNS_V2_FK" delete-action="cascade"/>
Review comment:
```
If a foreign-key is specified (in MetaData) for the relation field then leave any deletion to the datastore to perform
```
I don't see any such relationship cascading relationship defined in the schema for Derby or MySQL, so DN should be doing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr closed pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr closed pull request #1118:
URL: https://github.com/apache/hive/pull/1118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr edited a comment on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr edited a comment on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-656722071
> Foreign Keys
>
> So we now have given the datastore control over the cascade deletion strategy for objects stored in these tables. Please be aware that JDO provides Dependent Fields as a way of allowing cascade deletion. The difference here is that Dependent Fields is controlled by DataNucleus, whereas foreign key delete actions are controlled by the datastore (assuming the datastore supports it even)
```
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#fk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-656722071
```
Foreign Keys
So we now have given the datastore control over the cascade deletion strategy for objects stored in these tables. Please be aware that JDO provides Dependent Fields as a way of allowing cascade deletion. The difference here is that Dependent Fields is controlled by DataNucleus, whereas foreign key delete actions are controlled by the datastore (assuming the datastore supports it even)
```
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#fk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-652546609
@ashutoshc Let me see if I can address all of your questions with some background and context. It took me a long time to get these changes to pass the unit tests.
So, these mappings, in some respect, don't really matter. When HMS is started, users use the `schema-tool` to create the HMS schema for real. Some of these mappings in the `jdo` file (like indexes) are only applied when unit testing because the unit tests build the schema via DN and `datanucleus.schema.autoCreateAll`. For unit testing, the database backend is Apache Derby. I changed the name of the index to match the Derby schema more closely. In trying to debug these various errors, I was very confused at first about it complaining about "COLUMNS_PK".
https://github.com/apache/hive/blob/4942a7c0b4be3a5b0c889a89b903e9a70c57d494/standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql#L364
With that said, when I upgraded to DN 5.x, the unit tests would not pass. I narrowed the issue down to this one table definition. I tried several iterations to get success, but this is the one that worked. I derived this solution by closely examining the docs on this topic. It has an example that very closely aligns with this use case:
http://www.datanucleus.org/products/accessplatform/jpa/mapping.html#embedded_collection
It is a bit of a wonder looking at the existing JDO definition how this ever worked.
```
<primary-key name="COLUMNS_PK">
<column name="COLUMN_NAME"/>
</primary-key>
```
This is not correct, this should be a compound primary key of CD_ID *and* COLUMN_NAME. This is enforced by `SQL110922153006740` in the full schema. As things currently stand, the COLUMN_NAME definition in the `jdo` file says that the COLUMN_NAME is not defined to be non-null. This caused an error with Derby as it didn't allow creating a PRIMARY KEY on a field that could be null.
So, putting it all together, I came to the current solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-658213248
Complete and merged through manual patch process
https://issues.apache.org/jira/browse/HIVE-23363
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #1118:
URL: https://github.com/apache/hive/pull/1118#discussion_r448563837
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
Review comment:
Just following the directions here:
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#embedded_collection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr closed pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr closed pull request #1118:
URL: https://github.com/apache/hive/pull/1118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr closed pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr closed pull request #1118:
URL: https://github.com/apache/hive/pull/1118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #1118:
URL: https://github.com/apache/hive/pull/1118#discussion_r448560561
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
Review comment:
Changed to:
```
<field name="cols" persistence-modifier="persistent" table="COLUMNS_V2">
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashutoshc commented on a change in pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
ashutoshc commented on a change in pull request #1118:
URL: https://github.com/apache/hive/pull/1118#discussion_r448480980
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
Review comment:
Can you describe the need for this change?
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
+ <field name="cols" persistence-modifier="persistent" table="COLUMNS_V2">
<collection element-type="MFieldSchema"/>
- <foreign-key name="COLUMNS_V2_FK" delete-action="cascade"/>
<join>
- <primary-key name="COLUMNS_PK">
Review comment:
Any reason to change the name here?
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
+ <field name="cols" persistence-modifier="persistent" table="COLUMNS_V2">
<collection element-type="MFieldSchema"/>
- <foreign-key name="COLUMNS_V2_FK" delete-action="cascade"/>
Review comment:
We do want cascade-delete here. Any reason to remove it?
##########
File path: standalone-metastore/metastore-server/src/main/resources/package.jdo
##########
@@ -345,20 +345,20 @@
<datastore-identity>
<column name="CD_ID"/>
</datastore-identity>
- <field name="cols" table="COLUMNS_V2" >
+ <field name="cols" persistence-modifier="persistent" table="COLUMNS_V2">
<collection element-type="MFieldSchema"/>
- <foreign-key name="COLUMNS_V2_FK" delete-action="cascade"/>
<join>
- <primary-key name="COLUMNS_PK">
+ <primary-key name="SQL110922153006740">
+ <column name="CD_ID"/>
Review comment:
This probably is fine to do. Though, was it necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr commented on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-656718317
```
ALTER TABLE "APP"."COLUMNS_V2" ADD CONSTRAINT "COLUMNS_V2_FK1" FOREIGN KEY ("CD_ID") REFERENCES "APP"."CDS" ("CD_ID") ON DELETE NO ACTION ON UPDATE NO ACTION;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr edited a comment on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr edited a comment on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-652546609
@ashutoshc Let me see if I can address all of your questions with some background and context. It took me a long time to get these changes to pass the unit tests.
So, these mappings, in some respect, don't really matter. When HMS is started, users use the `schema-tool` to create the HMS schema for real. Some of these mappings in the `jdo` file (like indexes) are only applied when unit testing because the unit tests build the schema via DN and `datanucleus.schema.autoCreateAll`. For unit testing, the database backend is Apache Derby. I changed the name of the index to match the Derby schema more closely. In trying to debug these various errors, I was very confused at first about it complaining about "COLUMNS_PK".
https://github.com/apache/hive/blob/4942a7c0b4be3a5b0c889a89b903e9a70c57d494/standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql#L364
With that said, when I upgraded to DN 5.x, the unit tests would not pass. I narrowed the issue down to this one table definition. I tried several iterations to get success, but this is the one that worked. I derived this solution by closely examining the docs on this topic. It has an example that very closely aligns with this use case:
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#embedded_collection
It is a bit of a wonder looking at the existing JDO definition how this ever worked.
```
<primary-key name="COLUMNS_PK">
<column name="COLUMN_NAME"/>
</primary-key>
```
This is not correct, this should be a compound primary key of CD_ID *and* COLUMN_NAME. This exact scenario is covered in the second half of:
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#embedded_collection
In the official schema (hive-schema-4.0.0.derby.sql), the primary key is enforced by the `SQL110922153006740` index. As things currently stand, the COLUMN_NAME definition in the `jdo` file says that the COLUMN_NAME is not defined to be non-null. This caused an error with Derby as it didn't allow creating a PRIMARY KEY on a field that could be null.
So, putting it all together, I came to the current solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr edited a comment on pull request #1118: HIVE-23363: Upgrade DataNucleus dependency to 5.2
Posted by GitBox <gi...@apache.org>.
belugabehr edited a comment on pull request #1118:
URL: https://github.com/apache/hive/pull/1118#issuecomment-652546609
@ashutoshc Let me see if I can address all of your questions with some background and context. It took me a long time to get these changes to pass the unit tests.
So, these mappings, in some respect, don't really matter. When HMS is started, users use the `schema-tool` to create the HMS schema for real. Some of these mappings in the `jdo` file (like indexes) are only applied when unit testing because the unit tests build the schema via DN and `datanucleus.schema.autoCreateAll`. For unit testing, the database backend is Apache Derby. I changed the name of the index to match the Derby schema more closely. In trying to debug these various errors, I was very confused at first about it complaining about "COLUMNS_PK".
https://github.com/apache/hive/blob/4942a7c0b4be3a5b0c889a89b903e9a70c57d494/standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql#L364
With that said, when I upgraded to DN 5.x, the unit tests would not pass. I narrowed the issue down to this one table definition. I tried several iterations to get success, but this is the one that worked. I derived this solution by closely examining the docs on this topic. It has an example that very closely aligns with this use case:
http://www.datanucleus.org/products/accessplatform/jdo/mapping.html#embedded_collection
It is a bit of a wonder looking at the existing JDO definition how this ever worked.
```
<primary-key name="COLUMNS_PK">
<column name="COLUMN_NAME"/>
</primary-key>
```
This is not correct, this should be a compound primary key of CD_ID *and* COLUMN_NAME. This is enforced by `SQL110922153006740` in the full schema. As things currently stand, the COLUMN_NAME definition in the `jdo` file says that the COLUMN_NAME is not defined to be non-null. This caused an error with Derby as it didn't allow creating a PRIMARY KEY on a field that could be null.
So, putting it all together, I came to the current solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org