You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/01/11 22:25:41 UTC
[GitHub] [druid] jihoonson opened a new pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
jihoonson opened a new pull request #12144:
URL: https://github.com/apache/druid/pull/12144
### Description
The `gcs-connector` library is currently in `lib` for integration tests which fails the test with `NoClassDefFoundError` while loading `GoogleHadoopFileSystem` when you load the hdfs extension.
<hr>
##### Key changed/added classes in this PR
* `copy_resources_template.sh`
<hr>
This PR has:
- [x] been self-reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kphan102 commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
kphan102 commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1019550368
@jihoonson I am having an issue `NoClassDefFoundError while loading GoogleHadoopFileSystem (class ..GoogleHadoopFileSystem not found)` when running druid with `gcs-connector`, `druid-hdfs-extensions` and `druid-parquet-extensions` altogether. I saw this PR you moved the `gcs-connector` to `hadoop-extensions` folder for the integration test to avoid that error.
Would you happen to know how to resolve the issue for the druid itself? (For e.g include extra jars to any of the extensions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kphan102 edited a comment on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
kphan102 edited a comment on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1019550368
@jihoonson I am having an issue `NoClassDefFoundError while loading GoogleHadoopFileSystem (class ..GoogleHadoopFileSystem not found)` when running druid with `gcs-connector`, `druid-hdfs-extensions` and `druid-parquet-extensions` altogether. I saw this PR you moved the `gcs-connector` to `hadoop-extensions` folder for the integration test to avoid that error.
Would you happen to know how to resolve the issue for the druid itself? (For e.g include extra jars to any of the extensions). I'm not familiar with java so it's hard for me to debug the jar dependencies. Please kindly help me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kphan102 edited a comment on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
kphan102 edited a comment on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1019550368
@jihoonson I am having an issue `NoClassDefFoundError while loading GoogleHadoopFileSystem (class ..GoogleHadoopFileSystem not found)` when running druid with `gcs-connector`, `druid-hdfs-extensions` and `druid-parquet-extensions` altogether. The druid services with version 0.22.1 will keep crashing with the above error.
I saw this PR you moved the `gcs-connector` to `hadoop-extensions` folder for the integration test to avoid that error.
Would you happen to know how to resolve the issue for the druid itself? (For e.g include extra jars to any of the extensions). I'm not familiar with java so it's hard for me to debug the jar dependencies. Please kindly help me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
jihoonson merged pull request #12144:
URL: https://github.com/apache/druid/pull/12144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kphan102 commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
kphan102 commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1020429693
@jihoonson Thanks for your reply. I tried something similar earlier today by putting the gcs-connector.jar inside a version-number folder in the `hadoop-gcs` directory and then use it in my batch ingestion spec with `hadoopCoordinates` property. Actually, I was inspired by your fix with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kphan102 commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
kphan102 commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1020429693
@jihoonson Thanks for your reply. I tried something similar earlier today by putting the gcs-connector.jar inside a version-number folder in the `hadoop-gcs` directory and then use it in my batch ingestion spec with `hadoopCoordinates` property. Actually, I was inspired by your fix with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1020399266
@kphan102 in my case, `NoClassDefFoundError` was thrown for `GoogleHadoopFileSystem` because `GooglehadoopFileSystem` was in Druid's main class loader via `gcs-connector` being in the `$DRUID/lib` directory while `FileSystem` was not in the main class loader but in an extension class loader via `druid-hdfs-extensions`. If you are seeing the same issue, you can make a `hadoop-gcs` directory under `$DRUID/hadoop-dependencies` and put `gcs-connector` there to resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1011595965
@clintropolis thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on pull request #12144: Move gcs-connector from lib to hadoop-dependencies for integration test
Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #12144:
URL: https://github.com/apache/druid/pull/12144#issuecomment-1020399266
@kphan102 in my case, `NoClassDefFoundError` was thrown for `GoogleHadoopFileSystem` because `GooglehadoopFileSystem` was in Druid's main class loader via `gcs-connector` being in the `$DRUID/lib` directory while `FileSystem` was not in the main class loader but in an extension class loader via `druid-hdfs-extensions`. If you are seeing the same issue, you can make a `hadoop-gcs` directory under `$DRUID/hadoop-dependencies` and put `gcs-connector` there to resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org