You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2025/06/08 21:48:34 UTC

[PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

jojochuang opened a new pull request, #46:
URL: https://github.com/apache/ozone-docker/pull/46

   The change was copied from https://github.com/apache/ozone/pull/8556/files and verified to work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #46:
URL: https://github.com/apache/ozone-docker/pull/46#discussion_r2135244609


##########
docker-compose.yaml:
##########
@@ -73,3 +73,14 @@ services:
       environment:
          <<: *common-config
       command: ["ozone","s3g"]
+   httpfs:
+     <<: *image
+     ports:
+       - 14000:14000
+     environment:
+       CORE-SITE.XML_fs.defaultFS: "ofs://om"
+       CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*"
+       CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*"

Review Comment:
   These need to be added in `common-config`.  Otherwise I got the following error when trying to put a file:
   
   ```
   httpfs-1  | 07:44:03,141 ERROR OzoneClientFactory:267 - Couldn't create RpcClient protocol exception: 
   httpfs-1  | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop
   httpfs-1  | 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1584)
   ...
   httpfs-1  | 	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1880)
   httpfs-1  | 	at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:260)
   ...
   httpfs-1  | 	at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:50)
   httpfs-1  | 	at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:108)
   httpfs-1  | 	at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:194)
   ...
   httpfs-1  | 	at org.apache.ozone.lib.service.hadoop.FileSystemAccessService.createFileSystem(FileSystemAccessService.java:339)
   ...
   httpfs-1  | 	at org.apache.ozone.fs.http.server.HttpFSServer.handleCreate(HttpFSServer.java:1229)
   httpfs-1  | 	at org.apache.ozone.fs.http.server.HttpFSServer.put(HttpFSServer.java:901)
   ```



##########
docker-compose.yaml:
##########
@@ -73,3 +73,14 @@ services:
       environment:
          <<: *common-config
       command: ["ozone","s3g"]
+   httpfs:
+     <<: *image
+     ports:
+       - 14000:14000

Review Comment:
   nit: the rest of this file has indentation of 3 spaces, please use the same for the new service.



##########
docker-compose.yaml:
##########
@@ -73,3 +73,14 @@ services:
       environment:
          <<: *common-config
       command: ["ozone","s3g"]
+   httpfs:
+     <<: *image
+     ports:
+       - 14000:14000
+     environment:
+       CORE-SITE.XML_fs.defaultFS: "ofs://om"
+       CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*"
+       CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*"
+       OZONE-SITE.XML_hdds.scm.safemode.min.datanode: ${OZONE_SAFEMODE_MIN_DATANODES:-1}

Review Comment:
   `hdds.scm.safemode.min.datanode` is not needed for `httpfs`, can be removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #46:
URL: https://github.com/apache/ozone-docker/pull/46#issuecomment-2960170946

   > Also I'm wondering instead of putting it in common-config, how about moving the proxy user configs to OM and have fs.defaultFS set at httpfs? I tested and that also works for me.
   
   Yes, that's probably the most specific working config.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #46:
URL: https://github.com/apache/ozone-docker/pull/46#issuecomment-2960157037

   Thanks @adoroszlai i had initially added them to common-config and then figured i should made it role specific.
   Updated according to your suggestion.
   
   Also I'm wondering instead of putting it in common-config, how about moving the proxy user configs to OM and have fs.defaultFS set at httpfs? I tested and that also works for me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #46:
URL: https://github.com/apache/ozone-docker/pull/46#issuecomment-2961232957

   Merged. Thanks @adoroszlai for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13210. Add httpfs container to docker compose file. [ozone-docker]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #46:
URL: https://github.com/apache/ozone-docker/pull/46


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org