You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2025/06/03 23:12:57 UTC
[PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
jojochuang opened a new pull request, #8556:
URL: https://github.com/apache/ozone/pull/8556
## What changes were proposed in this pull request?
HDDS-13165. [Docs] Python client developer guide.
Please describe your PR in detail:
* Added interface/Python.md: overall Python client access introduction.
* Recipe: Access Ozone using PyArrow (Docker Quickstart)
* Recipe: Access Ozone using Boto3 (Docker Quickstart)
* Recipe: Access Ozone using HTTPFS REST API (Docker + Python Requests)
For interface/Python.md, the draft was generated using ChatGPT 4o using the prompt:
```
Create a user document in Markdown format for Python developers who want to access Apache Ozone. This document will be part of the Ozone Client Interfaces page: https://ozone.apache.org/docs/edge/interface.html.
📌 *Audience*: Python developers familiar with Python integration and Ozone. Skip the introduction.
📌 *Structure*:
Setup and Prerequisites:
Required libraries (PyArrow, Boto3, WebHDFS)
Required configurations (e.g., HADOOP_CONF_DIR, Ozone URIs, credentials, authentication)
Access Method 1: PyArrow with libhdfs
Setup steps (including any system paths or environment variables)
Python code sample (validate for correctness)
Access Method 2: Boto3 with Ozone S3 Gateway
Setup steps (including Ozone S3 endpoint format, bucket naming conventions, credentials)
Python code sample (validate for correctness)
Access Method 3: WebHDFS/HttpFS or REST API
Setup steps (including endpoint URL, authentication)
Python code sample (using requests or webhdfs)
Access from PySpark
Configuration settings in Spark (fs.ozone. settings)
Python code sample for reading/writing data to Ozone
Troubleshooting Tips
Common issues (e.g., authentication failures, connection errors)
Suggested debugging techniques
References and Further Resources
Links to official Ozone documentation, PyArrow, Boto3, WebHDFS, PySpark
📌 *Markdown Format*:
Use proper headers (##, ###) for each section.
Include Python syntax highlighting in code blocks (```python).
Use clear formatting and spacing for readability.
Include warnings or notes where appropriate (e.g., > *Note:*).
If applicable, include a simple diagram showing connection flows.
📌 *Quality Checks*:
Validate all code samples for correctness.
Ensure the document is clear and concise.
Focus only on actionable instructions and setup information.
Generate the complete Markdown document in response. Include a Hugo header. Include Apache License header
```
The PyArrow recipe draft was generated using ChatGPT 4o prompt:
```
I personally verified the following steps using Ozone's Docker image. Please rewrite in a user tutorial format.
PyArrow to access Ozone
# Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
docker compose up -d --scale datanode=3
connect to the SCM container:
docker exec -it weichiu-scm-1 bash
ozone sh volume create volume
ozone sh bucket create volume/bucket
pip install pyarrow
curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*’
or
curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*’
export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
export CLASSPATH=$(ozone classpath ozone-tools)
Add to /etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>ofs://om:9862</value>
<description>Where HDFS NameNode can be found on the network</description>
</property>
</configuration>
Code:
#!/usr/bin/python
import pyarrow.fs as pafs
# Create Hadoop FileSystem object
fs = pafs.HadoopFileSystem("default", 9864)
fs.create_dir("volume/bucket/aaa")
path = "volume/bucket/file1"
with fs.open_output_stream(path) as stream:
stream.write(b'data')
```
The Boto3 recipe draft was generated using ChatGPT 4o prompt:
```
Following the similar PyArrow using Ozone Docker image tutorial, create a similar one for boto3 using the following instructions:
ozone sh bucket create s3v/bucket
Code
#!/usr/bin/python
import boto3
# Create a local file to upload
with open("localfile.txt", "w") as f:
f.write("Hello from Ozone via Boto3!\n")
# Configure Boto3 client
s3 = boto3.client(
's3',
endpoint_url='http://weichiu-s3g-1:9878',
aws_access_key_id='ozone-access-key',
aws_secret_access_key='ozone-secret-key'
)
# List buckets
response = s3.list_buckets()
print(response['Buckets'])
# Upload a file
s3.upload_file('localfile.txt', 'bucket', 'file.txt')
# Download a file
s3.download_file('bucket', 'file.txt', 'downloaded.txt’)
```
The httpfs receipe draft was generated using ChatGPT 4o, prompt:
```
Use the following instructions to create a tutorial of accessing Ozone using HttpFS REST API via requests library
Ozone httpfs using Python requests
# Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
add to docker-compose.yaml: CORE-SITE.XML_fs.defaultFS: "ofs://om"
CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*"
CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*"
docker compose up -d --scale datanode=3
connect to the SCM container:
ozone sh volume create vol1
ozone sh bucket create vol1/bucket1
pip install requests
#!/usr/bin/python
import requests
# Ozone HTTPFS endpoint and file path
host = "http://weichiu-httpfs-1:14000"
volume = “vol1"
bucket = "bucket1"
filename = "hello.txt"
path = f"/webhdfs/v1/{volume}/{bucket}/{filename}"
user = "ozone" # can be any value in simple auth mode
# Step 1: Initiate file creation (responds with 307 redirect)
params_create = {
"op": "CREATE",
"overwrite": "true",
"user.name": user
}
print("Creating file...")
resp_create = requests.put(host + path, params=params_create, allow_redirects=False)
if resp_create.status_code != 307:
print(f"Unexpected response: {resp_create.status_code}")
print(resp_create.text)
exit(1)
redirect_url = resp_create.headers['Location']
print(f"Redirected to: {redirect_url}")
# Step 2: Write data to the redirected location with correct headers
headers = {"Content-Type": "application/octet-stream"}
content = b"Hello from Ozone HTTPFS!\n"
resp_upload = requests.put(redirect_url, data=content, headers=headers)
if resp_upload.status_code != 201:
print(f"Upload failed: {resp_upload.status_code}")
print(resp_upload.text)
exit(1)
print("File created successfully.")
# Step 3: Read the file back
params_open = {
"op": "OPEN",
"user.name": user
}
print("Reading file...")
resp_read = requests.get(host + path, params=params_open, allow_redirects=True)
if resp_read.ok:
print("File contents:")
print(resp_read.text)
else:
print(f"Read failed: {resp_read.status_code}")
print(resp_read.text)
```
Note: initially the draft had PySpark content. Due to the length of the content, I decided to leave it out. Will work on it in a follow-up task.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13165?filter=-1
## How was this patch tested?
After Gemini/ChatGPT generated the user doc draft, I manually followed the code samples and verified the steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8556:
URL: https://github.com/apache/ozone/pull/8556#discussion_r2125290504
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
Review Comment:
The Ozone docker image does not have hadoop executables.
In the tutorial, it downloads just the libhdfs.so files nothing else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8556:
URL: https://github.com/apache/ozone/pull/8556#issuecomment-2938695401
Open up for review.
Please also check out the review comments previously made by Gemini: https://github.com/jojochuang/ozone/pull/566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8556:
URL: https://github.com/apache/ozone/pull/8556#discussion_r2125261292
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
+```
Review Comment:
Can tell user there are more variable they can set: https://lidavidm.github.io/arrow-docs-next/python/filesystems.html#hadoop-file-system-hdfs
> The libhdfs library is loaded at runtime (rather than at link / library load time, since the library may not be in your LD_LIBRARY_PATH), and relies on some environment variables.
>
> `HADOOP_HOME`: the root of your installed Hadoop distribution. Often has lib/native/libhdfs.so.
>
> `JAVA_HOME`: the location of your Java SDK installation.
>
> `ARROW_LIBHDFS_DIR` (optional): explicit location of libhdfs.so if it is installed somewhere other than $HADOOP_HOME/lib/native.
>
> `CLASSPATH`: must contain the Hadoop jars. You can set these using:
>
> ```bash
> export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`
> ```
> If `CLASSPATH` is not set, then it will be set automatically if the `hadoop` executable is in your system path, or if `HADOOP_HOME` is set.
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
Review Comment:
I think this should be:
```bash
export CLASSPATH=$(hadoop-3.4.0/lib/native/bin/hdfs classpath --glob)
```
Otherwise users need to download both hadoop and ozone build. I'm not really sure this is good? maybe ozone's client has better performance or compatibility？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8556:
URL: https://github.com/apache/ozone/pull/8556#discussion_r2125288306
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
+```
Review Comment:
The Ozone docker image does not have hadoop nor hdfs executables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8556:
URL: https://github.com/apache/ozone/pull/8556#discussion_r2125296060
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
Review Comment:
The classpath only requires Ozone Java libs. It does not need HDFS Java libs.
The ozone file system jar shades everything it depends on, including hadoop common classes, so we don't need to include hdfs classpath at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8556:
URL: https://github.com/apache/ozone/pull/8556#discussion_r2125296798
##########
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md:
##########
@@ -0,0 +1,141 @@
+---
+title: Access Ozone using PyArrow (Docker Quickstart)
+linkTitle: PyArrow Access (Docker)
+summary: Step-by-step tutorial for accessing Ozone from Python using PyArrow in a Docker environment.
+weight: 11
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This tutorial demonstrates how to access Apache Ozone from Python using **PyArrow**, with Ozone running in Docker.
+
+## Prerequisites
+
+- Docker and Docker Compose installed.
+- Python 3.x environment.
+
+## Steps
+
+### 1️⃣ Start Ozone in Docker
+
+Download the latest Docker Compose file for Ozone and start the cluster with 3 DataNodes:
+
+```bash
+curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
+docker compose up -d --scale datanode=3
+```
+
+### 2️⃣ Connect to the SCM Container
+
+```bash
+docker exec -it <your-scm-container-name-or-id> bash
+```
+> Change the container id `<your-scm-container-name-or-id>` to your actual container id.
+
+The rest of the tutorial will run on this container.
+
+Create a volume and a bucket inside Ozone:
+
+```bash
+ozone sh volume create volume
+ozone sh bucket create volume/bucket
+```
+
+### 3️⃣ Install PyArrow in Your Python Environment
+
+```bash
+pip install pyarrow
+```
+
+### 4️⃣ Download Hadoop Native Libraries for libhdfs Support
+
+Depending on your system architecture, run one of the following:
+
+For ARM64 (Apple Silicon, ARM servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+For x86_64 (most desktops and servers):
+```bash
+curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*'
+```
+
+Set environment variables to point to the native libraries and Ozone classpath:
+
+```bash
+export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
+export CLASSPATH=$(ozone classpath ozone-tools)
Review Comment:
Ah ok, I see, thanks for clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8556:
URL: https://github.com/apache/ozone/pull/8556#issuecomment-2949918206
add @adoroszlai for the parts regarding httpfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13165. [Docs] Python client developer guide. [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli merged PR #8556:
URL: https://github.com/apache/ozone/pull/8556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org