You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@storm.apache.org by bi...@apache.org on 2023/06/30 15:19:24 UTC
[storm] branch master updated: [STORM-3929] Update documentation to refer to python3 (#3551)
This is an automated email from the ASF dual-hosted git repository.
bipinprasad pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git
The following commit(s) were added to refs/heads/master by this push:
new 1636bb318 [STORM-3929] Update documentation to refer to python3 (#3551)
1636bb318 is described below
commit 1636bb318ca3e2e6228c909d3f7541064575f065
Author: Bipin Prasad <bi...@apache.org>
AuthorDate: Fri Jun 30 08:19:18 2023 -0700
[STORM-3929] Update documentation to refer to python3 (#3551)
---
DEVELOPER.md | 4 ++--
dev-tools/jira-github-join.py | 2 +-
dev-tools/report/__init__.py | 2 +-
docs/Clojure-DSL.md | 2 +-
docs/Command-line-client.md | 2 +-
docs/Defining-a-non-jvm-language-dsl-for-storm.md | 13 ++++++++++---
docs/Multilang-protocol.md | 8 ++++----
docs/OCI-support.md | 4 ++--
docs/Setting-up-a-Storm-cluster.md | 2 +-
docs/Tutorial.md | 6 +++---
docs/Using-non-JVM-languages-with-Storm.md | 12 ++++++------
docs/flux.md | 6 +++---
docs/storm-sql-example.md | 12 +++++-------
storm-client/src/jvm/org/apache/storm/task/ShellBolt.java | 2 +-
14 files changed, 41 insertions(+), 36 deletions(-)
diff --git a/DEVELOPER.md b/DEVELOPER.md
index 825986702..698cec835 100644
--- a/DEVELOPER.md
+++ b/DEVELOPER.md
@@ -247,7 +247,7 @@ To pull in a merge request you should generally follow the command line instruct
## Prerequisites
-In order to build `storm` you need `python`, `ruby` and `nodejs`. In order to avoid an overfull page we don't provide platform/OS specific installation instructions for those here. Please refer to you platform's/OS' documentation for support.
+In order to build `storm` you need `python3`, `ruby` and `nodejs`. In order to avoid an overfull page we don't provide platform/OS specific installation instructions for those here. Please refer to you platform's/OS' documentation for support.
The `ruby` package manager `rvm` and `nodejs` package manager `nvm` are for convenience and are used in the tests which run on [GitHub actions](https://github.com/apache/storm/actions). They can be installed using `curl -L https://get.rvm.io | bash -s stable --autolibs=enabled && source ~/.profile` (see the [rvm installation instructions](https://github.com/rvm/rvm) for details) and `wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash && source ~/.bashrc` [...]
@@ -279,7 +279,7 @@ If you wish to skip the unit tests you can do this by adding `-DskipTests` to th
If you wish to skip the examples and external modules, you can do this by adding `-P '!examples,!externals'` to the command line.
-In case you modified `storm.thrift`, you have to regenerate thrift code as java and python code before compiling whole project.
+In case you modified `storm.thrift`, you have to regenerate thrift code as Java and Python code before compiling whole project.
```sh
cd storm-client/src
diff --git a/dev-tools/jira-github-join.py b/dev-tools/jira-github-join.py
index 3e23ea1e5..a6e1580b6 100755
--- a/dev-tools/jira-github-join.py
+++ b/dev-tools/jira-github-join.py
@@ -23,7 +23,7 @@ If you get certificate error when running on Mac,
(https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org)
Go to Macintosh HD
> Applications
- > Python3.9 folder (or whatever version of python you're using)
+ > Python3.9 folder (or whatever version of Python you're using)
> double click on "Install Certificates.command" file.
"""
diff --git a/dev-tools/report/__init__.py b/dev-tools/report/__init__.py
index e931fe08f..81a0e86e2 100644
--- a/dev-tools/report/__init__.py
+++ b/dev-tools/report/__init__.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/docs/Clojure-DSL.md b/docs/Clojure-DSL.md
index 56bb54f8d..e8485079d 100644
--- a/docs/Clojure-DSL.md
+++ b/docs/Clojure-DSL.md
@@ -77,7 +77,7 @@ Here's an example `shell-bolt-spec`:
```clojure
(shell-bolt-spec {"1" :shuffle "2" ["id"]}
- "python"
+ "python3"
"mybolt.py"
["outfield1" "outfield2"]
:p 25)
diff --git a/docs/Command-line-client.md b/docs/Command-line-client.md
index e2b34625f..352e1519e 100644
--- a/docs/Command-line-client.md
+++ b/docs/Command-line-client.md
@@ -313,7 +313,7 @@ Syntax: `storm shell resourcesdir command args`
Makes constructing jar and uploading to nimbus for using non JVM languages
-eg: `storm shell resources/ python topology.py arg1 arg2`
+eg: `storm shell resources/ python3 topology.py arg1 arg2`
### upload-credentials
diff --git a/docs/Defining-a-non-jvm-language-dsl-for-storm.md b/docs/Defining-a-non-jvm-language-dsl-for-storm.md
index 311577b64..aad552508 100644
--- a/docs/Defining-a-non-jvm-language-dsl-for-storm.md
+++ b/docs/Defining-a-non-jvm-language-dsl-for-storm.md
@@ -20,19 +20,26 @@ For a Python DSL, you would want to make use of "2" and "3". ShellComponent lets
There's a "storm shell" command that will help with submitting a topology. Its usage is like this:
```
-storm shell resources/ python topology.py arg1 arg2
+storm shell resources/ python3 topology.py arg1 arg2
```
storm shell will then package resources/ into a jar, upload the jar to Nimbus, and call your topology.py script like this:
```
-python topology.py arg1 arg2 {nimbus-host} {nimbus-port} {uploaded-jar-location}
+python3 topology.py arg1 arg2 {nimbus-host} {nimbus-port} {uploaded-jar-location}
```
Then you can connect to Nimbus using the Thrift API and submit the topology, passing {uploaded-jar-location} into the submitTopology method. For reference, here's the submitTopology definition:
```java
-void submitTopology(1: string name, 2: string uploadedJarLocation, 3: string jsonConf, 4: StormTopology topology) throws (1: AlreadyAliveException e, 2: InvalidTopologyException ite);
+void submitTopology(
+ 1: string name,
+ 2: string uploadedJarLocation,
+ 3: string jsonConf,
+ 4: StormTopology topology)
+ throws (
+ 1: AlreadyAliveException e,
+ 2: InvalidTopologyException ite);
```
Finally, one of the key things to do in a non-JVM DSL is make it easy to define the entire topology in one file (the bolts, spouts, and the definition of the topology).
diff --git a/docs/Multilang-protocol.md b/docs/Multilang-protocol.md
index ae77b1990..77cb9314c 100644
--- a/docs/Multilang-protocol.md
+++ b/docs/Multilang-protocol.md
@@ -30,10 +30,10 @@ program via the shell using Java's ProcessBuilder class.
By default the ShellProcess assumes that your code is packaged inside of your topology jar under the resources subdirectory of your jar and by default will change the current working directory of
the executable process to be that resources directory extracted from the jar.
-A jar file does not store permissions of the files in it. This includes the execute bit that would allow a shell script to be laoded and run by the operating systme.
-As such in most examples the scripts are of the form `python mybolt.py` because the python executable is already on the supervisor and mybolt is packaged in the resources directory of the jar.
+A jar file does not store permissions of the files in it. This includes the execute bit that would allow a shell script to be loaded and run by the operating systme.
+As such in most examples the scripts are of the form `python3 mybolt.py` because the Python executable is already on the supervisor and mybolt is packaged in the resources directory of the jar.
-If you want to package something more complicated, like a new version of python itself, you need to instead use the blob store for this and a `.tgz` archive that does support permissions.
+If you want to package something more complicated, like a new version of Python itself, you need to instead use the blob store for this and a `.tgz` archive that does support permissions.
See the docs on the [Blob Store](distcache-blobstore.html) for more details on how to ship a jar.
@@ -49,7 +49,7 @@ So if I shipped python with a symlink named `newPython` and a python ShellSpout
```
public MyShellSpout() {
- super("./newPython/bin/python", "./shell_spout.py");
+ super("./newPython/bin/python3", "./shell_spout.py");
changeChildCWD(false);
}
```
diff --git a/docs/OCI-support.md b/docs/OCI-support.md
index c73fcb5f0..a44959ea1 100644
--- a/docs/OCI-support.md
+++ b/docs/OCI-support.md
@@ -46,7 +46,7 @@ Docker does not need to be installed on each node, nor is there a dependency on
by an admin before containers can be launched. All that is required to be present on each node is an OCI-compatible runtime like
`runc`.
-##### Leverages Distributed File Sytems For Scale
+##### Leverages Distributed File Systems For Scale
Image can be fetched via HDFS or other distributed file systems instead of the Docker registry. This prevents a large cluster from
overwhelming a Docker registry when a big topology causes all of the nodes to request an image at once. This also allows large clusters
@@ -66,7 +66,7 @@ localization is also faster, as the layers no longer need to be unpacked into a
First you need to use the`docker-to-squash.py` script to download docker images and configs, convert layers to squashfs files and put them to a directory in HDFS, for example
```bash
-python docker-to-squash.py pull-build-push-update --hdfs-root hdfs://hostname:port/containers \
+python3 docker-to-squash.py pull-build-push-update --hdfs-root hdfs://hostname:port/containers \
docker.xxx.com:4443/hadoop-user-images/storm/rhel7:20201202-232133,storm/rhel7:dev_current --log DEBUG --bootstrap
```
diff --git a/docs/Setting-up-a-Storm-cluster.md b/docs/Setting-up-a-Storm-cluster.md
index f4a26d2c3..b86d47f7c 100644
--- a/docs/Setting-up-a-Storm-cluster.md
+++ b/docs/Setting-up-a-Storm-cluster.md
@@ -30,7 +30,7 @@ A few notes about Zookeeper deployment:
Next you need to install Storm's dependencies on Nimbus and the worker machines. These are:
1. Java 8+ (Apache Storm 2.x is tested through GitHub actions against Java 8 and Java 11)
-2. Python 2.7.x or Python 3.x
+2. Python 3.x
These are the versions of the dependencies that have been tested with Storm. Storm may or may not work with different versions of Java and/or Python.
diff --git a/docs/Tutorial.md b/docs/Tutorial.md
index 1dd891ba4..c4ebffaec 100644
--- a/docs/Tutorial.md
+++ b/docs/Tutorial.md
@@ -251,7 +251,7 @@ Here's the definition of the `SplitSentence` bolt from `WordCountTopology`:
```java
public static class SplitSentence extends ShellBolt implements IRichBolt {
public SplitSentence() {
- super("python", "splitsentence.py");
+ super("python3", "splitsentence.py");
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
@@ -260,7 +260,7 @@ public static class SplitSentence extends ShellBolt implements IRichBolt {
}
```
-`SplitSentence` overrides `ShellBolt` and declares it as running using `python` with the arguments `splitsentence.py`. Here's the implementation of `splitsentence.py`:
+`SplitSentence` overrides `ShellBolt` and declares it as running using `python3` with the arguments `splitsentence.py`. Here's the implementation of `splitsentence.py`:
```python
import storm
@@ -282,7 +282,7 @@ Earlier on in this tutorial, we skipped over a few aspects of how tuples are emi
## Trident
-Storm guarantees that every message will be played through the topology at least once. A common question asked is "how do you do things like counting on top of Storm? Won't you overcount?" Storm has a higher level API called Trudent that let you achieve exactly-once messaging semantics for most computations. Read more about Trident [here](Trident-tutorial.html).
+Storm guarantees that every message will be played through the topology at least once. A common question asked is "how do you do things like counting on top of Storm? Won't you overcount?" Storm has a higher level API called Trident that let you achieve exactly-once messaging semantics for most computations. Read more about Trident [here](Trident-tutorial.html).
## Distributed RPC
diff --git a/docs/Using-non-JVM-languages-with-Storm.md b/docs/Using-non-JVM-languages-with-Storm.md
index 1b3ae451b..da8093409 100644
--- a/docs/Using-non-JVM-languages-with-Storm.md
+++ b/docs/Using-non-JVM-languages-with-Storm.md
@@ -6,13 +6,13 @@ layout: documentation
- creating topologies in another language is easy since topologies are just thrift structures (link to storm.thrift)
- implementing spouts and bolts in another language is called a "multilang components" or "shelling"
- Here's a specification of the protocol: [Multilang protocol](Multilang-protocol.html)
- - the thrift structure lets you define multilang components explicitly as a program and a script (e.g., python and the file implementing your bolt)
+ - the thrift structure lets you define multilang components explicitly as a program and a script (e.g., python3 and the file implementing your bolt)
- In Java, you override ShellBolt or ShellSpout to create multilang components
- note that output fields declarations happens in the thrift structure, so in Java you create multilang components like the following:
- declare fields in java, processing code in the other language by specifying it in constructor of shellbolt
- multilang uses json messages over stdin/stdout to communicate with the subprocess
- - storm comes with ruby, python, and fancy adapters that implement the protocol. show an example of python
- - python supports emitting, anchoring, acking, and logging
+ - storm comes with Ruby, Python, and Fancy adapters that implement the protocol. show an example of Python
+ - Python supports emitting, anchoring, acking, and logging
- "storm shell" command makes constructing jar and uploading to nimbus easy
- makes jar and uploads it
- calls your program with host/port of nimbus and the jarfile id
@@ -31,18 +31,18 @@ union ComponentObject {
}
```
-For a non-JVM DSL, you would want to make use of "2" and "3". ShellComponent lets you specify a script to run that component (e.g., your python code). And JavaObject lets you specify native java spouts and bolts for the component (and Storm will use reflection to create that spout or bolt).
+For a non-JVM DSL, you would want to make use of "2" and "3". ShellComponent lets you specify a script to run that component (e.g., your Python code). And JavaObject lets you specify native java spouts and bolts for the component (and Storm will use reflection to create that spout or bolt).
There's a "storm shell" command that will help with submitting a topology. Its usage is like this:
```
-storm shell resources/ python topology.py arg1 arg2
+storm shell resources/ python3 topology.py arg1 arg2
```
storm shell will then package resources/ into a jar, upload the jar to Nimbus, and call your topology.py script like this:
```
-python topology.py arg1 arg2 {nimbus-host} {nimbus-port} {uploaded-jar-location}
+python3 topology.py arg1 arg2 {nimbus-host} {nimbus-port} {uploaded-jar-location}
```
Then you can connect to Nimbus using the Thrift API and submit the topology, passing {uploaded-jar-location} into the submitTopology method. For reference, here's the submitTopology definition:
diff --git a/docs/flux.md b/docs/flux.md
index 000270f77..cf83741c4 100644
--- a/docs/flux.md
+++ b/docs/flux.md
@@ -47,7 +47,7 @@ The easiest way to use Flux, is to add it as a Maven dependency in you project a
If you would like to build Flux from source and run the unit/integration tests, you will need the following installed
on your system:
-* Python 2.7.x or later
+* Python 3.0.x or later
* Node.js 0.10.x or later
#### Building with unit tests enabled:
@@ -649,7 +649,7 @@ bolts:
className: "org.apache.storm.flux.wrappers.bolts.FluxShellBolt"
constructorArgs:
# command line
- - ["python", "splitsentence.py"]
+ - ["python3", "splitsentence.py"]
# output fields
- ["word"]
parallelism: 1
@@ -796,7 +796,7 @@ bolts:
className: "org.apache.storm.flux.wrappers.bolts.FluxShellBolt"
constructorArgs:
# command line
- - ["python", "splitsentence.py"]
+ - ["python3", "splitsentence.py"]
# output fields
- ["word"]
parallelism: 1
diff --git a/docs/storm-sql-example.md b/docs/storm-sql-example.md
index 61f255b90..fbd53185f 100644
--- a/docs/storm-sql-example.md
+++ b/docs/storm-sql-example.md
@@ -13,8 +13,7 @@ This page assumes that Apache Zookeeper, Apache Storm and Apache Kafka are insta
For convenience, this page assumes that Apache Kafka 0.10.0 is installed via `brew`.
We'll use below tools to prepare the JSON data which will be fed to the input data source.
-Since they're Python projects, this page assumes Python 2.7 with `pip`, `virtualenv` is installed locally.
-If you're using Python 3, you may need to convert some places to be compatible with 3 manually while feeding data.
+Since they're Python projects, this page assumes Python 3.0.x with `pip3`, `virtualenv` is installed locally.
* https://github.com/kiritbasu/Fake-Apache-Log-Generator
* https://github.com/rory/apache-log-parser
@@ -92,10 +91,10 @@ For convenience, you can skip cloning project and download modified file from he
`apache-log-parser` can be installed via `pip`.
```
-$ pip install apache-log-parser
+$ pip3 install apache-log-parser
```
-Since apache-log-parser is a library, in order to parse fake log we need to write small python script.
+Since apache-log-parser is a library, in order to parse fake log we need to write small Python script.
Let's create file `parse-fake-log-gen-to-json-with-incrementing-id.py` with below content:
```
@@ -115,9 +114,8 @@ while True:
parsed_dict['id'] = auto_incr_id
auto_incr_id += 1
- # works only python 2, but I don't care cause it's just a test module :)
parsed_dict = {k.upper(): v for k, v in parsed_dict.iteritems() if not k.endswith('datetimeobj')}
- print json.dumps(parsed_dict)
+ print(json.dumps(parsed_dict))
```
### Feed parsed JSON Apache Log to Kafka
@@ -125,7 +123,7 @@ while True:
OK! We're prepared to feed the data to Kafka topic. Let's use `kafka-console-producer` to feed parsed JSON.
```
-$ python apache-fake-log-gen.py -n 0 | python parse-fake-log-gen-to-json-with-incrementing-id.py | kafka-console-producer --broker-list localhost:9092 --topic apache-logs
+$ python3 apache-fake-log-gen.py -n 0 | python3 parse-fake-log-gen-to-json-with-incrementing-id.py | kafka-console-producer --broker-list localhost:9092 --topic apache-logs
```
and execute below to another terminal session to confirm data is being fed.
diff --git a/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java b/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java
index fca718ee8..6800e625a 100644
--- a/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java
+++ b/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java
@@ -63,7 +63,7 @@ import org.slf4j.LoggerFactory;
* topology that use other languages. For example:
*
*
- * <p>```java public class MyBolt extends ShellBolt implements IRichBolt { public MyBolt() { super("python", "mybolt.py"); }
+ * <p>```java public class MyBolt extends ShellBolt implements IRichBolt { public MyBolt() { super("python3", "mybolt.py"); }
*
* <p>public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("field1", "field2")); } } ```
*/