You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/24 12:47:46 UTC
[GitHub] [iceberg] rajarshisarkar opened a new pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
rajarshisarkar opened a new pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393
This PR adds metrics tracking for `OSSFileIO`. This would help reporting IO metrics via the Hadoop `FileSystem.Statistics`.
---
cc: @danielcweeks @rdblue @kbendick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836570972
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSOutputFile.java
##########
@@ -25,15 +25,17 @@
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.metrics.MetricsContext;
class OSSOutputFile extends BaseOSSFile implements OutputFile {
- OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties) {
- super(client, uri, aliyunProperties);
+ OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties, MetricsContext metrics) {
+ super(client, uri, aliyunProperties, metrics);
}
- static OSSOutputFile fromLocation(OSS client, String location, AliyunProperties aliyunProperties) {
- return new OSSOutputFile(client, new OSSURI(location), aliyunProperties);
+ static OSSOutputFile fromLocation(OSS client, String location, AliyunProperties aliyunProperties,
Review comment:
This is only used in tests as far as I can tell. How about not updating it and passing `nullMetrics` here? That will cut down on unnecessary test changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r837041758
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSOutputFile.java
##########
@@ -25,15 +25,17 @@
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.metrics.MetricsContext;
class OSSOutputFile extends BaseOSSFile implements OutputFile {
- OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties) {
- super(client, uri, aliyunProperties);
+ OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties, MetricsContext metrics) {
+ super(client, uri, aliyunProperties, metrics);
}
- static OSSOutputFile fromLocation(OSS client, String location, AliyunProperties aliyunProperties) {
- return new OSSOutputFile(client, new OSSURI(location), aliyunProperties);
+ static OSSOutputFile fromLocation(OSS client, String location, AliyunProperties aliyunProperties,
Review comment:
Nice catch, I have passed `MetricsContext.nullMetrics()` to reduce the test changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#issuecomment-1083549743
Thanks, @rajarshisarkar!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836569501
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,27 @@
private long next = 0;
private boolean closed = false;
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
public OSSInputStream(OSS client, OSSURI uri) {
this.client = client;
this.uri = uri;
this.createStack = Thread.currentThread().getStackTrace();
+
+ this.readBytes = MetricsContext.nullMetrics().counter(
+ FileIOMetricsContext.READ_BYTES, Long.class, Unit.BYTES);
+ this.readOperations = MetricsContext.nullMetrics().counter(
+ FileIOMetricsContext.READ_OPERATIONS, Integer.class, Unit.COUNT);
Review comment:
Can this constructor call `this(client, uri, MetricsContext.nullMetrics());` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836568512
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
##########
@@ -24,17 +24,19 @@
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.SeekableInputStream;
+import org.apache.iceberg.metrics.MetricsContext;
+@Deprecated
Review comment:
Can you add some context for this? Usually, we add javadoc to explain the deprecation:
```java
/**
* @deprecated moving to package-private in 0.15.0; use OSSFileIO to create InputFile instances
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836568512
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
##########
@@ -24,17 +24,19 @@
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.SeekableInputStream;
+import org.apache.iceberg.metrics.MetricsContext;
+@Deprecated
Review comment:
Can you add some context for this? Usually, we add javadoc to explain the deprecation:
```java
/**
* @deprecated moving to package-private in 0.15.0; use OSSFileIO to create InputFile instances.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r837041368
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
##########
@@ -24,17 +24,19 @@
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.SeekableInputStream;
+import org.apache.iceberg.metrics.MetricsContext;
+@Deprecated
Review comment:
Added the javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r837041460
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,27 @@
private long next = 0;
private boolean closed = false;
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
public OSSInputStream(OSS client, OSSURI uri) {
this.client = client;
this.uri = uri;
this.createStack = Thread.currentThread().getStackTrace();
+
+ this.readBytes = MetricsContext.nullMetrics().counter(
+ FileIOMetricsContext.READ_BYTES, Long.class, Unit.BYTES);
+ this.readOperations = MetricsContext.nullMetrics().counter(
+ FileIOMetricsContext.READ_OPERATIONS, Integer.class, Unit.COUNT);
Review comment:
Thanks, I have made the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r839513540
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,20 @@
private long next = 0;
private boolean closed = false;
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
public OSSInputStream(OSS client, OSSURI uri) {
Review comment:
Yes, I agree. Deprecating this would maintain the parity with the other InputStream classes (which have a package private constructor).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r835980956
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,16 @@
private long next = 0;
private boolean closed = false;
- public OSSInputStream(OSS client, OSSURI uri) {
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
+ public OSSInputStream(OSS client, OSSURI uri, MetricsContext metrics) {
Review comment:
Since the old constructor was public, should we keep it and pass `nullMetrics`?
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSOutputFile.java
##########
@@ -25,15 +25,17 @@
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.io.PositionOutputStream;
+import org.apache.iceberg.metrics.MetricsContext;
class OSSOutputFile extends BaseOSSFile implements OutputFile {
- OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties) {
- super(client, uri, aliyunProperties);
+ OSSOutputFile(OSS client, OSSURI uri, AliyunProperties aliyunProperties, MetricsContext metrics) {
Review comment:
Similar here. Should we keep the old constructor and pass `nullMetrics`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836569859
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,16 @@
private long next = 0;
private boolean closed = false;
- public OSSInputStream(OSS client, OSSURI uri) {
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
+ public OSSInputStream(OSS client, OSSURI uri, MetricsContext metrics) {
Review comment:
Can you make this one package-private? I don't think we need to expose it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836303839
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,16 @@
private long next = 0;
private boolean closed = false;
- public OSSInputStream(OSS client, OSSURI uri) {
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
+ public OSSInputStream(OSS client, OSSURI uri, MetricsContext metrics) {
Review comment:
Thanks, for the feedback. I was retained the old constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r838900710
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,20 @@
private long next = 0;
private boolean closed = false;
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
public OSSInputStream(OSS client, OSSURI uri) {
Review comment:
Should we deprecate this, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836304635
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
##########
@@ -24,17 +24,18 @@
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.SeekableInputStream;
+import org.apache.iceberg.metrics.MetricsContext;
-public class OSSInputFile extends BaseOSSFile implements InputFile {
+class OSSInputFile extends BaseOSSFile implements InputFile {
Review comment:
I have rolled back the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rajarshisarkar commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rajarshisarkar commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r836306188
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputStream.java
##########
@@ -44,10 +48,16 @@
private long next = 0;
private boolean closed = false;
- public OSSInputStream(OSS client, OSSURI uri) {
+ private final Counter<Long> readBytes;
+ private final Counter<Integer> readOperations;
+
+ public OSSInputStream(OSS client, OSSURI uri, MetricsContext metrics) {
Review comment:
Thanks, for the feedback. I have retained the old constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4393: Aliyun: Add metrics tracking for OSSFileIO
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4393:
URL: https://github.com/apache/iceberg/pull/4393#discussion_r835980861
##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/oss/OSSInputFile.java
##########
@@ -24,17 +24,18 @@
import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.SeekableInputStream;
+import org.apache.iceberg.metrics.MetricsContext;
-public class OSSInputFile extends BaseOSSFile implements InputFile {
+class OSSInputFile extends BaseOSSFile implements InputFile {
Review comment:
This is technically a breaking change. I think we should roll it back in case anyone references this file. We can deprecate the class so that we can make this change in a release or two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org