You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/01/31 21:05:25 UTC
[GitHub] [incubator-iceberg] sudssf opened a new issue #764: make
org.apache.iceberg. PartitionData public
sudssf opened a new issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764
PartitionData.java is using default classifier for defining class which makes it inaccessible from any other package.
we need this class to be public so that external code and perform operations such as append to table, add files etc for partitioned table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue edited a comment on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
rdblue edited a comment on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581183953
I don't think that this necessarily needs to be public. It's an internal implementation that is not part of the public API. What are you trying to do? Can you pass a `StructLike` instead? The Iceberg generic records in org.apache.iceberg.data can be passed, so you can build a tuple of data using the generic record.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] sudssf edited a comment on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
sudssf edited a comment on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581615179
@rdsr yes interface can be implemented, which code would you prefer implementing `StructLike` or reusing boilerplate from `PartitionData`?
also partition logic is much cleaner using `PartitionSpec`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdsr commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
rdsr commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581593929
@sudssf you should be able to implement `StructLike` e.g
```
return new StructLike() {
@Override
public int size() {
return // return number of your partition columns
}
@Override
public <T> T get(int pos, Class<T> javaClass) {
// At this point you should have all partition column values. You may return the one based on `pos`;
}
@Override
public <T> void set(int pos, T value) {
// maybe just throw?
throw new UnsupportedOperationException("Set not supported");
}
};
```
What's your concern?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] sudssf commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
sudssf commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581615179
@rdsr yes interface can be implemented which code would you prefer implementing `StructLike` or reusing boilerplate from `PartitionData`?
also partition logic is much cleaner using `PartitionSpec`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] sudssf commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
sudssf commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581646436
thanks @rdblue I test using `GenericRecord `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] sudssf commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
sudssf commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581543316
Thanks @rdblue for reply. in one of our ingestion flow, we are writing files directly to iceberg table using java API. when we build location string and datafiles I am not sure how to pass partition information using `StructLike` as its interface.
for now I am using following API when creating data files
` new DataFiles.Builder(table.spec).withPartition(partLike)`
there is helper method which returns PartitionData
```
def getPartitionData(spec: PartitionSpec, par : String) : PartitionData = {
val partition = new PartitionData(spec.partitionType())
partition.set(0, par)
partition
}
```
can you point me to example where we can use `StructLike` directly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581618827
We don't want to open up `PartitionData` because it includes quite a bit that we don't want to make part of our public API. For example, it implements Avro's `IndexedRecord` API that we may not support indefinitely. The guarantee we provide is `StructLike`.
But, you should be able to use [`GenericRecord`](https://github.com/apache/incubator-iceberg/blob/master/data/src/main/java/org/apache/iceberg/data/GenericRecord.java) with `PartitionSpec.partitionType()` to get a `StructLike` that works for you without implementing that interface directly. Of course, you're welcome to implement that interface if you'd like more control over the implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764#issuecomment-581183953
I don't think that this necessarily needs to be public. It's an internal implementation that is not part of the public API. What are you trying to do? Can you pass `StructLike` instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] sudssf closed issue #764: make
org.apache.iceberg. PartitionData public
Posted by GitBox <gi...@apache.org>.
sudssf closed issue #764: make org.apache.iceberg. PartitionData public
URL: https://github.com/apache/incubator-iceberg/issues/764
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org