You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/05/08 06:34:38 UTC
[PR] [codex] Support catalog-aware blob view function [paimon]
leaves12138 opened a new pull request, #7786:
URL: https://github.com/apache/paimon/pull/7786
## Summary
- Inject Paimon catalog context into catalog-aware Flink functions.
- Make `sys.blob_view` resolve blob fields by name through the catalog while keeping the legacy `(identifier, fieldId, rowId)` overload.
- Add a flat `unionAll` helper for BTree index topology and cover it with a regression test.
## Root Cause
The stash conflicted with an upstream BlobViewStruct change that now stores table identifiers as `Identifier`. The catalog-aware BlobView path also needed the Flink catalog context during function instantiation so it can resolve field IDs from table metadata at runtime.
## Validation
- `mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -DskipTests compile`
- `mvn -pl paimon-flink/paimon-flink-common -DfailIfNoTests=false -Dtest=BlobTableITCase#testWriteBlobViewWithBuiltInFunction,BTreeIndexTopoBuilderTest test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [codex] Support catalog-aware blob view function [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 merged PR #7786:
URL: https://github.com/apache/paimon/pull/7786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [codex] Support catalog-aware blob view function [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #7786:
URL: https://github.com/apache/paimon/pull/7786#discussion_r3208000785
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/function/BlobViewFunction.java:
##########
@@ -18,18 +18,157 @@
package org.apache.paimon.flink.function;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.data.BlobViewStruct;
+import org.apache.paimon.flink.FlinkFileIOLoader;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.Table;
+import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.ScalarFunction;
+import javax.annotation.Nullable;
+
+import java.util.HashMap;
+import java.util.Map;
+
/** Flink scalar function that constructs a serialized {@link BlobViewStruct}. */
-public class BlobViewFunction extends ScalarFunction {
+public class BlobViewFunction extends ScalarFunction implements CatalogAwareFunction {
+
+ @Nullable private String catalogName;
+ @Nullable private Options catalogOptions;
+
+ private transient Catalog catalog;
+ private transient Map<String, Map<String, BlobViewField>> blobViewFieldCache;
+
+ @Override
+ public void setCatalogContext(
+ String catalogName, String defaultDatabase, Options catalogOptions) {
+ this.catalogName = catalogName;
+ this.catalogOptions = new Options(catalogOptions.toMap());
+ this.catalog = null;
+ this.blobViewFieldCache = null;
+ }
+
+ @Override
+ public void open(FunctionContext context) {
+ openCatalog(context.getUserCodeClassLoader());
Review Comment:
It is CatalogAwareFunction, so it must set catalog context.
<img width="1740" height="854" alt="image" src="https://github.com/user-attachments/assets/aced42f2-2eb4-43a9-95ef-baa630350480" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [codex] Support catalog-aware blob view function [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #7786:
URL: https://github.com/apache/paimon/pull/7786#discussion_r3207805790
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/function/BlobViewFunction.java:
##########
@@ -18,18 +18,157 @@
package org.apache.paimon.flink.function;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.data.BlobViewStruct;
+import org.apache.paimon.flink.FlinkFileIOLoader;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.Table;
+import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.ScalarFunction;
+import javax.annotation.Nullable;
+
+import java.util.HashMap;
+import java.util.Map;
+
/** Flink scalar function that constructs a serialized {@link BlobViewStruct}. */
-public class BlobViewFunction extends ScalarFunction {
+public class BlobViewFunction extends ScalarFunction implements CatalogAwareFunction {
+
+ @Nullable private String catalogName;
+ @Nullable private Options catalogOptions;
+
+ private transient Catalog catalog;
+ private transient Map<String, Map<String, BlobViewField>> blobViewFieldCache;
+
+ @Override
+ public void setCatalogContext(
+ String catalogName, String defaultDatabase, Options catalogOptions) {
+ this.catalogName = catalogName;
+ this.catalogOptions = new Options(catalogOptions.toMap());
+ this.catalog = null;
+ this.blobViewFieldCache = null;
+ }
+
+ @Override
+ public void open(FunctionContext context) {
+ openCatalog(context.getUserCodeClassLoader());
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (catalog != null) {
+ catalog.close();
+ catalog = null;
+ }
+ }
public byte[] eval(String identifier, int fieldId, long rowId) {
if (identifier == null) {
return null;
}
return new BlobViewStruct(Identifier.fromString(identifier), fieldId, rowId).serialize();
}
+
+ public byte[] eval(String tableName, String fieldName, long rowId) {
+ if (tableName == null || fieldName == null) {
+ return null;
+ }
+
+ BlobViewField field = blobViewField(tableName, fieldName);
+ return new BlobViewStruct(field.tableName, field.fieldId, rowId).serialize();
+ }
+
+ private BlobViewField blobViewField(String tableName, String fieldName) {
+ Map<String, BlobViewField> tableCache = blobViewFieldCache().get(tableName);
+ if (tableCache != null) {
+ BlobViewField cached = tableCache.get(fieldName);
+ if (cached != null) {
+ return cached;
+ }
+ }
+
+ Identifier identifier = toIdentifier(tableName);
+ BlobViewField field = new BlobViewField(identifier, fieldId(identifier, fieldName));
+ if (tableCache == null) {
+ tableCache = new HashMap<>();
+ blobViewFieldCache().put(tableName, tableCache);
+ }
+ tableCache.put(fieldName, field);
+ return field;
+ }
+
+ private int fieldId(Identifier identifier, String fieldName) {
+ try {
+ Table table = catalog().getTable(identifier);
+ if (!table.rowType().containsField(fieldName)) {
Review Comment:
Should we also check that the resolved field is a BLOB field here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [codex] Support catalog-aware blob view function [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #7786:
URL: https://github.com/apache/paimon/pull/7786#issuecomment-4411390029
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [codex] Support catalog-aware blob view function [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #7786:
URL: https://github.com/apache/paimon/pull/7786#discussion_r3207785604
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/function/BlobViewFunction.java:
##########
@@ -18,18 +18,157 @@
package org.apache.paimon.flink.function;
+import org.apache.paimon.catalog.Catalog;
+import org.apache.paimon.catalog.CatalogContext;
+import org.apache.paimon.catalog.CatalogFactory;
import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.data.BlobViewStruct;
+import org.apache.paimon.flink.FlinkFileIOLoader;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.table.Table;
+import org.apache.flink.table.functions.FunctionContext;
import org.apache.flink.table.functions.ScalarFunction;
+import javax.annotation.Nullable;
+
+import java.util.HashMap;
+import java.util.Map;
+
/** Flink scalar function that constructs a serialized {@link BlobViewStruct}. */
-public class BlobViewFunction extends ScalarFunction {
+public class BlobViewFunction extends ScalarFunction implements CatalogAwareFunction {
+
+ @Nullable private String catalogName;
+ @Nullable private Options catalogOptions;
+
+ private transient Catalog catalog;
+ private transient Map<String, Map<String, BlobViewField>> blobViewFieldCache;
+
+ @Override
+ public void setCatalogContext(
+ String catalogName, String defaultDatabase, Options catalogOptions) {
+ this.catalogName = catalogName;
+ this.catalogOptions = new Options(catalogOptions.toMap());
+ this.catalog = null;
+ this.blobViewFieldCache = null;
+ }
+
+ @Override
+ public void open(FunctionContext context) {
+ openCatalog(context.getUserCodeClassLoader());
Review Comment:
This will fail if blob view function without catalog context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org