You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/08/20 10:06:33 UTC

[PR] Support OBJECT data type in tablet write [iotdb-client-go]

shuwenwei opened a new pull request, #175:
URL: https://github.com/apache/iotdb-client-go/pull/175

   ## Description
   
   Adds support for the **OBJECT** data type (TSDataType `12`) in the table-model tablet write path of the Go client, following the Java `TableSession` / tsfile implementation.
   
   ### Write path
   - Add `OBJECT` to the `TSDataType` enum and the string/byte type maps (`client/protocol.go`)
   - Treat OBJECT as a binary column (same as BLOB) in `Tablet` (`client/tablet.go`): value storage, `SetValueAt`/`GetValueAt`, `Swap`, `getValuesBytes`, `NewTablet`
   - Add `Tablet.SetObjectValueAt(isEOF, offset, content, columnIndex, rowIndex)` for **segmented OBJECT writes**: each segment is wrapped with a 1-byte `isEOF` flag + 8-byte big-endian offset, matching Java `Tablet.addValue(rowIndex, columnIndex, isEOF, offset, content)`
   
   ### Read path
   - Decode OBJECT (and BLOB/STRING) binary columns in `BinaryArrayColumnDecoder`, and handle OBJECT like BLOB in `IoTDBRpcDataSet` getters, so written objects can be read back (e.g. `select READ_OBJECT(file)` + `GetBlob`)
   
   ### Tests
   - Unit tests: `TestTablet_OBJECT` (whole-object write), `TestTablet_SetObjectValueAt` (segmented write) in `client/tablet_test.go`
   - E2E table test: `Test_InsertObjectTablet` in `test/e2e/e2e_table_test.go`, covering whole-object, segmented (512B), and null-object rows, verified via `READ_OBJECT`/`GetBlob`, `count(*)`, null check, and tag/field round-trip
   
   Also includes the regenerated thrift `common` code (new aggregation types, `pipeRecentFailureList` field) produced by `make all`.
   
   ## Verification
   - `go build ./...` / `go test ./client/...` pass
   - E2E suite requires Docker (`make e2e_test`)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support OBJECT data type in tablet write [iotdb-client-go]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #175:
URL: https://github.com/apache/iotdb-client-go/pull/175


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support OBJECT data type in tablet write [iotdb-client-go]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #175:
URL: https://github.com/apache/iotdb-client-go/pull/175#discussion_r3820607232


##########
common/common.go:
##########
@@ -6709,28 +6823,37 @@ func (p *TPipeHeartbeatResp) Equals(other *TPipeHeartbeatResp) bool {
 	}
 	if len(p.PipeMetaList) != len(other.PipeMetaList) { return false }
 	for i, _tgt := range p.PipeMetaList {
-		_src29 := other.PipeMetaList[i]
-		if bytes.Compare(_tgt, _src29) != 0 { return false }
+		_src32 := other.PipeMetaList[i]
+		if bytes.Compare(_tgt, _src32) != 0 { return false }
 	}
 	if len(p.PipeCompletedList) != len(other.PipeCompletedList) { return false }
 	for i, _tgt := range p.PipeCompletedList {
-		_src30 := other.PipeCompletedList[i]
-		if _tgt != _src30 { return false }
+		_src33 := other.PipeCompletedList[i]
+		if _tgt != _src33 { return false }
 	}
 	if len(p.PipeRemainingEventCountList) != len(other.PipeRemainingEventCountList) { return false }
 	for i, _tgt := range p.PipeRemainingEventCountList {
-		_src31 := other.PipeRemainingEventCountList[i]
-		if _tgt != _src31 { return false }
+		_src34 := other.PipeRemainingEventCountList[i]
+		if _tgt != _src34 { return false }
 	}
 	if len(p.PipeRemainingTimeList) != len(other.PipeRemainingTimeList) { return false }
 	for i, _tgt := range p.PipeRemainingTimeList {
-		_src32 := other.PipeRemainingTimeList[i]
-		if _tgt != _src32 { return false }
+		_src35 := other.PipeRemainingTimeList[i]
+		if _tgt != _src35 { return false }
 	}
 	if len(p.PipeDegradedStatusList) != len(other.PipeDegradedStatusList) { return false }
 	for i, _tgt := range p.PipeDegradedStatusList {
-		_src33 := other.PipeDegradedStatusList[i]
-		if _tgt != _src33 { return false }
+		_src36 := other.PipeDegradedStatusList[i]
+		if _tgt != _src36 { return false }
+	}
+	if len(p.PipeRecentFailureList) != len(other.PipeRecentFailureList) { return false }
+	for i, _tgt := range p.PipeRecentFailureList {
+		_src37 := other.PipeRecentFailureList[i]
+		if len(_tgt) != len(_src37) { return false }
+		for k, _tgt := range _tgt {
+			_src38 := _src37[k]
+			if _tgt != _src38 { return false }

Review Comment:
   This equality check does not verify that the key exists in the other map. Because a missing Go map key reads as the value type's zero value, maps such as `{"a": 0}` and `{"b": 0}` have equal lengths and are incorrectly reported as equal. Check the lookup boolean as well (and make the corresponding generator/template fix so regeneration preserves it).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support OBJECT data type in tablet write [iotdb-client-go]

Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #175:
URL: https://github.com/apache/iotdb-client-go/pull/175#discussion_r3820930145


##########
client/tablet.go:
##########
@@ -195,7 +195,7 @@ func (t *Tablet) SetValueAt(value interface{}, columnIndex, rowIndex int) error
 		default:
 			return fmt.Errorf("illegal argument value %v %v", value, reflect.TypeOf(value))
 		}
-	case TEXT, STRING:
+	case TEXT, STRING, OBJECT:

Review Comment:
   [P1] Do not accept unframed OBJECT payloads here
   
   For TableSession OBJECT writes, the server interprets every value as a segment envelope: one EOF byte, an eight-byte big-endian offset, then content. This branch accepts an arbitrary `string` or `[]byte` and serializes it without that envelope; a live insertion of a string accepted here failed with status 741 because the server interpreted the string bytes as the offset. Please remove OBJECT from this generic branch and have `SetObjectValueAt` assign through an internal framed-value helper, or expose a separate, clearly documented API if another valid pre-encoded/object-path representation is required. The unit test should not assert that raw `"hello"` is a valid TableSession OBJECT value.



##########
client/tablet.go:
##########
@@ -229,6 +229,42 @@ func (t *Tablet) SetValueAt(value interface{}, columnIndex, rowIndex int) error
 	return nil
 }
 
+// SetObjectValueAt writes a segment of an OBJECT column value. An OBJECT value can be
+// written in multiple segments so that a large object does not need to be fully loaded
+// into memory: each segment is wrapped into a 9-byte header (1 byte isEOF flag followed
+// by an 8-byte big-endian offset) and then the raw content, consistent with the Java
+// Tablet.addValue(rowIndex, columnIndex, isEOF, offset, content). Segments of the same
+// object must be written in order with ascending offsets, and the last segment must set
+// isEOF to true.
+//
+// Parameters:
+//   - isEOF: Whether this segment is the last one of the object.
+//   - offset: The offset of this segment within the whole object.
+//   - content: The raw bytes of this segment.
+//   - columnIndex: The column index of the OBJECT column.
+//   - rowIndex: The row index to write the segment into.
+//
+// Returns:
+//   - err: An error if the column/row index is invalid or the column is not of type OBJECT.
+func (t *Tablet) SetObjectValueAt(isEOF bool, offset int64, content []byte, columnIndex, rowIndex int) error {
+	if columnIndex < 0 || columnIndex >= len(t.measurementSchemas) {
+		return fmt.Errorf("illegal argument columnIndex %d", columnIndex)
+	}
+	if rowIndex < 0 || rowIndex >= t.maxRowNumber {
+		return fmt.Errorf("illegal argument rowIndex %d", rowIndex)
+	}
+	if t.measurementSchemas[columnIndex].DataType != OBJECT {
+		return fmt.Errorf("column %d must be of type OBJECT", columnIndex)
+	}
+	value := make([]byte, len(content)+9)
+	if isEOF {
+		value[0] = 1
+	}
+	binary.BigEndian.PutUint64(value[1:9], uint64(offset))
+	copy(value[9:], content)
+	return t.SetValueAt(value, columnIndex, rowIndex)

Review Comment:
   [P2] Clear a previously marked NULL bit before storing the segment
   
   `SetValueAt(nil, ...)` marks this cell in the bitmap, but the non-nil path never unmarks it. Consequently, calling `SetValueAt(nil, ...)` and then `SetObjectValueAt(...)` for the same cell still writes a NULL row; this was reproducible against TimechoDB. The equivalent Java overload calls `updateBitMap(rowIndex, columnIndex, false)`. Please clear the bit for non-nil assignments (ideally in the shared `SetValueAt` path) and add a regression test for overwriting NULL with an OBJECT value.



##########
client/rpcdataset.go:
##########
@@ -542,7 +542,7 @@ func (s *IoTDBRpcDataSet) getObjectByTsBlockIndex(tsBlockColumnIndex int32) (int
 		} else {
 			return binary.GetStringValue(), nil
 		}
-	case BLOB:
+	case BLOB, OBJECT:

Review Comment:
   [P1] Decode OBJECT results with OBJECT semantics
   
   Grouping `OBJECT` with `BLOB` exposes server-side OBJECT metadata instead of the public object representation. In a live TimechoDB test, `SELECT file` returned an OBJECT payload whose first eight bytes are the object size and whose remaining bytes contain the internal object path; this branch makes `GetObject` return those raw bytes, while the matching `GetString` branch hex-encodes them. The Java client uses `BytesUtils.parseObjectByteArrayToString(...)` for both getters (for example, `(Object) 1.00 KB`) and rejects `getBlob` for OBJECT. Please split OBJECT from BLOB, implement the same formatter/getter contract, add direct `SELECT file` tests, and add the corresponding `database/sql` column mapping so a non-null OBJECT is not silently returned as `nil`.



##########
common/common.go:
##########
@@ -475,6 +475,10 @@ const (
 	TAggregationType_SKEWNESS TAggregationType = 38
 	TAggregationType_KURTOSIS TAggregationType = 39
 	TAggregationType_PERCENTILE TAggregationType = 40
+	TAggregationType_RATE TAggregationType = 41

Review Comment:
   [P2] Keep unrelated generated protocol updates out of this OBJECT PR
   
   These aggregation constants and the `pipeRecentFailureList` additions are generated from a newer IoTDB protocol and are unrelated to OBJECT tablet support. Bundling them expands the public Thrift/protocol surface, creates version drift, and also brings in the correctness issue already reported in the `Equals` implementation below. Please revert `common/common.go` from this PR and submit any pinned, reviewed IDL regeneration as a separate change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org