You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@tsfile.apache.org by "ColinLeeo (via GitHub)" <gi...@apache.org> on 2025/11/20 02:42:53 UTC
[PR] Feature/python new datatypes (#641) [tsfile]
ColinLeeo opened a new pull request, #644:
URL: https://github.com/apache/tsfile/pull/644
* [Python] new datatypes
* add cwrapper ut for python new datatypes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/python new datatypes (#641) [tsfile]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #644:
URL: https://github.com/apache/tsfile/pull/644#discussion_r2554567473
##########
cpp/src/cwrapper/tsfile_cwrapper.h:
##########
@@ -623,6 +626,12 @@ INSERT_DATA_INTO_TS_RECORD_BY_NAME(int64_t);
INSERT_DATA_INTO_TS_RECORD_BY_NAME(bool);
INSERT_DATA_INTO_TS_RECORD_BY_NAME(float);
INSERT_DATA_INTO_TS_RECORD_BY_NAME(double);
+ERRNO _insert_data_into_ts_record_by_name_string(TsRecord data,
+ const char* measurement_name,
+ const char* value);
Review Comment:
Do we really need to provide such an insecure method?
##########
python/tsfile/tsfile_cpp.pxd:
##########
@@ -159,6 +161,12 @@ cdef extern from "./tsfile_cwrapper.h":
ErrorCode _insert_data_into_ts_record_by_name_double(TsRecord data, const char *measurement_name,
const double value);
ErrorCode _insert_data_into_ts_record_by_name_bool(TsRecord data, const char *measurement_name, const bint value);
+ ErrorCode _insert_data_into_ts_record_by_name_string(TsRecord data, const char *measurement_name,
+ const char *value);
+ ErrorCode _insert_data_into_ts_record_by_name_string_with_len(TsRecord data, const char *measurement_name,
+ const char *value,
+ const uint32_t value_len);
+ #ErrorCode _insert_data_into_ts_record_by_name_blob(record, PyUnicode_AsUTF8(field.get_field_name()), field.get_bytes_value())
Review Comment:
Remove
##########
python/tsfile/tsfile_py_cpp.pyx:
##########
@@ -266,8 +268,8 @@ cdef Tablet to_c_tablet(object tablet):
if value[row] is not None:
tablet_add_value_by_index_double(ctablet, row, col, value[row])
- # STRING
- elif data_type == TS_DATATYPE_STRING:
+ # STRING or TEXT
+ elif data_type == TS_DATATYPE_STRING or data_type == TS_DATATYPE_TEXT or data_type == TS_DATATYPE_BLOB:
for row in range(max_row_num):
Review Comment:
or BLOB
##########
cpp/test/cwrapper/cwrapper_test.cc:
##########
@@ -28,23 +39,145 @@ extern "C" {
#include "utils/errno_define.h"
namespace cwrapper {
-class CWrapperTest : public testing::Test {};
-
-// TEST_F(CWrapperTest, RegisterTimeSeries) {
-// ERRNO code = 0;
-// char* temperature = strdup("temperature");
-// TimeseriesSchema ts_schema{temperature, TS_DATATYPE_INT32,
-// TS_ENCODING_PLAIN,
-// TS_COMPRESSION_UNCOMPRESSED};
-// remove("cwrapper_register_timeseries.tsfile");
-// TsFileWriter writer =
-// tsfile_writer_new("cwrapper_register_timeseries.tsfile", &code);
-// ASSERT_EQ(code, 0);
-// code = tsfile_writer_register_timeseries(writer, "device1", &ts_schema);
-// ASSERT_EQ(code, 0);
-// free(temperature);
-// tsfile_writer_close(writer);
-// }
+class CWrapperTest : public testing::Test {
+ public:
+ static void ASSERT_OK(ERRNO code, const char* msg = "") {
+ ASSERT_EQ(code, RET_OK) << msg;
+ }
+};
+
+TEST_F(CWrapperTest, TestForPythonInterfaceInsert) {
+ ERRNO code = 0;
+ const int column_num = 10;
+ char* filename = "cwrapper_for_python.tsfile";
+ remove(filename); // Clean up any existing file
+
+ // Device and measurement definitions
+ char* device_id = "root.device1";
+ char* str_measurement_id = "str_measurement";
+ char* text_measurement_id = "text_measurement";
+ char* date_measurement_id = "date_measurement";
+
+ // Define time series schemas for different data types
+ timeseries_schema str_measurement;
+ str_measurement.timeseries_name = str_measurement_id;
+ str_measurement.compression = TS_COMPRESSION_UNCOMPRESSED;
+ str_measurement.data_type = TS_DATATYPE_STRING;
+ str_measurement.encoding = TS_ENCODING_PLAIN;
+
+ timeseries_schema text_measurement;
+ text_measurement.timeseries_name = text_measurement_id;
+ text_measurement.compression = TS_COMPRESSION_UNCOMPRESSED;
+ text_measurement.data_type = TS_DATATYPE_TEXT;
+ text_measurement.encoding = TS_ENCODING_PLAIN;
+
+ timeseries_schema date_measurement;
+ date_measurement.timeseries_name = date_measurement_id;
+ date_measurement.compression = TS_COMPRESSION_UNCOMPRESSED;
+ date_measurement.data_type = TS_DATATYPE_DATE;
+ date_measurement.encoding = TS_ENCODING_PLAIN;
+
+ // Create TsFile writer
+ auto* writer = (storage::TsFileWriter*)_tsfile_writer_new(
+ filename, 128 * 1024 * 1024, &code);
+ ASSERT_OK(code, "create writer failed");
+
+ // Register time series with the writer
+ ASSERT_OK(
+ _tsfile_writer_register_timeseries(writer, device_id, &str_measurement),
+ "register timeseries failed");
+
+ ASSERT_OK(_tsfile_writer_register_timeseries(writer, device_id,
+ &text_measurement),
+ "register timeseries failed");
+
+ ASSERT_OK(_tsfile_writer_register_timeseries(writer, device_id,
+ &date_measurement),
+ "register timeseries failed");
+
+ // Create a new time series record
+ auto* record = _ts_record_new(device_id, 0, 3);
+
+ // Insert string data
+ char* test_str = "test_string";
+ ASSERT_OK(_insert_data_into_ts_record_by_name_string(
+ record, str_measurement_id, test_str),
+ "insert data failed");
+
+ // Insert text data
+ char* test_text = "test_text";
+ ASSERT_OK(_insert_data_into_ts_record_by_name_string(
+ record, text_measurement_id, test_text),
+ "insert data failed");
+
+ // Insert date data - NOTE: There's a bug here, should use
+ // date_measurement_id
+ int32_t test_date = 20251118;
+ ASSERT_OK(_insert_data_into_ts_record_by_name_int32_t(
+ record, date_measurement_id, test_date),
+ "insert data failed");
+
+ // Write the record to file and close writer
+ ASSERT_OK(_tsfile_writer_write_ts_record(writer, record),
+ "write record failed");
+ ASSERT_OK(_tsfile_writer_flush(writer), "flush failed");
+ ASSERT_OK(_tsfile_writer_close(writer), "close writer failed");
+ // _tsfile_writer_close already deletes the writer, no need to free
+
+ // Free the record after writing
+ _free_tsfile_ts_record(&record);
+
+ // Create reader to verify the written data
+ auto* reader = (storage::TsFileReader*)tsfile_reader_new(filename, &code);
+ ASSERT_OK(code, "create reader failed");
+
+ // Query the data we just wrote
+ char* sensors[] = {str_measurement_id, text_measurement_id,
+ date_measurement_id};
+ auto* result = (storage::ResultSet*)_tsfile_reader_query_device(
+ reader, device_id, sensors, 3, 0, 100, &code);
+ ASSERT_OK(code, "query device failed");
+
+ // Verify the retrieved data matches what we inserted
+ bool has_next = false;
+ int row_count = 0;
+ while (!result->next(has_next) && has_next) {
Review Comment:
Rewrite `!result->next(has_next)` to make it clear that it is checking the status code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/python new datatypes (#641) [tsfile]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #644:
URL: https://github.com/apache/tsfile/pull/644#issuecomment-3555568159
## [Codecov](https://app.codecov.io/gh/apache/tsfile/pull/644?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `87.50000%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 62.41%. Comparing base ([`be7bd7e`](https://app.codecov.io/gh/apache/tsfile/commit/be7bd7e56c4c609f529dd8b396bd0d52213c594f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c234bb0`](https://app.codecov.io/gh/apache/tsfile/commit/c234bb0013198d8c2d118975721027ac9101e0d4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/tsfile/pull/644?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [cpp/src/cwrapper/tsfile\_cwrapper.cc](https://app.codecov.io/gh/apache/tsfile/pull/644?src=pr&el=tree&filepath=cpp%2Fsrc%2Fcwrapper%2Ftsfile_cwrapper.cc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3BwL3NyYy9jd3JhcHBlci90c2ZpbGVfY3dyYXBwZXIuY2M=) | 75.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/tsfile/pull/644?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #644 +/- ##
===========================================
+ Coverage 62.34% 62.41% +0.07%
===========================================
Files 703 703
Lines 39628 39654 +26
Branches 5537 5547 +10
===========================================
+ Hits 24705 24750 +45
+ Misses 14261 14236 -25
- Partials 662 668 +6
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/tsfile/pull/644?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/python new datatypes (#641) [tsfile]
Posted by "hongzhi-gao (via GitHub)" <gi...@apache.org>.
hongzhi-gao commented on PR #644:
URL: https://github.com/apache/tsfile/pull/644#issuecomment-3600703602
Perhaps we still need to supplement some content regarding the implementation and testing of the new data types for **tablet** and **aligned time series**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/python new datatypes (#641) [tsfile]
Posted by "ColinLeeo (via GitHub)" <gi...@apache.org>.
ColinLeeo merged PR #644:
URL: https://github.com/apache/tsfile/pull/644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@tsfile.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org