You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/05/11 02:00:24 UTC
Apache Pinot Daily Email Digest (2021-05-10)
### _#general_
**@humengyuk18:** Can multiple table consume from the same kafka topic using
low level consumer in Pinot?
**@mayanks:** Yes. Each table is an independent entity. You can configure the
same kafka topic for multiple of them
**@mayanks:** Added to faq:
**@humengyuk18:** Thanks, I will try it.
**@patidar.rahul8392:** Hi Team, I am trying to use hdfs as deepatorage and
created controller.conf server.conf and broker.conf files. Kindly review and
Provide your valuable suggestions on attached file. Please suggest/guide if
anything needs to change or add.
**@sleepythread:** @patidar.rahul8392 you have added script in wrong file.
e.g: For Broker create a file broker.conf and add following in that file.
```pinot.set.instance.id.to.hostname=true pinot.server.grpc.enable=true```
After this, run the following in your terminal ( or you have also create a
shell script file) to start the broker ```exportHADOOP_HOME=/user/local/hadoop
exportHADOOP_VERSION=2.7.1 exportHADOOP_GUAVA_VERSION=11.0.2
exportHADOOP_GSON_VERSION=2.2.4 exportGC_LOG_LOCATION=/tmp/gc_log/
exportPINOT_VERSION=0.7.1 exportPINOT_DISTRIBUTION_DIR=/tmp/my_data_2/apache-
pinot-incubating-0.7.1-bin exportSERVER_CONF_DIR=/tmp/my_data_2/apache-pinot-
incubating-0.7.1-bin/bin exportZOOKEEPER_ADDRESS=00.000.00.000:2181 export
CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-
${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-
annotations-
${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-
${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-
${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-
${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-
${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms4G -Xmx4G -XX:+UseG1GC
-XX:MaxGCPauseMillis=200 - XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCApplicationStoppedTime - XX:+PrintGCApplicationConcurrentTime-
Xloggc:${GC_LOG_LOCATION}/gc-pinot-broker.log"
${PINOT_DISTRIBUTION_DIR}/bin/start-broker.sh -zkAddress ${ZOOKEEPER_ADDRESS}
- configFileName ${SERVER_CONF_DIR}/broker.con```
**@sleepythread:** Perform similar operation for almost all the other services
as well
**@patidar.rahul8392:** Thanks alot @sleepythread
**@patidar.rahul8392:** @tingchen @sleepythread @ken @slack1 @mayanks kindly
help.
**@syedakram93:** Hi, I am running a query which involves AND , OR and with
some filters on string and long values. It has basically 34Million rows , and
querying(selecting few columns for an ID) takes almost 2 sec and
numEntriesScannedInFilter(89Million) & numEntriesScannedPostFilter are bigger
values. Can someone help me to understand, how come this many entries scanned
in filter, where i am using Inverted index...?
**@mayanks:** Are there any ORs that can be changed into INs? If so try that
**@syedakram93:** yeah we can do that
**@syedakram93:** after changing it from OR to IN wherever possible giving
same time taken, but record scanned is more this time
**@mayanks:** Can you paste the new query and the records scanned?
**@mayanks:** and also all columns that have sorted/inv index?
**@syedakram93:** Query: select
AUDITLOGID,RECORDID,RECORDNAME,AUDITEDTIME,USERID,ACTIONTYPE,SOURCE,ACTIONINFO,OTHERDETAILS,DONEBY
FROM auditlog WHERE ((((((RELATEDID = 553493000165096765) AND (RELATEDMODULE =
'Contacts')) AND (AUDITEDTIME >= 1588214155000)) AND ((((((((MODULE =
'Potentials') AND ((ACTIONTYPE = 19) OR (ACTIONTYPE = 20))) OR ((MODULE =
'Potentials') AND (((((OTHERDETAILS = 'Unattended Dialled') OR (OTHERDETAILS =
'Attended Dialled')) OR (OTHERDETAILS = 'Scheduled Attended')) OR
(OTHERDETAILS = 'Received')) OR (OTHERDETAILS = 'Missed')))) OR (ACTIONTYPE IN
(36,34,19))) OR ((ACTIONTYPE IN (10,11,1)) AND (SOURCE = 19))) OR ((ACTIONTYPE
IN (10,11,19)) AND (SOURCE = 10))) OR ((ACTIONTYPE = 1) AND (MODULE =
'Potentials'))) OR (ACTIONTYPE = 69))))) limit 5000 table config: { "OFFLINE":
{ "tableName": "auditlog_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": {
"replication": "1", "segmentPushType": "REFRESH" }, "tenants": { "broker":
"DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": {
"invertedIndexColumns": [ "RELATEDID", "AUDITLOGID" ],
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false, "loadMode": "MMAP",
"enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false,
"aggregateMetrics": false, "nullHandlingEnabled": false }, "metadata": {
"customConfigs": {} }, "ingestionConfig": {}, "isDimTable": false } }
**@mayanks:** You didn’t change the otherDetails to IN?
**@syedakram93:** totalDocs : 34Milion num ofsegments: 1 num of servers: 1
numEntriesScannedInFilter : 89627266
**@syedakram93:** otherDetails is basically a json string
**@syedakram93:** i will change that too
**@syedakram93:** i changed otherDetails too, but giving same result
/timetaken and stats
**@syedakram93:** my main concern is, how come *numEntriesScannedInFilter* is
89Million which is very high in number
**@mayanks:** You have inv index only on 2 columns?
**@syedakram93:** yes
**@syedakram93:** but when i checked metadata.properties, it shows almost all
columns have invertedindex
**@mayanks:** Yeah that is misleading. It is only for the two columns you
specified
**@syedakram93:** ohh
**@syedakram93:** shall i specify the columns which i used to filter in
invertedindex?
**@syedakram93:** ACTIONTYPE, OTHERDETAILS, MODULE, SOURCE
**@syedakram93:**?
**@mayanks:** Which ones of them have high cardinality?
**@syedakram93:** unique values will be more in this in case of OTHERDETAILS
column
**@mayanks:** Ok but that is json?
**@mayanks:** As in configured as json for Pinot or Pinot thinks it is just a
string?
**@syedakram93:** its string
**@syedakram93:** toString()
**@mayanks:** Ok, let’s start with that
**@mayanks:** And if that data size is not that much, may be set for all
**@syedakram93:** what do u mean by data size?
**@mayanks:** Total data size in Pinot
**@syedakram93:**
**@syedakram93:** segment tar.gz size is 639M
**@syedakram93:** i applied inv index , now timetaken to finish the query took
393ms(earlier it was almost 2sec), but numOfEntriesScanned is 2140274...
**@syedakram93:** can i reduce numOfEntriesScanned it some more
**@mayanks:** Is this with inv index on all columns?
**@mayanks:** Also I do not see a sorted column
**@syedakram93:** Yes with 12 columns inv index and no sorted column
**@syedakram93:** Will apply pk as sorted index and check if entries scanned
can reduced
**@mayanks:** Yeah, sorted column will help reduce further.
**@mayanks:** Will the pk be part of most queries? And what kind of predicate
would you have on that pk in the queries?
**@syedakram93:** check this for ref
**@pedro.cls93:** Hello, Is there a way to check the current kafka offset
that a realtime table is reading on at a given point in time?
**@mayanks:** Not at the moment, but you can check the previous saved offset
from segment Zk metadata
**@pedro.cls93:** Thought as much, thank you @mayanks
**@mayanks:** Could you tell us a bit more on how you wanted to use it?
**@pedro.cls93:** My realtime table wasn't consuming from kafka. These
messages are stringified jsons which frequently change schemas or have may
occasionally have some errors, I wanted to understand what was the last read
offsets for each kafka partition to be able to accurately debug where Pinot
stopped.
**@mayanks:** Got it. There are metrics emitted for consumption. You could see
those going to zero when consumption would have stopped. That would give you
the time-frame.
**@npawar:** You can use the consumingSegmentsInfo Api
**@pedro.cls93:** In my use-case ingestion time-frame in Pinot does not
correlate kafka's message time. The kafka messages are computed in batch and
sent in bulk to the pinot ingestion topic.
**@npawar:** You'll find it on swagger under segments section
**@pedro.cls93:** There seems to an issue, the request does not reach the
cluster?
**@npawar:** On the top left hand corner change http to https
**@pedro.cls93:** Brilliant! Thank you very much Neha!
**@patidar.rahul8392:** What is process to setup a 3 node pinot cluster. With
2 controller 2 server and 2 controller.I have separate servers for zookeeper
and kafka and already up and running. Now on remaining 3 servers I want to set
2 controllers , 2 brokers and 2 server. Kindly suggest. Total available
servers=3 Total components needs to install= 6
**@pedro.cls93:** Hello, Can I combine within groovy ?
**@npawar:** No that's not supported. However, a groovy script should be able
to do all of the logic you need rt? What are you trying to achieve?
**@pedro.cls93:** Groovy should be sufficient, yes, I'm actively trying it
now. For context: A have a json field in my table, let's call it `result` I'm
trying to retrieve a field from the `result` payload, called `duration` and
parse it into a Long field in the Pinot Table to represent the duration in
milliseconds. The value of the `duration` property in the `result` field can
have multiple forms: `HH:MM:SS.SSSS / HH:MM:SS / HH:MM:SS.SSSSSSS` or simply
not exist. To complicate matters, the key of `duration` is not consistent,
when it appears, in some messages it may come as `Duration` and others as
`duration` This is legacy and not something I can fix upstream unfortunately.
**@g.kishore:** You can technically combine it but we don’t let groovy access
any other classes for security reasons
### _#troubleshooting_
**@jmeyer:** Hello What is the recommended approach to getting the "last non-
null value" ? Use a UDF ?
**@pedro.cls93:** Hello, does anyone know the meaning of the following warn
log in the server: `2021/05/10 09:11:48.182 WARN
[LLRealtimeSegmentDataManager_HitExecutionView__4__5__20210430T1719Z]
[HitExecutionView__10__4__20210429T1719Z] Commit failed with response
{"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":false,"status":"FAILED","offset":-1}`
This is a realtime table, consuming messages from kafka with upsert enabled.
I've noticed that the table hasn't been updating offsets for the past 3 days.
**@pedro.cls93:** I'm also getting the following exception from the server
instances: ```2021/05/10 10:29:48.876 ERROR
[ServerSegmentCompletionProtocolHandler]
[HitExecutionView__13__6__20210510T1029Z] Could not send request
java.net.SocketTimeoutException: Read timed out at
java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_282] at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
~[?:1.8.0_282] at java.net.SocketInputStream.read(SocketInputStream.java:171)
~[?:1.8.0_282] at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[?:1.8.0_282] at
shaded.org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:383)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:675)
~[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:207)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:174)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:949)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:559)
[pinot-all-0.7.1-jar-with-
dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed] at
java.lang.Thread.run(Thread.java:748) [?:1.8.0_282] 2021/05/10 10:29:48.877
ERROR [LLRealtimeSegmentDataManager_HitExecutionView__13__6__20210510T1029Z]
[HitExecutionView__13__6__20210510T1029Z] Holding after response from
Controller:
{"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":false,"status":"NOT_SENT","offset":-1}```
This seems related, the server not being able to contact the controller?
**@pedro.cls93:** Controller is logging the following warnings at the same
time: ```2021/05/10 10:32:07.285 WARN [TopStateHandoffReportStage]
[HelixController-pipeline-default-pinot-data-(3c494c91_DEFAULT)] Event
3c494c91_DEFAULT : Cannot confirm top state missing start time. Use the
current system time as the start time.```
**@mayanks:** Yeah, seems server is unable to connect to controller
**@mayanks:** Is controller accessible otherwise?
**@pedro.cls93:** Yes, the process has not suffered restarts and grafana
monitoring does not report anything in particular. Java memory is manageable
(84.3% of total), GC is working, CPU load is ok (1% of total)
**@mayanks:** Hmm can you log on to server and try to connect to controller?
**@pedro.cls93:** I can connect to the controller and get a bash shell yes
**@tamas.nadudvari:** Hello, we’re using `RealtimeToOfflineSegmentsTask` for
our hybrid table and we ran into a problem in our dev environment. We have
time gaps in our data ingest and when it’s larger than the bucket time period,
the minion task runs into an error of creating empty segment. After exception
the minion fails to update the watermark, thus we’re ending up with a stuck
task (trying to create an empty segment over an over again for this specific
period). While it’s unlikely to run into this empty segment problem in
production, we’re wondering what’s the recommended way to overcome this issue
in a dev environment?
**@mayanks:** @jackie.jxt
**@laxman:** I’m also trying this feature in our test environment. Watermark
is stored as ZK metadata. You can try updating the metadata manually. Try
setting it to the next available segment start time stamp.
**@npawar:** thanks for the suggestion Laxman! @tamas.nadudvari do you mind
creating a github issue for this? will see if it can be fixed easily
**@npawar:** or if you’d like to give it a shot to fix, that would be great
too
**@npawar:** another thing i can think of is to increase the bucket time ?
**@tamas.nadudvari:** We’re also running into a strange problem with hybrid
tables that might be connected to completely empty realtime table(part):
querying the hybrid table gives no records while querying the offline
table(part) does. Filtering the time column to an interval for records in the
offline table doesn’t help either. Failed to find any relevant log message
that can help me identify, what could be the issue.
**@aaron:** I'm trying to track down where my SegmentCreation job is spending
time. It's been running for a few hours, but here are the only time
measurements I see: ```Assembled and processed 7990100 records from 17 columns
in 121766 ms: 65.618484 rec/ms, 1115.5142 cell/ms time spent so far 0% reading
(237 ms) and 99% processing (121766 ms) at row 7990100. reading next block
block read in memory in 44 ms. row count = 1418384 Finished building
StatsCollector! Collected stats for 9408484 documents Created dictionary for
INT column: ... ... RecordReader initialized will read a total of 9408484
records. at row 0. reading next block Got brand-new decompressor [.gz] block
read in memory in 133 ms. row count = 7990100 Start building IndexCreator!
Assembled and processed 7990100 records from 17 columns in 127060 ms:
62.884464 rec/ms, 1069.0359 cell/ms time spent so far 0% reading (133 ms) and
99% processing (127060 ms) at row 7990100. reading next block block read in
memory in 26 ms. row count = 1418384 Finished records indexing in
IndexCreator! Finished segment seal! ... Generated 25884 star-tree records
from 9408484 segment records Finished constructing star-tree, got 1228 tree
nodes and 2058 records under star-node Finished creating aggregated documents,
got 1227 aggregated records Finished building star-tree in 276631ms Starting
building star-tree with config: StarTreeV2BuilderConfig[...] Generated 9408484
star-tree records from 9408484 segment records``` It's been stuck at that last
line for a long time but still seems to be doing something. All the time
measurements only add up to a few minutes. Any idea where it's spending time?
**@ken:** There was a similar question recently in <#CDRCA57FC|general>, where
GCing could be a potential source of slowness (trying to include a ref, let’s
see if this works: )
**@fx19880617:** I feel it’s mostly spent on startree generation
**@avasudevan:** Hi, Could you guys help me with the Avro -> Pinot Schema
Conversion? `Avro Schema:` ```{ "type": "record", "name": "Clickstream",
"namespace": "com.acme.event.clickstream.business", "fields": [ { "name":
"event_header", "type": { "type": "record", "name": "EventHeader",
"namespace": "com.acme.event", "fields": [ { "name": "event_uuid", "type": {
"type": "string", "avro.java.string": "String", "logicalType": "uuid" },
"doc": "Universally Unique Identifier for this event " }, { "name":
"published_timestamp", "type": { "type": "long", "logicalType": "timestamp-
millis" }, "doc": "Timestamp in milliseconds since the epoch that the event
occurred on its producing device. e.g.
<code>System.currentTimeMillis()</code>." }] } } }``` The corresponding Pinot
Schema i have is: ```{ "schemaName": "user_clickstream_v1",
"dimensionFieldSpecs": [ { "name": "event_header.event_uuid", "dataType":
"STRING" } ], "dateTimeFieldSpecs": [{ "name":
"event_header.published_timestamp", "dataType": "LONG", "format" :
"1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] }``` In the created
Pinot Table i see all the values as null….I suspect the issue is in the
Schema… Any idea?
**@npawar:** did you set transform functions in the table config?
**@npawar:** this demo explains what transform function you’d have to set:
**@npawar:** treat the avro hierarchy similar to the json as shown in the demo
### _#pinot-dev_
**@syedakram93:** Hi, I am running a query which involves AND , OR and with
some filters on string and long values. It has basically 34Million rows , and
querying(selecting few columns for an ID) takes almost 2 sec and
numEntriesScannedInFilter(89Million) & numEntriesScannedPostFilter are bigger
values. Can someone help me to understand, how come this many entries scanned
in filter, where i am using Inverted index...?
**@syedakram93:**
**@ssahay:** @ssahay has joined the channel
**@snlee:** Hi all, I would like to use this thread for discussing about
0.8.0 release. Some points that we need to discuss: • Should we cut the
release before or after the graduation? • Let’s call the timeline so that
people can check-in the features to be included to the release I think that we
should make the call first on the before/after graduation and then decide the
timeline.
**@mayanks:** Thanks @snlee, my thoughts: ```1. If the graduation process
requires a release, let's cut it before. 2\. If it does not, then lets fix
things like licensing at least. 3\. Let's collect data if we have enough
features since 0.7.1 that calls for a new release.```
### _#minion-improvements_
**@jackie.jxt:** The `NullValueTransformer` is the one to fill default value
for `null` values. Currently in the `PinotSegmentRecordReader` we don't fill
the `null` values back to the record, and that causes the null value vector
not being reconstructed.
**@jackie.jxt:** In `MutableSegmentImpl.getRecord()` we have logic to
backfill the `null` values, and we should do similar operation for
`PinotSegmentRecordReader`
**@laxman:** I spent some time over the weekend and today. I think I got the
exact root cause. Testing a patch.
**@laxman:** I think the major root cause for these null issues is pinot to
avro schema conversion. While converting we treat all avro fields as non-
nullable fields.
**@laxman:**
**@laxman:** Fixes needed in this schema conversion • Nullable fields we need
to use unions (union {null, int} for a nullable int type) • Default values to
be set properly
**@laxman:** @jackie.jxt /@fx19880617: Does this make sense? I’m not very sure
whether this fixes the null vector problem. But it will fix these NPEs in avro
and rightly.
**@fx19880617:** I see, then shall we always convert a filed to nullable in
avro then?
**@laxman:** Yes. Always a union {null, actual type}
**@npawar:** i dont understand why a field in the segmnt will be nullable.
When the segment was created in the first place, the NullValueTransformer
would have put the default value correctly rt?
**@laxman:** @npawar: Following is one case where a field value can be null.
> We are seeing issues with nullable fields added recently. These fields were
not there in the old segments. And we are seeing NPE issues while converting
old REALTIME segments to OFFLINE.
**@npawar:** i see. so it doesnt exist in the segment at all
**@laxman:** Yes. Examples & issue
**@npawar:** in an ideal scenario, after you add the new field, you would
invoke a reloadSegments, which would generate the new column in the old
segments too
**@npawar:** i think your solution is also fine. But we might have to be
careful if the filter/record transformation/aggregation steps need to happen
on the new null field. We’d have to handle null in each of those steps
**@laxman:** Does reload segments updates the deep store as well? iiuc,
conversion task loads the segments from deep store. Hence the question.
**@npawar:** ah, it does not. Hmm this is going to be a problem then..
Especially now that we’ve added derived fields
**@laxman:** Can you please explain little more? Can I proceed with union
approach during pinot->avro schema conversion? Or that will cause other
problems?
**@npawar:** you’re right. the conversion task will download from deep store.
That segment copy will not have the default values for new column. If you
proceed with your approach, we will face problems in the further steps of the
SegmentProcessor, like filtering, partitioning, etc, in case they start using
the new column. To avoid this, Jackie’s suggestion is better, of adding the
Default values for null values, inside the PinotSegmentRecordReader. The
bigger problem I’m thinking about is that we’ve introduced the concept of
derived columns. So lets say segment has columns `x,y`. Now someone can add
new column `z=x+y` for example, wherein value of new column is derived from
the existing values. But this new value is also not updated into the deep
store. In this case, during the RealtimeToOffline job, it is not correct to
treat it as null, neither is it correct to give it default value
**@jackie.jxt:** After transforming the input record with
`CompositeTransformer`, the `GenericRow` should have `_nullValueFields` set
for `null` fields. We should store them into the avro, then read them out in
the reducer
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org