You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/09 23:35:59 UTC
[GitHub] [pinot] mayankshriv opened a new issue, #8500: Bad table config may result in Zombie tables
mayankshriv opened a new issue, #8500:
URL: https://github.com/apache/pinot/issues/8500
Issue faced by user:
```
hey folks, does anyone have experience with “zombie” tables? I attempted to add a tableSpec but was missing a field in the config. This resulted in a null pointer exception when Pinot processed the config. As a result, the table doesn’t exist in the UI or by querying the list API, but I can’t create a table with the same name due to receiving an error that “table already exists”. Presumably the spec reached ZK or something, but no where else? What can be done to remove the zombie table?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1097427420
Fixed by #8509
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] timsants commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
timsants commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1095670691
When a realtime table is being created, the table config is first uploaded to the ZK property store. After that, we try to create the realtime consumer. But as pointed out, if there is an invalid config for the consumer, an exception is thrown. The next time you try to create the table (with correct configs), it checks the ZK property store for the existence of the table config and throws an exception saying that the table already exists (although the table creation process did not finish).
**Why does the table not show up with GET `/tables`?**
It checks for the tables existence in the ideal state [resources](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java#L604). Because the realtime table creation process never completed, the realtime table was not added to the ZK ideal state.
**Why did deleting the table not fix the issue?**
Table deletion should have actually fixed the issue because it deletes the config from the property store (verified that this was the case). The return message from the operation was misleading because it only returns a success message if the table was originally in the ideal state.
**Proposed solution**
The table existence check for table creation is inconsistent with the other table operations (GET and DELETE). For table creation, we should check the table existence in the ideal state instead of checking for the table config in the property store. If the table config already existed in the property store, it should be ok to overwrite since it was never fully created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] timsants commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
timsants commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1096000480
@npawar, this was a change that was merged in Feb [#8103](https://github.com/apache/pinot/pull/8103/files). Before that, realtime tables didn't have this check. Previously, the table config was probably being overwritten but I don't think the other ZK metadata was being cleaned up correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1095728788
Checking table config existence is intentional to prevent race condition of re-creating a recently not fully deleted table (see #8103 for details)
The correct fix here should be:
- If adding a table fails, delete the table config and all entries already persisted
- Enhance the exception message to inform the user to try delete the table which can delete the remaining entries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1095889721
@timsants there was a report that this worked fine in 0.9.3 (i.e. table creation failed cleanly) and in 0.10.0 this fails with zombie states. Any idea why that is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] timsants commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
timsants commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1094156084
Investigation in progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] nicovak commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
nicovak commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1094811870
I had the same issue multiple time last week :
- Try to addTable if config error or internal error happens (specify an SSL path for example that does not exist)
- It returns an error in logs - sometime you also can have cannot build kafka consumer error
- Using rest api / ui or pinot tools does not show any tables
- Deleting with rest api doesn’t work (as GET return empty array)
- Try to create the table again give an error
- In my case I’m on k8s so I delete all PVC (data) and deploy It again to get rid of the issue.
Example of config that would raise the issue
```json
"tableIndexConfig": {
"loadMode": "MMAP",
"nullHandlingEnabled": true,
"streamConfigs": {
"streamType": "kafka",
"security.protocol": "SSL",
"ssl.truststore.type": "PKCS12",
"ssl.keystore.type": "PKCS12",
"ssl.truststore.location": "/I/DONT_EXIST.p12",
"ssl.keystore.location": "/I/DONT_EXIST.p12"
"ssl.truststore.password": "${SSL_TRUSTSTORE_PASSWORD}",
"ssl.keystore.password": "${SSL_KEYSTORE_PASSWORD}"
},
}
```
(I removed broker list, etc but you can take the `ssl.` block )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1094141692
@timsants could you please take a look? Some how I am unable to assign this to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] timsants commented on issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
timsants commented on issue #8500:
URL: https://github.com/apache/pinot/issues/8500#issuecomment-1098415425
Fix has been merged. We now do a full ZK metadata cleanup in case there is any failure during the table creation process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #8500: Bad table config may result in Zombie tables
URL: https://github.com/apache/pinot/issues/8500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv closed issue #8500: Bad table config may result in Zombie tables
Posted by GitBox <gi...@apache.org>.
mayankshriv closed issue #8500: Bad table config may result in Zombie tables
URL: https://github.com/apache/pinot/issues/8500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org