You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "John Omernik (JIRA)" <ji...@apache.org> on 2016/02/15 20:15:18 UTC
[jira] [Commented] (DRILL-4143) REFRESH TABLE METADATA - Permission
Issues with metadata files
[ https://issues.apache.org/jira/browse/DRILL-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147712#comment-15147712 ]
John Omernik commented on DRILL-4143:
-------------------------------------
After checking on this issue on Drill 1.4, and running things and having it worked, I thought all was well, until I loaded more data in the table, and then it broke again. Here are some of the steps I took:
I tried to refresh table metadata on the table, using the dataadm user (basically the user who owns the data). Note all directories and files are owned by dataadm:dataadm and the permissions are 770.
On a fresh table, I ran
REFRESH TABLE METADATA mytable;
And it worked.
Then I loaded another day worth of data, 2016-02-15, and I try to refresh the table only to get a permisisons error on 2015-11-12 (this day was not changed/affected by the load)
Error:
"false| Error: 2126.29602.2546226 /data/prod/mytable/2015-011-12/.drill.parquet_metadata (Permission denied)12:44
This is the SAME shell where I ran it before, and I loaded more data (note the directory in question was already loaded, that was no touched).
Then I use the find command to remove all the .drill.parquet_metadata files. and run the REFRESH TABLE METADATA command again:
This time the command works. Great.
If I run it again, right after: It runs successfully again.
12:35 Ran it a third time, and it worked.
12:37 Ran it a fourth time: and it worked. (Note all the parquet_metadata files are owned by my drillbituser: drillbitgroup (in this case, mapr:mapr) despite the metadata operation being done by the data owner in the drill shell)
12:39 Another process *running as dataadm* loaded a new day of data (2016-02-12) No other data was altered here.
12:40 Ran REFRESH TABLE METADATA a fifth time: Got the error. Maybe it has to do with adding data? Error on 2015-11-12 again....
12:41 A new Process loaded more data. (2016-02-11, and 2016-02-10 loaded) Process completes successfully, disabled at this time. for troubleshooting (not more data being loaded)
12:42 Attempt REFRESH TABLE METADATA again, same error on 2015-11-12
12:43 Removed all .drill.parquet_metadata files using find command
12:44 Ran REFRESH TABLE METADATA - This time ran with success. Will now run and check without data loading. May have to do with data loading...
12:52 Ran REFRESH: Success
12:58 Ran REFRESH: Success
1:00 Forced Reload of 2016-02-15. Basically making it so the folder "2016-02-15" did not have a .drill.parquet_metadata file (while the other days did)
1:01 Ran REFRESH : Error: 2126.27460.2555888 /data/prod/mytable/2015-11-12/.drill.parquet_metadata (Permission denied) (Same file, not sure why it picks on this file, nothing is changed there) (Even validated, no files modified since 12:58 when the parquet_metadata file was modified, all parquet files still have the same modified times of when they were loaded, Feb 9th)
So thoughts:
1. When running REFRESH TABLE METADATA, it checks to see if all the files in the subdirectories exist, if they don't it starts to "do things"
2. The date 2015-11-12 probably keeps coming out is because it's first in .drill.parquet_metadata located in /mytable (not in the individual directories)
3. After the REFRESH failed, I checked some files. 2015-11-12/.drill.parquet_metadata was a 0 size files. (Like it was attempted to be rewritten and failed) Looking in 2016-11-13, the .drill.parquet_metadata file has data in it.
4. To test #3, I rm .drill.parquet_metadata from 2015-11-12, and run the refresh command again. Interesting... when I do that, I get permission denied on the 2015-11-12 directory again, this time, instead of the file owned by the driillbit user (and having the drillbit user group, in this case mapr) I have a file of 0 bytes, with "dataadm:datareaders" as the owner. That's interesting... shouldn't it be mapr:mapr (the drillbit user?)
So this seems to be the crux of the issue... what should happen here? all metadata operations be checked to see if the user issuing it has permissions, and then writes happening as the drillbit user?
> REFRESH TABLE METADATA - Permission Issues with metadata files
> --------------------------------------------------------------
>
> Key: DRILL-4143
> URL: https://issues.apache.org/jira/browse/DRILL-4143
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: 1.3.0, 1.4.0
> Reporter: John Omernik
> Labels: Metadata, Parquet, Permissions
> Fix For: Future
>
>
> Summary of Refresh Metadata Issues confirmed by two different users on Drill User Mailing list. (Title: REFRESH TABLE METADATA - Access Denied)
> This issue pertains to table METADATA and revolves around user authentication.
> Basically, when the drill bits are running as one user, and the data is owned by another user, there can be access denied issues on subsequent queries after issuing a REFRESH TABLE METADATA command.
> To troubleshoot what is actually happening, I turned on MapR Auditing (This is a handy feature) and found that when I run a query (that is giving me access denied.. my query is select count(1) from testtable ) Per MapR the user I am logged in as (dataowner) is trying to do a create operation on the .drill.parquet_metadata file and it's failing with status: 17. Per Keys at MapR, "status 17 means errno 17 which means EEXIST. Looks like Drill is trying to create a file that already exists." This seems to indicate that drill is perhaps trying to create the .drill.parquet_metadata on each select as the dataowner user, but the permissions (as seen below) don't allow it.
> Here are the steps to reproduce:
> Enable Authentication.
> Run all drill bits in the cluster as "drillbituser", then have the files owned by "dataowner". Note the root of the table permissions are drwxrwxr-x but as Drill loads each partition it loads them as drwxr-xr-x (all with dataowner:dataowner ownership). That may be something too, the default permissions when creating a table? Another note, in my setup, drillbituser is in the group for dataowner. Thus, they should always have read access.
> # Authenticated as dataowner (this should have full permissions to all the data)
> Enter username for jdbc:drill:zk=zknode1:5181: dataowner
> Enter password for jdbc:drill:zk=zknode1:5181: **********
> 0: jdbc:drill:zk=zknode1> use dfs.dev;
> +-------+--------------------------------------+
> | ok | summary |
> +-------+--------------------------------------+
> | true | Default schema changed to [dfs.dev] |
> +-------+--------------------------------------+
> 1 row selected (0.307 seconds)
> # The query works fine with no table metadata
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> +-----------+
> | EXPR$0 |
> +-----------+
> | 24565203 |
> +-----------+
> 1 row selected (3.392 seconds)
> # Refresh of metadata works under with no errors
> 0: jdbc:drill:zk=zknode1> refresh table metadata `testtable`;
> +-------+-------------------------------------------------------+
> | ok | summary |
> +-------+-------------------------------------------------------+
> | true | Successfully updated metadata for table testtable. |
> +-------+-------------------------------------------------------+
> 1 row selected (5.767 seconds)
>
> # Trying to run the same query, it returns a access denied issue.
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> Error: SYSTEM ERROR: IOException: 2127.7646.2950962 /data/dev/testtable/2015-11-12/.drill.parquet_metadata (Permission denied)
>
>
> [Error Id: 7bfce2e7-f78d-4fba-b047-f4c85b471de4 on node1:31010] (state=,code=0)
>
>
> # Note how all the files are owned by the drillbituser. Per discussion on list, this is normal
>
> $ find ./ -type f -name ".drill.parquet_metadata" -exec ls -ls {} \;
> 726 -rwxr-xr-x 1 drillbituser drillbituser 742837 Nov 30 14:27 ./2015-11-12/.drill.parquet_metadata
> 583 -rwxr-xr-x 1 drillbituser drillbituser 596146 Nov 30 14:27 ./2015-11-29/.drill.parquet_metadata
> 756 -rwxr-xr-x 1 drillbituser drillbituser 773811 Nov 30 14:27 ./2015-11-11/.drill.parquet_metadata
> 763 -rwxr-xr-x 1 drillbituser drillbituser 780829 Nov 30 14:27 ./2015-11-04/.drill.parquet_metadata
> 632 -rwxr-xr-x 1 drillbituser drillbituser 646851 Nov 30 14:27 ./2015-11-08/.drill.parquet_metadata
> 845 -rwxr-xr-x 1 drillbituser drillbituser 864421 Nov 30 14:27 ./2015-11-05/.drill.parquet_metadata
> 771 -rwxr-xr-x 1 drillbituser drillbituser 788823 Nov 30 14:27 ./2015-11-28/.drill.parquet_metadata
> 1273 -rwxr-xr-x 1 drillbituser drillbituser 1303168 Nov 30 14:27 ./2015-11-10/.drill.parquet_metadata
> 645 -rwxr-xr-x 1 drillbituser drillbituser 660028 Nov 30 14:27 ./2015-11-22/.drill.parquet_metadata
> 1017 -rwxr-xr-x 1 drillbituser drillbituser 1040469 Nov 30 14:27 ./2015-11-13/.drill.parquet_metadata
> 1280 -rwxr-xr-x 1 drillbituser drillbituser 1310552 Nov 30 14:27 ./2015-11-03/.drill.parquet_metadata
> 585 -rwxr-xr-x 1 drillbituser drillbituser 598973 Nov 30 14:27 ./2015-11-07/.drill.parquet_metadata
> 737 -rwxr-xr-x 1 drillbituser drillbituser 754103 Nov 30 14:27 ./2015-11-20/.drill.parquet_metadata
> 836 -rwxr-xr-x 1 drillbituser drillbituser 855363 Nov 30 14:27 ./2015-11-16/.drill.parquet_metadata
> 646 -rwxr-xr-x 1 drillbituser drillbituser 661134 Nov 30 14:27 ./2015-11-27/.drill.parquet_metadata
> 1156 -rwxr-xr-x 1 drillbituser drillbituser 1183378 Nov 30 14:27 ./2015-11-21/.drill.parquet_metadata
> 679 -rwxr-xr-x 1 drillbituser drillbituser 694551 Nov 30 14:27 ./2015-11-09/.drill.parquet_metadata
> 631 -rwxr-xr-x 1 drillbituser drillbituser 645500 Nov 30 14:27 ./2015-11-26/.drill.parquet_metadata
> 190 -rwxr-xr-x 1 drillbituser drillbituser 193798 Nov 30 14:27 ./2015-11-18/.drill.parquet_metadata
> 221 -rwxr-xr-x 1 drillbituser drillbituser 225342 Nov 30 14:27 ./2015-11-23/.drill.parquet_metadata
> 269 -rwxr-xr-x 1 drillbituser drillbituser 274988 Nov 30 14:27 ./2015-11-25/.drill.parquet_metadata
> 845 -rwxr-xr-x 1 drillbituser drillbituser 864615 Nov 30 14:27 ./2015-11-06/.drill.parquet_metadata
> 32 -rwxr-xr-x 1 drillbituser drillbituser 32221 Nov 30 14:27 ./2015-11-19/.drill.parquet_metadata
> 16468 -rwxr-xr-x 1 drillbituser drillbituser 16862958 Nov 30 14:27 ./.drill.parquet_metadata
>
> # Once the files are deleted...
> $ sudo find ./ -type f -name ".drill.parquet_metadata" -exec rm {} \;
> # Then the query works again
> 0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
> +-----------+
> | EXPR$0 |
> +-----------+
> | 24567700 |
> +-----------+
> 1 row selected (1.353 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)