You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2021/12/13 15:53:44 UTC

Review Request 73758: RANGER-3548: Update performance engine test scripts

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/
-----------------------------------------------------------

Review request for ranger.


Bugs: RANGER-3548
    https://issues.apache.org/jira/browse/RANGER-3548


Repository: ranger


Description
-------

It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.


Diffs
-----

  distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
  ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
  ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 


Diff: https://reviews.apache.org/r/73758/diff/1/


Testing
-------

- Created a file containing tagged hive entities
- Uploaded generated file into Ranger using tagsync's file-based interface
- Ensured that the tagged entities are downloaded to hive service


Thanks,

Abhay Kulkarni

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Abhay Kulkarni <ak...@hortonworks.com>.


> On Dec. 13, 2021, 7:01 p.m., Madhan Neethiraj wrote:
> > ranger-tools/scripts/create_tags_file.sh
> > Lines 39 (patched)
> > <https://reviews.apache.org/r/73758/diff/1/?file=2256592#file2256592line39>
> >
> >     output_file is overwritten in each iteration inside this for ever loop - see #37. Is the script supposed to create a single file? If not, consider writing to different file in each iteration.

Updating the same file is by design. With a File-based tag upload using tagsync, without changing the configuration of tagsync - such as source-file-name, successive set of generaated tagged entities can be uploaded to Ranger. (File-Based tag-enricher will re-read the file specified in its configuration periodically and find different set of entities to upload to Ranger).


- Abhay


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/#review223846
-----------------------------------------------------------


On Dec. 13, 2021, 3:53 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73758/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2021, 3:53 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-3548
>     https://issues.apache.org/jira/browse/RANGER-3548
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.
> 
> 
> Diffs
> -----
> 
>   distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
>   ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
>   ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 
> 
> 
> Diff: https://reviews.apache.org/r/73758/diff/1/
> 
> 
> Testing
> -------
> 
> - Created a file containing tagged hive entities
> - Uploaded generated file into Ranger using tagsync's file-based interface
> - Ensured that the tagged entities are downloaded to hive service
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/#review223846
-----------------------------------------------------------




ranger-tools/scripts/create_tags_file.sh
Lines 39 (patched)
<https://reviews.apache.org/r/73758/#comment312889>

    output_file is overwritten in each iteration inside this for ever loop - see #37. Is the script supposed to create a single file? If not, consider writing to different file in each iteration.


- Madhan Neethiraj


On Dec. 13, 2021, 3:53 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73758/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2021, 3:53 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-3548
>     https://issues.apache.org/jira/browse/RANGER-3548
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.
> 
> 
> Diffs
> -----
> 
>   distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
>   ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
>   ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 
> 
> 
> Diff: https://reviews.apache.org/r/73758/diff/1/
> 
> 
> Testing
> -------
> 
> - Created a file containing tagged hive entities
> - Uploaded generated file into Ranger using tagsync's file-based interface
> - Ensured that the tagged entities are downloaded to hive service
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/#review223848
-----------------------------------------------------------


Fix it, then Ship it!





ranger-tools/scripts/create_tags_file.sh
Lines 39 (patched)
<https://reviews.apache.org/r/73758/#comment312891>

    ok. Thanks for the details.
    
    I suggest one change: instead of writting to same file, it might help to write a /tmp/ file and then move to output_file.
      ./gen_service_tags.sh cm_hive ${num_of_tags} ${initial_id} > /tmp/$$-${output_file}
      mv /tmp/$$-${output_file} ${output_file}
    
    This can help avoid plugin to load empty/partial file.


- Madhan Neethiraj


On Dec. 13, 2021, 3:53 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73758/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2021, 3:53 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-3548
>     https://issues.apache.org/jira/browse/RANGER-3548
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.
> 
> 
> Diffs
> -----
> 
>   distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
>   ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
>   ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 
> 
> 
> Diff: https://reviews.apache.org/r/73758/diff/1/
> 
> 
> Testing
> -------
> 
> - Created a file containing tagged hive entities
> - Uploaded generated file into Ranger using tagsync's file-based interface
> - Ensured that the tagged entities are downloaded to hive service
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Ramesh Mani <rm...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/#review223845
-----------------------------------------------------------


Ship it!




Ship It!

- Ramesh Mani


On Dec. 13, 2021, 3:53 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73758/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2021, 3:53 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-3548
>     https://issues.apache.org/jira/browse/RANGER-3548
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.
> 
> 
> Diffs
> -----
> 
>   distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
>   ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
>   ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 
> 
> 
> Diff: https://reviews.apache.org/r/73758/diff/1/
> 
> 
> Testing
> -------
> 
> - Created a file containing tagged hive entities
> - Uploaded generated file into Ranger using tagsync's file-based interface
> - Ensured that the tagged entities are downloaded to hive service
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/#review223849
-----------------------------------------------------------


Ship it!




Ship It!

- Madhan Neethiraj


On Dec. 13, 2021, 8:41 p.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73758/
> -----------------------------------------------------------
> 
> (Updated Dec. 13, 2021, 8:41 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-3548
>     https://issues.apache.org/jira/browse/RANGER-3548
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.
> 
> 
> Diffs
> -----
> 
>   distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
>   ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
>   ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 
> 
> 
> Diff: https://reviews.apache.org/r/73758/diff/2/
> 
> 
> Testing
> -------
> 
> - Created a file containing tagged hive entities
> - Uploaded generated file into Ranger using tagsync's file-based interface
> - Ensured that the tagged entities are downloaded to hive service
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 73758: RANGER-3548: Update performance engine test scripts

Posted by Abhay Kulkarni <ak...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73758/
-----------------------------------------------------------

(Updated Dec. 13, 2021, 8:41 p.m.)


Review request for ranger.


Changes
-------

Addressed comment


Bugs: RANGER-3548
    https://issues.apache.org/jira/browse/RANGER-3548


Repository: ranger


Description
-------

It is useful to have scripts to create a file containing large number of tagged entities that can be uploaded into Ranger using file-based interface supported by tagsync. Such scripts are useful in populating a large number of tagged entities in Ranger for testing performance/memory characteristics of Ranger admin and the services using Ranger as an authorizer.


Diffs (updated)
-----

  distro/src/main/assembly/ranger-tools.xml 1eb9104b5 
  ranger-tools/scripts/create_tags_file.sh PRE-CREATION 
  ranger-tools/scripts/gen_service_tags.sh 9a81a0a19 


Diff: https://reviews.apache.org/r/73758/diff/2/

Changes: https://reviews.apache.org/r/73758/diff/1-2/


Testing
-------

- Created a file containing tagged hive entities
- Uploaded generated file into Ranger using tagsync's file-based interface
- Ensured that the tagged entities are downloaded to hive service


Thanks,

Abhay Kulkarni