You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hawq.apache.org by yo...@apache.org on 2017/01/10 23:54:39 UTC
[48/57] [abbrv] [partial] incubator-hawq-docs git commit: HAWQ-1254
Fix/remove book branching on incubator-hawq-docs
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/bestpractices/querying_data_bestpractices.html.md.erb
----------------------------------------------------------------------
diff --git a/bestpractices/querying_data_bestpractices.html.md.erb b/bestpractices/querying_data_bestpractices.html.md.erb
deleted file mode 100644
index 3efe569..0000000
--- a/bestpractices/querying_data_bestpractices.html.md.erb
+++ /dev/null
@@ -1,43 +0,0 @@
----
-title: Best Practices for Querying Data
----
-
-To obtain the best results when querying data in HAWQ, review the best practices described in this topic.
-
-## <a id="virtual_seg_performance"></a>Factors Impacting Query Performance
-
-The number of virtual segments used for a query directly impacts the query's performance. The following factors can impact the degree of parallelism of a query:
-
-- **Cost of the query**. Small queries use fewer segments and larger queries use more segments. Some techniques used in defining resource queues can influence the number of both virtual segments and general resources allocated to queries. For more information, see [Best Practices for Using Resource Queues](managing_resources_bestpractices.html#topic_hvd_pls_wv).
-- **Available resources at query time**. If more resources are available in the resource queue, those resources will be used.
-- **Hash table and bucket number**. If the query involves only hash-distributed tables, the query's parallelism is fixed (equal to the hash table bucket number) under the following conditions:
-
- - The bucket number (bucketnum) configured for all the hash tables is the same for all tables
- - The table size for random tables is no more than 1.5 times the size allotted for the hash tables.
-
- Otherwise, the number of virtual segments depends on the query's cost: hash-distributed table queries behave like queries on randomly-distributed tables.
-
-- **Query Type**: It can be difficult to calculate resource costs for queries with some user-defined functions or for queries to external tables. With these queries, the number of virtual segments is controlled by the `hawq_rm_nvseg_perquery_limit `and `hawq_rm_nvseg_perquery_perseg_limit` parameters, as well as by the ON clause and the location list of external tables. If the query has a hash result table (e.g. `INSERT into hash_table`), the number of virtual segments must be equal to the bucket number of the resulting hash table. If the query is performed in utility mode, such as for `COPY` and `ANALYZE` operations, the virtual segment number is calculated by different policies.
-
- ***Note:*** PXF external tables use the `default_hash_table_bucket_number` parameter, not the `hawq_rm_nvseg_perquery_perseg_limit` parameter, to control the number of virtual segments.
-
-See [Query Performance](../query/query-performance.html#topic38) for more details.
-
-## <a id="id_xtk_jmq_1v"></a>Examining Query Plans to Solve Problems
-
-If a query performs poorly, examine its query plan and ask the following questions:
-
-- **Do operations in the plan take an exceptionally long time?** Look for an operation that consumes the majority of query processing time. For example, if a scan on a hash table takes longer than expected, the data locality may be low; reloading the data can increase the data locality and speed up the query. Or, adjust `enable_<operator>` parameters to see if you can force the legacy query optimizer (planner) to choose a different plan by disabling a particular query plan operator for that query.
-- **Are the optimizer's estimates close to reality?** Run `EXPLAIN ANALYZE` and see if the number of rows the optimizer estimates is close to the number of rows the query operation actually returns. If there is a large discrepancy, collect more statistics on the relevant columns.
-- **Are selective predicates applied early in the plan?** Apply the most selective filters early in the plan so fewer rows move up the plan tree. If the query plan does not correctly estimate query predicate selectivity, collect more statistics on the relevant columns. You can also try reordering the `WHERE` clause of your SQL statement.
-- **Does the optimizer choose the best join order?** When you have a query that joins multiple tables, make sure that the optimizer chooses the most selective join order. Joins that eliminate the largest number of rows should be done earlier in the plan so fewer rows move up the plan tree.
-
- If the plan is not choosing the optimal join order, set `join_collapse_limit=1` and use explicit `JOIN` syntax in your SQL statement to force the legacy query optimizer (planner) to the specified join order. You can also collect more statistics on the relevant join columns.
-
-- **Does the optimizer selectively scan partitioned tables?** If you use table partitioning, is the optimizer selectively scanning only the child tables required to satisfy the query predicates? Scans of the parent tables should return 0 rows since the parent tables do not contain any data. See [Verifying Your Partition Strategy](../ddl/ddl-partition.html#topic74) for an example of a query plan that shows a selective partition scan.
-- **Does the optimizer choose hash aggregate and hash join operations where applicable?** Hash operations are typically much faster than other types of joins or aggregations. Row comparison and sorting is done in memory rather than reading/writing from disk. To enable the query optimizer to choose hash operations, there must be sufficient memory available to hold the estimated number of rows. Run an `EXPLAIN ANALYZE` for the query to show which plan operations spilled to disk, how much work memory they used, and how much memory was required to avoid spilling to disk. For example:
-
- `Work_mem used: 23430K bytes avg, 23430K bytes max (seg0). Work_mem wanted: 33649K bytes avg, 33649K bytes max (seg0) to lessen workfile I/O affecting 2 workers.`
-
- **Note:** The "bytes wanted" (*work\_mem* property) is based on the amount of data written to work files and is not exact. This property is not configurable. Use resource queues to manage memory use. For more information on resource queues, see [Configuring Resource Management](../resourcemgmt/ConfigureResourceManagement.html) and [Working with Hierarchical Resource Queues](../resourcemgmt/ResourceQueues.html).
-
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/bestpractices/secure_bestpractices.html.md.erb
----------------------------------------------------------------------
diff --git a/bestpractices/secure_bestpractices.html.md.erb b/bestpractices/secure_bestpractices.html.md.erb
deleted file mode 100644
index 04c5343..0000000
--- a/bestpractices/secure_bestpractices.html.md.erb
+++ /dev/null
@@ -1,11 +0,0 @@
----
-title: Best Practices for Securing HAWQ
----
-
-To secure your HAWQ deployment, review the recommendations listed in this topic.
-
-- Set up SSL to encrypt your client server communication channel. See [Encrypting Client/Server Connections](../clientaccess/client_auth.html#topic5).
-- Configure `pg_hba.conf` only on HAWQ master. Do not configure it on segments.
- **Note:** For a more secure system, consider removing all connections that use trust authentication from your master `pg_hba.conf`. Trust authentication means the role is granted access without any authentication, therefore bypassing all security. Replace trust entries with ident authentication if your system has an ident service available.
-
-
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/Gemfile
----------------------------------------------------------------------
diff --git a/book/Gemfile b/book/Gemfile
new file mode 100644
index 0000000..f66d333
--- /dev/null
+++ b/book/Gemfile
@@ -0,0 +1,5 @@
+source "https://rubygems.org"
+
+gem 'bookbindery'
+
+gem 'libv8', '3.16.14.7'
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/Gemfile.lock
----------------------------------------------------------------------
diff --git a/book/Gemfile.lock b/book/Gemfile.lock
new file mode 100644
index 0000000..3c483c0
--- /dev/null
+++ b/book/Gemfile.lock
@@ -0,0 +1,203 @@
+GEM
+ remote: https://rubygems.org/
+ specs:
+ activesupport (4.2.7.1)
+ i18n (~> 0.7)
+ json (~> 1.7, >= 1.7.7)
+ minitest (~> 5.1)
+ thread_safe (~> 0.3, >= 0.3.4)
+ tzinfo (~> 1.1)
+ addressable (2.4.0)
+ ansi (1.5.0)
+ bookbindery (9.12.0)
+ ansi (~> 1.4)
+ css_parser
+ elasticsearch
+ fog-aws (~> 0.7.1)
+ font-awesome-sass
+ git (~> 1.2.8)
+ middleman (~> 3.4.0)
+ middleman-livereload (~> 3.4.3)
+ middleman-syntax (~> 2.0)
+ nokogiri (= 1.6.7.2)
+ puma
+ rack-rewrite
+ redcarpet (~> 3.2.3)
+ rouge (!= 1.9.1)
+ therubyracer
+ thor
+ builder (3.2.2)
+ capybara (2.4.4)
+ mime-types (>= 1.16)
+ nokogiri (>= 1.3.3)
+ rack (>= 1.0.0)
+ rack-test (>= 0.5.4)
+ xpath (~> 2.0)
+ chunky_png (1.3.6)
+ coffee-script (2.4.1)
+ coffee-script-source
+ execjs
+ coffee-script-source (1.10.0)
+ compass (1.0.3)
+ chunky_png (~> 1.2)
+ compass-core (~> 1.0.2)
+ compass-import-once (~> 1.0.5)
+ rb-fsevent (>= 0.9.3)
+ rb-inotify (>= 0.9)
+ sass (>= 3.3.13, < 3.5)
+ compass-core (1.0.3)
+ multi_json (~> 1.0)
+ sass (>= 3.3.0, < 3.5)
+ compass-import-once (1.0.5)
+ sass (>= 3.2, < 3.5)
+ css_parser (1.4.5)
+ addressable
+ elasticsearch (2.0.0)
+ elasticsearch-api (= 2.0.0)
+ elasticsearch-transport (= 2.0.0)
+ elasticsearch-api (2.0.0)
+ multi_json
+ elasticsearch-transport (2.0.0)
+ faraday
+ multi_json
+ em-websocket (0.5.1)
+ eventmachine (>= 0.12.9)
+ http_parser.rb (~> 0.6.0)
+ erubis (2.7.0)
+ eventmachine (1.2.0.1)
+ excon (0.51.0)
+ execjs (2.7.0)
+ faraday (0.9.2)
+ multipart-post (>= 1.2, < 3)
+ ffi (1.9.14)
+ fog-aws (0.7.6)
+ fog-core (~> 1.27)
+ fog-json (~> 1.0)
+ fog-xml (~> 0.1)
+ ipaddress (~> 0.8)
+ fog-core (1.42.0)
+ builder
+ excon (~> 0.49)
+ formatador (~> 0.2)
+ fog-json (1.0.2)
+ fog-core (~> 1.0)
+ multi_json (~> 1.10)
+ fog-xml (0.1.2)
+ fog-core
+ nokogiri (~> 1.5, >= 1.5.11)
+ font-awesome-sass (4.6.2)
+ sass (>= 3.2)
+ formatador (0.2.5)
+ git (1.2.9.1)
+ haml (4.0.7)
+ tilt
+ hike (1.2.3)
+ hooks (0.4.1)
+ uber (~> 0.0.14)
+ http_parser.rb (0.6.0)
+ i18n (0.7.0)
+ ipaddress (0.8.3)
+ json (1.8.3)
+ kramdown (1.12.0)
+ libv8 (3.16.14.7)
+ listen (3.0.8)
+ rb-fsevent (~> 0.9, >= 0.9.4)
+ rb-inotify (~> 0.9, >= 0.9.7)
+ middleman (3.4.1)
+ coffee-script (~> 2.2)
+ compass (>= 1.0.0, < 2.0.0)
+ compass-import-once (= 1.0.5)
+ execjs (~> 2.0)
+ haml (>= 4.0.5)
+ kramdown (~> 1.2)
+ middleman-core (= 3.4.1)
+ middleman-sprockets (>= 3.1.2)
+ sass (>= 3.4.0, < 4.0)
+ uglifier (~> 2.5)
+ middleman-core (3.4.1)
+ activesupport (~> 4.1)
+ bundler (~> 1.1)
+ capybara (~> 2.4.4)
+ erubis
+ hooks (~> 0.3)
+ i18n (~> 0.7.0)
+ listen (~> 3.0.3)
+ padrino-helpers (~> 0.12.3)
+ rack (>= 1.4.5, < 2.0)
+ thor (>= 0.15.2, < 2.0)
+ tilt (~> 1.4.1, < 2.0)
+ middleman-livereload (3.4.6)
+ em-websocket (~> 0.5.1)
+ middleman-core (>= 3.3)
+ rack-livereload (~> 0.3.15)
+ middleman-sprockets (3.4.2)
+ middleman-core (>= 3.3)
+ sprockets (~> 2.12.1)
+ sprockets-helpers (~> 1.1.0)
+ sprockets-sass (~> 1.3.0)
+ middleman-syntax (2.1.0)
+ middleman-core (>= 3.2)
+ rouge (~> 1.0)
+ mime-types (3.1)
+ mime-types-data (~> 3.2015)
+ mime-types-data (3.2016.0521)
+ mini_portile2 (2.0.0)
+ minitest (5.9.0)
+ multi_json (1.12.1)
+ multipart-post (2.0.0)
+ nokogiri (1.6.7.2)
+ mini_portile2 (~> 2.0.0.rc2)
+ padrino-helpers (0.12.8)
+ i18n (~> 0.6, >= 0.6.7)
+ padrino-support (= 0.12.8)
+ tilt (~> 1.4.1)
+ padrino-support (0.12.8)
+ activesupport (>= 3.1)
+ puma (3.6.0)
+ rack (1.6.4)
+ rack-livereload (0.3.16)
+ rack
+ rack-rewrite (1.5.1)
+ rack-test (0.6.3)
+ rack (>= 1.0)
+ rb-fsevent (0.9.7)
+ rb-inotify (0.9.7)
+ ffi (>= 0.5.0)
+ redcarpet (3.2.3)
+ ref (2.0.0)
+ rouge (1.11.1)
+ sass (3.4.22)
+ sprockets (2.12.4)
+ hike (~> 1.2)
+ multi_json (~> 1.0)
+ rack (~> 1.0)
+ tilt (~> 1.1, != 1.3.0)
+ sprockets-helpers (1.1.0)
+ sprockets (~> 2.0)
+ sprockets-sass (1.3.1)
+ sprockets (~> 2.0)
+ tilt (~> 1.1)
+ therubyracer (0.12.2)
+ libv8 (~> 3.16.14.0)
+ ref
+ thor (0.19.1)
+ thread_safe (0.3.5)
+ tilt (1.4.1)
+ tzinfo (1.2.2)
+ thread_safe (~> 0.1)
+ uber (0.0.15)
+ uglifier (2.7.2)
+ execjs (>= 0.3.0)
+ json (>= 1.8.0)
+ xpath (2.0.0)
+ nokogiri (~> 1.3)
+
+PLATFORMS
+ ruby
+
+DEPENDENCIES
+ bookbindery
+ libv8 (= 3.16.14.7)
+
+BUNDLED WITH
+ 1.11.2
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/config.yml
----------------------------------------------------------------------
diff --git a/book/config.yml b/book/config.yml
new file mode 100644
index 0000000..22d2799
--- /dev/null
+++ b/book/config.yml
@@ -0,0 +1,21 @@
+book_repo: incubator-hawq-docs/book
+
+public_host: http://localhost:9292/
+
+sections:
+ - repository:
+ name: incubator-hawq-docs/markdown
+ directory: docs/userguide/2.1.0.0-incubating
+ subnav_template: apache-hawq-nav
+
+template_variables:
+ use_global_header: true
+ global_header_product_href: https://github.com/apache/incubator-hawq
+ global_header_product_link_text: Downloads
+ support_url: http://mail-archives.apache.org/mod_mbox/incubator-hawq-dev/
+ product_url: http://hawq.incubator.apache.org/
+ book_title: Apache HAWQ (incubating) Documentation
+ support_link: <a href="https://issues.apache.org/jira/browse/HAWQ" target="_blank">Issues</a>
+ support_call_to_action: <a href="https://issues.apache.org/jira/browse/HAWQ" target="_blank">Need Help?</a>
+ product_link: <div class="header-item"><a href="http://hawq.incubator.apache.org/">Back to Apache HAWQ Page</a></div>
+ book_title_short: Apache HAWQ (Incubating) Docs
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/images/favicon.ico
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/images/favicon.ico b/book/master_middleman/source/images/favicon.ico
new file mode 100644
index 0000000..b2c3a0c
Binary files /dev/null and b/book/master_middleman/source/images/favicon.ico differ
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/book.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/book.js b/book/master_middleman/source/javascripts/book.js
new file mode 100644
index 0000000..90879c4
--- /dev/null
+++ b/book/master_middleman/source/javascripts/book.js
@@ -0,0 +1,16 @@
+// Declare your book-specific javascript overrides in this file.
+//= require 'waypoints/waypoint'
+//= require 'waypoints/context'
+//= require 'waypoints/group'
+//= require 'waypoints/noframeworkAdapter'
+//= require 'waypoints/sticky'
+
+window.onload = function() {
+ Bookbinder.boot();
+ var sticky = new Waypoint.Sticky({
+ element: document.querySelector('#js-to-top'),
+ wrapper: '<div class="sticky-wrapper" />',
+ stuckClass: 'sticky',
+ offset: 100
+ });
+}
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/waypoints/context.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/waypoints/context.js b/book/master_middleman/source/javascripts/waypoints/context.js
new file mode 100644
index 0000000..5e3551b
--- /dev/null
+++ b/book/master_middleman/source/javascripts/waypoints/context.js
@@ -0,0 +1,300 @@
+(function() {
+ 'use strict'
+
+ function requestAnimationFrameShim(callback) {
+ window.setTimeout(callback, 1000 / 60)
+ }
+
+ var keyCounter = 0
+ var contexts = {}
+ var Waypoint = window.Waypoint
+ var oldWindowLoad = window.onload
+
+ /* http://imakewebthings.com/waypoints/api/context */
+ function Context(element) {
+ this.element = element
+ this.Adapter = Waypoint.Adapter
+ this.adapter = new this.Adapter(element)
+ this.key = 'waypoint-context-' + keyCounter
+ this.didScroll = false
+ this.didResize = false
+ this.oldScroll = {
+ x: this.adapter.scrollLeft(),
+ y: this.adapter.scrollTop()
+ }
+ this.waypoints = {
+ vertical: {},
+ horizontal: {}
+ }
+
+ element.waypointContextKey = this.key
+ contexts[element.waypointContextKey] = this
+ keyCounter += 1
+
+ this.createThrottledScrollHandler()
+ this.createThrottledResizeHandler()
+ }
+
+ /* Private */
+ Context.prototype.add = function(waypoint) {
+ var axis = waypoint.options.horizontal ? 'horizontal' : 'vertical'
+ this.waypoints[axis][waypoint.key] = waypoint
+ this.refresh()
+ }
+
+ /* Private */
+ Context.prototype.checkEmpty = function() {
+ var horizontalEmpty = this.Adapter.isEmptyObject(this.waypoints.horizontal)
+ var verticalEmpty = this.Adapter.isEmptyObject(this.waypoints.vertical)
+ if (horizontalEmpty && verticalEmpty) {
+ this.adapter.off('.waypoints')
+ delete contexts[this.key]
+ }
+ }
+
+ /* Private */
+ Context.prototype.createThrottledResizeHandler = function() {
+ var self = this
+
+ function resizeHandler() {
+ self.handleResize()
+ self.didResize = false
+ }
+
+ this.adapter.on('resize.waypoints', function() {
+ if (!self.didResize) {
+ self.didResize = true
+ Waypoint.requestAnimationFrame(resizeHandler)
+ }
+ })
+ }
+
+ /* Private */
+ Context.prototype.createThrottledScrollHandler = function() {
+ var self = this
+ function scrollHandler() {
+ self.handleScroll()
+ self.didScroll = false
+ }
+
+ this.adapter.on('scroll.waypoints', function() {
+ if (!self.didScroll || Waypoint.isTouch) {
+ self.didScroll = true
+ Waypoint.requestAnimationFrame(scrollHandler)
+ }
+ })
+ }
+
+ /* Private */
+ Context.prototype.handleResize = function() {
+ Waypoint.Context.refreshAll()
+ }
+
+ /* Private */
+ Context.prototype.handleScroll = function() {
+ var triggeredGroups = {}
+ var axes = {
+ horizontal: {
+ newScroll: this.adapter.scrollLeft(),
+ oldScroll: this.oldScroll.x,
+ forward: 'right',
+ backward: 'left'
+ },
+ vertical: {
+ newScroll: this.adapter.scrollTop(),
+ oldScroll: this.oldScroll.y,
+ forward: 'down',
+ backward: 'up'
+ }
+ }
+
+ for (var axisKey in axes) {
+ var axis = axes[axisKey]
+ var isForward = axis.newScroll > axis.oldScroll
+ var direction = isForward ? axis.forward : axis.backward
+
+ for (var waypointKey in this.waypoints[axisKey]) {
+ var waypoint = this.waypoints[axisKey][waypointKey]
+ var wasBeforeTriggerPoint = axis.oldScroll < waypoint.triggerPoint
+ var nowAfterTriggerPoint = axis.newScroll >= waypoint.triggerPoint
+ var crossedForward = wasBeforeTriggerPoint && nowAfterTriggerPoint
+ var crossedBackward = !wasBeforeTriggerPoint && !nowAfterTriggerPoint
+ if (crossedForward || crossedBackward) {
+ waypoint.queueTrigger(direction)
+ triggeredGroups[waypoint.group.id] = waypoint.group
+ }
+ }
+ }
+
+ for (var groupKey in triggeredGroups) {
+ triggeredGroups[groupKey].flushTriggers()
+ }
+
+ this.oldScroll = {
+ x: axes.horizontal.newScroll,
+ y: axes.vertical.newScroll
+ }
+ }
+
+ /* Private */
+ Context.prototype.innerHeight = function() {
+ /*eslint-disable eqeqeq */
+ if (this.element == this.element.window) {
+ return Waypoint.viewportHeight()
+ }
+ /*eslint-enable eqeqeq */
+ return this.adapter.innerHeight()
+ }
+
+ /* Private */
+ Context.prototype.remove = function(waypoint) {
+ delete this.waypoints[waypoint.axis][waypoint.key]
+ this.checkEmpty()
+ }
+
+ /* Private */
+ Context.prototype.innerWidth = function() {
+ /*eslint-disable eqeqeq */
+ if (this.element == this.element.window) {
+ return Waypoint.viewportWidth()
+ }
+ /*eslint-enable eqeqeq */
+ return this.adapter.innerWidth()
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/context-destroy */
+ Context.prototype.destroy = function() {
+ var allWaypoints = []
+ for (var axis in this.waypoints) {
+ for (var waypointKey in this.waypoints[axis]) {
+ allWaypoints.push(this.waypoints[axis][waypointKey])
+ }
+ }
+ for (var i = 0, end = allWaypoints.length; i < end; i++) {
+ allWaypoints[i].destroy()
+ }
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/context-refresh */
+ Context.prototype.refresh = function() {
+ /*eslint-disable eqeqeq */
+ var isWindow = this.element == this.element.window
+ /*eslint-enable eqeqeq */
+ var contextOffset = isWindow ? undefined : this.adapter.offset()
+ var triggeredGroups = {}
+ var axes
+
+ this.handleScroll()
+ axes = {
+ horizontal: {
+ contextOffset: isWindow ? 0 : contextOffset.left,
+ contextScroll: isWindow ? 0 : this.oldScroll.x,
+ contextDimension: this.innerWidth(),
+ oldScroll: this.oldScroll.x,
+ forward: 'right',
+ backward: 'left',
+ offsetProp: 'left'
+ },
+ vertical: {
+ contextOffset: isWindow ? 0 : contextOffset.top,
+ contextScroll: isWindow ? 0 : this.oldScroll.y,
+ contextDimension: this.innerHeight(),
+ oldScroll: this.oldScroll.y,
+ forward: 'down',
+ backward: 'up',
+ offsetProp: 'top'
+ }
+ }
+
+ for (var axisKey in axes) {
+ var axis = axes[axisKey]
+ for (var waypointKey in this.waypoints[axisKey]) {
+ var waypoint = this.waypoints[axisKey][waypointKey]
+ var adjustment = waypoint.options.offset
+ var oldTriggerPoint = waypoint.triggerPoint
+ var elementOffset = 0
+ var freshWaypoint = oldTriggerPoint == null
+ var contextModifier, wasBeforeScroll, nowAfterScroll
+ var triggeredBackward, triggeredForward
+
+ if (waypoint.element !== waypoint.element.window) {
+ elementOffset = waypoint.adapter.offset()[axis.offsetProp]
+ }
+
+ if (typeof adjustment === 'function') {
+ adjustment = adjustment.apply(waypoint)
+ }
+ else if (typeof adjustment === 'string') {
+ adjustment = parseFloat(adjustment)
+ if (waypoint.options.offset.indexOf('%') > - 1) {
+ adjustment = Math.ceil(axis.contextDimension * adjustment / 100)
+ }
+ }
+
+ contextModifier = axis.contextScroll - axis.contextOffset
+ waypoint.triggerPoint = elementOffset + contextModifier - adjustment
+ wasBeforeScroll = oldTriggerPoint < axis.oldScroll
+ nowAfterScroll = waypoint.triggerPoint >= axis.oldScroll
+ triggeredBackward = wasBeforeScroll && nowAfterScroll
+ triggeredForward = !wasBeforeScroll && !nowAfterScroll
+
+ if (!freshWaypoint && triggeredBackward) {
+ waypoint.queueTrigger(axis.backward)
+ triggeredGroups[waypoint.group.id] = waypoint.group
+ }
+ else if (!freshWaypoint && triggeredForward) {
+ waypoint.queueTrigger(axis.forward)
+ triggeredGroups[waypoint.group.id] = waypoint.group
+ }
+ else if (freshWaypoint && axis.oldScroll >= waypoint.triggerPoint) {
+ waypoint.queueTrigger(axis.forward)
+ triggeredGroups[waypoint.group.id] = waypoint.group
+ }
+ }
+ }
+
+ Waypoint.requestAnimationFrame(function() {
+ for (var groupKey in triggeredGroups) {
+ triggeredGroups[groupKey].flushTriggers()
+ }
+ })
+
+ return this
+ }
+
+ /* Private */
+ Context.findOrCreateByElement = function(element) {
+ return Context.findByElement(element) || new Context(element)
+ }
+
+ /* Private */
+ Context.refreshAll = function() {
+ for (var contextId in contexts) {
+ contexts[contextId].refresh()
+ }
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/context-find-by-element */
+ Context.findByElement = function(element) {
+ return contexts[element.waypointContextKey]
+ }
+
+ window.onload = function() {
+ if (oldWindowLoad) {
+ oldWindowLoad()
+ }
+ Context.refreshAll()
+ }
+
+ Waypoint.requestAnimationFrame = function(callback) {
+ var requestFn = window.requestAnimationFrame ||
+ window.mozRequestAnimationFrame ||
+ window.webkitRequestAnimationFrame ||
+ requestAnimationFrameShim
+ requestFn.call(window, callback)
+ }
+ Waypoint.Context = Context
+}())
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/waypoints/group.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/waypoints/group.js b/book/master_middleman/source/javascripts/waypoints/group.js
new file mode 100644
index 0000000..57c3038
--- /dev/null
+++ b/book/master_middleman/source/javascripts/waypoints/group.js
@@ -0,0 +1,105 @@
+(function() {
+ 'use strict'
+
+ function byTriggerPoint(a, b) {
+ return a.triggerPoint - b.triggerPoint
+ }
+
+ function byReverseTriggerPoint(a, b) {
+ return b.triggerPoint - a.triggerPoint
+ }
+
+ var groups = {
+ vertical: {},
+ horizontal: {}
+ }
+ var Waypoint = window.Waypoint
+
+ /* http://imakewebthings.com/waypoints/api/group */
+ function Group(options) {
+ this.name = options.name
+ this.axis = options.axis
+ this.id = this.name + '-' + this.axis
+ this.waypoints = []
+ this.clearTriggerQueues()
+ groups[this.axis][this.name] = this
+ }
+
+ /* Private */
+ Group.prototype.add = function(waypoint) {
+ this.waypoints.push(waypoint)
+ }
+
+ /* Private */
+ Group.prototype.clearTriggerQueues = function() {
+ this.triggerQueues = {
+ up: [],
+ down: [],
+ left: [],
+ right: []
+ }
+ }
+
+ /* Private */
+ Group.prototype.flushTriggers = function() {
+ for (var direction in this.triggerQueues) {
+ var waypoints = this.triggerQueues[direction]
+ var reverse = direction === 'up' || direction === 'left'
+ waypoints.sort(reverse ? byReverseTriggerPoint : byTriggerPoint)
+ for (var i = 0, end = waypoints.length; i < end; i += 1) {
+ var waypoint = waypoints[i]
+ if (waypoint.options.continuous || i === waypoints.length - 1) {
+ waypoint.trigger([direction])
+ }
+ }
+ }
+ this.clearTriggerQueues()
+ }
+
+ /* Private */
+ Group.prototype.next = function(waypoint) {
+ this.waypoints.sort(byTriggerPoint)
+ var index = Waypoint.Adapter.inArray(waypoint, this.waypoints)
+ var isLast = index === this.waypoints.length - 1
+ return isLast ? null : this.waypoints[index + 1]
+ }
+
+ /* Private */
+ Group.prototype.previous = function(waypoint) {
+ this.waypoints.sort(byTriggerPoint)
+ var index = Waypoint.Adapter.inArray(waypoint, this.waypoints)
+ return index ? this.waypoints[index - 1] : null
+ }
+
+ /* Private */
+ Group.prototype.queueTrigger = function(waypoint, direction) {
+ this.triggerQueues[direction].push(waypoint)
+ }
+
+ /* Private */
+ Group.prototype.remove = function(waypoint) {
+ var index = Waypoint.Adapter.inArray(waypoint, this.waypoints)
+ if (index > -1) {
+ this.waypoints.splice(index, 1)
+ }
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/first */
+ Group.prototype.first = function() {
+ return this.waypoints[0]
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/last */
+ Group.prototype.last = function() {
+ return this.waypoints[this.waypoints.length - 1]
+ }
+
+ /* Private */
+ Group.findOrCreate = function(options) {
+ return groups[options.axis][options.name] || new Group(options)
+ }
+
+ Waypoint.Group = Group
+}())
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js b/book/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js
new file mode 100644
index 0000000..99abcb5
--- /dev/null
+++ b/book/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js
@@ -0,0 +1,213 @@
+(function() {
+ 'use strict'
+
+ var Waypoint = window.Waypoint
+
+ function isWindow(element) {
+ return element === element.window
+ }
+
+ function getWindow(element) {
+ if (isWindow(element)) {
+ return element
+ }
+ return element.defaultView
+ }
+
+ function classNameRegExp(className) {
+ return new RegExp("\\b" + className + "\\b");
+ }
+
+ function NoFrameworkAdapter(element) {
+ this.element = element
+ this.handlers = {}
+ }
+
+ NoFrameworkAdapter.prototype.innerHeight = function() {
+ var isWin = isWindow(this.element)
+ return isWin ? this.element.innerHeight : this.element.clientHeight
+ }
+
+ NoFrameworkAdapter.prototype.innerWidth = function() {
+ var isWin = isWindow(this.element)
+ return isWin ? this.element.innerWidth : this.element.clientWidth
+ }
+
+ NoFrameworkAdapter.prototype.off = function(event, handler) {
+ function removeListeners(element, listeners, handler) {
+ for (var i = 0, end = listeners.length - 1; i < end; i++) {
+ var listener = listeners[i]
+ if (!handler || handler === listener) {
+ element.removeEventListener(listener)
+ }
+ }
+ }
+
+ var eventParts = event.split('.')
+ var eventType = eventParts[0]
+ var namespace = eventParts[1]
+ var element = this.element
+
+ if (namespace && this.handlers[namespace] && eventType) {
+ removeListeners(element, this.handlers[namespace][eventType], handler)
+ this.handlers[namespace][eventType] = []
+ }
+ else if (eventType) {
+ for (var ns in this.handlers) {
+ removeListeners(element, this.handlers[ns][eventType] || [], handler)
+ this.handlers[ns][eventType] = []
+ }
+ }
+ else if (namespace && this.handlers[namespace]) {
+ for (var type in this.handlers[namespace]) {
+ removeListeners(element, this.handlers[namespace][type], handler)
+ }
+ this.handlers[namespace] = {}
+ }
+ }
+
+ /* Adapted from jQuery 1.x offset() */
+ NoFrameworkAdapter.prototype.offset = function() {
+ if (!this.element.ownerDocument) {
+ return null
+ }
+
+ var documentElement = this.element.ownerDocument.documentElement
+ var win = getWindow(this.element.ownerDocument)
+ var rect = {
+ top: 0,
+ left: 0
+ }
+
+ if (this.element.getBoundingClientRect) {
+ rect = this.element.getBoundingClientRect()
+ }
+
+ return {
+ top: rect.top + win.pageYOffset - documentElement.clientTop,
+ left: rect.left + win.pageXOffset - documentElement.clientLeft
+ }
+ }
+
+ NoFrameworkAdapter.prototype.on = function(event, handler) {
+ var eventParts = event.split('.')
+ var eventType = eventParts[0]
+ var namespace = eventParts[1] || '__default'
+ var nsHandlers = this.handlers[namespace] = this.handlers[namespace] || {}
+ var nsTypeList = nsHandlers[eventType] = nsHandlers[eventType] || []
+
+ nsTypeList.push(handler)
+ this.element.addEventListener(eventType, handler)
+ }
+
+ NoFrameworkAdapter.prototype.outerHeight = function(includeMargin) {
+ var height = this.innerHeight()
+ var computedStyle
+
+ if (includeMargin && !isWindow(this.element)) {
+ computedStyle = window.getComputedStyle(this.element)
+ height += parseInt(computedStyle.marginTop, 10)
+ height += parseInt(computedStyle.marginBottom, 10)
+ }
+
+ return height
+ }
+
+ NoFrameworkAdapter.prototype.outerWidth = function(includeMargin) {
+ var width = this.innerWidth()
+ var computedStyle
+
+ if (includeMargin && !isWindow(this.element)) {
+ computedStyle = window.getComputedStyle(this.element)
+ width += parseInt(computedStyle.marginLeft, 10)
+ width += parseInt(computedStyle.marginRight, 10)
+ }
+
+ return width
+ }
+
+ NoFrameworkAdapter.prototype.scrollLeft = function() {
+ var win = getWindow(this.element)
+ return win ? win.pageXOffset : this.element.scrollLeft
+ }
+
+ NoFrameworkAdapter.prototype.scrollTop = function() {
+ var win = getWindow(this.element)
+ return win ? win.pageYOffset : this.element.scrollTop
+ }
+
+ NoFrameworkAdapter.prototype.height = function(newHeight) {
+ this.element.style.height = newHeight;
+ }
+
+ NoFrameworkAdapter.prototype.removeClass = function(className) {
+ this.element.className = this.element.className.replace(classNameRegExp(className), '');
+ }
+
+ NoFrameworkAdapter.prototype.toggleClass = function(className, addClass) {
+ var check = classNameRegExp(className);
+ if (check.test(this.element.className)) {
+ if (!addClass) {
+ this.removeClass(className);
+ }
+ } else {
+ this.element.className += ' ' + className;
+ }
+ }
+
+ NoFrameworkAdapter.prototype.parent = function() {
+ return new NoFrameworkAdapter(this.element.parentNode);
+ }
+
+ NoFrameworkAdapter.prototype.wrap = function(wrapper) {
+ this.element.insertAdjacentHTML('beforebegin', wrapper)
+ var wrapperNode = this.element.previousSibling
+ this.element.parentNode.removeChild(this.element)
+ wrapperNode.appendChild(this.element)
+ }
+
+ NoFrameworkAdapter.extend = function() {
+ var args = Array.prototype.slice.call(arguments)
+
+ function merge(target, obj) {
+ if (typeof target === 'object' && typeof obj === 'object') {
+ for (var key in obj) {
+ if (obj.hasOwnProperty(key)) {
+ target[key] = obj[key]
+ }
+ }
+ }
+
+ return target
+ }
+
+ for (var i = 1, end = args.length; i < end; i++) {
+ merge(args[0], args[i])
+ }
+ return args[0]
+ }
+
+ NoFrameworkAdapter.inArray = function(element, array, i) {
+ return array == null ? -1 : array.indexOf(element, i)
+ }
+
+ NoFrameworkAdapter.isEmptyObject = function(obj) {
+ /* eslint no-unused-vars: 0 */
+ for (var name in obj) {
+ return false
+ }
+ return true
+ }
+
+ NoFrameworkAdapter.proxy = function(func, obj) {
+ return function() {
+ return func.apply(obj, arguments);
+ }
+ }
+
+ Waypoint.adapters.push({
+ name: 'noframework',
+ Adapter: NoFrameworkAdapter
+ })
+ Waypoint.Adapter = NoFrameworkAdapter
+}())
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/waypoints/sticky.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/waypoints/sticky.js b/book/master_middleman/source/javascripts/waypoints/sticky.js
new file mode 100644
index 0000000..569fcdb
--- /dev/null
+++ b/book/master_middleman/source/javascripts/waypoints/sticky.js
@@ -0,0 +1,63 @@
+(function() {
+ 'use strict'
+
+ var Waypoint = window.Waypoint;
+ var adapter = Waypoint.Adapter;
+
+ /* http://imakewebthings.com/waypoints/shortcuts/sticky-elements */
+ function Sticky(options) {
+ this.options = adapter.extend({}, Waypoint.defaults, Sticky.defaults, options)
+ this.element = this.options.element
+ this.$element = new adapter(this.element)
+ this.createWrapper()
+ this.createWaypoint()
+ }
+
+ /* Private */
+ Sticky.prototype.createWaypoint = function() {
+ var originalHandler = this.options.handler
+
+ this.waypoint = new Waypoint(adapter.extend({}, this.options, {
+ element: this.wrapper,
+ handler: adapter.proxy(function(direction) {
+ var shouldBeStuck = this.options.direction.indexOf(direction) > -1
+ var wrapperHeight = shouldBeStuck ? this.$element.outerHeight(true) : ''
+
+ this.$wrapper.height(wrapperHeight)
+ this.$element.toggleClass(this.options.stuckClass, shouldBeStuck)
+
+ if (originalHandler) {
+ originalHandler.call(this, direction)
+ }
+ }, this)
+ }))
+ }
+
+ /* Private */
+ Sticky.prototype.createWrapper = function() {
+ if (this.options.wrapper) {
+ this.$element.wrap(this.options.wrapper)
+ }
+ this.$wrapper = this.$element.parent()
+ this.wrapper = this.$wrapper.element
+ }
+
+ /* Public */
+ Sticky.prototype.destroy = function() {
+ if (this.$element.parent().element === this.wrapper) {
+ this.waypoint.destroy()
+ this.$element.removeClass(this.options.stuckClass)
+ if (this.options.wrapper) {
+ this.$element.unwrap()
+ }
+ }
+ }
+
+ Sticky.defaults = {
+ wrapper: '<div class="sticky-wrapper" />',
+ stuckClass: 'stuck',
+ direction: 'down right'
+ }
+
+ Waypoint.Sticky = Sticky
+}())
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/javascripts/waypoints/waypoint.js
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/javascripts/waypoints/waypoint.js b/book/master_middleman/source/javascripts/waypoints/waypoint.js
new file mode 100644
index 0000000..7f76f1d
--- /dev/null
+++ b/book/master_middleman/source/javascripts/waypoints/waypoint.js
@@ -0,0 +1,160 @@
+(function() {
+ 'use strict'
+
+ var keyCounter = 0
+ var allWaypoints = {}
+
+ /* http://imakewebthings.com/waypoints/api/waypoint */
+ function Waypoint(options) {
+ if (!options) {
+ throw new Error('No options passed to Waypoint constructor')
+ }
+ if (!options.element) {
+ throw new Error('No element option passed to Waypoint constructor')
+ }
+ if (!options.handler) {
+ throw new Error('No handler option passed to Waypoint constructor')
+ }
+
+ this.key = 'waypoint-' + keyCounter
+ this.options = Waypoint.Adapter.extend({}, Waypoint.defaults, options)
+ this.element = this.options.element
+ this.adapter = new Waypoint.Adapter(this.element)
+ this.callback = options.handler
+ this.axis = this.options.horizontal ? 'horizontal' : 'vertical'
+ this.enabled = this.options.enabled
+ this.triggerPoint = null
+ this.group = Waypoint.Group.findOrCreate({
+ name: this.options.group,
+ axis: this.axis
+ })
+ this.context = Waypoint.Context.findOrCreateByElement(this.options.context)
+
+ if (Waypoint.offsetAliases[this.options.offset]) {
+ this.options.offset = Waypoint.offsetAliases[this.options.offset]
+ }
+ this.group.add(this)
+ this.context.add(this)
+ allWaypoints[this.key] = this
+ keyCounter += 1
+ }
+
+ /* Private */
+ Waypoint.prototype.queueTrigger = function(direction) {
+ this.group.queueTrigger(this, direction)
+ }
+
+ /* Private */
+ Waypoint.prototype.trigger = function(args) {
+ if (!this.enabled) {
+ return
+ }
+ if (this.callback) {
+ this.callback.apply(this, args)
+ }
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/destroy */
+ Waypoint.prototype.destroy = function() {
+ this.context.remove(this)
+ this.group.remove(this)
+ delete allWaypoints[this.key]
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/disable */
+ Waypoint.prototype.disable = function() {
+ this.enabled = false
+ return this
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/enable */
+ Waypoint.prototype.enable = function() {
+ this.context.refresh()
+ this.enabled = true
+ return this
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/next */
+ Waypoint.prototype.next = function() {
+ return this.group.next(this)
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/previous */
+ Waypoint.prototype.previous = function() {
+ return this.group.previous(this)
+ }
+
+ /* Private */
+ Waypoint.invokeAll = function(method) {
+ var allWaypointsArray = []
+ for (var waypointKey in allWaypoints) {
+ allWaypointsArray.push(allWaypoints[waypointKey])
+ }
+ for (var i = 0, end = allWaypointsArray.length; i < end; i++) {
+ allWaypointsArray[i][method]()
+ }
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/destroy-all */
+ Waypoint.destroyAll = function() {
+ Waypoint.invokeAll('destroy')
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/disable-all */
+ Waypoint.disableAll = function() {
+ Waypoint.invokeAll('disable')
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/enable-all */
+ Waypoint.enableAll = function() {
+ Waypoint.invokeAll('enable')
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/refresh-all */
+ Waypoint.refreshAll = function() {
+ Waypoint.Context.refreshAll()
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/viewport-height */
+ Waypoint.viewportHeight = function() {
+ return window.innerHeight || document.documentElement.clientHeight
+ }
+
+ /* Public */
+ /* http://imakewebthings.com/waypoints/api/viewport-width */
+ Waypoint.viewportWidth = function() {
+ return document.documentElement.clientWidth
+ }
+
+ Waypoint.adapters = []
+
+ Waypoint.defaults = {
+ context: window,
+ continuous: true,
+ enabled: true,
+ group: 'default',
+ horizontal: false,
+ offset: 0
+ }
+
+ Waypoint.offsetAliases = {
+ 'bottom-in-view': function() {
+ return this.context.innerHeight() - this.adapter.outerHeight()
+ },
+ 'right-in-view': function() {
+ return this.context.innerWidth() - this.adapter.outerWidth()
+ }
+ }
+
+ window.Waypoint = Waypoint
+}())
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/layouts/_title.erb
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/layouts/_title.erb b/book/master_middleman/source/layouts/_title.erb
new file mode 100644
index 0000000..ea744d9
--- /dev/null
+++ b/book/master_middleman/source/layouts/_title.erb
@@ -0,0 +1,6 @@
+<% if current_page.data.title %>
+ <h1 class="title-container" <%= current_page.data.dita ? 'style="display: none;"' : '' %>>
+ <%= current_page.data.title %>
+ </h1>
+<% end %>
+
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/patch/dynamic_variable_interpretation.py
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/patch/dynamic_variable_interpretation.py b/book/master_middleman/source/patch/dynamic_variable_interpretation.py
new file mode 100644
index 0000000..66df9ff
--- /dev/null
+++ b/book/master_middleman/source/patch/dynamic_variable_interpretation.py
@@ -0,0 +1,192 @@
+#!/usr/bin/env python
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+__all__ = ["copy_tarballs_to_hdfs", ]
+import os
+import glob
+import re
+import tempfile
+from resource_management.libraries.functions.default import default
+from resource_management.libraries.functions.format import format
+from resource_management.libraries.resources.copy_from_local import CopyFromLocal
+from resource_management.libraries.resources.execute_hadoop import ExecuteHadoop
+from resource_management.core.resources.system import Execute
+from resource_management.core.exceptions import Fail
+from resource_management.core.logger import Logger
+from resource_management.core import shell
+
+"""
+This file provides helper methods needed for the versioning of RPMs. Specifically, it does dynamic variable
+interpretation to replace strings like {{ hdp_stack_version }} where the value of the
+variables cannot be determined ahead of time, but rather, depends on what files are found.
+
+It assumes that {{ hdp_stack_version }} is constructed as ${major.minor.patch.rev}-${build_number}
+E.g., 998.2.2.1.0-998
+Please note that "-${build_number}" is optional.
+"""
+
+# These values must be the suffix of the properties in cluster-env.xml
+TAR_SOURCE_SUFFIX = "_tar_source"
+TAR_DESTINATION_FOLDER_SUFFIX = "_tar_destination_folder"
+
+
+def _get_tar_source_and_dest_folder(tarball_prefix):
+ """
+ :param tarball_prefix: Prefix of the tarball must be one of tez, hive, mr, pig
+ :return: Returns a tuple of (x, y) after verifying the properties
+ """
+ component_tar_source_file = default("/configurations/cluster-env/%s%s" % (tarball_prefix.lower(), TAR_SOURCE_SUFFIX), None)
+ # E.g., /usr/hdp/current/hadoop-client/tez-{{ hdp_stack_version }}.tar.gz
+
+ component_tar_destination_folder = default("/configurations/cluster-env/%s%s" % (tarball_prefix.lower(), TAR_DESTINATION_FOLDER_SUFFIX), None)
+ # E.g., hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/
+
+ if not component_tar_source_file or not component_tar_destination_folder:
+ Logger.warning("Did not find %s tar source file and destination folder properties in cluster-env.xml" %
+ tarball_prefix)
+ return None, None
+
+ if component_tar_source_file.find("/") == -1:
+ Logger.warning("The tar file path %s is not valid" % str(component_tar_source_file))
+ return None, None
+
+ if not component_tar_destination_folder.endswith("/"):
+ component_tar_destination_folder = component_tar_destination_folder + "/"
+
+ if not component_tar_destination_folder.startswith("hdfs://"):
+ return None, None
+
+ return component_tar_source_file, component_tar_destination_folder
+
+
+def _copy_files(source_and_dest_pairs, file_owner, group_owner, kinit_if_needed):
+ """
+ :param source_and_dest_pairs: List of tuples (x, y), where x is the source file in the local file system,
+ and y is the destination file path in HDFS
+ :param file_owner: Owner to set for the file copied to HDFS (typically hdfs account)
+ :param group_owner: Owning group to set for the file copied to HDFS (typically hadoop group)
+ :param kinit_if_needed: kinit command if it is needed, otherwise an empty string
+ :return: Returns 0 if at least one file was copied and no exceptions occurred, and 1 otherwise.
+
+ Must kinit before calling this function.
+ """
+ import params
+
+ return_value = 1
+ if source_and_dest_pairs and len(source_and_dest_pairs) > 0:
+ return_value = 0
+ for (source, destination) in source_and_dest_pairs:
+ try:
+ destination_dir = os.path.dirname(destination)
+
+ params.HdfsDirectory(destination_dir,
+ action="create",
+ owner=file_owner,
+ mode=0555
+ )
+
+ CopyFromLocal(source,
+ mode=0444,
+ owner=file_owner,
+ group=group_owner,
+ dest_dir=destination_dir,
+ kinnit_if_needed=kinit_if_needed,
+ hdfs_user=params.hdfs_user,
+ hadoop_bin_dir=params.hadoop_bin_dir,
+ hadoop_conf_dir=params.hadoop_conf_dir
+ )
+ except:
+ return_value = 1
+ return return_value
+
+
+def copy_tarballs_to_hdfs(tarball_prefix, component_user, file_owner, group_owner):
+ """
+ :param tarball_prefix: Prefix of the tarball must be one of tez, hive, mr, pig
+ :param component_user: User that will execute the Hadoop commands
+ :param file_owner: Owner of the files copied to HDFS (typically hdfs account)
+ :param group_owner: Group owner of the files copied to HDFS (typically hadoop group)
+ :return: Returns 0 on success, 1 if no files were copied, and in some cases may raise an exception.
+
+ In order to call this function, params.py must have all of the following,
+ hdp_stack_version, kinit_path_local, security_enabled, hdfs_user, hdfs_principal_name, hdfs_user_keytab,
+ hadoop_bin_dir, hadoop_conf_dir, and HdfsDirectory as a partial function.
+ """
+ import params
+
+ if not hasattr(params, "hdp_stack_version") or params.hdp_stack_version is None:
+ Logger.warning("Could not find hdp_stack_version")
+ return 1
+
+ component_tar_source_file, component_tar_destination_folder = _get_tar_source_and_dest_folder(tarball_prefix)
+ if not component_tar_source_file or not component_tar_destination_folder:
+ Logger.warning("Could not retrieve properties for tarball with prefix: %s" % str(tarball_prefix))
+ return 1
+
+ if not os.path.exists(component_tar_source_file):
+ Logger.warning("Could not find file: %s" % str(component_tar_source_file))
+ return 1
+
+ # Ubuntu returns: "stdin: is not a tty", as subprocess output.
+ tmpfile = tempfile.NamedTemporaryFile()
+ with open(tmpfile.name, 'r+') as file:
+ get_hdp_version_cmd = '/usr/bin/hdp-select versions > %s' % tmpfile.name
+ code, stdoutdata = shell.call(get_hdp_version_cmd)
+ out = file.read()
+ pass
+ if code != 0 or out is None:
+ Logger.warning("Could not verify HDP version by calling '%s'. Return Code: %s, Output: %s." %
+ (get_hdp_version_cmd, str(code), str(out)))
+ return 1
+
+ hdp_version = out.strip() # this should include the build number
+
+ file_name = os.path.basename(component_tar_source_file)
+ destination_file = os.path.join(component_tar_destination_folder, file_name)
+ destination_file = destination_file.replace("{{ hdp_stack_version }}", hdp_version)
+
+ does_hdfs_file_exist_cmd = "fs -ls %s" % destination_file
+
+ kinit_if_needed = ""
+ if params.security_enabled:
+ kinit_if_needed = format("{kinit_path_local} -kt {hdfs_user_keytab} {hdfs_principal_name};")
+
+ if kinit_if_needed:
+ Execute(kinit_if_needed,
+ user=component_user,
+ path='/bin'
+ )
+
+ does_hdfs_file_exist = False
+ try:
+ ExecuteHadoop(does_hdfs_file_exist_cmd,
+ user=component_user,
+ logoutput=True,
+ conf_dir=params.hadoop_conf_dir,
+ bin_dir=params.hadoop_bin_dir
+ )
+ does_hdfs_file_exist = True
+ except Fail:
+ pass
+
+ if not does_hdfs_file_exist:
+ source_and_dest_pairs = [(component_tar_source_file, destination_file), ]
+ return _copy_files(source_and_dest_pairs, file_owner, group_owner, kinit_if_needed)
+ return 1
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/stylesheets/book-styles.css.scss
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/stylesheets/book-styles.css.scss b/book/master_middleman/source/stylesheets/book-styles.css.scss
new file mode 100644
index 0000000..1236d8e
--- /dev/null
+++ b/book/master_middleman/source/stylesheets/book-styles.css.scss
@@ -0,0 +1,3 @@
+* {
+ box-sizing: border-box;
+}
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/stylesheets/partials/_book-base-values.scss
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/stylesheets/partials/_book-base-values.scss b/book/master_middleman/source/stylesheets/partials/_book-base-values.scss
new file mode 100644
index 0000000..e69de29
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/de1e2e07/book/master_middleman/source/stylesheets/partials/_book-vars.scss
----------------------------------------------------------------------
diff --git a/book/master_middleman/source/stylesheets/partials/_book-vars.scss b/book/master_middleman/source/stylesheets/partials/_book-vars.scss
new file mode 100644
index 0000000..4245d57
--- /dev/null
+++ b/book/master_middleman/source/stylesheets/partials/_book-vars.scss
@@ -0,0 +1,19 @@
+$navy: #243640;
+$blue1: #2185c5;
+$blue2: #a7cae1;
+$bluegray1: #4b6475;
+$teal1: #03786D;
+$teal2: #00a79d;
+
+$color-accent: $teal1;
+$color-accent-bright: $teal2;
+
+// link colors
+$color-link: $blue1;
+$color-link-border: $blue2;
+
+$color-border-tip: $blue2;
+
+$color-bg-header: $navy;
+$color-bg-dark: $bluegray1;
+