#user admin

Hue is the web interface that improves the Apache Hadoop user experience. It’s a Django driven application and manages users accordingly. In this tutorial, we’ll be exploring the different options available for altering passwords in Hue using the default authentication backend (AllowFirstUserBackend).

User Interface

Users can change their passwords via the “<User Name>” -> “Edit Profile” found in the top-right corner of Hue.

If a user cannot remember their password, the Hue administrator can change it for them via the user manager.

If the Hue administrator loses their password, then a more technical approach must be taken.

Programmatic

When a Hue administrator loses their password, a more programmatic approach is required to secure the administrator again. Hue comes with a wrapper around the python interpreter called the “shell” command. It loads all the libraries required to work with Hue at a programmatic level. To start the Hue shell, type the following command from the Hue installation root:

cd /usr/share/hue(/opt/cloudera/parcels/CDH-XXXXX/share/hue if using parcels) build/env/bin/hue shell

The following is a small script, that can be executed within the Hue shell, to change the password for a user named “example”:

from django.contrib.auth.models import User user = User.objects.get(username='example') user.set_password('some password') user.save()

The script can also be invoked in the shell by using input redirection (assuming the script is in a file named script.py):

build/env/bin/hue shell < script.py

For more information on developing in Hue, checkout the Hue SDK and the Django documentation.

Summary

We hope this helps you manage your password and assists administrators when they’ve lost their own passwords. In a future blog post, we will detail other ways to authenticate with Hue.

Hue is the turn-key solution for Apache Hadoop. It hides the complexity of the ecosystem including HDFS, Oozie, MapReduce, etc. Hue provides authentication and integrates with SAML, LDAP, and other systems. A new feature added in Hue is the ability to synchronize groups with a third party authority provider. In this blog post, we’ll be covering the basics of creating a Group Synchronization Backend.

  The Design

The purpose of the group synchronization backends are to keep Hue’s internal group lists fresh. The design was separated into two functional parts:

A way to synchronize on every request.

A definition of how and what to synchronize.

Image 1: Request cycle in Hue with a synchronization backend.

The first function is a Django middleware that is called on every request. It is intended to be immutable, but configurable. The second function is a backend that can be customized. This gives developers the ability to choose how their groups and user-group memberships can be synchronized. The middleware can be configured to use a particular synchronization backend and will call it on every request. If no backend is configured, then the middleware is disabled.

  Creating Your Own Backend

A synchronization backend can be created by extending a class and providing your own logic. Here is an example backend that comes packaged with Hue:

class LdapSynchronizationBackend(DesktopSynchronizationBackendBase): USER_CACHE_NAME = 'ldap_use_group_sync_cache' def sync(self, request): user = request.user if not user or not user.is_authenticated(): return if not User.objects.filter(username=user.username, userprofile__creation_method=str(UserProfile.CreationMethod.EXTERNAL)).exists(): LOG.warn("User %s is not an Ldap user" % user.username) return # Cache should be cleared when user logs out. if self.USER_CACHE_NAME not in request.session: request.session[self.USER_CACHE_NAME] = import_ldap_users(user.username, sync_groups=True, import_by_dn=False) request.session.modified = True

In the above code snippet, the synchronization backend is defined by extending “DesktopSynchronizationBackendBase”. Then, the method “sync(self, request)” is overridden and provides the syncing logic.

Configuration

The synchronization middleware can be configured to use a backend by changing “desktop -> auth -> user_group_membership_synchronization_backend” to the full import path of your class. For example, setting this config to “desktop.auth.backend.LdapSynchronizationBackend” configures Hue to synchronize with the configured LDAP authority.

Design Intelligently

Backends in Hue are extremely powerful and can affect the performance of the server. So, they should be designed in such a fashion that they do not do any operations that block for long periods of time. Also, they should manage the following appropriately:

Throttling requests to whatever service contains the group information.

Ensuring users are authenticated.

Caching if appropriate.

  Summary

Hue is enterprise grade software ready to integrate with LDAP, SAML, etc. The newest feature, Group Synchronization, ensures corporate authority is fresh in Hue. It’s easy to configure and create backends and Hue comes with an LDAP backend.

This post is about the new release of Hue, an open source web-based interface that makes Apache Hadoop easier to use, that’s included in CDH4.2.

Hue lets you interact with Hadoop services from within your browser without having to go to a command-line interface. It features a file browser for HDFS, an Apache Oozie Application for creating workflows of data processing jobs, a job designer/browser for MapReduce, Apache Hive and Cloudera Impala query editors, a Shell, and a collection of Hadoop APIs.

The goal of this release was to add a set of new features and improve the user experience. Read on for a list of the major changes (from 304 commits).

Oozie Application

With the Oozie Application you can chain jobs and schedule them repeatedly without having to write XML anymore. Workflow and Coordinator management got extra focus and now matches all the Oozie functionalities:

The workflow editor supports Drag & Drop and was restyled.

The coordinator page was redesigned with a wizard and data can be specified by range.

The dashboard displays any workflow as a graph and refreshes itself dynamically.

All the Oozie actions are supported (addition of Sub-workflow, DistCp, Email, Fs, Generic).

Forks can be converted to decision nodes.

A read-only user can access the dashboard.

A workflow or a coordinator can be resubmitted from specific steps.

Existing XML workflow definition can be imported.

The dashboard provides direct access to task logs of any action.  

Drag & Drop Workflow Editor  

Workflow Dashboard

  Coordinator Wizard

Rerun a Workflow (left) or a Coordinator (right)

Beeswax/Hive Query Editor

A number of user experience improvements make it simpler to query your data with SQL:

Multiple databases are supported (tackling one of the most popular requests HUE-535).

Query editor is bigger, has line numbers and shows lines with error(s).

Running queries shows logs in Ajax and lets you scroll through them.

Query results page has a horizontal scroll bar and a quick column name lookup for accessing a certain column when they are many.

  Query Editor  

Wide result page with column lookup

Impala Editor

Impala can now be queried from a new interface. More features will be supported when Impala is GA.

Cloudera Impala query

FileBrowser

FileBrowser lets you navigate and manage HDFS files in a UI. Its front end was totally redesigned and new filesystem operations were added. You do not need to use the hadoop fs command anymore!

Bulk operations for multiple deletions, changing of permissions or owner

Supports bulks operation recursively or not (e.g. chmod recursively a folder)

Upload archives (e.g. upload multiple files at once like the Oozie sharelib)

Create a file and edit it

Bulk editing

JobBrowser

JobBrowser lists MapReduce jobs with their statistics and statuses. It was prettified and now supports MR2 jobs running over YARN:

MR2 jobs and their logs can be browsed.

Job logs can be accessed with one click.

Other apps like Beeswax and Oozie can now show the MR2 logs.

MR1/MR2 job list and direct log access

UserAdmin

Groups and Hue permissions can be assigned to the users through the UserAdmin application. Access to Hue applications can be customized for each user. (For example, Bob can only see and use the Oozie and Impala applications.) The application has been restyled and simplified, and is no longer accessible by default to non-superuser:

HDFS home of first/new/imported users is created automatically.

LDAP support now has wildcard search, user import by wildcard expression and group syncing by distinguished name.

Group permission editing

Desktop

Desktop is the core library of Hue and every application is built on top of it. In this release, the user experience has been improved with more informative errors (now with stack traces and line numbers) and new status messages (such as when critical services like Oozie are down).

On the technical side, users can now upload files to a federated cluster, some XSS vulnerabilities were fixed, and database greenlet support was introduced for more performance. Hue now fully supports transactional databases like MySQL InnoDB and PostgreSQL.

Hue is also internationalized and available in English, French, German, Spanish, Portuguese (and Brazilian Portuguese), Korean, Japanese and simplified Chinese.

Conclusion

With this 2.2 release, a big part of the Hadoop user experience gap was filled in.

The next 2.3 release will target users who wish to better leverage the multiple query solutions in CDH (Beeswax/Hive, Impala, and Pig). A new document model (HUE-950) would make each query (e.g. Hive query) searchable and shareable with your colleagues or importable into an Oozie workflow without any duplication. A trash and source control versioning system (HUE-951) is also discussed as well as Oozie bundle (HUE-869) integrations.

Hue 2.2 , the open source web-based interface that makes Apache Hadoop easier to use, lets you interact with Hadoop services from within your browser without having to go to a command-line interface. It features different applications like an Apache Hive editor and Apache Oozie dashboard and workflow builder.

This post is based on our “Analyzing Twitter Data with Hadoop” sample app and details how the same results can be achieved through Hue in a simpler way. Moreover, all the code and examples of the previous series have been updated to the recent CDH4.2 release.

Collecting Data

The first step is to create the “flume” user and his home on the HDFS where the data will be stored. This can be done via the User Admin application.

The second step consists of collecting some tweet data from the live Twitter stream.

Apache Flume is an elegant solution for taking care of this. The configuration of Flume is detailed in the readme and previous blog post. However, if you want to skip this step, some data is available on GitHub. Just upload it as a zip file in the home directory of the flume user and the “tweets” directory will show up after a few seconds.

If you are not taking this shortcut, create the tweets directory in the File Browser with the New Folder action.

Then, when the Flume agent is started, the data will start appearing:

Clicking on a file will display its content in the built-in viewer:

Preparing Hive

It is time to prepare the analysis of the tweet data. We’ll use Apache Hive, which can query the data with SQL-like syntax in a scalable way. The detailed description of the Hive setup is detailed in the readme.

When Hive is ready, the tweet table can be created in the query editor of Beeswax. Notice that the Hive SerDe (to download or compile here) must be included as a jar in the query. You can read more about Hive SerDe in this previous post.

To do this, just click on “Add” > “File Resources”, click on the path chooser button, click on the “Home” button, and upload hive-serdes-1.0-SNAPSHOT.jar.

Then just enter the CREATE TABLE statement and execute it:

CREATE EXTERNAL TABLE tweets ( id BIGINT, created_at STRING, source STRING, favorited BOOLEAN, retweet_count INT, retweeted_status STRUCT< text:STRING, user:STRUCT<screen_name:STRING,name:STRING>>, entities STRUCT< urls:ARRAY<STRUCT<expanded_url:STRING>>, user_mentions:ARRAY<STRUCT<screen_name:STRING,name:STRING>>, hashtags:ARRAY<STRUCT<text:STRING>>>, text STRING, user STRUCT< screen_name:STRING, name:STRING, friends_count:INT, followers_count:INT, statuses_count:INT, verified:BOOLEAN, utc_offset:INT, time_zone:STRING>, in_reply_to_screen_name STRING ) PARTITIONED BY (datehour INT) ROW FORMAT SERDE 'com.cloudera.hive.serde.JSONSerDe' LOCATION '/user/flume/tweets'

  Now that the table is created, let’s insert some data in the table. First, select the table in the “Table” tab and click “Import data”. Enter the path “/user/flume/tweets/2013/02/25/17″ and “201302251″ as the key:

Depending on the partition picked, a query similar to this will be generated:

LOAD DATA INPATH '/user/flume/tweets/2013/02/25/16' INTO TABLE `default.tweets` PARTITION (datehour='2013022516')

  After the query executes, the table ‘tweets’ will be available.

Beeswax can access the Hive metastore and its list of tables. A description of their schema and partitions with some example of data contained in each table are helpful while designing your queries. Moreover, a wizard can guide you step-by-step to create new tables.

Analysis with Beeswax

It becomes now possible to perform some SELECT queries on the data. Here is an example below but most of interesting ones are described in Parts 1 and 3 of the “Analyzing Twitter with Hadoop” series.

SELECT   t.retweeted_screen_name,   sum(retweets) AS total_retweets,   count(*) AS tweet_count FROM (SELECT         retweeted_status.user.screen_name as retweeted_screen_name,             retweeted_status.text,             max(retweet_count) as retweets       FROM tweets       GROUP BY retweeted_status.user.screen_name,                retweeted_status.text) t GROUP BY t.retweeted_screen_name ORDER BY total_retweets DESC LIMIT 10;

  Beeswax possesses multiple features for providing a better user experience than the command line shell. For example you can save queries and share them with other users. The result of a query can be exported into a new table or an HDFS file or downloaded to your desktop. Some other good examples are:

Ajax refresh of the logs

Quick column navigation on the result page

MapReduce jobs listing with a direct access to their logs

‘Email me on completion’ setting

Multi-database support

Example of the screen while running query:

Seeing the result of the query:

Note: if your queries are failing and you are seeing an error like below, it means that you forgot to add the ‘/user/flume/hive-serdes-1.0-SNAPSHOT.jar’ to the query:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

  Conclusion

In this post we focused on how the Beeswax application can make it easy to execute Hive queries. New features such as multi-query (HUE-159), autocomplete, and syntax highlighting (HUE-1063) are going to improve the usability even more.

The next article in this series will elaborate on this topic and describe how Hue’s Apache Oozie application can be used for scheduling Hive queries in a few clicks.

Hue is a web interface for Apache Hadoop that makes common Hadoop tasks such as running MapReduce jobs, browsing HDFS, and creating Apache Oozie workflows, easier. (To learn more about the integration of Oozie and Hue, see this blog post.) In this post, we’re going to focus on how one of the fundamental components in Hue, Useradmin, has matured.

New User and Permission Features

User and permission management in Hue has changed drastically over the past year. Oozie workflows, Apache Hive queries, and MapReduce jobs can be shared with other users or kept private. Permissions exist at the app level. Access to particular apps can be restricted, as well as certain sections of the apps. For instance, access to the shell app can be restricted, as well as access to the Apache HBase, Apache Pig, and Apache Flume shells themselves. Access privileges are defined for groups and users can be members of one or more groups.

Changes to Users, Groups, and Permissions

Hue now supports authentication against PAM, Spnego, and an LDAP server. Users and groups can be imported from LDAP and be treated like their non-external counterparts. The import is manual and is on a per user/group basis. Users can authenticate using different backends such as LDAP. Using the LDAP authentication backend will allow users to login using their LDAP password. This can be configured in /etc/hue/hue.ini by changing the ‘desktop.auth.backend’ setting to ‘desktop.auth.backend.LdapBackend’. The LDAP server to authenticate against can be configured through the settings under ‘desktop.ldap’.

Here’s an example:

A company would like to use the following LDAP users and groups in Hue:

John Smith belonging to team A

Helen Taylor belonging to team B

Assuming the following access requirements:

Team A should be able to use Beeswax, but nothing else.

Team B should only be able to see the Oozie dashboard with readonly permissions.

In Hue 1 the scenarios cannot be realistically addressed given the lack of groups.

In Hue 2 the scenarios can be addressed more appropriately. Users can be imported from LDAP by clicking “Add/Sync LDAP user” in Useradmin > Users:

Similarly, groups can be imported from LDAP by clicking “Add/Sync LDAP group” in Useradmin > Groups.

If a previously imported user’s information was updated recently, the information in Hue will need to be resynchronized. This can be achieved through the LDAP sync feature:

Part A of the example can be addressed by explicitly allowing access Beeswax for Team A. This is managed in the “Groups” tab of the Useradmin app:

The Team A group can be edited by clicking on its name, where access privileges for the group are selectable. Here, the “beeswax.access” permission would be selected and the others would be unselected:

Part B of the example can be handled by explicitly defining access for Team B. This can be accomplished by following the same steps in part A, except for Team B. Every permission would be unselected except “oozie.dashboard_jobs_access”:

By explicitly setting the app level permissions, the apps that these users will be able to see will change. For instance, Helen, who is a member of Team B, will only see the Oozie app available:

Summary

User management has been revamped, groups were added, and various backends are exposed. One such backend, LDAP, facilitates synchronization of users and groups. App-level permissions allow administrators to control who can access certain apps and what documents can be shared.

Hue is maturing quickly and many more features are on their way. Hue will soon have document-level permissions (workflows, queries, and so on), trash functionality, and improvements to the existing editors.