#amazonemr

Security for EMR Notebooks

Recent Amazon EMR documentation highlights numerous built-in options to increase EMR Notebook security that are now available in the AWS dashboard as EMR Studio Workspaces. These capabilities are aimed to give users precise control so that only authorised users may access and interact with these notebooks and, most crucially, use the notebook editor to run code on linked clusters.

The security measures for Amazon EMR and its clusters complement those for EMR Notebooks. Tiered security allows for additional thoroughness. Many important processes for restricting access and securing notebook environments are mentioned in the documentation:

AWS IAM Integration: Integrated Identity and Access Management is crucial. Use IAM policy statements. In these policies, AWS defines permissions, including who can access what resources and do what. The documentation suggests using policy statements with notebook tags to restrict access.

This solution lets you tag EMR notebooks with key-value labels and build IAM policies that allow or deny access based on these tags. These extracts do not include the tagging methods, however this allows more granular control than providing access to all notebooks. Certain projects, teams, or data sensitivity levels may allow access control.

Amazon EC2 security groups are highlighted. They function as virtual firewalls. They control network traffic between the notebook editor and the cluster's primary instance in EMR Notebooks.

This basic network security solution restricts network connectivity between the real computing resources (the principal instance of the EMR cluster), where code execution begins, and the notebook environment, where the user interacts. According to the documentation, customers can adjust EMR Notebook security groups to meet their network isolation needs or use the default settings. EMR Notebook EC2 security group configuration instructions are available.

An AWS Service Role is utilised for setup. Highlights your responsibility to define this job. This Service Role is necessary to grant EMR notebooks authorisation to communicate with other AWS services. This Service Role allows notebook code to interface with databases, access S3 data, and call other AWS APIs.

The least privilege principle requires that a position only have the access needed to complete their tasks.

AWS console access requires additional permissions to access EMR Notebooks. Console users can access EMR Notebooks as EMR Studio Workspaces. You require extra IAM role rights to access or create these Workspaces. Use of the “Create Workspace” button requires this. This adds access control to the console interface, unlike the notebook's execution permissions or Service Role for communicating with other services. It indicates that basic EMR console rights and console access to EMR Studio Workspaces are covered elsewhere.

Together, EC2 security groups act as virtual firewalls to regulate network traffic, IAM policies with notebook tags limit access, a specific AWS Service Role defines interaction permissions with other services, and additional IAM permissions for console access to EMR Studio Workspaces allow administrators to customise the security posture of their EMR Notebook environments.

EMR Notebook Programming Commands

Programmatic Amazon EMR Notebook interaction.

How to leverage execution APIs from a script or command line to control EMR notebook executions outside the AWS UI. This lets you list, characterise, halt, and start EMR notebook executions.

The following examples demonstrate these abilities:

AWS CLI: Amazon EMR clusters on Amazon EC2 and EMR Notebooks clusters (EMR on EKS) with notebooks in EMR Studio Workspaces are shown. An Amazon S3 location-based notebook execution sample is also provided. The displayed instructions can list executions by start time or start time and status, halt an ongoing execution, and describe a notebook execution.

Boto3 SDK (Python): Demo.py uses boto3 to interface with EMR notebook execution APIs. The script explains how to initiate a notebook execution, get the execution ID, describe it, list all running instances, and stop it after a short pause. Status updates and execution IDs are shown in this script's output.

Ruby SDK: Sample Ruby code shows notebook execution API calls and Amazon EMR connection setup. Example: describe execution, print information, halt notebook execution, start notebook execution, and get execution ID. Predicted Ruby notebook run outcomes are also shown.

Programmatic command parameters

Important parameters in these programming instructions are:

EditorId: EMR Studio workspace.

relative-path or RelativePath: The notebook file's path to the workspace's home directory. Pathways include my_folder/python3.ipynb and demo_pyspark.ipynb.

execution-engine or ExecutionEngine: EMR cluster ID (j-1234ABCD123) or EMR on EKS endpoint ARN and type to choose engine.

The IAM service role, such as EMR_Notebooks_DefaultRole, is defined.

notebook-params or notebook_params: Allows a notebook to receive multiple parameter values, eliminating the need for multiple copies. Typically, parameters are JSON strings.

The input notebook file's S3 bucket and key are supplied.

The S3 bucket and key where the output notebook will be stored.

notebook-execution-name: Names the performance.

This identifies an execution when describing, halting, or listing.

–from and –status: Status and start time filters for executions.

The console can also access EMR Notebooks as EMR Studio Workspaces, according to documentation. Workspace access and creation require additional IAM role rights. Programmatic execution requires IAM policies like StartNotebookExecution, DescribeNotebookExecution, ListNotebookExecutions, and iam:PassRole. EMR Notebooks clusters (EMR on EKS) require emr-container permissions.

The AWS Region per account maximum is 100 concurrent executions, and executions that last more than 30 days are terminated. Interactive Amazon EMR Serverless apps cannot execute programs.

How to Make EMR Notebook?

Amazon Web Services (AWS) has incorporated Amazon EMR Notebooks into Amazon EMR Studio Workspaces on the new Amazon EMR interface. Integration aims to provide a single environment for notebook creation and massive data processing. However, the new console's “Create Workspace” button usually creates notebooks.

Users must visit the Amazon EMR console at the supplied web URL and complete the previous console's procedures to create an EMR notebook. Users usually select “Notebooks” and “Create notebook” from this interface.

When creating a Notebook, users choose a name and a description. The next critical step is connecting the notebook to an Amazon EMR cluster to run the code.

There are two basic ways users associate clusters:

Select an existing cluster

If an appropriate EMR cluster is already operating, users can click “Choose,” select it from a list, and click “Choose cluster” to confirm. EMR Notebooks have cluster requirements, per documentation. These prerequisites, EMR release versions, and security problems are detailed in specialised sections.

Create a cluster

Users can also “Create a cluster” to have Amazon EMR create a laptop-specific cluster. This method lets users name their clusters. This workflow defaults to the latest supported EMR release version and essential apps like Hadoop, Spark, and Livy, however some configuration variables, such as the Release version and pre-selected apps, may not be modifiable.

Users can customise instance parameters by selecting EC2 Instance and entering the appropriate number of instances. A primary node and core nodes are identified. The instance type determines the maximum number of notebooks that can connect to the cluster, subject to constraints.

The EC2 instance profile and EMR role, which users can choose custom or default roles for, are also defined during cluster setup. Links to more information about these service roles are supplied. An EC2 key pair for cluster instance SSH connections can also be chosen.

Amazon EMR versions 5.30.0 and 6.1.0 and later allow optional but helpful auto-termination. For inactivity, users can click the box to shut down the cluster automatically. Users can specify security groups for the primary instance and notebook client instance, use default security groups, or use custom ones from the cluster's VPC.

Cluster settings and notebook-specific configuration are part of notebook creation. Choose a custom or default AWS Service Role for the notebook client instance. The Amazon S3 Notebook location will store the notebook file. If no bucket or folder exists, Amazon EMR can create one, or users can choose their own. A folder with the Notebook ID and NotebookName and.ipynb extension is created in the S3 location to store the notebook file.

If an encrypted Amazon S3 location is used, the Service role for EMR Notebooks (EMR_Notebooks_DefaultRole) must be set up as a key user for the AWS KMS key used for encryption. To add key users to key policies, see AWS KMS documentation and support pages.

Users can link a Git-based repository to a notebook in Amazon EMR. After selecting “Git repository” and “Choose repository”, pick from the list.

Finally, notebook users can add Tags as key-value pairs. The documentation includes an Important Note about a default tag with the key creatorUserID and the value set to the user's IAM user ID. Users should not change or delete this tag, which is automatically applied for access control, because IAM policies can use it. After configuring all options, clicking “Create Notebook” finishes notebook creation.

Users should note that these instructions are for the old console, while the new console now uses EMR Notebooks as EMR Studio Workspaces. To access existing notebooks as Workspaces or create new ones using the “Create Workspace” option in the new UI, EMR Notebooks users need extra IAM role rights. Users should not change or delete the notebook's default access control tag, which contains the creator's user ID. No notebooks can be created with the Amazon EMR API or CLI.

Amazon EMR Studio features, specs, and limitations:

Amazon EMR Studio describes an IDE for data preparation and visualisation, departmental collaboration, and application debugging. When utilising EMR Studio, consider tool usage, cluster demands, known issues, feature constraints, service limits, and regional availability.

Features of Amazon EMR Studio

Service Catalogue lets administrators connect EMR Studio to cluster templates. This lets users create Amazon EC2 EMR clusters for workspaces. Administrators can grant or deny Studio users access to cluster templates.

The Amazon EMR service role is needed to define access permissions to Amazon S3 notebook files or AWS Secrets Manager secrets because session policies do not allow them.

Multiple EMR Studios can control access to EMR clusters in different VPCs.

Use the AWS CLI to configure Amazon EMR on EKS clusters. Connect these clusters to Workspaces via a controlled API in Studio to run notebook jobs.

Amazon EMR and EMR Studio use trusted identity propagation, which has extra considerations. IAM Identity Centre and trusted identity propagation are required for EMR Studio to connect to EMR clusters that use it.

To secure Amazon EMR off-console applications, application hosting domains list their apps in the Public Suffix List (PSL). Examples are emrappui-prod.us-east-1.amazonaws.com, emrnotebooks-prod.us-east-1.amazonaws.com, and emrstudio-prod.us-east-1.amazonaws.com. For sensitive cookies in the default domain name, a __Host- prefix can prevent CSRF and add security.

EMR Studio Workspaces and Persistent UI endpoints use FIPS 140-certified cryptographic modules for encryption-in-transit, making the service suitable for regulated workloads.

Amazon EMR Studio requirements and compatibility

EMR Studio supports Amazon EMR Software versions 5.32.0 and 6.2.0.

EMR clusters using IAM Identity Centre with trusted identity propagation must use it.

Before setting up a Studio, disable browser proxy control applications like FoxyProxy or SwitchyOmega. Active proxies can cause Studio creation network failures.

Amazon EMR Studio restrictions and issues

EMR Studio does not support Python magic commands %alias, %alias_magic, %automagic, %macro, %%js, and %%javascript. Changing KERNEL_USERNAME or proxy_user using %env or %set_env or %configure is not supported.

Amazon EMR on EKS clusters does not support SparkMagic commands in EMR Studio.

All multi-line Scala statements in notebook cells must end with a period except the last.

Amazon EMR kernels on EKS clusters may timeout and fail to start. Should this happen, restart the kernel and close and reopen the notebook file. The Restart kernel operation requires restarting the Workspace, and EMR on EKS clusters may not work.

If a workspace is not connected to a cluster, starting a notebook and choosing a kernel fails. Choose a kernel and attach the workspace to run code, but ignore this error.

With Amazon EMR 6.2.0 security, the Workspace interface may be blank. For security-configured EMRFS S3 authorisation or data encryption, choose a different supported version. Troubleshooting EMR on EC2 tasks may disable on-cluster Spark UI connectivity. Run %%info in a new cell to regenerate these links.

5.32.0, 5.33.0, 6.2.0, and 6.3.0 Amazon EMR primary nodes do not have idle kernels cleaned away by Jupyter Enterprise Gateway. This may drain resources and crash long-running clusters. A script in the sources configures idle kernel cleanup for certain versions.

If the auto-termination policy is enabled on Amazon EMR versions 5.32.0, 5.33.0, 6.2.0, or 6.3.0, a cluster with an active Python3 kernel may be designated as inactive and terminated since it does not submit a Spark task. Amazon EMR 6.4.0 or later is recommended for Python3 kernel auto-termination.

Displaying a Spark DataFrame using %%display may truncate wide tables. Create a scrollable view by right-clicking the output and selecting Create New View for Output.

If you interrupt a running cell in a Spark-based kernel (PySpark, Spark, SparkR), the Spark task stays running. The on-cluster Spark UI is needed to end the job.

EMR Studio Workspaces as the root user in an AWS account causes a 403: Forbidden error because Jupyter Enterprise Gateway settings disallow root user access. Instead of root, employ alternate authentication methods for normal activities.

EMR Studio does not support Amazon EMR features:

connecting to and running tasks on Kerberos-secured clusters.

multi-node clusters.

AWS Graviton2-based EC2 clusters for EMR 6.x releases below 6.9.0 and 5.x releases below 5.36.1.

A studio utilising trusted identity propagation cannot provide these features:

Building EMR clusters without templates using serverless applications.

Amazon EMR launches on EKS clusters.

Use a runtime role.

Supporting SQL Explorer or Workspace collaboration.

Limited Amazon EMR Studio Service

Service Restriction The sources list EMR Studio service limits:

EMR Studios:

Each AWS account can have 100 max.

Maximum five subnets per EMR Studio.

IAM Identity Centre Groups are limited to five per EMR Studio.

Benefits of Amazon EMR

Amazon EMR has many benefits. These include AWS's flexibility and cost savings over on-premises resource development.

Cost-saving

Amazon EMR costs depend on instance type, number of Amazon EC2 instances, and cluster launch area. On-demand pricing is low, but Reserved or Spot Instances save much more. Spot instances can save up to a tenth of on-demand costs.

Note

Using Amazon S3, Kinesis, or DynamoDB with your EMR cluster incurs expenses irrespective of Amazon EMR usage.

Note

Set up Amazon S3 VPC endpoints when creating an Amazon EMR cluster in a private subnet. If your EMR cluster is on a private subnet without Amazon S3 VPC endpoints, you will be charged extra for S3 traffic NAT gates.

AWS integration

Amazon EMR integrates with other AWS services for cluster networking, storage, security, and more. The following list shows many examples of this integration:

Use Amazon EC2 for cluster nodes.

Amazon VPC creates the virtual network where your instances start.

Amazon S3 input/output data storage

Set alarms and monitor cluster performance with Amazon CloudWatch.

AWS IAM permissions setting

Audit service requests with AWS CloudTrail.

Cluster scheduling and launch with AWS Data Pipeline

AWS Lake Formation searches, categorises, and secures Amazon S3 data lakes.

Its deployment

The EC2 instances in your EMR cluster do the tasks you designate. When you launch your cluster, Amazon EMR configures instances using Spark or Apache Hadoop. Choose the instance size and type that best suits your cluster's processing needs: streaming data, low-latency queries, batch processing, or big data storage.

Amazon EMR cluster software setup has many options. For example, an Amazon EMR version can be loaded with Hive, Pig, Spark, and flexible frameworks like Hadoop. Installing a MapR distribution is another alternative. Since Amazon EMR runs on Amazon Linux, you can manually install software on your cluster using yum or the source code.

Flexibility and scalability

Amazon EMR lets you scale your cluster as your computing needs vary. Resizing your cluster lets you add instances during peak workloads and remove them to cut costs.

Amazon EMR supports multiple instance groups. This lets you employ Spot Instances in one group to perform jobs faster and cheaper and On-Demand Instances in another for guaranteed processing power. Multiple Spot Instance types might be mixed to take advantage of a better price.

Amazon EMR lets you use several file systems for input, output, and intermediate data. HDFS on your cluster's primary and core nodes can handle data you don't need to store beyond its lifecycle.

Amazon S3 can be used as a data layer for EMR File System applications to decouple computation and storage and store data outside of your cluster's lifespan. EMRFS lets you scale up or down to meet storage and processing needs independently. Amazon S3 lets you adjust storage and cluster size to meet growing processing needs.

Reliability

Amazon EMR monitors cluster nodes and shuts down and replaces instances as needed.

Amazon EMR lets you configure automated or manual cluster termination. Automatic cluster termination occurs after all procedures are complete. Transitory cluster. After processing, you can set up the cluster to continue running so you can manually stop it. You can also construct a cluster, use the installed apps, and manually terminate it. These clusters are “long-running clusters.”

Termination prevention can prevent processing errors from terminating cluster instances. With termination protection, you can retrieve data from instances before termination. Whether you activate your cluster by console, CLI, or API changes these features' default settings.

Security

Amazon EMR uses Amazon EC2 key pairs, IAM, and VPC to safeguard data and clusters.

IAM

Amazon EMR uses IAM for permissions. Person or group permissions are set by IAM policies. Users and groups can access resources and activities through policies.

The Amazon EMR service uses IAM roles, while instances use the EC2 instance profile. These roles allow the service and instances to access other AWS services for you. Amazon EMR and EC2 instance profiles have default roles. By default, roles use AWS managed policies generated when you launch an EMR cluster from the console and select default permissions. Additionally, the AWS CLI may construct default IAM roles. Custom service and instance profile roles can be created to govern rights outside of AWS.

Security groups

Amazon EMR employs security groups to control EC2 instance traffic. Amazon EMR shares a security group for your primary instance and core/task instances when your cluster is deployed. Amazon EMR creates security group rules to ensure cluster instance communication. Extra security groups can be added to your primary and core/task instances for more advanced restrictions.

Encryption

Amazon EMR enables optional server-side and client-side encryption using EMRFS to protect Amazon S3 data. After submission, Amazon S3 encrypts data server-side.

The EMRFS client on your EMR cluster encrypts and decrypts client-side encryption. AWS KMS or your key management system can handle client-side encryption root keys.

Amazon VPC

Amazon EMR launches clusters in Amazon VPCs. VPCs in AWS allow you to manage sophisticated network settings and access functionalities.

AWS CloudTrail

Amazon EMR and CloudTrail record AWS account requests. This data shows who accesses your cluster, when, and from what IP.

Amazon EC2 key pairs

A secure link between the primary node and your remote computer lets you monitor and communicate with your cluster. SSH or Kerberos can authenticate this connection. SSH requires an Amazon EC2 key pair.

Monitoring

Debug cluster issues like faults or failures utilising log files and Amazon EMR management interfaces. Amazon EMR can archive log files on Amazon S3 to save records and solve problems after your cluster ends. The Amazon EMR UI also has a task, job, and step-specific debugging tool for log files.

Amazon EMR connects to CloudWatch for cluster and job performance monitoring. Alarms can be set based on cluster idle state and storage use %.

Management interfaces

There are numerous Amazon EMR access methods:

The console provides a graphical interface for cluster launch and management. You may examine, debug, terminate, and describe clusters to launch via online forms. Amazon EMR is easiest to use via the console, requiring no scripting.

Installing the AWS Command Line Interface (AWS CLI) on your computer lets you connect to Amazon EMR and manage clusters. The broad AWS CLI includes Amazon EMR-specific commands. You can automate cluster administration and initialisation with scripts. If you prefer command line operations, utilise the AWS CLI.

SDK allows cluster creation and management for Amazon EMR calls. They enable cluster formation and management automation systems. This SDK is best for customising Amazon EMR. Amazon EMR supports Go, Java,.NET (C# and VB.NET), Node.js, PHP, Python, and Ruby SDKs.

A Web Service API lets you call a web service using JSON. A custom SDK that calls Amazon EMR is best done utilising the API.

Complexity:

EMR cluster setup and maintenance are more involved than with AWS Glue and require framework knowledge.

Learning curve

Setting up and optimising EMR clusters may require adjusting settings and parameters.

Possible Performance Issues:

Incorrect instance types or under-provisioned clusters might slow task execution and other performance.

Depends on AWS:

Describe Amazon EMR.

Amazon EMR, previously Amazon Elastic MapReduce, allows Apache Hadoop and Apache Spark easy to run on AWS for processing and analysing enormous amounts of data. These frameworks and open-source apps process data for corporate intelligence and analytics. Amazon EMR transforms and transfers massive volumes of data between Amazon DynamoDB and Amazon S3..

Amazon EMR cluster setup and operation

A detailed overview of Amazon EMR clusters, including how to submit work, how data is handled, and the cluster's processing phases.

Learning nodes and clusters

Main component of Amazon EMR is cluster. Amazon EC2 clusters are groups of instances. Every cluster instance is a node. Each cluster node type has a role. Amazon EMR puts software components on each node type to assign it a function in a distributed application like Apache Hadoop.

Types of Amazon EMR nodes:

The primary node runs software to coordinate work and data allocation across processing nodes, administering the cluster. The primary node monitors cluster health and tasks. Every cluster has a primary node that can form a single-node cluster.

The core node contains the software needed to run operations and store data in your cluster's Hadoop Distributed File System. Core nodes are present in multi-node clusters.

Task nodes: Software-equipped nodes that execute tasks without storing data in HDFS. Task nodes are optional.

Submitted work to cluster

When running an Amazon EMR cluster, you may specify tasks in several ways.

Provide clear instructions for cluster construction phases. This is frequently done to clusters that process a particular amount of data and then shut down.

Submit steps, including jobs, using the Amazon EMR UI, API, or CLI after constructing a long-running cluster. Check out Submit work to an Amazon EMR cluster.

Establish a cluster, connect to the primary node and other nodes via SSH, then complete tasks and send interactive or scripted queries using the installed apps' interfaces. Learn more from the Amazon EMR Release Guide.

Data processing

When you launch your cluster, you choose data processing frameworks and apps. You can process data in your Amazon EMR cluster by performing steps in the cluster or sending jobs or queries to installed apps.

Jobs posted directly to applications

Your Amazon EMR cluster's software lets you submit jobs and communicate with it. This is usually done by connecting securely to the primary node and utilising the tools and interfaces for your cluster's software.

Executing data processing procedures

Amazon EMR clusters can receive ordered steps. Each stage contains data modification instructions for the cluster's software.

The following procedure has four steps:

Submit a dataset for processing.

Process first-stage output with Pig.

Hive can process a second input dataset.

Make an output dataset.

Amazon EMR usually processes data from your chosen file system, such as HDFS or Amazon S3. This data progresses via processing. The output data is written to an Amazon S3 bucket in the last stage.

Steps are performed in this order:

Start processing is requested.

All actions are pending.

It becomes RUNNING when the sequence starts. The remaining steps are PENDING.

After the first stage, it becomes COMPLETED.

Once the sequence continues, its status becomes RUNNING. Its condition is COMPLETED when done.

This cycle continues until all stages are completed and processing is complete.

The following diagram shows processing steps and state changes.

Failure while processing marks a step as FAILED. Choose a follow-up for each stage. If a previous step fails, the remaining steps are set to CANCELLED and do not execute. Other alternatives include stopping the cluster immediately or disregarding the failure and continuing.

The figure shows the default state change and step sequence when a processing step fails.

Understanding cluster lifespan

Successful Amazon EMR clusters work like this:

Amazon EMR creates EC2 instances in the cluster for each instance based on your requirements. See Amazon EMR cluster hardware and networking configuration for more. Amazon EMR always utilises the default AMI or your custom Amazon Linux AMI. For more, see Using a custom AMI to increase Amazon EMR cluster configuration flexibility. The cluster state is just beginning.

You can configure bootstrap activities for each Amazon EMR instance. Custom apps can be installed and customised using bootstrap activities. Read Create bootstrap actions for Amazon EMR cluster software installation. Currently, the cluster is BOOTSTRAPPING.

Amazon EMR may install native apps like Hive, Hadoop, Spark, and others when you establish the cluster. After startup and native application installation, the cluster is RUNNING. After connecting to cluster instances, the cluster will execute the sequential steps you selected when you established it. Submit further actions after prior steps are complete. Check out Submit work to an Amazon EMR cluster.

A successful step puts the cluster in WAITING.

Following the last phase, an auto-terminating cluster enters TERMINATING before terminating. Waiting requires manually shutting down the cluster. After a manual shutdown, the cluster enters TERMINATING before TERMINATED.

Amazon EMR terminates the cluster and all instances if a cluster lifecycle failure occurs without termination protection. If a cluster fails, its data is destroyed and its status changed to TERMINATED_WITH_ERRORS. If configured, you can restore data, deactivate termination protection, and end the cluster. Find out how termination protection can prevent unintended shutdown of Amazon EMR clusters.