#emrstudio

EMR Notebook Programming Commands

Programmatic Amazon EMR Notebook interaction.

How to leverage execution APIs from a script or command line to control EMR notebook executions outside the AWS UI. This lets you list, characterise, halt, and start EMR notebook executions.

The following examples demonstrate these abilities:

AWS CLI: Amazon EMR clusters on Amazon EC2 and EMR Notebooks clusters (EMR on EKS) with notebooks in EMR Studio Workspaces are shown. An Amazon S3 location-based notebook execution sample is also provided. The displayed instructions can list executions by start time or start time and status, halt an ongoing execution, and describe a notebook execution.

Boto3 SDK (Python): Demo.py uses boto3 to interface with EMR notebook execution APIs. The script explains how to initiate a notebook execution, get the execution ID, describe it, list all running instances, and stop it after a short pause. Status updates and execution IDs are shown in this script's output.

Ruby SDK: Sample Ruby code shows notebook execution API calls and Amazon EMR connection setup. Example: describe execution, print information, halt notebook execution, start notebook execution, and get execution ID. Predicted Ruby notebook run outcomes are also shown.

Programmatic command parameters

Important parameters in these programming instructions are:

EditorId: EMR Studio workspace.

relative-path or RelativePath: The notebook file's path to the workspace's home directory. Pathways include my_folder/python3.ipynb and demo_pyspark.ipynb.

execution-engine or ExecutionEngine: EMR cluster ID (j-1234ABCD123) or EMR on EKS endpoint ARN and type to choose engine.

The IAM service role, such as EMR_Notebooks_DefaultRole, is defined.

notebook-params or notebook_params: Allows a notebook to receive multiple parameter values, eliminating the need for multiple copies. Typically, parameters are JSON strings.

The input notebook file's S3 bucket and key are supplied.

The S3 bucket and key where the output notebook will be stored.

notebook-execution-name: Names the performance.

This identifies an execution when describing, halting, or listing.

–from and –status: Status and start time filters for executions.

The console can also access EMR Notebooks as EMR Studio Workspaces, according to documentation. Workspace access and creation require additional IAM role rights. Programmatic execution requires IAM policies like StartNotebookExecution, DescribeNotebookExecution, ListNotebookExecutions, and iam:PassRole. EMR Notebooks clusters (EMR on EKS) require emr-container permissions.

The AWS Region per account maximum is 100 concurrent executions, and executions that last more than 30 days are terminated. Interactive Amazon EMR Serverless apps cannot execute programs.

Design and customise workspaces in an EMR studio to organise and operate notebooks. This section covers workspace construction and use.

Helpful EMR Studio Workspace topics:

Make an EMR Studio Workspace

Start an EMR Studio Workspace

Learn EMR Studio's Workspace UI.

See EMR Studio notebook examples.

Save EMR Studio Workspace content.

EMR Studio Workspace and notebook deletion

Know workspace status

Fix Workspace connectivity.

Make an EMR Studio Workspace

Create EMR Studio Workspaces to run notebook code.

To create an EMR studio workplace

Log into EMR Studio.

Select “Create a Workspace.”

Enter a workspace description and name. Naming workspaces helps find them on the Workspaces page.

Workspace collaboration allows Studio users to collaborate in real time on this workspace. Create collaborators after starting the Workspace.

Joining a cluster to a workspace requires expanding Advanced setup. You can add a cluster later. Refer to Attach CPU to EMR Studio Workspace for information.

Provisioning a new cluster requires administrator access.

After choosing a workspace cluster, attach the cluster.

Click Create a Workspace at the bottom.

After creating a workspace, EMR Studio opens the Workspaces page. The freshly established workspace is listed with a green success banner at the top.

Any Studio user can see shared workspaces by default. However, only one individual can utilise a workstation. You can collaborate with other users in EMR Studio using workspace collaboration.

Launch of EMR Studio Workspace

The notebook editor in a Workspace lets you deal with notebook files. The Workspaces page of a studio lists all accessible workspaces, along with their Name, Status, Creation time, and Last Modified.

Note

Your EMR notebooks from the previous Amazon EMR console may be in the console as EMR Studio Workspaces. IAM role rights are needed to access or create Workspaces in EMR Notebooks. You may need to refresh the Workspaces list to see a notebook you made in the last terminal.

To create a notepad and editing workspace

Your Studio's Workspaces page has the workspace. Keywords and column values can filter the list.

Select the workspace name to open it in a new browser tab. It may take several minutes to open the workspace if idle. Click Launch Workspace after selecting the Workspace row.

These launch options are available:

Quick launch: Use default settings to launch your workspace. Select Quick launch to connect clusters to JupyterLab.

Start your workstation with customisable settings. Launch Jupyter or JupyterLab, connect to an EMR cluster, and select security groups.

Note

How to Make EMR Notebook?

Amazon Web Services (AWS) has incorporated Amazon EMR Notebooks into Amazon EMR Studio Workspaces on the new Amazon EMR interface. Integration aims to provide a single environment for notebook creation and massive data processing. However, the new console's “Create Workspace” button usually creates notebooks.

Users must visit the Amazon EMR console at the supplied web URL and complete the previous console's procedures to create an EMR notebook. Users usually select “Notebooks” and “Create notebook” from this interface.

When creating a Notebook, users choose a name and a description. The next critical step is connecting the notebook to an Amazon EMR cluster to run the code.

There are two basic ways users associate clusters:

Select an existing cluster

If an appropriate EMR cluster is already operating, users can click “Choose,” select it from a list, and click “Choose cluster” to confirm. EMR Notebooks have cluster requirements, per documentation. These prerequisites, EMR release versions, and security problems are detailed in specialised sections.

Create a cluster

Users can also “Create a cluster” to have Amazon EMR create a laptop-specific cluster. This method lets users name their clusters. This workflow defaults to the latest supported EMR release version and essential apps like Hadoop, Spark, and Livy, however some configuration variables, such as the Release version and pre-selected apps, may not be modifiable.

Users can customise instance parameters by selecting EC2 Instance and entering the appropriate number of instances. A primary node and core nodes are identified. The instance type determines the maximum number of notebooks that can connect to the cluster, subject to constraints.

The EC2 instance profile and EMR role, which users can choose custom or default roles for, are also defined during cluster setup. Links to more information about these service roles are supplied. An EC2 key pair for cluster instance SSH connections can also be chosen.

Amazon EMR versions 5.30.0 and 6.1.0 and later allow optional but helpful auto-termination. For inactivity, users can click the box to shut down the cluster automatically. Users can specify security groups for the primary instance and notebook client instance, use default security groups, or use custom ones from the cluster's VPC.

Cluster settings and notebook-specific configuration are part of notebook creation. Choose a custom or default AWS Service Role for the notebook client instance. The Amazon S3 Notebook location will store the notebook file. If no bucket or folder exists, Amazon EMR can create one, or users can choose their own. A folder with the Notebook ID and NotebookName and.ipynb extension is created in the S3 location to store the notebook file.

If an encrypted Amazon S3 location is used, the Service role for EMR Notebooks (EMR_Notebooks_DefaultRole) must be set up as a key user for the AWS KMS key used for encryption. To add key users to key policies, see AWS KMS documentation and support pages.

Users can link a Git-based repository to a notebook in Amazon EMR. After selecting “Git repository” and “Choose repository”, pick from the list.

Finally, notebook users can add Tags as key-value pairs. The documentation includes an Important Note about a default tag with the key creatorUserID and the value set to the user's IAM user ID. Users should not change or delete this tag, which is automatically applied for access control, because IAM policies can use it. After configuring all options, clicking “Create Notebook” finishes notebook creation.

Users should note that these instructions are for the old console, while the new console now uses EMR Notebooks as EMR Studio Workspaces. To access existing notebooks as Workspaces or create new ones using the “Create Workspace” option in the new UI, EMR Notebooks users need extra IAM role rights. Users should not change or delete the notebook's default access control tag, which contains the creator's user ID. No notebooks can be created with the Amazon EMR API or CLI.

To ensure users can access and use the environment properly, Amazon EMR Studio setup involves many steps. Once you meet prerequisites, the process begins.

Setting up an EMR studio

Setup requirements for EMR Studio Before setting up, you need:

An AWS account

Establishing and running an EMR Studio.

A dedicated Amazon S3 bucket for EMR Studio notebook and workspace backups.

Five subnets and an Amazon VPC are recommended for Git repositories and connecting to Amazon EMR on EC2 or EKS clusters. EMR Studio works with EMR Serverless without VPC.

Setup steps Setup often involves these steps:

Choose an Authentication Mode: Choose IAM Identity Centre or IAM for your studio. User and permission management is affected by this decision. AWS IAM authenticates and IAM Identity Centre stores identities. Like IAM authentication or federation, IAM mode is compatible with many identity providers and straightforward to set up for identity management. IAM Identity Centre mode simplifies user and group assignment for Amazon EMR and AWS beginners. SAML 2.0 and Microsoft Active Directory integration simplifies multi-account federation.

Create the EMR Studio Service Role: An EMR Studio needs an IAM service role to create a secure network channel between Workspaces and clusters, store notebook files in Amazon S3, and access AWS Secrets Manager for Git repositories. This service role should describe all Amazon S3 notebook storage and AWS Secrets Manager Git repository access rights.

This role requires a trust policy from AWS to allow elasticmapreduce.amazonaws.com to play:AWS:SourceArn and SourceAccount settings for confused deputy prevention. After trust policy creation, you link an IAM permissions policy to the role. This policy must include permissions for Amazon EC2 tag-based access control and specific S3 read/write operations for your assigned S3 bucket. If your S3 bucket is encrypted, you need AWS KMS permissions. Some policy claims concerning tagging network interfaces and default security groups must remain unaltered for the service role to work.

Set EMR Studio user permissions: Set up user access policies to fine-tune Studio user access.

Create an EMR Studio user role to leverage IAM Identity Centre authentication. Sts:SetContext and AssumeRole allow elasticmapreduce.amazonaws.com to assume this role's trust relationship policy. You assign EMR Studio session policies to this user role before assigning users. Session policies provide Studio users fine-grained rights like creating new EMR clusters. The final permissions of a user depend on their session policy and EMR Studio user role. If a person belongs to multiple Studio groups, their permissions are a mix of group policies.

IAM authentication mode grants studio access via ABAC and IAM permissions policies. Allowing elasticmapreduce:CreateStudioPresignedUrl in a user's IAM permissions policy lets you use ARN or ABAC tags to limit the user to a Studio.

You specify one or more IAM permissions policies to describe user behaviours regardless of authentication mode. Workspace creation, cluster attachment and detachment, Git repository management, and cluster formation are basic, intermediate, and advanced rules with different authority. Clusters set data access control rights, not Studio user permissions.

(Optional) Create custom security groups to handle EMR Studio network traffic. If no custom security groups are selected, Studio uses defaults. When using custom security groups, specify a Workspace security group for outgoing access to clusters and Git repositories and an engine security group for inbound access.

Create an EMR Studio using the AWS CLI or Amazon EMR console. The interface creates an EMR Serverless application and offers simple configurations for interactive or batch workloads. ‘Custom’ gives full control over settings. Custom parameters include studio name, S3 location, workspace count, IAM or IAM Identity Centre authentication, VPC, subnets, and security groups. IAM authentication for federated users can include an IdP login URL and RelayState parameter name.

You must select EMR Studio Service and User Roles for IAM Identity Centre authentication. For speedier sign-on, enable trusted identity propagation. The AWS CLI tool create-studio requires programmatic creation options based on authentication method.

After building an EMR Studio, you may assign users and groups. Approach depends on authentication mode.

In IAM authentication mode, user assignment and permissions may require your identity provider. Limiting Studio access with ARN or ABAC tags and configuring the user's IAM rights policy to allow CreateStudioPresignedUrl does this.

AWS Documentation Essential IAM Administrators' EMR Studio Permissions

AWS EMR Studio Admin Permissions

Amazon Web Services documentation describes the IAM privileges administrators need to create and manage Amazon EMR Studio installations. AWS accounts need appropriate permissions to access EMR Studio resources safely and securely. The documentation helps administrators set up IAM policies for EMR Studio management access.

Running an EMR Studio requires certain IAM permissions for critical tasks. Administrators require elasticmapreduce permissions for normal tasks, especially without IAM Identity Centre authentication. EMR Studio creation requires the “elasticmapreduce:CreateStudio” permission.

The “elasticmapreduce:DescribeStudio” permission is needed to investigate a Studio's settings or status. Administrators need the “elasticmapreduce:ListStudios” access to see all EMR Studios in their account. Deactivating a Studio requires the “elasticmapreduce:DeleteStudio” access. In addition to these EMR-specific tasks, the handbook emphasises that Studio creation requires “iam:PassRole” access. The EMR service needs this permission to assume the Studio's service role and user role to communicate with other AWS services on behalf of the user or Studio.

Importantly, the handbook states that EMR Studios using IAM Identity Centre authentication require specific rights. These additional permissions' main duties are managing Studio Session Mappings, which control how users and groups authenticated through IAM Identity Centre can access and interact with the EMR Studio and the AWS IAM Identity Centre (formerly known as AWS SSO) and related directory services.

EMR Studio in IAM Identity Centre mode requires more complex permissions and actions to restrict user and group access. Individuals or groups are assigned to Studios using permissions from many AWS services.

These include “sso:AssociateProfile”, “sso:CreateApplicationAssignment”, “sso-directory:SearchUsers”, and “sso:DescribeUser” as well as rights like “elasticmapreduce:CreateStudioSessionMapping”. Also included in the assignment operations list are organisations and iam, with permissions like “organizations:DescribeOrganization” and “iam:ListPolicies”.

To retrieve user or group assignments, permissions like “elasticmapreduce:GetStudioSessionMapping” are needed. SSO-directory actions (“sso:SearchUsers” and “sso:DescribeUser”) and sso actions (“sso:DescribeApplication”) are also needed. Users and groups assigned to an EMR Studio are listed using “elasticmapreduce:ListStudioSessionMappings”. “elasticmapreduce:UpdateStudioSessionMapping” and sso-directory and sso privileges like “sso:SearchUsers”, “sso:DescribeApplication”, and “sso:DescribeInstance” are needed to alter a user or group's session policy.

Finally, deleting a Studio user or group requires permissions from sso-directory (“SearchUsers”, “DescribeGroup”), elasticmapreduce (“DeleteStudioSessionMapping”), and sso.

The AWS documentation provides sample IAM policies for both traditional IAM authentication and IAM Identity Centre authentication to help administrators set up these permissions. These images help create distinctive policies.

Administrators should fill out policy templates with their account and resource details. The placeholder values for the AWS Region code where the Studio will be placed, the AWS account ID, the Amazon Resource Name (ARN) of the object or objects the policy statement covers, and the EMR Studio service role and user role names must be changed.

Resource descriptions for service activities are vital to documentation, notably for the IAM Identity Centre sample policy. Identity Centre and Identity Centre directory APIs do not permit naming ARNs in IAM policy statements' “Resource” section, according to the specification.

In the sample policy for IAM Identity Centre mode, the “Resource” element is set to “” for sso and sso-directory service activities, authorising these actions across all resources those services support. The policy can be applied to Studio ARNs (e.g., “arn:aws:elasticmapreduce:\region>::studio/”) or role ARNs (e.g., “arn:aws:iam:::role/\EMRStudio-Service-Role>”) to allow elasticmapreduce activities. These services can be controlled more precisely at the resource level.

Amazon EMR Studio features, specs, and limitations:

Amazon EMR Studio describes an IDE for data preparation and visualisation, departmental collaboration, and application debugging. When utilising EMR Studio, consider tool usage, cluster demands, known issues, feature constraints, service limits, and regional availability.

Features of Amazon EMR Studio

Service Catalogue lets administrators connect EMR Studio to cluster templates. This lets users create Amazon EC2 EMR clusters for workspaces. Administrators can grant or deny Studio users access to cluster templates.

The Amazon EMR service role is needed to define access permissions to Amazon S3 notebook files or AWS Secrets Manager secrets because session policies do not allow them.

Multiple EMR Studios can control access to EMR clusters in different VPCs.

Use the AWS CLI to configure Amazon EMR on EKS clusters. Connect these clusters to Workspaces via a controlled API in Studio to run notebook jobs.

Amazon EMR and EMR Studio use trusted identity propagation, which has extra considerations. IAM Identity Centre and trusted identity propagation are required for EMR Studio to connect to EMR clusters that use it.

To secure Amazon EMR off-console applications, application hosting domains list their apps in the Public Suffix List (PSL). Examples are emrappui-prod.us-east-1.amazonaws.com, emrnotebooks-prod.us-east-1.amazonaws.com, and emrstudio-prod.us-east-1.amazonaws.com. For sensitive cookies in the default domain name, a __Host- prefix can prevent CSRF and add security.

EMR Studio Workspaces and Persistent UI endpoints use FIPS 140-certified cryptographic modules for encryption-in-transit, making the service suitable for regulated workloads.

Amazon EMR Studio requirements and compatibility

EMR Studio supports Amazon EMR Software versions 5.32.0 and 6.2.0.

EMR clusters using IAM Identity Centre with trusted identity propagation must use it.

Before setting up a Studio, disable browser proxy control applications like FoxyProxy or SwitchyOmega. Active proxies can cause Studio creation network failures.

Amazon EMR Studio restrictions and issues

EMR Studio does not support Python magic commands %alias, %alias_magic, %automagic, %macro, %%js, and %%javascript. Changing KERNEL_USERNAME or proxy_user using %env or %set_env or %configure is not supported.

Amazon EMR on EKS clusters does not support SparkMagic commands in EMR Studio.

All multi-line Scala statements in notebook cells must end with a period except the last.

Amazon EMR kernels on EKS clusters may timeout and fail to start. Should this happen, restart the kernel and close and reopen the notebook file. The Restart kernel operation requires restarting the Workspace, and EMR on EKS clusters may not work.

If a workspace is not connected to a cluster, starting a notebook and choosing a kernel fails. Choose a kernel and attach the workspace to run code, but ignore this error.

With Amazon EMR 6.2.0 security, the Workspace interface may be blank. For security-configured EMRFS S3 authorisation or data encryption, choose a different supported version. Troubleshooting EMR on EC2 tasks may disable on-cluster Spark UI connectivity. Run %%info in a new cell to regenerate these links.

5.32.0, 5.33.0, 6.2.0, and 6.3.0 Amazon EMR primary nodes do not have idle kernels cleaned away by Jupyter Enterprise Gateway. This may drain resources and crash long-running clusters. A script in the sources configures idle kernel cleanup for certain versions.

If the auto-termination policy is enabled on Amazon EMR versions 5.32.0, 5.33.0, 6.2.0, or 6.3.0, a cluster with an active Python3 kernel may be designated as inactive and terminated since it does not submit a Spark task. Amazon EMR 6.4.0 or later is recommended for Python3 kernel auto-termination.

Displaying a Spark DataFrame using %%display may truncate wide tables. Create a scrollable view by right-clicking the output and selecting Create New View for Output.

If you interrupt a running cell in a Spark-based kernel (PySpark, Spark, SparkR), the Spark task stays running. The on-cluster Spark UI is needed to end the job.

EMR Studio Workspaces as the root user in an AWS account causes a 403: Forbidden error because Jupyter Enterprise Gateway settings disallow root user access. Instead of root, employ alternate authentication methods for normal activities.

EMR Studio does not support Amazon EMR features:

connecting to and running tasks on Kerberos-secured clusters.

multi-node clusters.

AWS Graviton2-based EC2 clusters for EMR 6.x releases below 6.9.0 and 5.x releases below 5.36.1.

A studio utilising trusted identity propagation cannot provide these features:

Building EMR clusters without templates using serverless applications.

Amazon EMR launches on EKS clusters.

Use a runtime role.

Supporting SQL Explorer or Workspace collaboration.

Limited Amazon EMR Studio Service

Service Restriction The sources list EMR Studio service limits:

EMR Studios:

Each AWS account can have 100 max.

Maximum five subnets per EMR Studio.

IAM Identity Centre Groups are limited to five per EMR Studio.