#jupyterlab

Data science in 2025 is thriving, driven by a robust ecosystem of open-source tools that empower professionals to extract insights, build predictive models, and deploy data-driven solutions at scale. This year, the landscape is more dynamic than ever, with established favorites and emerging contenders shaping how data scientists work. Here’s an in-depth look at the best open-source tools that are defining data science in 2025.

1. Python: The Universal Language of Data Science

Python remains the cornerstone of data science. Its intuitive syntax, extensive libraries, and active community make it the go-to language for everything from data wrangling to deep learning. Libraries such as NumPy and Pandas streamline numerical computations and data manipulation, while scikit-learn is the gold standard for classical machine learning tasks.

NumPy: Efficient array operations and mathematical functions.

Pandas: Powerful data structures (DataFrames) for cleaning, transforming, and analyzing structured data.

scikit-learn: Comprehensive suite for classification, regression, clustering, and model evaluation.

Python’s popularity is reflected in the 2025 Stack Overflow Developer Survey, with 53% of developers using it for data projects.

2. R and RStudio: Statistical Powerhouses

R continues to shine in academia and industries where statistical rigor is paramount. The RStudio IDE enhances productivity with features for scripting, debugging, and visualization. R’s package ecosystem—especially tidyverse for data manipulation and ggplot2 for visualization—remains unmatched for statistical analysis and custom plotting.

Shiny: Build interactive web applications directly from R.

CRAN: Over 18,000 packages for every conceivable statistical need.

R is favored by 36% of users, especially for advanced analytics and research.

3. Jupyter Notebooks and JupyterLab: Interactive Exploration

Jupyter Notebooks are indispensable for prototyping, sharing, and documenting data science workflows. They support live code (Python, R, Julia, and more), visualizations, and narrative text in a single document. JupyterLab, the next-generation interface, offers enhanced collaboration and modularity.

Over 15 million notebooks hosted as of 2025, with 80% of data analysts using them regularly.

4. Apache Spark: Big Data at Lightning Speed

As data volumes grow, Apache Spark stands out for its ability to process massive datasets rapidly, both in batch and real-time. Spark’s distributed architecture, support for SQL, machine learning (MLlib), and compatibility with Python, R, Scala, and Java make it a staple for big data analytics.

65% increase in Spark adoption since 2023, reflecting its scalability and performance.

5. TensorFlow and PyTorch: Deep Learning Titans

For machine learning and AI, TensorFlow and PyTorch dominate. Both offer flexible APIs for building and training neural networks, with strong community support and integration with cloud platforms.

TensorFlow: Preferred for production-grade models and scalability; used by over 33% of ML professionals.

PyTorch: Valued for its dynamic computation graph and ease of experimentation, especially in research settings.

6. Data Visualization: Plotly, D3.js, and Apache Superset

Effective data storytelling relies on compelling visualizations:

Plotly: Python-based, supports interactive and publication-quality charts; easy for both static and dynamic visualizations.

D3.js: JavaScript library for highly customizable, web-based visualizations; ideal for specialists seeking full control.

Apache Superset: Open-source dashboarding platform for interactive, scalable visual analytics; increasingly adopted for enterprise BI.

Tableau Public, though not fully open-source, is also popular for sharing interactive visualizations with a broad audience.

7. Pandas: The Data Wrangling Workhorse

Pandas remains the backbone of data manipulation in Python, powering up to 90% of data wrangling tasks. Its DataFrame structure simplifies complex operations, making it essential for cleaning, transforming, and analyzing large datasets.

8. Scikit-learn: Machine Learning Made Simple

scikit-learn is the default choice for classical machine learning. Its consistent API, extensive documentation, and wide range of algorithms make it ideal for tasks such as classification, regression, clustering, and model validation.

9. Apache Airflow: Workflow Orchestration

As data pipelines become more complex, Apache Airflow has emerged as the go-to tool for workflow automation and orchestration. Its user-friendly interface and scalability have driven a 35% surge in adoption among data engineers in the past year.

10. MLflow: Model Management and Experiment Tracking

MLflow streamlines the machine learning lifecycle, offering tools for experiment tracking, model packaging, and deployment. Over 60% of ML engineers use MLflow for its integration capabilities and ease of use in production environments.

11. Docker and Kubernetes: Reproducibility and Scalability

Containerization with Docker and orchestration via Kubernetes ensure that data science applications run consistently across environments. These tools are now standard for deploying models and scaling data-driven services in production.

12. Emerging Contenders: Streamlit and More

Streamlit: Rapidly build and deploy interactive data apps with minimal code, gaining popularity for internal dashboards and quick prototypes.

Redash: SQL-based visualization and dashboarding tool, ideal for teams needing quick insights from databases.

Kibana: Real-time data exploration and monitoring, especially for log analytics and anomaly detection.

Conclusion: The Open-Source Advantage in 2025

Open-source tools continue to drive innovation in data science, making advanced analytics accessible, scalable, and collaborative. Mastery of these tools is not just a technical advantage—it’s essential for staying competitive in a rapidly evolving field. Whether you’re a beginner or a seasoned professional, leveraging this ecosystem will unlock new possibilities and accelerate your journey from raw data to actionable insight.

Working with AWS EMR Notebooks

Amazon EMR Notebooks, renamed EMR Studio Workspaces, simplify data processing cluster interaction. They use the popular open-source Jupyter Notebook or JupyterLab editors and are available from Amazon EMR. This may be more efficient than EMR cluster notebooks. Users with suitable IAM rights can open the editor in the console.

Notebook statuses

When and how to communicate with EMR Notebooks requires knowing their status. The numerous states you may encounter are listed below:

The notebook is being produced and connected to the cluster. Launching, stopping, removing, or changing the editor's cluster is currently impossible. It starts rapidly but can take longer if a cluster forms.

You can access the fully prepared notebook in the notebook editor. Stop or remove the notebook in this state. Stop the notebook before altering the cluster. A Ready notebook will shut down after a long inactivity.

The notebook has been produced, however cluster integration may require resource provisioning or additional steps. In this case, you can launch the notebook editor in local mode, but cluster-dependent code will fail.

Stopping: Laptop or cluster shutdown. Like the ‘Starting’ state, the editor cannot be opened, stopped, deleted, or clusters altered while stopping.

The laptop shut down successfully. You can delete the laptop, swap clusters, or restart it on the same cluster (assuming the cluster is still operating).

Notebook is being removed from console list. Even after the notebook entry is erased, Amazon S3 will charge for the notebook file (NotebookName.ipynb). To retrieve the latest status, reload the console's notebook list.

Working in Notebook Editor

The notebook editor starts when the notebook is Ready or Pending. You choose Open in JupyterLab or Jupyter after choosing the notebook from the list. This opens a new browser tab with the editor. After opening, select your programming language's kernel from the Kernel menu.

The console-accessible editor's ability to limit EMR notebooks to one user is critical. Opening an already-used notebook will result in an error. Amazon EMR produces a unique pre-signed URL for each session that is only valid for a short time, displaying security.

This URL should not be shared since recipients could inherit your rights and be at risk. IAM permissions policies and granting EMR Notebooks service role access to the Amazon S3 location are two strategies to control access.

Preserving Work

While editing, your notebook cells and output are automatically and occasionally saved to the Amazon S3 notebook file. When there are no modifications since the last save, the editor displays “autosaved,” and otherwise, “unsaved.” You can manually save the notebook by pressing CTRL+S or choosing Save and Checkpoint from File. Manual saves create a checkpoint file (NotebookName.ipynb) in the notebook's principal Amazon S3 folder's checkpoints folder. This site stores only the latest checkpoint.

Attached Cluster Change

Switching the cluster to which an EMR notebook is linked without affecting its content is useful. Only Stopped notebooks can accomplish this. The approach involves selecting the paused notebook, viewing its data, selecting the Change cluster, and then choosing an existing Hadoop, Spark, and Livy cluster or creating a new one. Finally, select the security group and click Change cluster and start laptop to confirm.

Delete Notebooks and Files

The Amazon EMR interface lets you remove an EMR notebook from your list. Importantly, this approach does not delete Amazon S3 notebook files. These S3 data continue to accrue storage fees.

To remove the notebook entry and files, delete the notebook from the console and note its Amazon S3 location (in the notebook details). The AWS CLI or Amazon S3 interface must be used to manually remove the folder and its contents from the S3 location. An example CLI command removes the notebook directory and its contents.

Share and Use Notebook Files

Every EMR notebook has a NotebookName.ipynb file in Amazon S3. If it works with EMR Notebook Jupyter Notebook, you can open a notebook file as an EMR notebook. Saving the.ipynb file locally and uploading it to Jupyter or JupyterLab makes using a notebook file from another user straightforward. This method can recover a console-erased notebook or work with publicly published Jupyter notebooks if you have the file.

A new EMR notebook can be created by replacing the S3 notebook file. Stop all running EMR notebooks and close any open editor sessions.

sasUEを用いてデータ解析を行なう場合、dataFremaにすることでデータ処理が便利になります。

今回の記事ではSASUEにあるjupyterlabを用いてdataFremaオブジェクト指向について記事にします。

pandas.DataFrameおよびpandas.Seriesのメソッドdescribe()を使うと、各列ごとに平均や標準偏差、最大値、最小値、最頻値などの要約統計量を取得できます

https://www.capablog2020.com/2020/05/20/sas-university-edition/ サンプルコード

import pandas as pd df = pd.read_csv('test20200820.csv') type(df) df.describe sas =…

Python Data Science: Treating Missing Values Using NumPy and Pandas in JupyterLab https://morioh.com/p/db55686f078c?f=5c21fb01c16e2556b555ab32

Python Data Science: Treating Missing Values Using NumPy and Pandas in JupyterLab https://morioh.com/p/db55686f078c?f=5c21fb01c16e2556b555ab32