#cloudrun

Deployment times for Etsy’s Service Platform on Cloud Run are reduced from days to less than an hour.

Overview

Popular online business Etsy sells vintage, handcrafted, and unusual items and attempts to give excellent service. Etsy needs more people, technology, and resources like many fast-growing organizations. Over 1400% of its gross product sales climbed to $13.5 billion between 2012 and 2021.

Etsy moved all of its infrastructure from conventional data centers to Google Cloud in an attempt to keep up with this development. In addition to being a major technology advancement, this change forced Etsy to reconsider how it approaches service development. The process resulted in the establishment of “ESP” (Etsy’s Service Platform), a Google Cloud Run-based service platform specifically designed for Etsy that simplifies microservices development, deployment, and administration.

The need for change and architectural vision

The need for technical team to handle more sophisticated features and more traffic in Google cloud marketplace increased along with Etsy’s growth. Etsy developers were able to investigate and use Google Cloud-based service platforms with 2018 transfer to GCP. However, this surge of technological innovation also brought out some new difficulties, such as redundant code and scaffolding and unsupported infrastructure with unclear ownership.

In order to overcome these obstacles, Etsy brought together a group of architects to create a blueprint outlining the direction of the company’s future service growth. The objective was unambiguous: establish a platform that frees developers from the burden of backend complexity and enables them to swiftly and securely launch new services by separating service development from infrastructure.

Transforming vision into reality

The resultant architectural concept served as the foundation for Etsy’s Service Platform, or ESP, and a newly assembled team was tasked with the thrilling task of making the vision a reality. Putting together a dynamic team that could bridge the gap between application development and infrastructure was the first step. The team, which was made up of seasoned engineers with a variety of specialties, contributed a wide range of abilities.

Understanding how critical it was to connect with future platform users, the team worked closely with Etsy’s engineering and architecture. By consenting to embed one of their senior engineers in the service platform team, the Ads Platform Team, which was previously involved in service development, played a crucial role. As part of the Etsy’s Service Platform experiment, they jointly produced a Minimum Viable Platform (MVP) to facilitate the rollout of a new Ads Platform service.

Choosing Cloud Run for accelerated development 

By separating infrastructure and automating its provisioning, architectural vision for a successful service platform would simplify the developer experience. The team realized that the bigger engineering organization’s prospective clients also need a platform that could seamlessly integrate into their workflow. The service platform team decided to concentrate on Etsy-specific elements in order to do this, including observability, service catalog, security, compliance, CI/CD, connection with current services, developer experience and language support, and more.

It was a calculated move to use Google Cloud services, particularly Cloud Run. The team intended to provide value as soon as possible, even if options like GKE were alluring. The team was able to concentrate on core platform functionality because to Cloud Run’s strong and user-friendly architecture, which helped Cloud Run manage the more difficult and time-consuming parts of executing containerized services.

The Toolbox: A Closer Look

Etsy’s Service Platform uses a well chosen toolkit to provide a reliable and effective development and operational experience:

Developer Interface: A specially designed CLI tool to make developer interactions more efficient.

Protocols for standardized communication include protobuf and gRPC.

Supported languages include Go, Python, Node.js, PHP, Java, and Scala.

CI/CD: Use GitHub Actions to provide a seamless pipeline for integration and deployment.

Observability: Using Prometheus, AlertManager, Google Monitoring and Logging, and OTEL on Google Cloud services

Client Library: Artifactory has Etsy’s Service Platform-generated clients registered.

Service Catalog: Centralized service visibility via Backstage.

Cloud Run was selected as the runtime due to its compatibility and ease of use.

Navigating Challenges

There were challenges along the way to developing the service platform. Overloading occurred on the VPC connection, and in order to maximize resource allocation, some services needed to be adjusted. Future adopters will benefit from platform-level enhancements brought forth by these difficulties.

Flexibility was given top priority in Etsy’s Service Platform design to account for varied technological environment. Despite the team’s multi-technological experience, it was difficult to develop a platform that could accommodate a wide range of service and client languages and use cases. Based on customer input, Google cloud made the decision to first concentrate on a core feature set and then add incremental capabilities and workarounds.

Important lessons learned throughout ESP’s development influenced both its ongoing operations and its future direction.

Sandbox Feature: Developers were able to deploy development versions of new services on Cloud Run in less than five minutes, replete with CI/CD and observability, with a rapid iteration process provided by a “sandbox” environment.

Known Observability Tools: ESP simplified engineer processes by integrating with current tools, such as promQL and Grafana.

Security Considerations: Working with the Google Serverless Networking team guaranteed safe connection with the old apps, even though ESP preferred TLS and layer 7 authentication via Google IAM.

Encouraging AI/ML Innovation: ESP’s flexibility was shown at a company-wide hackathon when a service that interfaced with Google’s Vertex AI was quickly put into use.

Real-World Success: As client support in new languages became available, the Ads Platform service grew to three more systems. The increasing load was effortlessly managed by Cloud Run’s auto-scaling.

Conclusion and Future Outlook

Etsy’s Service Platform is being steadily and continuously adopted across the company, allowing engineers to be bold, quick, and safe. Collaboration between Google cloud internal GKE team and Google has been sparked by customer needs for workloads beyond the serverless approach. Extending ESP’s tools to accommodate a growing range of services while preserving a consistently high standard of developer and operational experience is the aim.

Google Cloud Run

It’s no secret that Cloud Run provides one of the easiest methods available for deploying AI-powered applications into production, freeing developers from the burden of managing the underlying infrastructure or scaling from a small number of users to millions. However, did you know that a lot of clients also choose Cloud Run as their go-to platform for giving their AI researchers the resources they require to carry out and scale up their experiments outside of their reliable Python notebooks?

Upon top of the container runtime, Cloud Run offers several services that provide an all-inclusive platform for developing and executing AI-powered apps. Google Cloud outlines several of Cloud Run’s primary capabilities in this blog post, which can expedite the creation of AI-powered applications:

Time to market: by quickly transitioning from Vertex AI Studio prototyping to a deployed containerised application

Observability: by the use of Google Cloud observability technologies and the integrated SLO monitoring of Cloud Run

Rate of innovation: test several iterations of your service concurrently with updates and traffic division

Building RAG implementations by securely and immediately connecting to cloud databases is a relevant and factual approach.

By placing several Cloud Run services in front of a single external global application load balancer, multi-regional deployments and HA are made possible.

From using AI Studio for prototyping to releasing a Cloud Run service

Vertex AI Studio is the starting point for many new AI-based products since it enables quick prototyping on a variety of models without requiring the creation of code. From there, a convenient shortcut for converting experiments into code in a number of well-known programming languages is provided by the “Generate Code” feature.

A script that calls the Vertex AI APIs that provide the AI service makes up the resulting code snippet. The process of converting that script into a web application may be as simple as transforming the hardcoded prompt into a templated string and enclosing everything in a web framework, depending on the kind of application you are attempting to develop. This may be accomplished, for instance, in Python by enclosing the prompt in a little Flask application and parameterizing the request with a straightforward Python f-string:

Google Cloud can already containerise and launch its application with the help of a straightforward package.txt file that contains the necessary requirements. Google Cloud doesn’t even need to supply a Dockerfile describing how Google Cloud containers should be generated because of Cloud Run’s support for Buildpacks.

Use telemetry and SLOs to track the performance of your application

Ensuring that the programme satisfies user expectations and determining the business impact it generates are dependent on the implementation of observability. Out of the box, Cloud Run provides both observability and monitoring of Service Level Objectives (SLOs).

In order to manage your application based on error budgets and use that measure to strike a balance between stability and rate of innovation, it is crucial to monitor SLOs. SLO monitoring can be established using Cloud Run based on configurable metrics, latency, and availability.

In order to gather all the necessary data in one location, traditional observability such as logging, monitoring, and tracing is also readily available out of the box and seamlessly integrates with Google Cloud Observability. In particular, tracing has shown to be quite useful when examining the latency decomposition of AI applications. It is frequently applied to enhance comprehension of intricate orchestration situations and RAG implementations.

Quick invention combined with simultaneous updates and cloud deployment

Numerous AI use cases drastically alter Google Cloud’s problem-solving methodology. The end result is frequently unpredictable due to the nature of LLMs and the effects of variables like temperature or subtleties in prompting. Thus, being able to conduct experiments concurrently can facilitate rapid iteration and innovation.

With Cloud Run, developers may run multiple concurrent versions of different service revisions at once and have fine-grained control over how traffic is shared among them thanks to the built-in traffic splitting feature. This could entail serving various prompt iterations to various user groups and comparing them based on a shared success metric, such as click-through rate or likelihood of purchase, for AI applications.

A managed service called Cloud Deploy can be used to automatically plan the release of several iterations of a Cloud Run service. Additionally, it connects with your current development routines such that push events in source control can initiate a deployment pipeline.

Establishing a connection to cloud databases to incorporate company data

A static pre-trained model may not always be able to produce accurate results due to the absence of the domain-specific context. Retrieval-augmented generation (RAG) and other methods of adding extra data to the prompt frequently help give the model adequate contextual information to improve the relevance of the model’s responses for a given use case.Image Credit to Google Cloud

In order to use cloud databases like AlloyDB or Cloud SQL as a vector store for RAG implementations, Cloud Run offers direct and private connectivity from the orchestrating AI application. Cloud Run may now connect to private database endpoints without the additional step of a serverless VPC connector thanks to direct VPC egress capabilities.

Deployments across several regions and custom domains

Every Cloud Run service by default gets a URL in the form of <service-name>.<project-region-hash>.a.run.app, which can be used to make HTTP queries to the service. Although this is useful for internal services and rapid prototyping, it frequently causes two issues.

Firstly, the domain suffix does not correspond to the service provider, and the URL is not very memorable. As a result, users of the service are unable to determine whether it is a genuine offering. Not even the SSL certificate, which is granted to Google, divulges who owns the said service.

The second issue is that various areas will have different URLs if you grow your service to multiple regions in order to offer HA and lower latency to your distributed user base. This implies that changing service regions is not transparent to users and must be handled at the client or DNS level.

Both of these issues may be resolved with Cloud Run’s support for custom domain names and its ability to combine deployments of Cloud Run across several regions under a single external IP address based on anycast, all behind a global external load balancer. After setting up the load balancer and turning on Cloud launch’s outlayer traffic detection feature, you can launch your AI service with a custom domain, your own certificate, and automated failover in the event of a regional outage.

Let your AI software be powered by Cloud Run

Five key areas were examined by Google Cloud, which makes Cloud Run an ideal place to start when developing AI-powered applications on top of Vertex AI’s robust services.

Direct VPC egress

Google Cloud is introducing Direct VPC egress for Google Cloud Run to the general public (GA). With the help of this functionality, traffic from your Cloud Run resources can reach a VPC network directly, saving time and money by avoiding the need for proxying via Serverless VPC Access connectors.

Actually, with up to 1 GB per second per instance, Direct VPC egress offers around twice the throughput of both VPC connectors and the standard Google Cloud Run internet egress method. Direct VPC egress enables greater throughput and lower latency for performance-sensitive apps, whether you’re delivering traffic to destinations on the VPC, to other Google Cloud services like Cloud Storage, or to other destinations on the public internet.

Cloud Run

What has changed since the teaser

Notable enhancements and additions:

Direct VPC egress is now supported in all regions where Google Cloud Run is accessible.

Now, under quota management, every Google Cloud Run service revision with Direct VPC can scale to more than 100 instances. If you require even greater scalability, there is a defined procedure for requesting quota increases.

Direct VPC egress traffic is now included in VPC Flow Logs and Firewall Rules Logging, and Cloud NAT is supported.

The primary concerns raised by Google Cloud preview users particularly bigger clients with complex networking, scalability, and security needs are addressed in these changes.

Google cloud Run

Encrypting Data Between Cloud Run and VPC

To create communication between Cloud Run Google Cloud Run and VPC resources prior to Direct VPC Egress, developers used SVPC. SVPC was useful, but it had a number of drawbacks.

Management Overhead: For developers, setting up and overseeing connection virtual machines (VMs) inside the VPC for SVPC introduced a new level of complexity.

Scalability Restrictions: Due to the limited number of outgoing connections available on SVPC connectors, applications with large concurrent traffic demands were hampered.

Cost Incurrence: Using connection virtual machines (VMs) led to ongoing expenses, even in times when application activity was minimal.

These restrictions made it difficult for Google Cloud Run apps to seamlessly integrate with private and protected resources inside a VPC.

Direct VPC Egress

A Simplified Approach

Direct VPC Egress, a game-changing method of tying Google Cloud Run services to VPC resources, was introduced in 2023 and is currently generally accessible. It eschews the requirement for overseeing connection virtual machines and yields several significant advantages:

Simplified Configuration

Connector virtual machines are a thing of the past. Developers may concentrate on creating their apps because Direct VPC egress makes it easy to enable access to a particular VPC network with little effort.

Improved Scalability

Direct VPC egress makes use of the strong internal network fabric of Google Cloud. This feature makes Google Cloud Run instances perfect for applications with high traffic volumes since it gives them access to a large pool of outbound connections.

Direct VPC egress uses a pay-per-use paradigm for cost optimisation. There is no set cost involved in running connection virtual machines in SVPC; instead, you simply pay for the resources that your Cloud Run service uses.

Enhanced Security

Routing internet traffic is a part of traditional cloud run egress, which may be vulnerable to breaches. By keeping all communication inside Google Cloud’s secure internal network, direct VPC egress reduces potential security issues.

Granular Control

Revisions of Cloud Run may be associated with network tags. This gives developers the ability to create fine-grained network access control, specifying exactly which VPC resources particular versions are allowed to access.

Direct VPC Egress Operates

Your Cloud Run instances are assigned internal IP addresses within the specified VPC network by Google Cloud when you enable Direct VPC egress on a Cloud Run service. These instances can then immediately connect to resources in the VPC over secure networks. This promotes a more secure and effective communication channel and removes internet egress traffic.

Realising Potential

Applications of Direct VPC Egress

Direct VPC egress provides access to private resources inside a VPC for a variety of applications. The following are some strong use cases:

Database Connectivity

To facilitate data persistence and retrieval within your secure environment, Cloud Run services can establish direct connections with databases housed inside a VPC.

Interaction Between Internal Microservices

Cloud Run services are able to communicate with other microservices that are set up inside the VPC. This makes it possible for microservices to work together effectively without sacrificing security in a well-integrated and safe application architecture.

Data stored in private buckets or databases inside the VPC can be accessed and processed by Cloud Run services using secure data processing pipelines. This reduces the possibility of unauthorised access by guaranteeing that data is safely segregated throughout the processing pipeline.

Machine Learning Workflows

Models and training data are safely stored inside a VPC, and this is accessible to Cloud Run services. As a result, safe and effective machine learning workflows are promoted, with data security maintained during the training and deployment phases.

Launching Direct VPC Egress

A Smooth Transition

YAML files, the Google Cloud Console, and the Google Cloud CLI are some of the ways that direct VPC egress can be configured. Here’s a condensed rundown of the procedure:

Add the VPC network and subnet that your service needs access to in your Cloud Run service setup to enable direct VPC egress.

Ascertain Permissions

Make sure the service account linked to your Cloud Run service has the authorizations required to utilise the selected VPC resources.

Deploy Your Service

With the Direct VPC egress configuration enabled, deploy your Cloud Run service. To ensure smooth operation, test connectivity to make sure your Cloud Run service can properly communicate with the required VPC resources.

Security Aspects

It’s important to follow security best practices even though Direct VPC egress provides a secure method of connecting Cloud Run services to VPC resources:

Cloud Runには設定がないため、利用するコンテナイメージで指定する必要がある。