#valkey

Migrating from Redis to Valkey? Read this before you break production.

Migrating enterprise caching layers isn't just about changing a port in your config file and spinning up a new container. It’s an exercise in extreme risk management.

With the recent licensing changes, everyone is rushing to Valkey. But here is the engineering reality: assuming absolute compatibility between Redis 7.2 and Valkey without an audit is a mistake. If your legacy instance relies on proprietary modules like RedisJSON, your replication will fail and Valkey will refuse to ingest the data.

Over on the ServerMO blog, we just dropped a FAANG-level SRE runbook for executing this migration safely on Ubuntu 24.04 LTS.

The Blueprint We Cover:

The Reality of RDB Snapshots: Why you must freeze AOF rewrites and verify your checksums before touching replication pipelines.

Deep TLS Enforcement: Moving beyond basic firewalls. We show you how to disable plaintext, enforce TLSv1.2/1.3, and validate client certificates natively.

The Cutover Pattern: Don't change your application configs and cause a massive cache-miss spike. We use HAProxy to shift traffic seamlessly at the network edge for zero-downtime execution.

Observability: Hooking up Prometheus Valkey exporters to Grafana so you can monitor p99 tail latency in real-time.

Stop relying on basic tutorials. Pair this architecture with ServerMO NVMe Bare Metal to bypass public cloud network jitter and get the raw memory bandwidth your caching layer needs.

Memorystore, Google Cloud's fully managed in-memory solution for Valkey, Redis, and Memcached, is used by over 90% of the top 100 clients. Client installs increasingly use memorystore. Google Cloud made Memorystore for Valkey publicly available, advancing open-source cloud in-memory data management. The GA supports Memorystore for Valkey for production workloads including Private Service Connect, multi-VPC access, cross-region replication, persistence, and more. Also has 99.99% availability SLA.

Memorystore for Valkey was adopted by hundreds of Google Cloud users, including MLB and Bandai Namco Studios Inc., after its August 2024 preview. They've provided us invaluable feedback in recent months, which has shaped their new service:

Memorystore has helped Major League Baseball enhance fan data transmission. Google Cloud makes Memorystore for Valkey, an open-source alternative, widely available. They believe its built-in flexibility and community-driven development will boost speed, scalability, and real-time data processing to better serve players, fans, and operations.

Memorystore supports Bandai Namco Studios' high-scale, low-latency performance, essential for many titles. Memorystore for Valkey's GA launch thrills them. We can increase global player base and real-time gameplay with its features, speed, and open-source nature. It's pleased to adopt Memorystore for Valkey's gaming innovation.

The latest at GA

Memorystore for Valkey has the following enterprise-grade features and a 99.99% SLA powered by Google's high availability and zonal placement algorithms at GA:

Customers can connect to 250 shards with two IP addresses using Valkey's Memorystore's Private Service Connect. Memorystore's discovery endpoint is highly available, so your cluster has no single point of failure.

Memorystore for Valkey lets your cluster grow to meet application demands with cost-optimized functionality and zero-downtime scalability. Supported cluster sizes are 1–250 nodes.

Integrated Google-built vector similarity search: Memorystore for Valkey provides in-memory vector search, over a billion vectors, 99% recall, and single-digit millisecond latency.

Memorystore for Valkey uses Google's vector search module, the Valkey OSS project's official search module, to achieve this performance. The module supports RAG, recommendation systems, semantic search, and other AI applications. Hybrid search improves user experience and application performance by providing more accurate and contextually relevant results.

Integrated managed backups automate migration, compliance, and disaster recovery.

CRR provides low-latency measurements and disaster recovery preparedness. Google Cloud now supports two clustered regions with varied replica counts in addition to the primary region. Valkey's memorystore syncs the control and data planes across geographic borders.

Multiple client-side VPCs can access a Valkey cluster Private Service Connection endpoint using Memorystore for Valkey. Customers may be safely linked across projects and VPCs utilising this technique.

Persistence: Memorystore for Valkey offers RDB-snapshot and AOF-logging persistence for diverse data durability demands.

Memorystore for Valkey supports Valkey 7.2 and its improved engine, Valkey 8.0:

Great performance: Memorystore for Valkey 8.0's asynchronous I/O improvements raise throughput and achieve 2x QPS of Memorystore for Redis Cluster at microsecond latency, making it easy to manage internet-scale workloads.

Even if its pricing is equal to Memorystore for Redis Cluster, Memorystore for Valkey's performance optimisations may result in considerable cost savings by requiring fewer nodes to handle the same workload.

Optimised memory efficiency: Valkey 8.0 optimises memory management to save memory and reduce operating costs for various workloads.

Scalability is more reliable with Valkey 8.0's Google-contributed automatic failover for empty shards and highly available migration states. They added migration states auto-reparing for system resilience.

Memorystore for Valkey offers maintenance windows, single zone and shard clusters, free inter-zone replication, and more.

Customer trust and Google Cloud's open source commitment After Redis modified Redis OSS's licensing in March 2024, the open-source community launched Valkey OSS, funded by Google, Amazon, Snap, and others.

Your trust is highly appreciated. Memorystore for Valkey on Google Cloud offers access to powerful, open technologies. The Linux Foundation endorses Valkey OSS, which, unlike Redis, is licensed under the BSD 3-clause. It has been watching Valkey build steam.

It supports and enhances Memorystore for Redis Cluster, Redis, and Valkey. When Memorystore for Redis clients are ready to transition to Valkey because to its open-source, reliability, and price-performance ratio, Google Cloud provides comprehensive migration aid. Memorystore for Valkey's compliance with Redis OSS 7.2 APIs and your favourite clients simplifies open source migration. You may also reuse Memorystore for Redis and cluster committed usage discounts to ease the transfer.

Now try Valkey's Memorystore

This article explains how to install Valkey on Ubuntu 24.04. Valkey is an in-memory key-value store database forked from the Redis code due to changes in the Redis license. Valkey can run as a standalone daemon or in a cluster and supports a collection of datatypes, including strings, numbers, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs, and more. The steps below walk you through…

Google is excited to launch Valkey 8.0 in preview form on Memorystore today, making Google Cloud the first significant cloud provider to provide Valkey 8.0 as a fully managed service. This strengthens Google Cloud’s dedication to open source, building on the release of Memorystore for Valkey 7.2 in August 2024 and giving you access to the newest and best features from the Valkey open-source community.

Major League Baseball (MLB) and other customers are examples of Google cloud dedication to customer service, as demonstrated by Valkey 8.0 on Memorystore. MLB, the most storied professional sports league, processes enormous volumes of data in real-time to give fans insights and statistics during games using Memorystore.

The release of Valkey 8.0

The open-source community banded together to establish Valkey a fully open-source alternative under the BSD 3-clause license, earlier this year after Redis Inc. altered the license of Redis OSS from the permissive BSD 3-Clause license to a restrictive Source Available License (RSAL). The open-source Valkey 8.0 was made available to the public in a matter of months by the Valkey community, demonstrating the strength of unrestricted innovation and open-source cooperation.

The latest version of the open-source NoSQL in-memory data store, Valkey 8.0, was released today, according to a statement from the Linux Foundation. With major upgrades intended to improve performance, dependability, and observability for all installations, Valkey 8.0 demonstrates quick innovation. By adding new features that elevate Valkey and building on the qualities of its past open source releases, this release strengthens the foundation.

The release’s main highlights are as follows:

Throughput on AWS r7g instances is improved up to 1.2 million requests per second with intelligent multi-core usage and asynchronous I/O threading, more than three times greater than the previous version.

Enhanced cluster scalability with replicated migration states and automated failover for additional shards

Dual-channel RDB and replica backlog streaming for faster replication

Granular visibility into performance and resource utilization is provided by extensive per-slot and per-client metrics, which include pubsub clients, rehash memory, event loop latency, and command-level heavy traffic recording.

Reduced memory overhead by up to 10% with improved key storage.

Valkey 8.0 may be downloaded right now at valkey.io, and those who already have it can update with a few quick keystrokes. Valkey can be installed using pre-made container images or constructed from source.

As a fully Google managed service, Memorystore for Valkey 8.0 offers better performance, increased dependability, and complete compatibility with Redis OSS.

Wth to recently introduced asynchronous I/O capabilities, the Valkey performance benchmarks have improved. By allowing the main thread and I/O threads to run simultaneously, the improved I/O threading system maximizes throughput by minimizing bottlenecks in handling incoming requests and enabling parallel processing of commands and I/O operations. Compared to Memorystore for Redis Cluster, Memorystore for Valkey 8.0 achieves up to a 2x Queries Per Second (QPS) at microsecond latency, enabling applications to handle higher throughput with clusters of same size. Because of this, Valkey 8.0 is an excellent option for real-time, high-throughput applications that need to deliver very rapid user experiences.

Valkey 8.0 has further optimizations in addition to the throughput gain that improve the service’s overall speed:

Set union procedures can be completed more quickly with the SUNION command.

To achieve faster execution speeds, changes have been made to the ZUNIONSTORE and SDIFF commands.

For expired keys, the DEL command prevents redundant deletions.

Responses from CLUSTER SLOTS are cached to improve throughput and lower latency during cluster activities.

Large data batches benefit from increased CRC64 performance, which is essential for RDB snapshot and slot migration scenarios.

Enhancements to key-memory efficiency are another feature of Valkey 8.0 that let you store more data without having to modify your application. Performance has improved and memory overhead has decreased because to the direct embedding of keys into the main dictionary. Furthermore, the main dictionary is divided into slots by the new per-slot dictionary, which reduces memory overhead by an additional 16 bytes per key-value pair without compromising performance.

Meanwhile, a number of Google-developed innovations that were later added to the project have increased the stability of Valkey 8.0 and greatly improved cluster resilience and availability:

Even in the early phases of scaling, automatic failover for empty shards contributes to high availability by facilitating the seamless failover of newly created, slotless shards.

Replicating slot migration states lowers the chance of data unavailability during failover events and allows new replicas to immediately inherit the right state. It also helps ensure that all CLUSTER SETSLOT commands are synchronized across replicas prior to execution on the main.

Furthermore, slot migration state recovery guarantees that, during a failover, the source and destination nodes are immediately updated, preserving precise request routing to the appropriate primary without the need for operator interaction.

These improvements increase the resilience of Valkey 8.0 clusters against failures during slot transfer, providing clients with the assurance that their data will remain accessible even during intricate scaling processes.

Compliant with Redis OSS version 7.2

Similar to Valkey 7.2, Redis OSS 7.2 APIs are fully backwards compatible with Valkey 8.0, facilitating a smooth transition away from Redis. Well-known Redis clients like Jedis, redis-py, node-redis, and go-redis are completely supported, negating the need for application code changes when transferring workloads to Valkey.

Because Valkey combines the flexibility of open-source software with the dependability of managed services, it offers you an ideal mix of control and simplicity when it comes to your Redis OSS workloads.

Start using Valkey 8.0 on Memorystore

Google cloud cordially encourage you to begin using Valkey 8.0 on Memorystore right now so you can see the aforementioned improvements. Memorystore’s Valkey 8.0 offers the performance, stability, and scalability that today’s high-demanding applications require with features like zero-downtime scaling, high availability, and RDB snapshot and AOF logging based persistence.

Begin by setting up a fully managed Valkey Cluster using the gcloud or Google Cloud console, and become a part of the expanding community that is influencing the direction of truly open-source data management.

Memorystore for Redis Cluster became the perfect platform for Gen AI application cases like Retrieval Augmented Generation (RAG), recommendation systems, semantic search, and more with the arrival of vector search earlier this year. Why? due to its exceptionally low latency vector search. Vector search over tens of millions of vectors may be done with a single Memorystore for Redis instance at a latency of one digit millisecond. But what happens if you need to store more vectors than a single virtual machine can hold?

Google is presenting vector search on the new Memorystore for Redis Cluster and Memorystore for Valkey, which combine three exciting features:

1) Zero-downtime scalability (in or out);

2) Ultra-low latency in-memory vector search;

3) Robust high performance vector search over millions or billions of vectors.

Vector support for these Memorystore products, which is now in preview, allows you to scale up your cluster by up to 250 shards, allowing you to store billions of vectors in a single instance. Indeed, vector search on over a billion vectors with more than 99% recall can be carried out in single-digit millisecond latency on a single Memorystore for Redis Cluster instance! Demanding enterprise applications, such semantic search over a worldwide corpus of data, are made possible by this scale.

Modular in-memory vector exploration

Partitioning the vector index among the cluster’s nodes is essential for both performance and scalability. Because Memorystore employs a local index partitioning technique, every node has an index partition corresponding to the fraction of the keyspace that is kept locally. The OSS cluster protocol has already uniformly sharded the keyspace, so each index split is about similar in size.

This architecture leads to index build times for all vector indices being improved linearly with the addition of nodes. Furthermore, adding nodes enhances the performance of brute-force searches linearly and Hierarchical Navigable Small World (HNSW) searches logarithmically, provided that the number of vectors remains constant. All in all, a single cluster may support billions of vectors that are searchable and indexable, all the while preserving quick index creation times and low search latencies at high recall.

Hybrid queries

Google is announcing support for hybrid queries on Memorystore for Valkey and Memorystore for Redis Cluster, in addition to better scalability. You can combine vector searches with filters on tag and numeric data in hybrid queries. Memorystore combines tag, vector, and numeric search to provide complicated query answers.

Additionally, these filter expressions feature boolean logic, allowing for the fine-tuning of search results to only include relevant fields by combining numerous fields. Applications can tailor vector search queries to their requirements with this new functionality, leading to considerably richer results than previously.

OSS Valkey in the background

The Valkey key-value datastore has generated a lot of interest in the open-source community. It has coauthored a Request for Comments (RFC) and are collaborating with the open source community to contribute its vector search capabilities to Valkey as part of its dedication to making it fantastic. The community alignment process begins with an RFC, and it encourage comments on its proposal and execution. Its main objective is to make it possible for Valkey developers worldwide to use Valkey vector search to build incredible next-generation AI applications.

The quest for a scalable and quick vector search is finished

In addition to the features already available on Memorystore for Redis, Memorystore now offers ultra-low latency across all of its most widely used engines with the addition of fast and scalable vector search on Memorystore for Redis Cluster and Memorystore for Valkey . Therefore, Memorystore will be difficult to beat for developing generative AI applications that need reliable and consistent low-latency vector search. To experience the speed of in-memory search, get started right now by starting a Memorystore for Valkey or Memorystore for Redis Cluster instance.