#scann

similar posts here

| Une Saison Graphique 17 | 

AlloyDB ScaNN index

When developing a general intelligence or search application, you require real-time, high-quality results based on the most recent data for your application. Additionally, you need solutions that are affordable to run at scale and simple for developers to create and maintain.

Google Cloud is excited to announce today the general availability of the ScaNN for AlloyDB index, which will enable businesses to create scalable, accurate, and high-performing gen AI and search applications. For demanding enterprise workloads, including 4x faster transactional workloads and up to 100x faster analytical queries than regular PostgreSQL, AlloyDB for PostgreSQL is Google Cloud fully managed PostgreSQL-compatible solution. The ScaNN index now supercharges vector database workloads in AlloyDB for PostgreSQL by bringing 12 years of Google research, the same technology that powers some of Google’s billion+ user services.

With the help of the ScaNN index, AlloyDB combines the business features, multi-workload capacity, and query flexibility of a relational database with the efficiency of a vector-optimized database, providing precise results in record time and scalability to accommodate over a billion vectors.

ScaNN for AlloyDB index

With its ability to accommodate over a billion vectors and retain state-of-the-art query performance across the board, the ScaNN for AlloyDB index is the first PostgreSQL-compatible index that offers high performance at <25ms p95 latency at 1B vectors with 95% recall. In comparison to the HNSW index in regular PostgreSQL, it offers up to 4 times faster vector queries. Furthermore, AlloyDB’s ScaNN index has significant advantages for real-world workloads, including quick index creation times that boost developer productivity and cut down on costly bottlenecks for heavy lifting. Additionally, it uses memory efficiently, usually 3–4 times less than the regular PostgreSQL HNSW index.

This allows larger workloads to operate on smaller computers and enhances performance for hybrid workloads. Ultimately, AlloyDB enables users to appropriately scale for their unique workloads by supporting indices that are significantly larger than what can fit in main memory. When combined, these enhancements significantly increase the range of vector database workloads that AlloyDB can handle effectively and minimize the requirement for specialist tools.

AlloyDB has all the advantages of a relational database of commercial caliber while yet being affordable. With full PostgreSQL compatibility, developers may query vectors and scalar data simultaneously using an easy-to-use and versatile query language that supports a wide range of operations, from straightforward filters to intricate joins to hybrid searches on a single database. High write throughput and complete transactional consistency are supported by AlloyDB.

With ScaNN for AlloyDB index having up to 10 times the write throughput of normal PostgreSQL’s HNSW index, users with extremely high update rates may be guaranteed that their writes will be performant and easily queryable. Lastly, AlloyDB offers a 99.99% uptime SLA that is pleasant to customers and covers maintenance in addition to essential enterprise features like high availability, disaster recovery, data security, and more.

Professional developers who now utilize PostgreSQL, the most widely used open source database, may now easily use this technology with the new ScaNN index for AlloyDB, which is fully compatible with both PostgreSQL and pgvector.

Create indexes

Before you begin

The following requirements must be fulfilled before you can begin building indexes.

Within your AlloyDB database, an embedded vector is added to a table.

Installed is the pgvector extension, which Google enhanced for AlloyDB and is at least version 0.5.0.

CREATE EXTENSION IF NOT EXISTS vector;

Install the alloydb_scann extension in addition to the pgvector extension in order to produce ScaNN indexes.

CREATE EXTENSION IF NOT EXISTS alloydb_scann;

Create ScaNN index

Alloy DB Google’s alloydb_scann is a PostgreSQL modification that uses the ScaNN algorithm to create an extremely effective nearest-neighbor index.

For the approximate closest neighbor search, the ScaNN index is a quantization index based on trees. Compared to HNSW, it offers a shorter index creation time and a smaller memory footprint. On the basis of workload, it also offers faster QPS in contrast to HNSW.

Two-level ScaNN index tree

Use the following DDL query to apply a two-level tree index using the ScaNN algorithm to a column holding stored vector embeddings.CREATE INDEX INDEX_NAME ON TABLE   USING scann (EMBEDDING_COLUMN DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE); Replace the following: INDEX_NAME: the name of the index you want to create for example, my-scann-index. The index names are shared across your database. Ensure that each index name is unique to each table in your database. TABLE: the table to add the index to. EMBEDDING_COLUMN: a column that stores vector data. DISTANCE_FUNCTION: the distance function to use with this index. Choose one of the following: L2 distance: l2 Dot product: dot_product Cosine distance: cosine NUM_LEAVES_VALUE: the number of partitions to apply to this index. Set to any value between 1 to 1048576. For more information about how to decide this value, see Tune a ScaNN index.

Three-level ScaNN index tree

Use the following DDL query to generate a three-level tree index to a column containing stored vector embeddings using the ScaNN algorithm:CREATE INDEX INDEX_NAME ON TABLE   USING scann (EMBEDDING_COLUMN DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE, max_num_levels = MAX_NUM_LEVELS); Replace the following: MAX_NUM_LEVELS: the maximum number of levels of the K-means clustering tree. Set to (default) for two-level tree-based quantization and to for three-level tree-based quantization.

Once the index is created, you can use it to execute nearest-neighbor search queries by following the guidelines in Make a nearest-neighbor query using specified text.

A proper balance between recall and QPS must be achieved while setting the index parameters. See Tune a ScaNN index for additional details on adjusting the ScaNN index.

Cast the column into the vector data type in order to build this index on an embedding column that utilizes the real data type rather than vector:CREATE INDEX INDEX_NAME ON TABLE   USING scann (CAST(EMBEDDING_COLUMN AS vector(DIMENSIONS)) DISTANCE_FUNCTION)   WITH (num_leaves=NUM_LEAVES_VALUE, max_num_levels = MAX_NUM_LEVELS);

Substitute DIMENSIONS with the embedding column’s dimensional width. See the vector_dims function in Vector functions for additional details on how to determine the dimensions.

To view the indexing progress, use the pg_stat_progress_create_index view:SELECT * FROM pg_stat_progress_create_index;

The phase column displays the stage at which your index is currently being built. After the index is created, the tree training phase ends.