#vectorsearch

How CURIE Finds The Right Information In Seconds

Neural search, semantic search, and AI-powered discovery made practical — a hands-on playbook for content teams and marketers. We break down RAG, vector search, chunking, metadata, and UX patterns into actionable steps and a 90-day roadmap so you can boost discoverability, reduce hallucinations, and drive real results. Read the full guide →

A detailed look at AlloyDB's vector search improvements

Intelligent Adaptive Filtering Improves Vector Search Performance in AlloyDB AI

Google Cloud Next 2025: Google Cloud announced new ScaNN index upgrades for AlloyDB AI to improve structured and unstructured data search quality and performance. The Google Cloud Next 2025 advancements meet the increased demand for developers to create generative AI apps and AI agents that explore many data kinds.

Modern relational databases like AlloyDB for PostgreSQL now manage unstructured data with vector search. Combining vector searches with SQL filters on structured data requires careful optimisation for high performance and quality.

Filtered Vector Search issues

Filtered vector search allows specified criteria to refine vector similarity searches. An online store managing a product catalogue with over 100,000 items in an AlloyDB table may need to search for certain items using structured information (like colour or size) and unstructured language descriptors (like “puffer jacket”). Standard queries look like this:

Selected items: * WHERE text_embedding <-> Color=maroon, text-embedding-005, puff jacket, google_ml.embedding LIMIT 100

In the second section, the vector-indexed text_embedding column is vector searched, while the B-tree-indexed colour column is treated to the structured filter color='maroon'.

This query's efficiency depends on the database's vector search and SQL filter sequence. The AlloyDB query planner optimises this ordering based on workload. The filter's selectivity heavily influences this decision. Selectivity measures how often a criterion appears in the dataset.

Optimising with Pre-, Post-, and Inline Filters

AlloyDB's query planner intelligently chooses techniques using filter selectivity:

High Selectivity: The planner often employs a pre-filter when a filter is exceedingly selective, such as 0.2% of items being "maroon." Only a small part of data meets the criterion. After applying the filter (e.g., WHERE color='maroon'), the computationally intensive vector search begins. Using a B-tree index, this shrinks the candidate set from 100,000 to 200 products. Only this smaller set is vector searched (also known as a K-Nearest Neighbours or KNN search), assuring 100% recall in the filtered results.

Low Selectivity: A pre-filter that doesn't narrow the search field (e.g., 90% of products are “blue”) is unsuccessful. Planners use post-filter methods in these cases. First, an Approximate Nearest Neighbours (ANN) vector search using indexes like ScaNN quickly identifies the top 100 candidates based on vector similarity. After retrieving candidates, the filter condition (e.g., WHERE color='blue') is applied. This strategy works effectively for filters with low selectivity because many initial candidates fit the criteria.

Medium Selectivity: AlloyDB provides inline filtering (in-filtering) for filters with medium selectivity (0.5–10%, like “purple”). This method uses vector search and filter criteria. A bitmap from a B-tree index helps AlloyDB find approximate neighbours and candidates that match the filter in one run. Pre-filtering narrows the search field, but post-filtering on a highly selective filter does not produce too few results.

Learn at query time with adaptive filtering

Complex real-world workloads and filter selectivities can change over time, causing the query planner to make inappropriate selectivity decisions based on outdated facts. Poor execution tactics and results may result.

AlloyDB ScaNN solves this using adaptive filtration. This latest update lets AlloyDB use real-time information to determine filter selectivity. This real-time data allows the database to change its execution schedule for better filter and vector search ranking. Adaptive filtering reduces planner miscalculations.

Get Started

These innovations, driven by an intelligent database engine, aim to provide outstanding search results as data evolves.

Data accessibility can hinder AI adoption since so much data is unstructured and unmanaged. Data should be accessible, actionable, and revolutionary for businesses. A data cloud based on open standards, that connects data to AI in real-time, and conversational data agents that stretch the limits of conventional AI are available today to help you do this.

An open real-time data ecosystem

Google Cloud announced intentions to combine BigQuery into a single data and AI use case platform earlier this year, including all data formats, numerous engines, governance, ML, and business intelligence. It also announces a managed Apache Iceberg experience for open-format customers. It adds document, audio, image, and video data processing to simplify multimodal data preparation.

Volkswagen bases AI models on car owner’s manuals, customer FAQs, help center articles, and official Volkswagen YouTube videos using BigQuery.

New managed services for Flink and Kafka enable customers to ingest, set up, tune, scale, monitor, and upgrade real-time applications. Data engineers can construct and execute data pipelines manually, via API, or on a schedule using BigQuery workflow previews.

Customers may now activate insights in real time using BigQuery continuous queries, another major addition. In the past, “real-time” meant examining minutes or hours old data. However, data ingestion and analysis are changing rapidly. Data, consumer engagement, decision-making, and AI-driven automation have substantially lowered the acceptable latency for decision-making. The demand for insights to activation must be smooth and take seconds, not minutes or hours. It has added real-time data sharing to the Analytics Hub data marketplace in preview.

Google Cloud launches BigQuery pipe syntax to enable customers manage, analyze, and gain value from log data. Data teams can simplify data conversions with SQL intended for semi-structured log data.

Connect all data to AI

BigQuery clients may produce and search embeddings at scale for semantic nearest-neighbor search, entity resolution, semantic search, similarity detection, RAG, and recommendations. Vertex AI integration makes integrating text, photos, video, multimodal data, and structured data easy. BigQuery integration with LangChain simplifies data pre-processing, embedding creation and storage, and vector search, now generally available.

It previews ScaNN searches for large queries to improve vector search. Google Search and YouTube use this technology. The ScaNN index supports over one billion vectors and provides top-notch query performance, enabling high-scale workloads for every enterprise.

It is also simplifying Python API data processing with BigQuery DataFrames. Synthetic data can replace ML model training and system testing. It teams with Gretel AI to generate synthetic data in BigQuery to expedite AI experiments. This data will closely resemble your actual data but won’t contain critical information.

Finer governance and data integration

Tens of thousands of companies fuel their data clouds with BigQuery and AI. However, in the data-driven AI era, enterprises must manage more data kinds and more tasks.

BigQuery’s serverless design helps Box process hundreds of thousands of events per second and manage petabyte-scale storage for billions of files and millions of users. Finer access control in BigQuery helps them locate, classify, and secure sensitive data fields.

Data management and governance become important with greater data-access and AI use cases. It unveils BigQuery’s unified catalog, which automatically harvests, ingests, and indexes information from data sources, AI models, and BI assets to help you discover your data and AI assets. BigQuery catalog semantic search in preview lets you find and query all those data assets, regardless of kind or location. Users may now ask natural language questions and BigQuery understands their purpose to retrieve the most relevant results and make it easier to locate what they need.

It enables more third-party data sources for your use cases and workflows. Equifax recently expanded its cooperation with Google Cloud to securely offer anonymized, differentiated loan, credit, and commercial marketing data using BigQuery.

Equifax believes more data leads to smarter decisions. By providing distinctive data on Google Cloud, it enables its clients to make predictive and informed decisions faster and more agilely by meeting them on their preferred channel.

Its new BigQuery metastore makes data available to many execution engines. Multiple engines can execute on a single copy of data across structured and unstructured object tables next month in preview, offering a unified view for policy, performance, and workload orchestration.

Looker lets you use BigQuery’s new governance capabilities for BI. You can leverage catalog metadata from Looker instances to collect Looker dashboards, exploration, and dimensions without setting up, maintaining, or operating your own connector.

Finally, BigQuery has catastrophe recovery for business continuity. This provides failover and redundant compute resources with a SLA for business-critical workloads. Besides your data, it enables BigQuery analytics workload failover.

Gemini conversational data agents

Global organizations demand LLM-powered data agents to conduct internal and customer-facing tasks, drive data access, deliver unique insights, and motivate action. It is developing new conversational APIs to enable developers to create data agents for self-service data access and monetize their data to differentiate their offerings.

Conversational analytics

It used these APIs to create Looker’s Gemini conversational analytics experience. Combine with Looker’s enterprise-scale semantic layer business logic models. You can root AI with a single source of truth and uniform metrics across the enterprise. You may then use natural language to explore your data like Google Search.

LookML semantic data models let you build regulated metrics and semantic relationships between data models for your data agents. LookML models don’t only describe your data; you can query them to obtain it.

Data agents run on a dynamic data knowledge graph. BigQuery powers the dynamic knowledge graph, which connects data, actions, and relationships using usage patterns, metadata, historical trends, and more.

Last but not least, Gemini in BigQuery is now broadly accessible, assisting data teams with data migration, preparation, code assist, and insights. Your business and analyst teams can now talk with your data and get insights in seconds, fostering a data-driven culture. Ready-to-run queries and AI-assisted data preparation in BigQuery Studio allow natural language pipeline building and decrease guesswork.

Connect all your data to AI by migrating it to BigQuery with the data migration application. This product roadmap webcast covers BigQuery platform updates.

Memorystore for Redis Cluster became the perfect platform for Gen AI application cases like Retrieval Augmented Generation (RAG), recommendation systems, semantic search, and more with the arrival of vector search earlier this year. Why? due to its exceptionally low latency vector search. Vector search over tens of millions of vectors may be done with a single Memorystore for Redis instance at a latency of one digit millisecond. But what happens if you need to store more vectors than a single virtual machine can hold?

Google is presenting vector search on the new Memorystore for Redis Cluster and Memorystore for Valkey, which combine three exciting features:

1) Zero-downtime scalability (in or out);

2) Ultra-low latency in-memory vector search;

3) Robust high performance vector search over millions or billions of vectors.

Vector support for these Memorystore products, which is now in preview, allows you to scale up your cluster by up to 250 shards, allowing you to store billions of vectors in a single instance. Indeed, vector search on over a billion vectors with more than 99% recall can be carried out in single-digit millisecond latency on a single Memorystore for Redis Cluster instance! Demanding enterprise applications, such semantic search over a worldwide corpus of data, are made possible by this scale.

Modular in-memory vector exploration

Partitioning the vector index among the cluster’s nodes is essential for both performance and scalability. Because Memorystore employs a local index partitioning technique, every node has an index partition corresponding to the fraction of the keyspace that is kept locally. The OSS cluster protocol has already uniformly sharded the keyspace, so each index split is about similar in size.

This architecture leads to index build times for all vector indices being improved linearly with the addition of nodes. Furthermore, adding nodes enhances the performance of brute-force searches linearly and Hierarchical Navigable Small World (HNSW) searches logarithmically, provided that the number of vectors remains constant. All in all, a single cluster may support billions of vectors that are searchable and indexable, all the while preserving quick index creation times and low search latencies at high recall.

Hybrid queries

Google is announcing support for hybrid queries on Memorystore for Valkey and Memorystore for Redis Cluster, in addition to better scalability. You can combine vector searches with filters on tag and numeric data in hybrid queries. Memorystore combines tag, vector, and numeric search to provide complicated query answers.

Additionally, these filter expressions feature boolean logic, allowing for the fine-tuning of search results to only include relevant fields by combining numerous fields. Applications can tailor vector search queries to their requirements with this new functionality, leading to considerably richer results than previously.

OSS Valkey in the background

The Valkey key-value datastore has generated a lot of interest in the open-source community. It has coauthored a Request for Comments (RFC) and are collaborating with the open source community to contribute its vector search capabilities to Valkey as part of its dedication to making it fantastic. The community alignment process begins with an RFC, and it encourage comments on its proposal and execution. Its main objective is to make it possible for Valkey developers worldwide to use Valkey vector search to build incredible next-generation AI applications.

The quest for a scalable and quick vector search is finished

In addition to the features already available on Memorystore for Redis, Memorystore now offers ultra-low latency across all of its most widely used engines with the addition of fast and scalable vector search on Memorystore for Redis Cluster and Memorystore for Valkey . Therefore, Memorystore will be difficult to beat for developing generative AI applications that need reliable and consistent low-latency vector search. To experience the speed of in-memory search, get started right now by starting a Memorystore for Valkey or Memorystore for Redis Cluster instance.

Spanner is the globally consistent, always-on database with almost infinite scalability that powers a plethora of contemporary applications in many industries. It is now the cornerstone of many forward-thinking companies’ data-driven transformations. Google is launching Spanner throughout graph today, along with new features like vector search, full-text search, and more. With the help of these advancements, businesses can create intelligent apps that seamlessly integrate several data models and condense their operational deployments onto a single, potent database.

From Spanner versions, you may choose the Standard Edition, Enterprise, and Enterprise Plus editions with the right set of capabilities to fit your needs and budget.

With Spanner Graph, you can easily and succinctly match patterns, navigate relationships, and filter results across interconnected data for popular graph use cases like fraud detection, community discovery, and personalised recommendations.

Building on Google’s many years of search experience, advanced full-text search offers stronger matching and relevance ranking than unstructured text.

Vector search builds on more than a decade of Google research and innovation in approximate closest neighbour algorithms to facilitate semantic information retrieval, the cornerstone of generative  AI applications.

Furthermore, in response to Google’s enterprise customers’ increasingly complicated needs regarding latency, cost, and compliance, Google has improved Spanner’s basic infrastructure.

With geo-partitioning, you can store some of your data in particular places to provide quick local access while still deploying globally.

Data sovereignty needs are respected and multi-regional availability attributes are provided via dual-region arrangements.

By dynamically adjusting the size of your Spanner deployment, auto-scaling frees you from having to overprovision and enables you to respond swiftly to changes in traffic patterns.

Presenting the Spanner editions

Google is pleased to present Spanner editions, a new tiered pricing structure that enables you to benefit from these advancements while improving cost transparency and cost-saving options. You can select the Standard, Enterprise, and Enterprise Plus editions from Spanner editions to match your demands and budget with the appropriate collection of features.

The Standard edition is a feature-rich package that comes with additional features like scheduled backups and reverse ETL from BigQuery in addition to all of the current capabilities that are generally available (GA) in single-region settings. The Enterprise edition leverages multi-model features, such as vector search, enhanced full-text search, and the new Spanner Graph, to enhance the Standard edition.

Moreover, it offers improved data security and operational ease with incremental backups and automated autoscaling. The new geo-partitioning feature and multi-region setups with 99.999% availability are included in the Enterprise Plus version for workloads that demand the greatest levels of availability, performance, compliance, and governance.

Let’s examine each of the Spanner editions in more detail

Standard Edition

Everything you adore about Spanner is present in the standard edition, and then some. Access to Spanner’s single-region configurations, which include transparent failover, synchronous replication, and automatic sharding for almost infinite scalability and 99.99% availability, is provided by the Standard edition. All of the features available in the GA today, including Google’s enterprise-grade security and data protection measures, are included in the standard edition.

Furthermore, Google is adding new features to the Standard edition. Google is growing their offering with additional features like reverse ETL and external datasets with BigQuery, in addition to Spanner’s strong connection with BigQuery, which includes features like Data Boost and BigQuery federation. Google is also added scheduled backups to further its data protection capabilities. These improved features in the Standard edition are available to all Spanner customers.

Customers can gain from Standard edition in the following ways:

Migrate your relational workloads to Spanner to save operational toil, minimise costly scheduled and unplanned downtime, and cut the overall cost of ownership.

Shard MySQL or other complicated application-level sharding using Spanner’s automatic sharding instead.

Spanner’s nativePostgreSQL interface lets you modernise your apps without being locked into outdated features.

Elevate your NoSQL workloads with multirow transactions and stable secondary indexes thanks to support for adapters for Cassandra and DynamoDB that are compatible with APIs.

Avoid complex ETL pipelines and data silos by consolidating your data into a single source-of-truth database and running analytics on up-to-date data with Spanner Data Boost and BigQuery federation.

Utilise reverse ETL to service complicated analytics use-cases from Spanner and BigQuery to compute them at high QPS and low latency.

Enterprise Edition

Expanded data modalities are available to you with the Enterprise edition, enabling you to combine your operational deployments and provide your applications with true multi-model access. Spanner Graph, full-text search, and vector search capabilities are available for your data with the Enterprise edition, in addition to the relational and NoSQL functionality that Spanner currently offers.

The best part is that by combining all of this functionality, Spanner’s primary advantages of high availability, scalability, and consistency can be preserved in a genuine, completely interoperable multi-model database. Having all of your data in one location eliminates the need to transfer data between transactional and speciality databases, allowing you to minimise data silos and duplication, streamline data governance, cut down on operational complexity, and eventually minimise total cost of ownership.

Customers can gain from Enterprise edition in the following ways:

By fusing the advantages of SQL and GQL, you can supercharge your applications with Spanner Graph capabilities at nearly infinite scale. This allows analysts and developers to query linked and structured data in a single action.

Use full-text search to modernise your search apps and streamline your operational pipelines. You may leverage sophisticated search features without copying and indexing data into a separate search solution.

By basing fundamental models on contextual, domain-specific, and real-time data with native vector search capability in your operational data, you may enhance the AI-assisted user experience.

Launch robust multi-model applications, such a generative  AI product that grounds LLMs using relational and knowledge graph RAG approaches, or a searchable product catalogue that mixes traditional and semantic information retrieval through the combination of full-text and vector search.

Enterprise Plus Edition

Spanner’s most feature-rich and widely available edition is called Enterprise Plus. Enterprise Plus includes new dual-region configurations in addition to our 99.999% multi-region options, which let you take advantage of Spanner’s unparalleled availability while also adhering to your data sovereignty needs. You may run a truly global deployment with Enterprise Plus and still retain low latency and data locality for parts of your data by utilising geo-partitioning.

Customers can gain from Enterprise Plus edition in the following ways:

By using Spanner’s multi-region settings for application deployment, you can achieve real 0-RTO and 0-RPO deployments along with 99.999% availability, eliminating the need to construct and manage complex disaster recovery solutions.

Use Spanner’s dual-region configurations in places like Australia, Germany, India, and Japan to meet data sovereignty needs without sacrificing availability.

Partition your data to make it always available where you need it, latency-free, and manage your worldwide deployment while preserving data locality.

Increased openness and cost efficiency

Google is also introducing a more straightforward pricing approach with the new Spanner editions, shifting to per-replica charging and separating the expenses of compute and data replication to improve efficiency and cost transparency. Each edition has various cost-saving features, and switching to a new edition is meant to be a smooth process.

Customers in a single region can purchase the standard edition for the same price. Instead than paying multiple prices for every function, customers can choose the Enterprise version, which has a single price point, if they want increased multi-model and search capabilities. They can also take advantage of the price-performance improvements like incremental backups and managed autoscaler.

Even more, the Enterprise Plus edition offers lower compute expenses, separates computation and replication, and does not charge for witness replica storage. Finally, since configurable read-only replicas will be priced the same as read-write replicas for each edition, customers employing these replicas will benefit from significant cost savings. Making the switch to Spanner editions is easy, and you can take advantage of all the improvements Spanner has to offer while cutting prices.

The availability of the Spanner versions is scheduled for September 24, 2024. See the overview page for Spanner versions for further details.

The Tower of Babel reborn? Using vector search and multilingual embeddings in BigQuery Finding and comprehending reviews in a customer’s favourite language across many languages can be difficult in today’s globalised marketplace. Large datasets, including reviews, may be managed and analysed with BigQuery.

In order to enable customers to search for products or company reviews in their preferred language and obtain results in that language, google cloud describe a solution in this blog post that makes use of BigQuery multilingual embeddings, vector index, and vector search. These technologies translate textual data into numerical vectors, enabling more sophisticated search functions than just matching keywords. This improves the relevancy and accuracy of search results.

Vector Index

A data structure called a Vector Index is intended to enable the vector index function to carry out a more effective vector search of embeddings. In order to enhance search performance when vector index is possible to employ a vector index, the function approximates nearest neighbour search method, which has the trade-off of decreasing recall and yielding more approximate results.

Authorizations and roles

You must have the bigquery tables createIndex IAM permission on the table where the vector index is to be created in order to create one. The bigquery tables deleteIndex permission is required in order to drop a vector index. The rights required to operate with vector indexes are included in each of the preset IAM roles listed below:

Establish a vector index

The build VECTOR INDEX data definition language (DDL) statement can be used to build a vector index.

Access the BigQuery webpage.

Run the subsequent SQL statement in the query editor

Swap out the following:

The vector index you’re creating’s name is vector index. The index and base table are always created in the same project and dataset, therefore these don’t need to be included in the name.

Dataset Name: The dataset name including the table.

Table Name: The column containing the embeddings data’s name in the table.

Column Name:The column name containing the embeddings data is called Column name. ARRAY is the required type for the column. No child fields may exist in the column. The array’s items must all be non null, and each column’s values must have the same array dimensions. Stored Column Name: the vector index’s storage of a top-level table column name. A column cannot have a range type. If a policy tag is present in a column or if the table has a row-level access policy, then stored columns are not used. See Store columns and pre-filter for instructions on turning on saved columns.

Index Type:The vector index building algorithm is denoted by Index type. There is only one supported value: IVF. By specifying IVF, the vector index is constructed as an inverted file index (IVF). An IVF splits the vector data according to the clusters it created using the k-means method. These partitions allow the vector search function to search the vector data more efficiently by limiting the amount of data it must read to provide a result.

Distance Type: When utilizing this index in a vector search, distance type designates the default distance type to be applied. COSINE and EUCLIDEAN are the supported values. The standard is EUCLIDEAN.

While the distance utilised in the vector search function may vary, the index building process always employs EUCLIDEAN distance for training.

The Diatance type value is not used if you supply a value for the distance type argument in the vector search function. Num Lists: an INT64 value that is equal to or less than 5,000 that controls the number of lists the IVF algorithm generates. The IVF method places data points that are closer to one another on the same list, dividing the entire data space into a number of lists equal to num lists. A smaller number for num lists results in fewer lists with more data points, whereas a bigger value produces more lists with fewer data points.

To generate an effective vector search, utilise num list in conjunction with the fraction lists to search argument in the vector list function. Provide a low fraction lists to search value to scan fewer lists in vector search and a high num lists value to generate an index with more lists if your data is dispersed among numerous small groups in the embedding space. When your data is dispersed in bigger, more manageable groups, use a fraction lists to search value that is higher than num lists. Building the vector index may take longer if you use a high num lists value.

In addition to adding another layer of refinement and streamlining the retrieval results for users, google cloud’s solution translates reviews from many languages into the user’s preferred language by utilising the Translation API, which is easily integrated into BigQuery. Users can read and comprehend evaluations in their preferred language, and organisations can readily evaluate and learn from reviews submitted in multiple languages. An illustration of this solution can be seen in the architecture diagram below.

Google cloud took business metadata (such address, category, and so on) and review data (like text, ratings, and other attributes) from Google Local for businesses in Texas up until September 2021. There are reviews in this dataset that are written in multiple languages. Google cloud’s approach allows consumers who would rather read reviews in their native tongue to ask inquiries in that language and obtain the evaluations that are most relevant to their query in that language even if the reviews were originally authored in a different language.

For example, in order to investigate bakeries in Texas, google cloud asked, “Where can I find Cantonese-style buns and authentic Egg Tarts in Houston?” It is difficult to find relevant reviews among thousands of business profiles for these two unique and frequently available bakery delicacies in Asia, but less popular in Houston.

Google cloud system allows users to ask questions in Chinese and get the most appropriate answers in Chinese, even if the reviews were written in other languages at first, such Japanese, English, and so on. This solution greatly improves the user’s ability to extract valuable insights from reviews authored by people speaking different languages by gathering the most pertinent information regardless of the language used in the reviews and translating them into the language requested by the user.

Consumers may browse and search for reviews in the language of their choice without encountering any language hurdles; you can then utilise Gemini to expand the solution by condensing or categorising the reviews that were sought for. By simply adding a search function, you may expand the application of this solution to any product, business reviews, or multilingual datasets, enabling customers to find the answers to their inquiries in the language of their choice. Try it out and think of additional useful data and AI tools you can create using BigQuery!