Incremental Enrichment In Azure Cognitive Search

Azure Cognitive Search is a cloud-based search engine with built-in AI features that enable users to search relevant content more easily. It uses Microsoft’s vast language stack in addition to AI features to facilitate advanced language and speech recognition-based searches. Recently, the Microsoft Azure team has introduced a new feature called Incremental Enrichment that brings a declarative approach to index search data. 

Turning on this new feature enables you to effortlessly perform document enrichment even as you continue to improve your skills. In this blog post, we cover this feature in detail.

More About Incremental Enrichment

Incremental Enrichment is a new feature in Azure Cognitive Search that improves search capabilities by extending change tracking in documents under the enrichment pipeline. With this update, indexers in Azure Cognitive Search can easily add documents from a given data source to the search index. As a result, they can track real-time document updates in the data source and update the index accordingly. 

In a nutshell, the incremental enrichment feature enables indexers to efficiently manage your documents in the given data source. The indexers are specific to data sources and can be configured to drive eventual consistency between the search index and data source.

Also read Keeping Users Connected Amid The COVID 19 Outbreak Using Cloud

 

How It Improves Azure Cognitive Search?

Previously, Azure Cognitive Search required re-enriching the entire corpus of documents each time a new skill was added. It essentially meant that the indexer required a complete reset. The same process was required for updating or deleting the existing skills. However, with the advent of the incremental enrichment feature, the indexer state management is successfully expanded to support skillsets, output field mappings, and projections. 

Incremental enrichment significantly improves the overall efficiency of your enrichment pipeline. With this new feature, you do not need to reset the entire indexer to update or add a new skill. In addition, you do not have to deal with the version drift instances where the updated documents vary in shape or quality.

Other Significant Benefits

Below are the main benefits of the incremental enrichment feature that was recently introduced in Azure Cognitive Search. 

Incremental Indexing

Incremental enrichment makes it possible to achieve incremental indexing by adding an indexer cache to the enrichment pipeline. It enables the indexer to cache results from each skill of the document. As a result, when re-indexing is required to update new skills, the indexer can read the recently enriched documents from the cache. In this way, the data source is updated with the new skills without resetting the entire state of the document. 

At the same time, the results are saved in cache as the document is successfully updated in the index. In this context, cache acts as a storage account for the indexer wherein each indexer holds a unique, immutable cache ID. 

Granular Controls 

Incremental enrichment provides granular controls to gain complete control over the dynamic changes happening in a data source. It provides an array of granular control features to monitor the indexer status and override change detection. As a result, you can successfully deal with the unfavorable results caused by a change in the data source. 

You may also be interested in reading Connecting Devices To The Cloud Using AWS IoT Core

 

The Final Word

Incremental enrichment provides an easy way to declaratively ensure that data present in the given data source is consistent with the data in its search index. Therefore, as you update your skills, the enrichment pipeline ensures that minimal efforts are required to drive the documents towards eventual consistency. The best approach to using this feature is configuring the cache property on a new indexer. Alternatively, you can reset the existing indexer and set the cache property. 

Avail Our Azure Cloud Development Services To Enhance Your Business Productivity

We, at Oodles Technologies, provide end-to-end Azure application development services to strengthen enterprise IT processes. Our development team is skilled at using Azure cloud services to enhance workforce productivity and aids better business efficiency. We formulate effective strategies to implement complex cloud features that improve workforce efficiency with increased security, transparency, accessibility, and interoperability. At Oodles, we have successfully completed several projects based on the Azure cloud for clients from across the globe.