overview

Krixik Search Pipelines

Search Pipeline Overview

Search pipelines are those that enable document search on textual documents. These documents may be the initial pipeline input or they may be generated mid-pipeline, as may for instance be the case in pipelines that begin with transcription (where audio is converted to text) or image captioning (image to text) modules.

Such search capabilities are often employed in RAG (Retrieval-Augmented Generation) systems today, but the pipelines described in this section can also have general application with recommendation systems, image and video retrieval based on content similarity, and personalized content delivery, to name a few possibilitiess.

Two types of document search can be enabled: semantic search and keyword search. Depending on which of these is sought, the final module of the pipeline must respectively be vector-db or keyword-db.

Search pipelines are more complex than other pipelines because they require an additional step.

Files must first be "loaded" into the the pipeline with the process method.
The keyword_search method or the semantic_search method can be invoked on a search pipeline once at least one file has been processed through it. Keep in mind that the keyword_search method can only be invoked on a pipeline that ends with keyword-db, and the semantic_search method can only be invoked on a pipeline that ends with vector-db.

Search Pipeline Examples

Semantic Search: Enables semantic search on an input text file.
Semantic Search on Snippets: Enables semantic search on snippets in an input JSON file.
Keyword Search: Enables keyword search on an input text file.
Semantically-Searchable Transcription: Transcribes an input audio file and then enables semantic search on the transcript.
Keyword-Searchable Transcription: Transcribes an input audio file and then enables keyword search on the transcript.
Semantically-Searchable Translation: Translates an input text file and then enables semantic search on the translation.
Semantically-Searchable Translated Transcription: Transcribes an input audio file, translates it into English, and then enables semantic search on the translation.
Semantically-Searchable OCR: Extracts text from an input image and then enables semantic search on the extracted text.
Keyword-Searchable Image Captions: Generates a textual caption for an input image and then enables keyword search on the caption.