🌟 DeepSearcher: AI Harvester for Your Data.

The project combines the use of LLM, vector databases to perform search, evaluation, and reasoning tasks based on the provided data (files, text, sources).

It is positioned by developers as a tool for enterprise knowledge management, intelligent QA systems and information search scenarios.

DeepSearcher can use information from the Internet if necessary, is compatible with Milvus vector databases and their service provider Zilliz Cloud, Pymilvus, OpenAI and VoyageAI embeddings. It is possible to connect LLM DeepSeek and OpenAI via API directly or through TogetherAI and SiliconFlow.
Local file download, connection of web crawlers FireCrawl, Crawl4AI and Jina Reader are supported.

Our immediate plans include adding a web clipper feature, expanding the list of supported vector databases, and creating a RESTful API interface.

▶️ Local installation and launch:

# Clone the repository

git clone https://github.com/zilliztech/deep-searcher.git

# Create a Python venv

python3 -m venv .venv
source .venv/bin/activate

# Install dependencies

cd deep searcher
pip install -e .

# Quick start demo

from deepsearcher.configuration import Configuration, init_config
from deepsearcher.online_query import query

config = Configuration()

# Customize your config here

config.set_provider_config("llm", "OpenAI", {"model": "gpt-4o-mini"})
init_config(config = config)

# Load your local data

from deepsearcher.offline_loading import load_from_local_files
load_from_local_files(paths_or_directory=your_local_path)

# (Optional) Load from web crawling (FIRECRAWL_API_KEY env variable required)

from deepsearcher.offline_loading import load_from_website
load_from_website(urls=website_url)

# Query

result = query("Write a report about xxx.") # Your question here

🌐 GitHub: https://github.com/zilliztech/deep-searcher