Elastic Semantic Search

Demo repo for: https://j.blaszyk.me/tech-blog/mcp-server-elasticsearch-semantic-search/

Table of Contents

• Overview
• Running the MCP Server
• Integrating with Claude Desktop
• Crawling Search Labs Blog Posts
  • 1. Verify Crawler Setup
  • 2. Configure Elasticsearch
  • 3. Update Index Mapping for Semantic Search
  • 4. Start Crawling
  • 5. Verify Indexed Documents

---

Overview

This repository provides a Python implementation of an MCP server for semantic search through Search Labs blog posts indexed in Elasticsearch.

It assumes you've crawled the blog posts and stored them in the search-labs-posts index using Elastic Open Crawler.

---

Running the MCP Server

Add ES_URL and ES_AP_KEY into .env file, (take a look here for generating api key with minimum permissions)

Start the server in MCP Inspector:

make dev

Once running, access the MCP Inspector at: http://localhost:5173

---

Integrating with Claude Desktop

To add the MCP server to Claude Desktop:

make install-claude-config

This updates claude_desktop_config.json in your home directory. On the next restart, the Claude app will detect the server and load the declared tool.

---

Crawling Search Labs Blog Posts

1. Verify Crawler Setup

To check if the Elastic Open Crawler works, run:

docker run --rm \
  --entrypoint /bin/bash \
  -v "$(pwd)/crawler-config:/app/config" \
  --network host \
  docker.elastic.co/integrations/crawler:latest \
  -c "bin/crawler crawl config/test-crawler.yml"

This should print crawled content from a single page.

---

2. Configure Elasticsearch

Set up Elasticsearch URL and API Key.

Generate an API key with minimum crawler permissions:

POST /_security/api_key
{
  "name": "crawler-search-labs",
  "role_descriptors": {
    "crawler-search-labs-role": {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["search-labs-posts"],
          "privileges": ["all"]
        }
      ]
    }
  },
  "metadata": {
    "application": "crawler"
  }
}

Copy the encoded value from the response and set it as API_KEY.

---

3. Update Index Mapping for Semantic Search

Ensure the search-labs-posts index exists. If not, create it:

PUT search-labs-posts

Update the mapping to enable semantic search:

PUT search-labs-posts/_mappings
{
  "properties": {
    "body": {
      "type": "text",
      "copy_to": "semantic_body"
    },
    "semantic_body": {
      "type": "semantic_text",
      "inference_id": ".elser-2-elasticsearch"
    }
  }
}

The body field is indexed as semantic text using Elasticsearch’s ELSER model.

---

4. Start Crawling

Run the crawler to populate the index:

docker run --rm \
  --entrypoint /bin/bash \
  -v "$(pwd)/crawler-config:/app/config" \
  --network host \
  docker.elastic.co/integrations/crawler:latest \
  -c "bin/crawler crawl config/elastic-search-labs-crawler.yml"

[!TIP] If using a fresh Elasticsearch cluster, wait for the ELSER model to start before indexing.

---

5. Verify Indexed Documents

Check if the documents were indexed:

GET search-labs-posts/_count

This will return the total document count in the index. You can also verify in Kibana.

---

Done! You can now perform semantic searches on Search Labs blog posts