Tarhche

Latest

Most viewed

  • X
    ###
    ###
    ###
  • X
    ###
    ###
    ###
  • X
    ###
    ###
    ###
Contact usOur Github
Tarhche | A fresh design

Comments

Authentication required
To submit your comment you must first log in to your account
No comments have been submitted for this article yet!

Indexing Data With Debezium on Typesense to Build a Fast Search Service | CDC Pipeline

user avatar

Mahdi.Kh

September 18, 2026

Indexing Data With Debezium on Typesense to Build a Fast Search Service | CDC Pipeline
Using Debezium, we read data changes from the primary database and put them on RabbitMQ, and finally build an indexer service that processes the messages placed on RabbitMQ and indexes the data on Typesense.

1

Introduction and Overview | CDC Pipeline

2

Indexing Data With Maxwell on Typesense to Build a Fast Search Service | CDC Pipeline

3

Indexing Data With Debezium on Typesense to Build a Fast Search Service | CDC Pipeline

In this part, we use a small prebuilt blog management service and implement fast search on this project using CDC.

Initial Overview

Our service is a simple blog that gives us the ability to create, delete, and edit content. This project uses Golang as the backend language and PostgreSQL as the primary database. Using CDC, we can read data changes from the blog table and store these changes in the typesense database.

You can see and download the project's final source code on GitHub. To run the project, first go to the project folder using the terminal, and then run the project with Docker using the command below:

docker compose up -d

And finally, you can shut down the project using the command below:

docker compose down --remove-orphans

If you're not familiar with Docker or haven't installed it, it's best to install Docker first, and this will also be a good opportunity to get a bit familiar with Docker.

And if you have make on your system, you can use the commands below inside the project folder to start up and shut down the services:

// start the project
make up

// shut down the services
make down

Project Architecture

Before we start implementing, let's look at the final structure of what we want to have.

Overall, we'll use the following tools to build our project:

  • We use PostgreSQL as the primary database for storing and maintaining content.

  • We use the Typesense database to full-text index the content for use in search. This way, our application can search the desired content very quickly and provide the results to our application's users.

  • We use RabbitMQ as a queue. This way, we put changes made on the primary database into this queue, and then, by processing the messages in the queue, we update the data on Typesense.

  • This way, we'll index all the data on Typesense instantly, without putting any load on our primary database.

  • We use Debezium to read PostgreSQL's logs and turn those logs into messages on RabbitMQ. This way, the changes we make to data in PostgreSQL are stored as messages inside RabbitMQ.

#cdc#postgresql
  • We build an indexer application that reads the messages written to RabbitMQ and, based on these messages, indexes and updates the data on Typesense.

  • Our main application, whose responsibilities are split into two main parts:

    • Dashboard-related functionality for managing blog posts, which stores and updates data on the primary database (PostgreSQL, in this case).

    • General functionality, including the home page, post detail pages, and searching content, which reads and displays data from Typesense.

  • The overall shape of our application will be as follows:

    Debezium CDC pipeline
    The project's final architecture

    Setting Up Debezium on PostgreSQL

    To set up Debezium on Postgres, according to the documentation, we need to go through the following steps:

    • Enabling WAL (write-ahead log) in PostgreSQL
    • Running Debezium

    Enabling WAL in PostgreSQL

    To do this, we can create a file with a conf extension at /etc/postgresql/. For example, I'll create this file named postgresql.conf at /etc/postgresql/postgresql.conf.

    Then we put the following settings inside it:

    wal_level = logical
    max_wal_senders = 4 # or more, depending on parallel replication needs
    max_replication_slots = 4 # at least one per Debezium connector
    
    # Logging
    log_destination = 'stderr'
    logging_collector = on
    log_directory = 'log'
    log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
    log_rotation_age = 1d
    log_rotation_size = 100MB
    
    # Enable CDC logging
    log_statement = 'all'
    log_min_duration_statement = 1000

    Running Debezium

    To run debezium, we use debezium-server, and following the documentation, we configure it for connecting to RabbitMQ and reading PostgreSQL's logs. All of these settings can also be done using environment variables. We pass the following environment variables to debezium-server:

    # postgres related configurations
    DEBEZIUM_SOURCE_CONNECTOR_CLASS: io.debezium.connector.postgresql.PostgresConnector
    DEBEZIUM_SOURCE_DATABASE_HOSTNAME: postgres
    DEBEZIUM_SOURCE_DATABASE_PORT: 5432
    DEBEZIUM_SOURCE_DATABASE_USER: bloguser
    DEBEZIUM_SOURCE_DATABASE_PASSWORD: blogpass
    DEBEZIUM_SOURCE_DATABASE_DBNAME: blog
    DEBEZIUM_SOURCE_TOPIC_PREFIX: blog
    DEBEZIUM_SOURCE_SCHEMA_INCLUDE_LIST: public
    DEBEZIUM_SOURCE_PLUGIN_NAME: pgoutput
    DEBEZIUM_SOURCE_SLOT_NAME: dbz_slot
    DEBEZIUM_SOURCE_PUBLICATION_AUTOCREATE_MODE: filtered
    DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME: /tmp/offsets.dat
    DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS: 1000
    
    # rabbitmq related configurations
    DEBEZIUM_SINK_TYPE: rabbitmq
    DEBEZIUM_SINK_RABBITMQ_CONNECTION_HOST: rabbitmq
    DEBEZIUM_SINK_RABBITMQ_CONNECTION_PORT: 5672
    DEBEZIUM_SINK_RABBITMQ_CONNECTION_USERNAME: admin
    DEBEZIUM_SINK_RABBITMQ_CONNECTION_PASSWORD: admin123
    DEBEZIUM_SINK_RABBITMQ_CONNECTION_VIRTUAL_HOST: /
    DEBEZIUM_SINK_RABBITMQ_ACK_TIMEOUT: 30000
    DEBEZIUM_SINK_RABBITMQ_EXCHANGE: debezium
    DEBEZIUM_SINK_RABBITMQ_ROUTING_KEY: blog.posts
    DEBEZIUM_SINK_RABBITMQ_AUTO_CREATE_ROUTING_KEY: true
    DEBEZIUM_SINK_RABBITMQ_ROUTING_KEY_DURABLE: true

    After Debezium runs, we should be able to see, in the RabbitMQ dashboard, the messages that are sent to RabbitMQ.

    article figures

    In the chart above, we can see that in a short time window, 60 messages per second were sent to RabbitMQ. Next, we'll build a service that processes these messages and indexes the result on Typesense.

    Note: If you're using the project created in our GitHub repo, after running the project you can go to http://localhost:15672/ and log in with the username admin and the password admin123.

    Building the Indexer Service

    This service is responsible for processing the messages that Debezium writes onto RabbitMQ. Processing each message consists of creating, editing, or deleting an index entry.

    • If the user creates new content, that content is placed as a message on RabbitMQ, and then the indexer service processes this message and creates a new index entry on Typesense.

    • If the user edits existing content, that edit is placed as a message on RabbitMQ, and then the indexer service processes this message and updates the current indexed content on Typesense.

    • And finally, if content is deleted, by processing the message placed on RabbitMQ, the index entry corresponding to that message is removed from typesense.

    // StartCDC starts the CDC pipeline
    func (s *CDCService) StartCDC(ctx context.Context, queueName string) error {
    	log.Printf("Starting CDC service, listening to queue: %s", queueName)
    
    	// Connect to RabbitMQ
    	if err := s.messageQueue.Connect(); err != nil {
    		return fmt.Errorf("failed to connect to RabbitMQ: %w", err)
    	}
    	defer s.messageQueue.Close()
    
    	// Connect to Typesense√
    	if err := s.searchIndex.Connect(); err != nil {
    		return fmt.Errorf("failed to connect to Typesense: %w", err)
    	}
    	defer s.searchIndex.Close()
    
    	// Ensure the posts collection exists
    	if err := s.ensurePostsCollection(); err != nil {
    		return fmt.Errorf("failed to ensure posts collection: %w", err)
    	}
    
    	// Start consuming messages
    	return s.messageQueue.ConsumeMessages(queueName, s.handleMessage)
    }
    
    // handleMessage processes individual messages from RabbitMQ
    func (s *CDCService) handleMessage(message []byte) error {
    	// Parse the CDC event (try Debezium format first, fallback to original format)
    	event, err := domain.FromDebeziumJSON(message)
    	if err != nil {
    		log.Printf("Failed to parse CDC event: %v", err)
    		return err
    	}
    
    	// Validate the event
    	if !event.IsValid() {
    		log.Printf("Invalid CDC event: %+v", event)
    		return fmt.Errorf("invalid CDC event")
    	}
    
    	// Only process posts table events
    	if event.Table != "posts" {
    		log.Printf("Skipping non-posts table event: %s.%s", event.Database, event.Table)
    		return nil
    	}
    
    	// Process the event based on its type
    	switch event.Type {
    	case domain.EventTypeInsert, domain.EventTypeUpdate, domain.EventTypeBootstrapInsert:
    		return s.handleUpsert(event)
    	case domain.EventTypeDelete:
    		return s.handleDelete(event)
    	case domain.EventTypeBootstrapStart:
    		log.Printf("Bootstrap insert event received")
    		return nil
    	case domain.EventTypeBootstrapComplete:
    		log.Printf("Bootstrap complete event received")
    		return nil
    	default:
    		log.Printf("Unknown event type: %s", event.Type)
    		return fmt.Errorf("unknown event type: %s", event.Type)
    	}
    }
    
    // convertTimestamps converts nanosecond timestamps to Unix timestamps in the event data
    func (s *CDCService) convertTimestamps(data map[string]interface{}) {
    	// Convert created_at if present
    	if createdVal, ok := data["created_at"]; ok {
    		switch v := createdVal.(type) {
    		case float64:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, int64(v))
    				data["created_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		case int64:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, v)
    				data["created_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		case int:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, int64(v))
    				data["created_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		}
    	}
    
    	// Convert updated_at if present
    	if updatedVal, ok := data["updated_at"]; ok {
    		switch v := updatedVal.(type) {
    		case float64:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, int64(v))
    				data["updated_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		case int64:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, v)
    				data["updated_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		case int:
    			if v > 1e12 { // If greater than 1 trillion, likely nanoseconds
    				// Use time package to convert nanoseconds to Unix timestamp
    				t := time.Unix(0, int64(v))
    				data["updated_at"] = t.Unix() // Extract Unix timestamp in seconds
    			}
    		}
    	}
    }
    
    // handleUpsert processes insert, update, and bootstrap events
    func (s *CDCService) handleUpsert(event *domain.CDCEvent) error {
    	eventType := event.Type
    	log.Printf("Processing %s event for post ID: %v", eventType, event.Data["id"])
    
    	// Convert timestamps from nanoseconds to Unix timestamps before processing
    	s.convertTimestamps(event.Data)
    
    	// Create search document from the event data
    	doc, err := domain.NewSearchDocumentFromMap(event.Data)
    	if err != nil {
    		log.Printf("Failed to create search document: %v", err)
    		return err
    	}
    
    	// Upsert the document to Typesense
    	collectionName := "posts"
    	if err := s.searchIndex.UpsertDocument(collectionName, doc); err != nil {
    		log.Printf("Failed to upsert document to Typesense: %v", err)
    		return err
    	}
    
    	log.Printf("Successfully indexed post ID: %s", doc.ID)
    	return nil
    }
    
    // handleDelete processes delete events
    func (s *CDCService) handleDelete(event *domain.CDCEvent) error {
    	log.Printf("Processing delete event for post ID: %v", event.Data["id"])
    
    	// Get the ID from the event data
    	id, exists := event.GetID()
    	if !exists {
    		log.Printf("Failed to extract ID from delete event")
    		return fmt.Errorf("failed to extract ID from delete event")
    	}
    
    	// Delete the document from Typesense
    	collectionName := "posts"
    	documentID := fmt.Sprintf("%d", id)
    	if err := s.searchIndex.DeleteDocument(collectionName, documentID); err != nil {
    		log.Printf("Failed to delete document from Typesense: %v", err)
    		return err
    	}
    
    	log.Printf("Successfully removed post ID: %d from index", id)
    	return nil
    }
    
    // ensurePostsCollection ensures the posts collection exists in Typesense
    func (s *CDCService) ensurePostsCollection() error {
    	schema := map[string]interface{}{
    		"name": "posts",
    		"fields": []map[string]interface{}{
    			{"name": "id", "type": "string"},
    			{"name": "title", "type": "string", "facet": false, "index": true},
    			{"name": "image", "type": "string", "optional": true, "facet": false, "index": false},
    			{"name": "excerpt", "type": "string", "optional": true, "facet": false, "index": true},
    			{"name": "body", "type": "string", "facet": false, "index": true},
    			{"name": "created_at", "type": "int64", "facet": false, "index": true},
    			{"name": "updated_at", "type": "int64", "facet": false, "index": false},
    		},
    		"default_sorting_field": "created_at",
    	}
    
    	// Try to create the collection first
    	err := s.searchIndex.CreateCollection(schema)
    	if err != nil {
    		// If collection already exists, we need to ensure it has the right schema
    		// For now, we'll just log this and continue
    		// In a production environment, you might want to handle schema updates
    		log.Printf("Collection creation failed (might already exist): %v", err)
    	}
    
    	// Note: In Typesense, when you upsert a document with the same ID field value,
    	// it will replace the existing document. The id field is used as the document
    	// identifier to ensure proper updates.
    
    	return nil
    }

    Based on the code above, depending on the type of message placed on RabbitMQ, we create, edit, or delete the index entry in Typesense. You can see the complete code on GitHub.

    By running the indexer service, the data should get indexed in Typesense, and you can see the result in the typesense dashboard.

    typesense dashboard
    The typesense dashboard for viewing indexed data.

    For example, in the image we can see that 290 items have been indexed in the posts collection.

    Then, our main application can search through the indexed data and show the results to users:

    article figures

    Note: If you're using the project created in our GitHub repo, after running the project you can go to http://localhost:8085/ to see the blog service.

    In the image above, whatever we search for in the search box, if it exists, the list of related posts will be shown to us.


    Previous part: Indexing Data With Maxwell on Typesense to Build a Fast Search Service | CDC Pipeline