Using Redis keyspace events with Golang for a poor man’s event-driven application

Hello there..

So recently I was tasked to notify a client if a request timeout they passed in expired. ( along with some other data ) Since this is to be run in a sandbox environment, I decided that storing the data in Redis only was a good and easy bet since there was really no need to keep state or persistence after the expiry period. The challenge however was getting the server to tell the client “Hey, your stuff expired”. A couple of options came up.

• Have a polling function that polls the server on a certain frequency
• Implement some queuing system that notifies after the timeout. ( like a delayed queue )

But these seemed a bit too complicated and I’m not a big fan of polling in this way. So I thought, there must be a way to know when a key expires in Redis, as I remembered using the Redis pubSub some time ago, and surely there was, and this is where Keyspace events come in.

What are KeySpace events?

Keyspace events are events that are fired for changes to Redis data. It utilizes the Pub/Sub model of Redis, so essentially Redis events are published to specific channels, and all we need to do is subscribe to them.

Let’s walk through an example:

package main

import (
	"context"
	"fmt"
	"os"
	"sync"
	"time"

	"github.com/go-redis/redis/v8"
)

func main() {
	// connect to redis
	redisDB := redis.NewClient(&redis.Options{
		Addr: "localhost:6379",
		DB:   0,
	})
          
	_, err := redisDB.Do(context.Background(), "CONFIG", "SET", "notify-keyspace-events", "KEA").Result() // this is telling redis to publish events since it's off by default.
	if err != nil {
		fmt.Printf("unable to set keyspace events %v", err.Error())
		os.Exit(1)
	}
	pubsub := redisDB.PSubscribe(context.Background(), "__keyevent@0__:expired") // this is telling redis to subscribe to events published in the keyevent channel, specifically for expired events
         // this is just to show publishing events and catching the expired events in the same codebase. 
	wg := &sync.WaitGroup{}
	wg.Add(2) // two goroutines are spawned

	go func(redis.PubSub) {
		exitLoopCounter := 0
		for { // infinite loop
                        // this listens in the background for messages. 
			message, err := pubsub.ReceiveMessage(context.Background())
			exitLoopCounter++
			if err != nil {
				fmt.Printf("error message - %v", err.Error())
				break
			}
			fmt.Printf("Keyspace event recieved %v  \n", message.String())
			if exitLoopCounter >= 10 {
				wg.Done()
			}
		}
	}(*pubsub)

	go func(redis.Client, *sync.WaitGroup) {
		for i := 0; i <= 10; i++ {
			dynamicKey := fmt.Sprintf("event_%v", i)
			redisDB.Set(context.Background(), dynamicKey, "someval", time.Second*time.Duration(i*2)).Result()
		}
		wg.Done()
	}(*redisDB, wg)

	wg.Wait()
	fmt.Println("exiting program")
}

matt@DESKTOP-FQKV7LP:~/projects/go-redis-key-space-events-eg$ go build 
matt@DESKTOP-FQKV7LP:~/projects/go-redis-key-space-events-eg$ ./go-redis-key-space-events-eg 
Keyspace event recieved Message<__keyevent@0__:expired: event_1>  
Keyspace event recieved Message<__keyevent@0__:expired: event_2>  
Keyspace event recieved Message<__keyevent@0__:expired: event_3>  
Keyspace event recieved Message<__keyevent@0__:expired: event_4>  
Keyspace event recieved Message<__keyevent@0__:expired: event_5>  
Keyspace event recieved Message<__keyevent@0__:expired: event_6>  
Keyspace event recieved Message<__keyevent@0__:expired: event_7>  
Keyspace event recieved Message<__keyevent@0__:expired: event_8>  
Keyspace event recieved Message<__keyevent@0__:expired: event_9>  
Keyspace event recieved Message<__keyevent@0__:expired: event_10>  
exiting program
matt@DESKTOP-FQKV7LP:~/projects/go-redis-key-space-events-eg$ 

Here we have a simple program in Golang that tells Redis to send keyspace events, subscribes to a specific channel ( for a specific event on that channel )

Some limitations of this approach.

1. You don’t get the data that was stored, only the key that expired. ( to my current knowledge )
2. According to the official documentation Because Redis Pub/Sub is fire and forget currently there is no way to use this feature if your application demands reliable notification of events, that is, if your Pub/Sub client disconnects, and reconnects later, all the events delivered during the time the client was disconnected are lost.

The repo with the example can be found here

If you have any questions feel free to comment below.

Oh..A by the way,

If you’re using Elasticache, I was not able to get the keyspace events turned on from the application, so it had to be turned on manually via the console or by running the command via terraform or whatever you use for IaC. ( I’m not exactly sure how to do this, so apologies I can’t go into detail about it )

References:

1. https://redis.io/topics/notifications
2. https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-redis-keyspace-notifications/