From Signal Handling to Traffic Draining: Go Service Graceful Shutdown and Lossless Offline in Practice

Hello everyone, I'm Lao Wang.

Today, let's talk about an extremely classic issue during backend deployments that many teams still stumble upon: Graceful Shutdown and Zero-Loss Traffic Draining.

Years ago when working on payment and order systems, we learned this the hard way: our early deployment scripts were extremely brute-force, running kill -9 directly or abruptly destroying Pods during rollouts. As a result, every release spawned a flurry of 502 Bad Gateway and Connection Reset by Peer on the monitoring dashboards, and even led to lost orders and data inconsistencies caused by interrupted transactions when payment callbacks were cut off midway.

To achieve true "zero-loss traffic draining", relying solely on Shutdown at the Go code level is not enough. You must consider the application layer lifecycle together with the routing deregistration of the infrastructure (K8s / Nginx / service registries).

1. Why Does Calling server.Shutdown() Alone Still Cause Errors?

Many developers writing Go assume that catching the SIGTERM signal and calling http.Server.Shutdown(ctx) is all it takes. But in microservice or K8s environments, alerts will still fire.

There is a critical chain of causality here:

1. Event Trigger: K8s deletes the Pod and simultaneously sends instructions to two components:
   • Sends an Endpoints update notification to kube-proxy / Ingress Controller, requesting the removal of the Pod's IP.
   • Sends a SIGTERM signal to the Pod.
2. Asynchronous Execution Causes Race Conditions:
   • Network components remove the IP asynchronously with network latency (iptables / IPVS rule synchronization typically takes several seconds).
   • The Go service inside the Pod responds extremely fast; upon receiving SIGTERM, it immediately closes the listening socket and rejects new incoming requests.
3. The Consequence: At this point, the route hasn't been completely removed yet, so upstream services (gateways or other microservices) continue routing new requests to this Pod, resulting directly in Connection Refused or 502 errors.

Therefore, zero-loss traffic draining must be split into two stages: first, shut off the tap at the network layer (drain traffic), and then clear the backlog at the application layer (process in-flight requests and clean up resources).

2. Standard Graceful Shutdown Implementation for Go Services

Inside a Go service, a standard shutdown sequence should follow this order:

Catch signal -> Stop accepting new requests -> Wait for in-flight requests and async tasks to complete -> Close database/connection pools -> Process exit

Here is a production-ready skeleton implementation (integrating an HTTP server, an asynchronous task queue, and resource cleanup):

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"
)

type AppServer struct {
	httpServer *http.Server
	workerWg   sync.WaitGroup
	quitChan   chan struct{}
}

func NewAppServer() *AppServer {
	mux := http.NewServeMux()
	mux.HandleFunc("/api/v1/order/create", func(w http.ResponseWriter, r *http.Request) {
		// 模拟耗时业务逻辑
		time.Sleep(2 * time.Second)
		w.WriteHeader(http.StatusOK)
		w.Write([]byte(`{"status":"success"}`))
	})

	return &AppServer{
		httpServer: &http.Server{
			Addr:    ":8080",
			Handler: mux,
		},
		quitChan: make(chan struct{}),
	}
}

// 模拟后台异步任务（如从 Kafka 消费订单事件）
func (s *AppServer) StartBackgroundWorker() {
	s.workerWg.Add(1)
	go func() {
		defer s.workerWg.Done()
		log.Println("[Worker] 异步任务处理中心启动...")
		for {
			select {
			case <-s.quitChan:
				log.Println("[Worker] 收到退出信号，停止消费新消息，等待在途任务完成...")
				return
			default:
				// 模拟处理业务
				time.Sleep(500 * time.Millisecond)
			}
		}
	}()
}

func main() {
	app := NewAppServer()
	app.StartBackgroundWorker()

	// 1. 异步启动 HTTP 服务
	go func() {
		log.Printf("[HTTP] 服务启动，监听端口: %s\n", app.httpServer.Addr)
		if err := app.httpServer.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("[HTTP] 服务启动异常: %v\n", err)
		}
	}()

	// 2. 监听系统停机信号
	// 必须同时监听 SIGINT (Ctrl+C) 和 SIGTERM (K8s/Docker 默认停机信号)
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)

	sig := <-sigChan
	log.Printf("[System] 接收到停机信号: %s，开始执行优雅下线流程...\n", sig.String())

	// 3. 设定整个下线的最大超时时间（兜底，防止任务挂起导致无法退出）
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	// 4. 阶段一：停止 HTTP 请求入口
	// http.Server.Shutdown 会先关闭 Listener 拒绝新请求，再阻塞等待所有活跃连接处理完毕
	if err := app.httpServer.Shutdown(shutdownCtx); err != nil {
		log.Printf("[HTTP] 服务强制关闭: %v\n", err)
	} else {
		log.Println("[HTTP] 所有在途 HTTP 请求已处理完毕")
	}

	// 5. 阶段二：停止后台 Worker
	close(app.quitChan)
	workerDone := make(chan struct{})
	go func() {
		app.workerWg.Wait()
		close(workerDone)
	}()

	select {
	case <-workerDone:
		log.Println("[Worker] 异步任务已全部安全退出")
	case <-shutdownCtx.Done():
		log.Println("[Worker] 异步任务退出超时，强制跳过")
	}

	// 6. 阶段三：清理基础资源（DB 连接池、Redis 客户端、注册中心反注册等）
	cleanupResources()

	log.Println("[System] 服务已完全无损退出")
}

func cleanupResources() {
	log.Println("[Resource] 正在关闭 MySQL/Redis 连接池...")
	// db.Close()
	// redisClient.Close()
	log.Println("[Resource] 资源清理完成")
}

3. Configuring Kubernetes for Zero-Downtime Traffic Draining

Updating the code alone leaves out the final piece of the puzzle: resolving the race where K8s removes Endpoints slower than Go receives SIGTERM.

The most effective approach is to leverage the Pod's lifecycle.preStop hook, letting the container "idle" and wait for a few seconds before actually receiving SIGTERM, ensuring all network rules have been synchronized.

Here is an example Deployment configuration snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: order-app
        image: order-service:v1.0.0
        lifecycle:
          preStop:
            exec:
              # 因果链：先 sleep 10s，给 kube-proxy / Ingress 留出更新路由表的时间
              # 期间 Pod 依然正常处理请求，10s 后再向 Go 进程发送 SIGTERM
              command: ["/bin/sh", "-c", "sleep 10"]
        # 必须大于 preStop sleep 时间 + Go Shutdown 超时时间
        # 否则时间一到，K8s 会直接发 SIGKILL (kill -9) 强杀进程
      terminationGracePeriodSeconds: 45

4. Lessons Learned & Pitfalls to Avoid (Lao Wang's Checklist)

1. Do not drop Context propagation: In HTTP handlers, time-consuming operations such as database queries and RPC calls must be bound to r.Context(). When http.Server.Shutdown() is invoked, if a timeout occurs, the Context triggers Canceled, causing downstream calls to roll back quickly and preventing transactions from hanging and locking tables indefinitely.
2. Pay attention to service registry deregistration: If using service discovery mechanisms like Consul, Nacos, or Eureka, do not rely solely on heartbeat timeouts for deregistration. As the very first step upon receiving a shutdown signal, explicitly call Deregister() and allow time for caches to refresh (typically 3–5 seconds).
3. Kafka/RocketMQ consumer offset commits: When handling critical asynchronous consumption such as order payment callbacks, once an exit signal is received, you must finish processing the current batch of messages and explicitly CommitSync() before exiting to prevent duplicate consumption or message loss.
4. Be mindful of dependencies in preStop: The sleep command in preStop depends on the container's shell environment (/bin/sh). If using scratch or minimalist distroless base images, execution may fail and be skipped entirely. It is recommended to use alpine or debian-slim which include a basic shell.

There is no silver bullet in architecture design; system stability often hides in these deployment-time details. Once you clearly map out the causal chain among OS signals, network routing, and pooled application resources, 502 errors will naturally stay away.