Most worker pools work fine until the deploy. Then the process gets SIGTERM, the jobs in flight vanish, and somebody reruns them by hand the next morning.
The whole thing
func Run(ctx context.Context, jobs <-chan Job, workers int) {
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for job := range jobs {
job.Do(context.WithoutCancel(ctx))
}
}()
}
<-ctx.Done()
wg.Wait()
}
Why it holds up
- Workers range over the channel, so they stop when the producer closes it, not when the context is cancelled.
- Each job gets a context that isn't cancelled mid-flight, so a half-written row doesn't happen.
wg.Wait()is the drain. Give your orchestrator a grace period longer than your slowest job.
One more thing
Close the jobs channel from the producer side only. Closing it from a worker is a panic waiting for a busy day.