Circuit Breaker Pattern

Integrating with an external service often means dealing with unexpected behavior that impacts our service’s performance. The previous article prevents the crash/OOM by adding a timeout. But a timeout only limits how long you wait per request, it doesn’t stop you from waiting at all.

If the partner has an outage for 20 minutes, do we actually need to keep sending requests to them? Every request still needs its full 3 seconds before it times out. It’s true that the timeout prevents our service from an OOM crash, but do we need to keep the 3-second wait for each request when we already know it’s broken?

Solution

Once you’ve had a handful of failures in a row, there’s no reason to keep requesting. That’s the idea behind a circuit breaker: stop sending requests to a partner that’s already failing, and fail immediately instead of waiting to find out again.

A circuit breaker has three states:

• Closed — normal operation, meaning the service allows connections through normally.
• Open — once failures exceed the threshold, meaning the service no longer allows connections and returns an error immediately without touching the partner.
• Half-Open — after a cooldown, the breaker lets a small number of test requests through. If they succeed, it closes again. If they fail, it stays open.

Same setup as the previous article’s Case 3 — a 3-second timeout. Every caller experiences a flat 3-second latency spike for the entire outage.

With the circuit breaker, the chart looks more like this:

There’s no more 3-second wait once the breaker has had enough failures to trip. Requests fail instantly, removing the unnecessary wait for an external call that we already know is broken.

var breaker = gobreaker.NewCircuitBreaker(gobreaker.Settings{
	Name:    "partner-service",
	Timeout: 30 * time.Second, // how long it stays open before probing again
	ReadyToTrip: func(counts gobreaker.Counts) bool {
		return counts.ConsecutiveFailures > 5
	},
})

func fetchFromPartner(ctx context.Context) (*Response, error) {
	result, err := breaker.Execute(func() (interface{}, error) {
		return fetchFromPartnerWithTimeout(ctx)
	})
	if err != nil {
		return nil, err // fails instantly once the breaker is open
	}
	return result.(*Response), nil
}

After the 30 second Timeout window, the breaker moves to half-open and lets one request through to check if the partner recovered. If it succeeds, the breaker closes and traffic resumes normally. If it fails, the breaker opens again and waits another 30 seconds.

Summary

The timeout bounds how long a single request can wait. The circuit breaker bounds how many requests you send to a partner that’s already failing. Once it’s open, you stop paying the timeout cost on every call, and stop sending requests to an external service that is broken.

The two work together: timeout limits the damage of any one request, and the breaker stops the damage from adding up once the partner is actually down.