I did some more reading including the source code of both CapturePanic and RecoveryHandler and I think I now understand why RecoveryHandler is needed. http.ListenAndServe has an internal panic recovery mechanism, so it will never crash even if the handlers panic (panics in goroutines aside, that's a different topic). Therefore, wrapping http.ListenAndServe in a CapturePanic will never hit the code of CapturePanic and we won't be able to log the exceptions.
The need to wrap RecoveryHandler on every single one of my http handler functions is still a big hassle. I was hoping there's a better way, but I don't know enough about Go to know how to convert it into a middleware.
In most middleware packages, the signature is a func (http.Handler) http.Handler, which works very well with alice and other middleware wrapper libraries. RecoveryHandler, however, has a signature that's incompatible.