No new events log - Event exceeded its configured lifetime

On-Premise
#1

Our server’s SSL/TLS certificate recently lapse unnoticed for a few days, when I brought the instance back online - it was no longer logging new errors. From the relay log:

ERROR: error processing event: event exceeded its configured lifetime

Any suggestions? I’ve tried restarting the container as well as the instance. I don’t know if it’s related to our certificate lapse, that could simply be a coincidence.

Running Sentry On-Premise 20.10.1, Ubuntu 20.04, Redis 6.0.8, Docker 1.27.4, HAProxy 2.2, Postgres 12.4

#2

Sounds like some time mismatch for me. Can you check if your Sentry/Relay instances have the right time and timezone configured?

#3

I’ve confirmed that Sentry/Relay are running on UTC, same as the host.

Here are a few more lines from the log that may paint a clearer picture:

snuba-outcomes-consumer_1      | 2021-02-12T00:13:56.461787568Z 2021-02-12 00:13:56,461 Completed processing <Batch: 4 messages, open for 1.18 seconds>.
relay_1                        | 2021-02-12T00:13:56.635662397Z 2021-02-12T00:13:56Z [relay_server::endpoints::common] ERROR: error handling request: failed to queue envelope
relay_1                        | 2021-02-12T00:13:56.635695518Z   caused by: Too many events (event_buffer_size reached)

EDIT: I’ve doubled the buffer size, but now Relay reports

relay_1                        | 2021-02-12T01:22:15.685481746Z 2021-02-12T01:22:15Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream
relay_1                        | 2021-02-12T01:22:15.685518927Z   caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                        | 2021-02-12T01:22:15.685523027Z   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
relay_1                        | 2021-02-12T01:22:15.685525817Z   caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN
#4

@damjess did you succeed to fix those errors? I’m interested a lot to know “how” if you did!

We have the same errors here, on-premise version 20.10 which was perfectly working on an internal server in-house but now that we have moved (re-installation) that into a new stack (same version), no events get processed, as soon as one of the sentry sdk send something to sentry server the docker logs output that:

{“log”:“2021-02-26T19:27:05Z [relay_server::actors::upstream] ERROR: authentication encountered error: could not send request to upstream\n”,“stream”:“stderr”,“time”:“2021-02-26T19:27:05.613792963Z”}
{“log”:" caused by: Failed to connect to host: Failed resolving hostname: no record found for name: web type: AAAA class: IN\n",“stream”:“stderr”,“time”:“2021-02-26T19:27:05.613834161Z”}
{“log”:" caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN\n",“stream”:“stderr”,“time”:“2021-02-26T19:27:05.613839988Z”}
{“log”:" caused by: Failed resolving hostname: no record found for name: web type: AAAA class: IN\n",“stream”:“stderr”,“time”:“2021-02-26T19:27:05.613844035Z”}