I’m running an on-premise Sentry instance set up using the Sentry docker installation. I also placed Caddy in front of it for simple SSL configuration and everything is going pretty well. However, one of my applications (Java SDK) often sends bursts of logs and I end up getting 502s from Sentry. It’s not insane traffic, but about 10 logs/sec max. I have 6 workers running and it’s on a fairly powerful EC2 instance. The logs always follow the same pattern. One EOF, followed by “read: connection reset by peer”. Here’s an example of the logs from Caddy, with public IPs obfuscated:
123.456.789.10 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
31/Mar/2020:17:49:03 +0000 [ERROR 502 /api/4/store/] EOF
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 502 16
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
31/Mar/2020:17:49:05 +0000 [ERROR 502 /api/4/store/] read tcp 172.19.0.4:50192->172.19.0.6:9000: read: connection reset by peer
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 502 16
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:07 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
123.456.789.10 - - [31/Mar/2020:17:49:07 +0000] "POST /api/4/store/ HTTP/1.1" 200 41
Has anyone ever seen this before? It sort of seems to be traffic related but the logs look nearly identical every time.
Upon further investigation, it seems like the web container is actually ok. The logs from sentry_web_1 during this time period are:
172.19.0.4 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:48:28 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:02 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:03 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:05 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:07 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
172.19.0.4 - - [31/Mar/2020:17:49:07 +0000] "POST /api/4/store/ HTTP/1.1" 200 680 "-" "sentry-java/1.7.30-7a445"
Perhaps it’s an issue with Caddy, but I’ve used it in the same way for many projects and never seen this sort of problem. Since Caddy itself is reporting the hangup, it seems unlikely that it’s Caddy’s issue. But sentry_web_1 is the container bound to port 9000 and it’s not reporting 502s.