Redis issue when mail should be sent

Feedback
1

Hello. We’re trying to setup sentry (v.8.8.0, but I saw this error with previous versions too) and everything looks good except for we can’t make mail notifications working. Every time new error message arrives and mail should be sent I can see in logs only redis (we’re using v3.0.7) errors:

Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker Traceback (most recent call last):
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/utils/safe.py", line 26, in safe_execute
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     result = func(*args, **kwargs)
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/plugins/bases/notify.py", line 88, in rule_notify
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     maximum_delay=get_digest_option('maximum_delay'),
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/digests/backends/redis.py", line 200, in add
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     results = pipeline.execute()
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/client.py", line 2626, in execute
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     return execute(conn, stack, raise_on_error)
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/client.py", line 2521, in _execute_transaction
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     self.immediate_execute_command('DISCARD')
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/client.py", line 2461, in immediate_execute_command
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     return self.parse_response(conn, command_name, **options)
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/client.py", line 2584, in parse_response
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     self, connection, command_name, **options)
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/client.py", line 585, in parse_response
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     response = connection.read_response()
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/../redis/connection.py", line 582, in read_response
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker     raise response
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker ResponseError: DISCARD without MULTI
Oct 14 14:36:43 ip-10-10-22-227 supervisord: sentry-worker 14:36:43 [ERROR] sentry.safe.mailplugin: rule_notify.process_error
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker Traceback (most recent call last):
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/utils/raven.py", line 110, in send
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     helper.insert_data_to_database(data)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/coreapi.py", line 714, in insert_data_to_database
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     default_cache.set(cache_key, data, timeout=3600)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/cache/redis.py", line 33, in set
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     v = json.dumps(value)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/utils/json.py", line 99, in dumps
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     return _default_encoder.encode(value)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib64/python2.7/site-packages/simplejson/encoder.py", line 275, in encode
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     chunks = self.iterencode(o, _one_shot=True)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib64/python2.7/site-packages/simplejson/encoder.py", line 357, in iterencode
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     return _iterencode(o, 0)
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker   File "/var/lib/sentry/local/lib/python2.7/site-packages/sentry/utils/json.py", line 40, in better_default_encoder
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker     raise TypeError(repr(o) + ' is not JSON serializable')
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker TypeError: ResponseError('DISCARD without MULTI',) is not JSON serializable
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker 14:36:44 [ERROR] sentry.errors: Unable to record event: ResponseError('DISCARD without MULTI',) is not JSON serializable
Oct 14 14:36:44 ip-10-10-22-227 supervisord: sentry-worker Event was: 'unknown error'
2

It seems like this is happening due to this call. It seems like this is called during recovery after a failed transaction, as described in the Redis documentation, so the actual error (the ExecAbortError) is never actually raised.

We haven’t seen this one before. From a little bit of digging it sounds like this is usually an operational error (out of memory, failure to write to disk, some other kind of broken install.)

Do any write operations succeed against this Redis server?

3

Hello and thank you. Problem is solved. It has happened because of redis misconfiguration: redis was configured both in sentry.conf.py (BROKER_URL = ‘redis://localhost:6379’) and in config.yml