Worker error, Unable to incr internal metric

Can someone tell me what is going wrong here ? Is it not possible to connect to redis ?
(redis is running in the same machine)

worker_1                                    | 21:33:13 [INFO] sentry.plugins.github: apps-not-configured
worker_1                                    | 21:33:13 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_dsymcache:clean_dsymcache')
worker_1                                    | 21:33:13 [INFO] sentry.bgtasks: bgtask.spawn (task_name='sentry.bgtasks.clean_releasefilecache:clean_releasefilecache')
worker_1                                    | * Unknown config option found: 'slack.legacy-app'
worker_1                                    |  
worker_1                                    |  -------------- celery@26b538eba836 v4.4.7 (cliffs)
worker_1                                    | --- ***** ----- 
worker_1                                    | -- ******* ---- Linux-4.15.0-135-generic-x86_64-with-debian-10.8 2021-03-02 21:33:17
worker_1                                    | - *** --- * --- 
worker_1                                    | - ** ---------- [config]
worker_1                                    | - ** ---------- .> app:         sentry:0x7fdb539dc208
worker_1                                    | - ** ---------- .> transport:   redis://redis:6379/0
worker_1                                    | - ** ---------- .> results:     disabled://
worker_1                                    | - *** --- * --- .> concurrency: 4 (prefork)
worker_1                                    | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
worker_1                                    | --- ***** ----- 
worker_1                                    |  -------------- [queues]
worker_1                                    |                 .> activity.notify  exchange=default(direct) key=activity.notify
worker_1                                    |                 .> alerts           exchange=default(direct) key=alerts
worker_1                                    |                 .> app_platform     exchange=default(direct) key=app_platform
worker_1                                    |                 .> assemble         exchange=default(direct) key=assemble
worker_1                                    |                 .> auth             exchange=default(direct) key=auth
worker_1                                    |                 .> buffers.process_pending exchange=default(direct) key=buffers.process_pending
worker_1                                    |                 .> cleanup          exchange=default(direct) key=cleanup
worker_1                                    |                 .> commits          exchange=default(direct) key=commits
worker_1                                    |                 .> counters-0       exchange=counters(direct) key=default
worker_1                                    |                 .> data_export      exchange=default(direct) key=data_export
worker_1                                    |                 .> default          exchange=default(direct) key=default
worker_1                                    |                 .> digests.delivery exchange=default(direct) key=digests.delivery
worker_1                                    |                 .> digests.scheduling exchange=default(direct) key=digests.scheduling
worker_1                                    |                 .> email            exchange=default(direct) key=email
worker_1                                    |                 .> events.preprocess_event exchange=default(direct) key=events.preprocess_event
worker_1                                    |                 .> events.process_event exchange=default(direct) key=events.process_event
worker_1                                    |                 .> events.reprocess_events exchange=default(direct) key=events.reprocess_events
worker_1                                    |                 .> events.reprocessing.preprocess_event exchange=default(direct) key=events.reprocessing.preprocess_event
worker_1                                    |                 .> events.reprocessing.process_event exchange=default(direct) key=events.reprocessing.process_event
worker_1                                    |                 .> events.reprocessing.symbolicate_event exchange=default(direct) key=events.reprocessing.symbolicate_event
worker_1                                    |                 .> events.save_event exchange=default(direct) key=events.save_event
worker_1                                    |                 .> events.symbolicate_event exchange=default(direct) key=events.symbolicate_event
worker_1                                    |                 .> files.delete     exchange=default(direct) key=files.delete
worker_1                                    |                 .> group_owners.process_suspect_commits exchange=default(direct) key=group_owners.process_suspect_commits
worker_1                                    |                 .> incident_snapshots exchange=default(direct) key=incident_snapshots
worker_1                                    |                 .> incidents        exchange=default(direct) key=incidents
worker_1                                    |                 .> integrations     exchange=default(direct) key=integrations
worker_1                                    |                 .> merge            exchange=default(direct) key=merge
worker_1                                    |                 .> options          exchange=default(direct) key=options
worker_1                                    |                 .> relay_config     exchange=default(direct) key=relay_config
worker_1                                    |                 .> reports.deliver  exchange=default(direct) key=reports.deliver
worker_1                                    |                 .> reports.prepare  exchange=default(direct) key=reports.prepare
worker_1                                    |                 .> search           exchange=default(direct) key=search
worker_1                                    |                 .> sleep            exchange=default(direct) key=sleep
worker_1                                    |                 .> stats            exchange=default(direct) key=stats
worker_1                                    |                 .> subscriptions    exchange=default(direct) key=subscriptions
worker_1                                    |                 .> triggers-0       exchange=triggers(direct) key=default
worker_1                                    |                 .> unmerge          exchange=default(direct) key=unmerge
worker_1                                    |                 .> update           exchange=default(direct) key=update
worker_1                                    | 
worker_1                                    | Traceback (most recent call last):
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/metrics.py", line 98, in worker
worker_1                                    |     tsdb.incr(tsdb.models.internal, full_key, count=amount)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/utils/services.py", line 102, in <lambda>
worker_1                                    |     context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/tsdb/redissnuba.py", line 84, in method
worker_1                                    |     return getattr(self.backends[backend], key)(*a, **kw)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/tsdb/redis.py", line 202, in incr
worker_1                                    |     self.incr_multi([(model, key)], timestamp, count, environment_id)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/sentry/tsdb/redis.py", line 261, in incr_multi
worker_1                                    |     client.expireat(hash_key, key_expiries.pop(hash_key))
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/rb/clients.py", line 637, in __exit__
worker_1                                    |     self.mapping_client.join(timeout=timeout)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/rb/clients.py", line 420, in join
worker_1                                    |     self._release_command_buffer(command_buffer)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/rb/clients.py", line 365, in _release_command_buffer
worker_1                                    |     self.connection_pool.release(command_buffer.connection)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/rb/clients.py", line 274, in release
worker_1                                    |     real_pool.release(connection)
worker_1                                    |   File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 1114, in release
worker_1                                    |     self._in_use_connections.remove(connection)
worker_1                                    | KeyError: Connection<host=redis,port=6379,db=0>
worker_1                                    | 21:33:27 [ERROR] sentry.errors: Unable to incr internal metric

Looks like an issue with your Redis instance? Is it up and running properly?

Yes it is up and running correctly.
The only thing I am not sure is that the redis is completely initialized when it is run for the first time. I think, the setup needs some sane wait times for the components to get initialized correctly. You can probably reduced some forum posts that way.
When I restarted the worker with docker-compose restart worker once all the components were up, I didn’t see the KeyError in the logs.

Ah, this looks like a redis-py bug that is fixed in a newer version: KeyError on self._in_use_connections.remove(connection) · Issue #1138 · andymccurdy/redis-py · GitHub

I submitted this to fix: upgrade(redis-py): Upgrade redis-py to 3.4.1 by BYK · Pull Request #24235 · getsentry/sentry · GitHub

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.