Background workers haven't checked in recently

Tibingeo · August 17, 2017, 9:53am

I get this error while loading sentry Background workers haven't checked in recently. This is likely an issue with your configuration or the workers aren't running.

These 3 processes are running in supervisor
sentry run web -w 4 and
sentry run worker -c 4 with 2 process and
sentry run cron

Http response on /api/0/internal/health/

{"healthy": {"WarningStatusCheck": true, "CeleryAppVersionCheck": true, "CeleryAliveCheck": false}, "problems": [{"url": "http://sentry.mydomain.com/manage/queue/", "message": "Background workers haven't checked in recently. This is likely an issue with your configuration or the workers aren't running.", "id": "e42fb36a2766cde885e217b8e4524e99", "severity": "critical"}]}

murchik · September 4, 2017, 7:28am

Same here. Any ideas how to investigate further?

Annoying part is that some events are being received, but not all of them, and there’s no clear pattern to it.

murchik · September 8, 2017, 8:35am

BUMP, I guess. Any tips at all? Losing exception information turned out to be really disruptive.

alexandrul · September 20, 2017, 7:17am

Could you share all your config files? Sentry + supervisor

murchik · September 20, 2017, 9:27am

mail.backend: 'smtp'  # Use dummy if you want to disable email entirely
mail.host: 'smtp.gmail.com'
mail.port: 587
mail.username: '***'
mail.password: '***'
mail.use-tls: true
# The email address to send on behalf of
mail.from: 'Sentry <***>'


# If you'd like to configure email replies, enable this.
# mail.enable-replies: false

# When email-replies are enabled, this value is used in the Reply-To header
# mail.reply-hostname: ''

# If you're using mailgun for inbound mail, set your API key and configure a
# route to forward to /api/hooks/mailgun/inbound/
# mail.mailgun-api-key: ''

###################
# System Settings #
###################

# If this file ever becomes compromised, it's important to regenerate your a new key
# Changing this value will result in all current sessions being invalidated.
# A new key can be generated with `$ sentry config generate-secret-key`
system.secret-key: '***'

# The ``redis.clusters`` setting is used, unsurprisingly, to configure Redis
# clusters. These clusters can be then referred to by name when configuring
# backends such as the cache, digests, or TSDB backend.
redis.clusters:
  default:
    hosts:
      0:
        host: 127.0.0.1
        port: 6379

################
# File storage #
################

# Uploaded media uses these `filestore` settings. The available
# backends are either `filesystem` or `s3`.

filestore.backend: 'filesystem'
filestore.options:
  location: '/opt/sentry/media'

# This file is just Python, with a touch of Django which means
# you can inherit and tweak settings to your hearts content.
from sentry.conf.server import *

import os.path

CONF_ROOT = os.path.dirname(__file__)

DATABASES = {
    'default': {
        'ENGINE': 'sentry.db.postgres',
        'NAME': 'sentry',
        'USER': 'sentry',
        'PASSWORD': '',
        'HOST': '',
        'PORT': '',
        'AUTOCOMMIT': True,
        'ATOMIC_REQUESTS': False,
    }
}

# You should not change this setting after your database has been created
# unless you have altered all schemas first
SENTRY_USE_BIG_INTS = True

# If you're expecting any kind of real traffic on Sentry, we highly recommend
# configuring the CACHES and Redis settings

###########
# General #
###########

# Instruct Sentry that this install intends to be run by a single organization
# and thus various UI optimizations should be enabled.
SENTRY_SINGLE_ORGANIZATION = True
DEBUG = True

#########
# Cache #
#########

# Sentry currently utilizes two separate mechanisms. While CACHES is not a
# requirement, it will optimize several high throughput patterns.

# If you wish to use memcached, install the dependencies and adjust the config
# as shown:
#
#   pip install python-memcached
#
# CACHES = {
#     'default': {
#         'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
#         'LOCATION': ['127.0.0.1:11211'],
#     }
# }

# A primary cache is required for things such as processing events
SENTRY_CACHE = 'sentry.cache.redis.RedisCache'

#########
# Queue #
#########

# See https://docs.sentry.io/on-premise/server/queue/ for more
# information on configuring your queue broker and workers. Sentry relies
# on a Python framework called Celery to manage queues.

BROKER_URL = 'redis://localhost:6379'

###############
# Rate Limits #
###############

# Rate limits apply to notification handlers and are enforced per-project
# automatically.

SENTRY_RATELIMITER = 'sentry.ratelimits.redis.RedisRateLimiter'

##################
# Update Buffers #
##################

# Buffers (combined with queueing) act as an intermediate layer between the
# database and the storage API. They will greatly improve efficiency on large
# numbers of the same events being sent to the API in a short amount of time.
# (read: if you send any kind of real data to Sentry, you should enable buffers)

SENTRY_BUFFER = 'sentry.buffer.redis.RedisBuffer'

##########
# Quotas #
##########

# Quotas allow you to rate limit individual projects or the Sentry install as
# a whole.

SENTRY_QUOTAS = 'sentry.quotas.redis.RedisQuota'

########
# TSDB #
########

# The TSDB is used for building charts as well as making things like per-rate
# alerts possible.

SENTRY_TSDB = 'sentry.tsdb.redis.RedisTSDB'

###########
# Digests #
###########

# The digest backend powers notification summaries.

SENTRY_DIGESTS = 'sentry.digests.backends.redis.RedisBackend'

##############
# Web Server #
##############

# If you're using a reverse SSL proxy, you should enable the X-Forwarded-Proto
# header and uncomment the following settings
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True

# If you're not hosting at the root of your web server,
# you need to uncomment and set it to the path where Sentry is hosted.
# FORCE_SCRIPT_NAME = '/sentry'

SENTRY_WEB_HOST = '127.0.0.1'
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    'workers': 1,  # the number of web workers
    # 'protocol': 'uwsgi',  # Enable uwsgi protocol instead of http
}

SENTRY_BEACON = False

SENTRY_FEATURES['organizations:sso'] = True

# plugins

GITHUB_APP_ID = 'GitHub Application Client ID'
GITHUB_API_SECRET = 'GitHub Application Client Secret'
GITHUB_EXTENDED_PERMISSIONS = ['repo']

alexandrul · September 20, 2017, 9:59am

So far the only difference is in sentry.conf.py, I’m using:

BROKER_URL = 'redis://127.0.0.1:6379/0'

However, 0 is the default value for the db, so it should have no impact (based on: https://redis-py.readthedocs.io/en/latest/#redis.StrictRedis.from_url )

Could you upload the supervisor config too?

murchik · September 20, 2017, 10:46am

Sure. I’m using systemd on Ubuntu 16 instead of supervisord:

[Unit]
Description=Sentry Background Worker
After=network.target                                                                                                                                                         
                                                                                                                                                                             
[Service]                                                                                                                                                                    
Type=simple                                                                                                                                                                  
User=sentry                                                                                                                                                                  
Group=sentry                                                                                                                                                                 
WorkingDirectory=/opt/sentry                                                                                                                                                 
Environment=SENTRY_CONF=/opt/sentry                                                                                                                                          
ExecStart=/opt/virtualenvs/sentry/bin/sentry run worker -l INFO                                                                                                              
                                                                                                                                                                             
[Install]                                                                                                                                                                    
WantedBy=multi-user.target

[Unit]                                                                                                                                                                       
Description=Sentry Beat Service                                                                                                                                              
After=network.target                                                                                                                                                         
                                                                                                                                                                             
[Service]                                                                                                                                                                    
Type=simple                                                                                                                                                                  
User=sentry                                                                                                                                                                  
Group=sentry                                                                                                                                                                 
WorkingDirectory=/opt/sentry                                                                                                                                                 
Environment=SENTRY_CONF=/etc/sentry                                                                                                                                          
ExecStart=/opt/virtualenvs/sentry/bin/sentry run cron                                                                                                                        
                                                                                                                                                                             
[Install]                                                                                                                                                                    
WantedBy=multi-user.target

[Unit]                                                                                                                                                                       
Description=Sentry Main Service                                                                                                                                              
After=network.target                                                                                                                                                         
Requires=sentry-worker.service                                                                                                                                               
Requires=sentry-cron.service                                                                                                                                                 
                                                                                                                                                                             
[Service]                                                                                                                                                                    
Type=simple                                                                                                                                                                  
User=sentry                                                                                                                                                                  
Group=sentry                                                                                                                                                                 
WorkingDirectory=/opt/sentry                                                                                                                                                 
Environment=SENTRY_CONF=/opt/sentry                                                                                                                                          
ExecStart=/opt/virtualenvs/sentry/bin/sentry run web                                                                                                                         
                                                                                                                                                                             
[Install]                                                                                                                                                                    
WantedBy=multi-user.target

alexandrul · September 20, 2017, 11:22am

The only unusual thing is your cron config:

Environment=SENTRY_CONF=/etc/sentry

The other 2 services are using:

Environment=SENTRY_CONF=/opt/sentry

murchik · September 20, 2017, 11:26am

Right. Well, changing it to /opt doesn’t help.

alexandrul · September 20, 2017, 11:56am

Other than running all components manually from the activated virtualenv, you can compare the versions:

OS:

redis: 3.2.10

Python:

celery (3.1.18)
hiredis (0.1.6)
redis (2.10.5)
redis-py-cluster (1.3.4)

At some point I had to downgrade the celery version due to high CPU usage, not sure if it’s still a issue with the above version.

murchik · September 20, 2017, 12:11pm

If I run them manually, what should I be looking for? Let’s say a task makes it to a worker, but doesn’t appear in Sentry, what does it mean? Or it doesn’t make it to a worker, what could that mean?

My Redis is 3.0.6 — latest in Ubuntu 16.04. Could that be a problem? Package versions are the same except redis-py-cluster which was not even installed. Where is it supposed to come from? Installing it just to try something out…

This is all especially difficult because failures are inconsistent: some events make it, some don’t. I was wondering if anybody axcept me and the thread starter had the same issue and could point out in the right direction.

alexandrul · September 20, 2017, 12:19pm

Redis itself should be ok, but an upgrade can’t hurt.

Running the components manually will check if there is any systemd issue related to the virtualenv.

I’ve also had this issue during the initial setup, but I never encountered it again after getting everything right (and my config is similar with yours).

As a last resort you can try using 127.0.0.1 instead of localhost in sentry.conf.py.

murchik · September 20, 2017, 12:48pm

I replaced Redis with RabbitMQ with zero effect.

Running Celery worker manually didn’t get me closer to the truth either. I can see all calls create a bunch of tasks each, but Sentry still claims workers haven’t checked it. Is this an issue with the web app then?

alexandrul · September 20, 2017, 1:27pm

I’m out of ideas

If you can’t find anything useful in the Sentry Internal project (/sentry/internal/), an issue on GitHub might have a better visibility.

For the record, right now I’m running 8.20.0 with a fresh DB (due to a Postgres upgrade required by the usage of the citext extension).

murchik · September 20, 2017, 1:32pm

Thanks anyway!

I was running 8.19 initially, later upgraded to 8.20, the issue was in effect across both versions.

alexandrul · September 20, 2017, 1:57pm

Regarding the OS, I’m using RHEL 7.3 and the firewall is disabled.

murchik · October 1, 2017, 8:00pm

So I just installed it on a server with 2Gb of RAM and that solved it. Or it’s just a coincidence, but I can’t see any other differences between the installations.

avantassel · October 19, 2017, 4:31pm

I’m having the same issue, what are the workers checking into? is this connecting to the web api? postgres? or redis? guessing it has something todo with CeleryAliveCheck: false, how can I fix this?

plarivee · December 6, 2018, 7:31pm

[ Sentry on-premise 8.20 , running in docker, redis backend ]

So I have been searching for the same Error and think I found out why.

if you check the check the code

github.com

getsentry/sentry/blob/master/src/sentry/status_checks/celery_alive.py

from time import time

from django.conf import settings

from sentry import options
from sentry.utils.http import absolute_uri

from .base import Problem, StatusCheck


class CeleryAliveCheck(StatusCheck):
    def check(self):
        # There is no queue, and celery is not running, so never show error
        if settings.CELERY_ALWAYS_EAGER:
            return []
        last_ping = options.get("sentry:last_worker_ping") or 0
        if last_ping >= time() - 300:
            return []

        backlogged, size = None, 0

This file has been truncated. show original

it checks if the key “default” is there. When it’s empty it should not be there. So by following the code, you see that you fall in the last case because the key exist and the size is 0.

That is because the check queue length does not check if exists before and consider the 0 length

def get_size(self, queue):
return self.client.llen(queue)

In redis-cli:

127.0.0.1:6379> KEYS _kombu*

“_kombu.binding.celery.pidbox”

“_kombu.binding.counters”

“_kombu.binding.triggers”

“_kombu.binding.default”

“_kombu.binding.celeryev”
127.0.0.1:6379> type _kumbu.binding.default
none
127.0.0.1:6379> exists _kumbu.binding.default
(integer) 0

I think this is a bug in the way Sentry monitor Redis backend.

Celery documentation Monitoring and Management Guide — Celery 5.3.6 documentation for the redis backend

Topic		Replies	Views
Alert when Background workers haven't checked in recently On-Premise	4	1148	September 11, 2020
Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks	3	2509	March 6, 2019
Sentry 20: Background workers haven't checked in recently On-Premise	7	1393	January 20, 2021
Red error messages after upgrading sentry On-Premise	1	2240	October 14, 2017
Sentry stopped showing errors , just after few days. ? no issues on internal On-Premise	4	4734	December 8, 2020

Background workers haven't checked in recently

Related topics