Celery Worker IndexError Bug

On-Premise
1

Hi guys, on my EC2 sentry installation the celery worker keeps raising the following error constantly every 30 seconds or so:

worker_1                   | Traceback (most recent call last):
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
worker_1                   |     R = retval = fun(*args, **kwargs)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
worker_1                   |     return self.run(*args, **kwargs)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 169, in _inner
worker_1                   |     reraise(*exc_info)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 164, in _inner
worker_1                   |     return f(*args, **kwargs)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py", line 48, in _wrapped
worker_1                   |     result = func(*args, **kwargs)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/digests.py", line 43, in deliver_digest
worker_1                   |     project, target_type, target_identifier = split_key(key)
worker_1                   |   File "/usr/local/lib/python2.7/site-packages/sentry/digests/notifications.py", line 26, in split_key
worker_1                   |     target_type = ActionTargetType(key_parts[3])
worker_1                   | IndexError: list index out of range
worker_1                   | 01:59:44 [ERROR] celery.worker.job: Task sentry.tasks.digests.deliver_digest[5a8ccadc-7bff-4b2f-897e-62f35fceae94] raised unexpected: IndexError('list index out of range',) (data={u'hostname': 'celery@c0a073200350', u'name': 'sentry.tasks.digests.deliver_digest', u'args': "('mail:p:2', 1588004504.934216)", u'internal': False, u'kwargs': '{}', u'id': '5a8ccadc-7bff-4b2f-897e-62f35fceae94'})

this error started showing up April 27th when i pulled from github’s onpremise repo and ran ./install.sh

perhaps i can manually clear some table(s) in postgres to stop this error from happening? any ideas would be appreciated, thanks

2

You may need to increase the available memory for your Redis service (or just in general, more memory).

3

i apologize but for me it seems like there’s some corrupted data or some bug in the code since it’s trying to parse a value and not finding it long enough

i have 4GB of memory on the server and the only thing running on it is sentry

here’s the error’s Event as JSON:

{
"event_id": "3d987bb34b194d6b81026a72859f3635",
"project": 1,
"release": "87c527dbc8d3966464dbce35a2d09ad382492c73",
"dist": null,
"platform": "python",
"message": "",
"datetime": "2020-05-06T14:37:22.358695Z",
"tags": [
    [
        "celery_task_id",
        "a646092b-9d3c-4374-9311-59c23c5c9ca2"
    ],
    [
        "environment",
        "production"
    ],
    [
        "handled",
        "no"
    ],
    [
        "level",
        "error"
    ],
    [
        "mechanism",
        "celery"
    ],
    [
        "runtime",
        "CPython 2.7.16"
    ],
    [
        "runtime.name",
        "CPython"
    ],
    [
        "release",
        "87c527dbc8d3966464dbce35a2d09ad382492c73"
    ],
    [
        "server_name",
        "7f33a5fcd539"
    ],
    [
        "task_name",
        "sentry.tasks.digests.deliver_digest"
    ],
    [
        "trace",
        "a93f86428ec842f78e848ac8ff811442"
    ],
    [
        "trace.ctx",
        "a93f86428ec842f78e848ac8ff811442-9aaa0a774f74ea20"
    ],
    [
        "trace.span",
        "9aaa0a774f74ea20"
    ],
    [
        "transaction",
        "sentry.tasks.digests.deliver_digest"
    ]
],
"_metrics": {
    "bytes.ingested.event": 6791,
    "bytes.stored.event": 8021
},
"contexts": {
    "runtime": {
        "version": "2.7.16",
        "type": "runtime",
        "name": "CPython",
        "build": "2.7.16 (default, Oct 17 2019, 07:39:30) \n[GCC 8.3.0]"
    },
    "trace": {
        "status": "internal_error",
        "parent_span_id": "abb41762d8524218",
        "trace_id": "a93f86428ec842f78e848ac8ff811442",
        "span_id": "9aaa0a774f74ea20",
        "type": "trace",
        "op": "celery.task"
    }
},
"culprit": "sentry.tasks.digests.deliver_digest",
"environment": "production",
"exception": {
    "values": [
        {
            "stacktrace": {
                "frames": [
                    {
                        "function": "_wrapped",
                        "abs_path": "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py",
                        "pre_context": [
                            "                scope.set_tag(\"transaction_id\", transaction_id)",
                            "",
                            "            with metrics.timer(key, instance=instance), track_memory_usage(",
                            "                \"jobs.memory_change\", instance=instance",
                            "            ):"
                        ],
                        "post_context": [
                            "",
                            "            return result",
                            "",
                            "        return app.task(name=name, **kwargs)(_wrapped)",
                            ""
                        ],
                        "vars": {
                            "func": "<function deliver_digest at 0x7f09fddaded8>",
                            "name": "'sentry.tasks.digests.deliver_digest'",
                            "args": [
                                "'mail:p:2'",
                                "1588004504.934216"
                            ],
                            "instance": "'sentry.tasks.digests.deliver_digest'",
                            "key": "'jobs.duration'",
                            "kwargs": {},
                            "scope": "<Scope id=0x7f09f734eb78 name=celery>",
                            "stat_suffix": "None",
                            "transaction_id": "None"
                        },
                        "module": "sentry.tasks.base",
                        "filename": "sentry/tasks/base.py",
                        "lineno": 48,
                        "in_app": false,
                        "data": {
                            "orig_in_app": 1
                        },
                        "context_line": "                result = func(*args, **kwargs)"
                    },
                    {
                        "function": "deliver_digest",
                        "abs_path": "/usr/local/lib/python2.7/site-packages/sentry/tasks/digests.py",
                        "pre_context": [
                            "def deliver_digest(key, schedule_timestamp=None):",
                            "    from sentry import digests",
                            "    from sentry.mail import mail_adapter",
                            "",
                            "    try:"
                        ],
                        "post_context": [
                            "    except Project.DoesNotExist as error:",
                            "        logger.info(\"Cannot deliver digest %r due to error: %s\", key, error)",
                            "        digests.delete(key)",
                            "        return",
                            ""
                        ],
                        "vars": {
                            "schedule_timestamp": "1588004504.934216",
                            "mail_adapter": "<sentry.mail.adapter.MailAdapter object at 0x7f09fdd98c10>",
                            "digests": "<module 'sentry.digests' from '/usr/local/lib/python2.7/site-packages/sentry/digests/__init__.py'>",
                            "key": "'mail:p:2'"
                        },
                        "module": "sentry.tasks.digests",
                        "filename": "sentry/tasks/digests.py",
                        "lineno": 43,
                        "in_app": false,
                        "data": {
                            "orig_in_app": 1
                        },
                        "context_line": "        project, target_type, target_identifier = split_key(key)"
                    },
                    {
                        "function": "split_key",
                        "abs_path": "/usr/local/lib/python2.7/site-packages/sentry/digests/notifications.py",
                        "pre_context": [
                            "def split_key(key):",
                            "    from sentry.mail.adapter import ActionTargetType",
                            "",
                            "    key_parts = key.split(\":\", 4)",
                            "    project_id = key_parts[2]"
                        ],
                        "post_context": [
                            "    target_identifier = key_parts[4] if key_parts[4] else None",
                            "    return Project.objects.get(pk=project_id), target_type, target_identifier",
                            "",
                            "",
                            "def unsplit_key(project, target_type, target_identifier):"
                        ],
                        "vars": {
                            "ActionTargetType": "<enum 'ActionTargetType'>",
                            "project_id": "'2'",
                            "key": "'mail:p:2'",
                            "key_parts": [
                                "'mail'",
                                "'p'",
                                "'2'"
                            ]
                        },
                        "module": "sentry.digests.notifications",
                        "filename": "sentry/digests/notifications.py",
                        "lineno": 26,
                        "in_app": false,
                        "data": {
                            "orig_in_app": 1
                        },
                        "context_line": "    target_type = ActionTargetType(key_parts[3])"
                    }
                ]
            },
            "type": "IndexError",
            "module": "exceptions",
            "value": "list index out of range",
            "mechanism": {
                "type": "celery",
                "handled": false
            }
        }
    ]
},
"extra": {
    "sys.argv": [
        "/usr/local/bin/sentry",
        "run",
        "worker"
    ],
    "celery-job": {
        "args": [
            "mail:p:2",
            1588004504.934216
        ],
        "task_name": "sentry.tasks.digests.deliver_digest",
        "kwargs": {}
    }
},
"fingerprint": [
    "{{ default }}"
],
"grouping_config": {
    "enhancements": "eJybzDhxY3J-bm5-npWRgaGlroGxrpHxBABcTQcY",
    "id": "newstyle:2019-10-29"
},
"hashes": [
    "419e5fa54b1d4aef4fa572c9e9f88b81"
],
"key_id": "1",
"level": "error",
"location": "sentry/digests/notifications.py",
"logger": "",
"metadata": {
    "function": "split_key",
    "type": "IndexError",
    "value": "list index out of range",
    "filename": "sentry/digests/notifications.py"
},
"modules": {
    "googleapis-common-protos": "1.6.0",
    "django-crispy-forms": "1.7.2",
    "cffi": "1.14.0",
    "percy": "2.0.2",
    "isodate": "0.6.0",
    "querystring-parser": "1.2.4",
    "phonenumberslite": "8.11.5",
    "python-u2flib-server": "5.0.0",
    "confluent-kafka": "0.11.5",
    "sqlparse": "0.2.4",
    "oauthlib": "3.1.0",
    "selenium": "3.141.0",
    "celery": "3.1.18",
    "sentry-relay": "0.5.8",
    "djangorestframework": "3.6.4",
    "mmh3": "2.3.1",
    "pyjwt": "1.5.3",
    "statsd": "3.1",
    "cryptography": "2.9.2",
    "python": "2.7.16",
    "symbolic": "7.3.0",
    "google-cloud-core": "0.29.1",
    "maxminddb": "1.4.1",
    "billiard": "3.3.0.23",
    "decorator": "4.4.2",
    "unidiff": "0.5.5",
    "requests-oauthlib": "1.2.0",
    "funcsigs": "1.0.2",
    "python-utils": "2.4.0",
    "jsonschema": "2.6.0",
    "jmespath": "0.9.5",
    "protobuf": "3.11.3",
    "datadog": "0.30.0",
    "msgpack": "0.6.2",
    "pyopenssl": "19.1.0",
    "redis": "2.10.5",
    "google-api-core": "1.14.3",
    "docutils": "0.16",
    "rb": "1.7",
    "milksnake": "0.1.5",
    "google-auth": "1.6.3",
    "uwsgi": "2.0.18",
    "google-cloud-storage": "1.13.3",
    "grpc-google-iam-v1": "0.11.4",
    "soupsieve": "1.9.5",
    "mistune": "0.8.4",
    "pyyaml": "5.3.1",
    "django-sudo": "3.1.0",
    "cachetools": "3.1.1",
    "wsgiref": "0.1.2",
    "cssselect": "1.0.3",
    "requests": "2.20.1",
    "anyjson": "0.3.3",
    "pyasn1": "0.4.8",
    "pillow": "6.2.2",
    "phabricator": "0.7.0",
    "cssutils": "1.0.2",
    "pycparser": "2.20",
    "boto3": "1.4.5",
    "ua-parser": "0.10.0",
    "qrcode": "6.1",
    "petname": "2.6",
    "botocore": "1.5.70",
    "setproctitle": "1.1.10",
    "redis-py-cluster": "1.3.4",
    "ipaddress": "1.0.23",
    "pkgconfig": "1.5.1",
    "certifi": "2020.4.5.1",
    "simplejson": "3.8.2",
    "rsa": "4.0",
    "pytz": "2020.1",
    "parsimonious": "0.8.0",
    "functools32": "3.2.3.post2",
    "python-dateutil": "2.8.1",
    "google-cloud-pubsub": "0.35.4",
    "sentry": "10.1.0.dev0",
    "progressbar2": "3.32.1",
    "chardet": "3.0.4",
    "hiredis": "0.1.6",
    "setuptools": "41.4.0",
    "google-resumable-media": "0.4.1",
    "grpcio": "1.28.1",
    "backports.functools-lru-cache": "1.6.1",
    "pip": "19.3.1",
    "lxml": "4.3.5",
    "amqp": "1.4.9",
    "six": "1.10.0",
    "django-picklefield": "1.0.0",
    "click": "6.7",
    "kombu": "3.0.35",
    "croniter": "0.3.31",
    "mock": "3.0.5",
    "wheel": "0.33.6",
    "python3-saml": "1.4.1",
    "google-cloud-bigtable": "0.32.2",
    "librabbitmq": "1.6.1",
    "email-reply-parser": "0.2.0",
    "urllib3": "1.24.2",
    "structlog": "16.1.0",
    "beautifulsoup4": "4.7.1",
    "pyasn1-modules": "0.2.8",
    "defusedxml": "0.5.0",
    "xmlsec": "1.3.3",
    "psycopg2-binary": "2.8.5",
    "enum34": "1.1.10",
    "futures": "3.3.0",
    "s3transfer": "0.1.13",
    "toronado": "0.0.11",
    "django": "1.11.29",
    "sentry-sdk": "0.14.3",
    "python-memcached": "1.59",
    "idna": "2.7"
},
"received": 1588775842.380758,
"sdk": {
    "version": "0.14.3",
    "name": "sentry.python",
    "packages": [
        {
            "version": "0.14.3",
            "name": "pypi:sentry-sdk"
        }
    ],
    "integrations": [
        "argv",
        "atexit",
        "celery",
        "dedupe",
        "django",
        "excepthook",
        "logging",
        "modules",
        "rust_info",
        "stdlib",
        "threading"
    ]
},
"timestamp": 1588775842.358695,
"title": "IndexError: list index out of range",
"transaction": "sentry.tasks.digests.deliver_digest",
"type": "error",
"version": "7"
}
4

UPDATE: this error happens exactly every 30 minutes not seconds

5

Seems like project with id 2 is missing or something. Could that be the case?

6

project with id 2 is present in the database and running

7

Okay, I think I know what’s happening. This certainly looks like a bug, gonna submit a fix and post back here.

Thanks a lot for the useful info.

8

This is the culprit: https://github.com/getsentry/sentry/pull/18429

We’re working on a solution that would be suitable for on-premise. For now you can ignore the error.

9

Should be fixed when this lands and gets build to Docker Hub:

Thanks again for the report and sorry for the issue.

10

Thanks a lot, i am glad i was able to contribute something to your awesome app, keep up the good work :+1:

1 Like