Hard time limit 65s exceeded on Worker 4

Getting this time and again on my logs,
May be causing issues in save,

These are my workers

worker1:
    << : *sentry_defaults
    command: run worker -Q events.process_event
  worker2:
    << : *sentry_defaults
    command: run worker -Q events.reprocessing.process_event
  worker3:
    << : *sentry_defaults
    command: run worker -Q events.reprocess_events
  worker4:
    << : *sentry_defaults
    command: run worker -Q events.save_event
  worker5:
    << : *sentry_defaults
    command: run worker -Q subscriptions
  worker6:
    << : *sentry_defaults
    command: run worker -Q integrations
  worker:
    << : *sentry_defaults
    command: run worker

celery.worker.request: Hard time limit (65s) exceeded for sentry.tasks.store.save_event

worker4_1                      | 06:28:16 [ERROR] celery.worker.request: Hard time limit (65s) exceeded for sentry.tasks.store.save_event[4207f0fc-f8ce-4156-a833-3bb03c0ffb28]
worker4_1                      | Traceback (most recent call last):
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 375, in trace_task
worker4_1                      |     R = retval = fun(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 632, in __protected_call__
worker4_1                      |     return self.run(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 171, in _inner
worker4_1                      |     reraise(*exc_info)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 166, in _inner
worker4_1                      |     return f(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py", line 48, in _wrapped
worker4_1                      |     result = func(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 842, in save_event
worker4_1                      |     _do_save_event(cache_key, data, start_time, event_id, project_id, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 763, in _do_save_event
worker4_1                      |     project_id, assume_normalized=True, start_time=start_time, cache_key=cache_key
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/utils/metrics.py", line 193, in inner
worker4_1                      |     return f(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/event_manager.py", line 351, in save
worker4_1                      |     hashes = job["event"].get_hashes()
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 344, in get_hashes
worker4_1                      |     for _f in [x.get_hash() for x in self.get_grouping_variants(force_config).values()]
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 382, in get_grouping_variants
worker4_1                      |     return get_grouping_variants_for_event(self, config)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 204, in get_grouping_variants_for_event
worker4_1                      |     components = _get_calculated_grouping_variants_for_event(event, config)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 143, in _get_calculated_grouping_variants_for_event
worker4_1                      |     rv = strategy.get_grouping_component_variants(event, config=config)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 122, in get_grouping_component_variants
worker4_1                      |     component = self.get_grouping_component(event, variant, config)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 102, in get_grouping_component
worker4_1                      |     return self(event=event, variant=variant, config=config, *args)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__
worker4_1                      |     return self._invoke(self.func, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke
worker4_1                      |     return func(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 515, in chained_exception
worker4_1                      |     return config.get_grouping_component(exceptions[0], **meta)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component
worker4_1                      |     return strategy(interface, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__
worker4_1                      |     return self._invoke(self.func, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke
worker4_1                      |     return func(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 504, in single_exception
worker4_1                      |     return single_exception_common(exception, config, meta, with_value=with_value)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 459, in single_exception_common
worker4_1                      |     stacktrace_component = config.get_grouping_component(exception.stacktrace, **meta)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component
worker4_1                      |     return strategy(interface, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__
worker4_1                      |     return self._invoke(self.func, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke
worker4_1                      |     return func(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 379, in stacktrace
worker4_1                      |     return get_stacktrace_component(stacktrace, config, variant, meta)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 395, in get_stacktrace_component
worker4_1                      |     frame_component = config.get_grouping_component(frame, variant=variant, **meta)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component
worker4_1                      |     return strategy(interface, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__
worker4_1                      |     return self._invoke(self.func, *args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke
worker4_1                      |     return func(*args, **kwargs)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 261, in frame
worker4_1                      |     php_detect_anonymous_classes=php_detect_anonymous_classes,
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 339, in get_frame_component
worker4_1                      |     php_detect_anonymous_classes=php_detect_anonymous_classes,
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 169, in get_function_component
worker4_1                      |     id="function", values=[func], similarity_encoder=ident_encoder
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 57, in __init__
worker4_1                      |     values=values,
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 109, in update
worker4_1                      |     contributes = _calculate_contributes(values)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 24, in _calculate_contributes
worker4_1                      |     for value in values or ():
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/billiard/pool.py", line 227, in soft_timeout_sighandler
worker4_1                      |     raise SoftTimeLimitExceeded()
worker4_1                      | SoftTimeLimitExceeded: SoftTimeLimitExceeded()
worker4_1                      | 06:28:17 [ERROR] celery.app.trace: Task sentry.tasks.store.save_event[4207f0fc-f8ce-4156-a833-3bb03c0ffb28] raised unexpected: SoftTimeLimitExceeded() (data={u'internal': False, u'traceback': u'Traceback (most recent call last):\n  File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 375, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 632, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 171, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 166, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py", line 48, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 842, in save_event\n    _do_save_event(cache_key, data, start_time, event_id, project_id, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 763, in _do_save_event\n    project_id, assume_normalized=True, start_time=start_time, cache_key=cache_key\n  File "/usr/local/lib/python2.7/site-packages/sentry/utils/metrics.py", line 193, in inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/event_manager.py", line 351, in save\n    hashes = job["event"].get_hashes()\n  File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 344, in get_hashes\n    for _f in [x.get_hash() for x in self.get_grouping_variants(force_config).values()]\n  File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 382, in get_grouping_variants\n    return get_grouping_variants_for_event(self, config)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 204, in get_grouping_variants_for_event\n    components = _get_calculated_grouping_variants_for_event(event, config)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 143, in _get_calculated_grouping_variants_for_event\n    rv = strategy.get_grouping_component_variants(event, config=config)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 122, in get_grouping_component_variants\n    component = self.get_grouping_component(event, variant, config)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 102, in get_grouping_component\n    return self(event=event, variant=variant, config=config, *args)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__\n    return self._invoke(self.func, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke\n    return func(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 515, in chained_exception\n    return config.get_grouping_component(exceptions[0], **meta)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component\n    return strategy(interface, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__\n    return self._invoke(self.func, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke\n    return func(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 504, in single_exception\n    return single_exception_common(exception, config, meta, with_value=with_value)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 459, in single_exception_common\n    stacktrace_component = config.get_grouping_component(exception.stacktrace, **meta)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component\n    return strategy(interface, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__\n    return self._invoke(self.func, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke\n    return func(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 379, in stacktrace\n    return get_stacktrace_component(stacktrace, config, variant, meta)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 395, in get_stacktrace_component\n    frame_component = config.get_grouping_component(frame, variant=variant, **meta)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 201, in get_grouping_component\n    return strategy(interface, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 84, in __call__\n    return self._invoke(self.func, *args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 81, in _invoke\n    return func(*args, **kwargs)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 261, in frame\n    php_detect_anonymous_classes=php_detect_anonymous_classes,\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 339, in get_frame_component\n    php_detect_anonymous_classes=php_detect_anonymous_classes,\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/newstyle.py", line 169, in get_function_component\n    id="function", values=[func], similarity_encoder=ident_encoder\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 57, in __init__\n    values=values,\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 109, in update\n    contributes = _calculate_contributes(values)\n  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/component.py", line 24, in _calculate_contributes\n    for value in values or ():\n  File "/usr/local/lib/python2.7/site-packages/billiard/pool.py", line 227, in soft_timeout_sighandler\n    raise SoftTimeLimitExceeded()\nSoftTimeLimitExceeded: SoftTimeLimitExceeded()\n', u'name': 'sentry.tasks.store.save_event', u'args': '()', u'kwargs': "{'event_id': '1a67a91714204ae7893aa136c224e1a4', 'project_id': 19L, 'start_time': 1612422921.0, 'cache_key': u'e:1a67a91714204ae7893aa136c224e1a4:19', 'data': None}", u'description': u'raised unexpected', u'hostname': u'celery@d32bf612a5dc', u'id': '4207f0fc-f8ce-4156-a833-3bb03c0ffb28', u'exc': 'SoftTimeLimitExceeded()'})
snuba-transactions-consumer_1  | 2021-02-05 06:28:17,228 Completed processing <Batch: 17 messages, open for 0.81 seconds>.
worker4_1                      | 06:28:17 [ERROR] multiprocessing: Process 'ForkPoolWorker-1' pid:25 exited with 'signal 9 (SIGKILL)'
snuba-consumer_1               | 2021-02-05 06:28:17,520 Completed processing <Batch: 18 messages, open for 0.79 seconds>.
snuba-outcomes-consumer_1      | 2021-02-05 06:28:17,684 Completed processing <Batch: 18 messages, open for 0.79 seconds>.
snuba-transactions-consumer_1  | 2021-02-05 06:28:18,229 Completed processing <Batch: 25 messages, open for 1.00 seconds>.
snuba-consumer_1               | 2021-02-05 06:28:18,319 Completed processing <Batch: 26 messages, open for 0.80 seconds>.
worker4_1                      | Traceback (most recent call last):
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 311, in send_event_wrapper
worker4_1                      |     self._send_event(event)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 231, in _send_event
worker4_1                      |     headers={"Content-Type": "application/json", "Content-Encoding": "gzip"},
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 178, in _send_request
worker4_1                      |     headers=headers,
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 72, in request
worker4_1                      |     **urlopen_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 150, in request_encode_body
worker4_1                      |     return self.urlopen(method, url, **extra_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 324, in urlopen
worker4_1                      |     response = conn.urlopen(method, u.request_uri, **kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
worker4_1                      |     _stacktrace=sys.exc_info()[2])
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 399, in increment
worker4_1                      |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
worker4_1                      | MaxRetryError: HTTPSConnectionPool(host='retail-sentry.apple.com', port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
worker4_1                      | 06:28:18 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk
worker4_1                      | Traceback (most recent call last):
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 311, in send_event_wrapper
worker4_1                      |     self._send_event(event)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 231, in _send_event
worker4_1                      |     headers={"Content-Type": "application/json", "Content-Encoding": "gzip"},
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/sentry_sdk/transport.py", line 178, in _send_request
worker4_1                      |     headers=headers,
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 72, in request
worker4_1                      |     **urlopen_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/request.py", line 150, in request_encode_body
worker4_1                      |     return self.urlopen(method, url, **extra_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/poolmanager.py", line 324, in urlopen
worker4_1                      |     response = conn.urlopen(method, u.request_uri, **kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 667, in urlopen
worker4_1                      |     **response_kw)
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
worker4_1                      |     _stacktrace=sys.exc_info()[2])
worker4_1                      |   File "/usr/local/lib/python2.7/site-packages/urllib3/util/retry.py", line 399, in increment
worker4_1                      |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
worker4_1                      | MaxRetryError: HTTPSConnectionPool(host='retail-sentry.apple.com', port=443): Max retries exceeded with url: /api/1/store/ (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
worker4_1                      | 06:28:18 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

Sounds like:

  1. You need to process events faster
  2. You need valid SSL certificates or make your CA trusted to the system/sentry for internal error reporting

@BYK , where in sentry are the CA certs referenced. I would like to replace root certificates in sentry to match our internal root ca’s

I think the best way would be to build your own image based on the Sentry image and add the certs through a bind mount or something. AFAIK Sentry does not look into any specific location for custom CAs.