What does it mean when a buffer.revoked message is sent?

ParthKolekar · September 27, 2018, 7:17pm

I tried to look into the source code and attempt to figure it out, but I could not understand what it means when Sentry emits the ${prefix}.buffer.revoked message to statsd.

Could I have help understanding what the metric means?

What happens when the metric is buffer.revoked.empty? Does it mean that Redis has LRU expired my keys?

What does buffer.revoked.locked mean? Does it mean that the entry in Redis can not be accessed by multiple celery workers at the same time?

zeeg · September 28, 2018, 3:50pm

It’s easiest to just look at what the code is doing:

github.com

getsentry/sentry/blob/master/src/sentry/buffer/redis.py#L259


def process(self, key=None, batch_keys=None):
    assert not (key is None and batch_keys is None)
    assert not (key is not None and batch_keys is not None)


    if key is not None:
        batch_keys = [key]


    for key in batch_keys:
        self._process_single_incr(key)


def _process_single_incr(self, key):
    client = self.cluster.get_routing_client()
    lock_key = self._make_lock_key(key)
    # prevent a stampede due to the way we use celery etas + duplicate
    # tasks
    if not client.set(lock_key, '1', nx=True, ex=10):
        metrics.incr('buffer.revoked', tags={'reason': 'locked'})
        self.logger.debug('buffer.revoked.locked', extra={'redis_key': key})
        return


    pending_key = self._make_pending_key_from_key(key)

ParthKolekar · September 28, 2018, 4:02pm

I did read through this code, but I could not figure out how exactly it works in any measure of certainty.

zeeg · September 28, 2018, 4:22pm

the “locked” reason means “its already processing another increment for the same key”

the “empty” reason seems like an LRU or some other unexpected issue

If you’re seeing either of this it could be an indicator of lacking capacity.

ParthKolekar · September 28, 2018, 4:47pm

That is the impression that I got too! Thanks for validating!

Topic		Replies	Views
Could not set key: u'l:buffer:process_pending' On-Premise	4	2841	November 11, 2016
Excessive redis key On-Premise	5	2458	April 15, 2021
Redis eviction policy On-Premise	1	1878	August 8, 2019
Sentry stop working On-Premise	10	1409	September 2, 2021
MISCONF Redis on fresh install On-Premise	3	1650	April 16, 2021

What does it mean when a buffer.revoked message is sent?

Related Topics