Hi Team,
We have installed on premise sentry with docker setup.
Earlier we were able to get the data, but from 3 days we are unable to get it.
Can anyone help me here. Thanks in advance.
Thanks
Vishwanath
We need to see your logs to provide any guidance.
@BYK
Thanks for the quick help.
What all logs you need? How you want to see those?
Thanks
Vishwanath
When i see docker-compose logs, seeing below errors:
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:cb2b92d45fc1468fa85e6b9cfdbf315c:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:d5e69d9736114bef9d7814bdf250a758:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b19753b066ab4dd288c77732144d7a4f:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4836b8fdda7e452b9ff237a0fef8e815:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7a1d2e723bac4242b0beba294344d170:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:c49acf98f3e34b6788cbdd886f91b3f9:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:32f0670dedfc44799b16518c073922d7:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:9c4f2723d98d49efb883e05d9be4771e:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:77609e7e226b4176bbf857c3fa28ab40:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:bb7a53774d2b4beab49191f0a46d6b24:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:12fa8c740bae4d45b7d620c0afbabcab:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:1fe8c5fe88c0407fb9ffed11c0792be9:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:d3468fde3b4e4a469ed33092ebee9ae0:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7aefafc0d4c24aa4abf63cf2ccddc4ea:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:c1d0ce0ce1074407beb54c621616d53a:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:468596b1f47643908de49c461215e991:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:d5515b41d44a4d29bd4339610fc2985c:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:33c8d20929a14831849f93f11d83867e:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b13c5809e1944a9eb403b8998e2121eb:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4074fbef63374f4d9db1e29021dd85dd:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:2785269d26524dc3a979cf29c9411fe8:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f1cdc57394ac41e5bf1faa4489faaed0:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:185f040ba179458a90a709a36a53796d:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:99a011414c804a72b3cebfc521c6ce32:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4eaf25b9a819482c811459d85000a010:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:0ee3c44a7a904d49915ab1a63dc3b263:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4d8bf99bd0624b1ca8d8d883583d3ec9:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b78fab30a6d24017b135d038fb513fae:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5dddae5f8ee14cd1afda884dd892ef7f:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5913fdb9f8c74f648e444392aa5f981d:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:e80e72dbb9774ae5b14c01f1ff94f504:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:97dff80699074a30aee48707264e2cde:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:338c867488944bf6a2289c94acb608f0:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:05588014edc843998efbc6f00ec98a66:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b076eb5339e7408c866e91faa2bdac79:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:c5edccd28ad84a32862283f1b5ddbe69:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:2ac886f0900d45ada856d7aea7d2dd4d:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:79ed9d02d0204cb2a986f1dc66a87dff:3’)
worker_1_f1b53494de26 | 02:49:12 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:ad0e5f91ffeb44e78ad02536f2175dea:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7512be0bf54440cdaccaf792952ad1c4:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:c506008c73bb4ad58cd5487939ffa479:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:9a177955050e49b68dd5eca07f847662:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:a061262518d9400aa6323a95f8b334d5:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:53ea3811a41c4af5afd3990340c1abf5:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:47f2a63d68034c419ed511a8993cff91:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:fee49ad2103d4f439e36c8482a581cae:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:6ab2c184afea4860ab986f8920031fc0:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:92c64f5eb21746808975c79d6e081ed4:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f53b38b2e5f045359ba7235f05546828:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:d0de65e18d0d4429bdd5bddb499bf315:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f1fd3a3197f44d358014d2444dce674c:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:ae1fb74da4f94763b9ffa3543d9c5980:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7786c42697ed49e3881ccc6ded19ef79:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:70e42241bbe642f0a266503db29f95d1:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:83b9a46969304753bd10b6b55aa95f5a:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:6accd09eb9c6498eb7249bf3f73007d3:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:6819f87d1d934137b41417ef6882d993:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5a6eda678a7e4c128e7fe994c2ce20cd:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5ba57ae969964b33bf55ede6236d21d2:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:098148f5c51a499083191b36a753ecb8:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:c5452790efd34136a0438926c5fdb8fb:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f6831322ef5348a499b132f57b0429a0:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:6bf0c7b95f9345aeb09bedc2c66a2043:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:e606b62c08f846f79ba06960c4bb7d6c:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:11f5e7a84a3441a1bd16c42bbc49fc3a:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7b2c476422cc4105a80c1ec607396b24:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:566b9b8e9bfd4a63a5a342b5b8e27281:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:8a331f43de884a1fb35ab536e31ce7a9:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:e182c731263d499fa03af1ce8c93e1c3:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:861b9298fa9d4c8684b2fff18f7f3445:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:01b4e6c22c1748d8bf0902a623723b76:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:76dcf2916b4348929c99c23e39d2fcc0:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5a3f21c96d9a4ba2b4d22a4d034143a0:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:0eb1a1014c3745b3854e14680288c2b4:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:359a796ae02649978252531d6a1cba0b:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:e7fd7183f4954935b7f917f220317b6d:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4711936d30e3435f801e0d1855e15863:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:5af22fdb7a3242848dd5d49b175cd44f:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:7f3dbdd8fa9c42fea83d024b1b538f49:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:ef2204d71ce74bb5915379e6cce23e79:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:a8dcfe22dec44847aec82eb520ad1f69:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f2edec5e233844699c238231eadc8637:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:d8b916168e6e416fa2edab214841b57c:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:17e7daad195b47559d4d9e4febefaed5:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f648bf96fce9472f9529936adaf35b23:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b26f35de8d2a4a7eb865decd5b321a51:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:f540d920f0c84847a19ee3dc9a92f06f:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:3c8c3668898549fba21e627b32d292d5:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:e457b5ccd9e243afb840adc2e3ed35e8:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:df752fa657e142cb84233d3e0c10e64b:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:bd0c5eb4f3d74c178ea470c89dd19c67:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:a22f1ef1473d4715bc2dd36003702362:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:ce9b2830500443f3a33eff37f013c6f4:3’)
worker_1_f1b53494de26 | 02:49:13 [INFO] sentry.mail: mail.sent (size=4939 message_id=u’20201014134159.22.22387@bankopen.com’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b188eb64bfc342dfafe35c6fed0aeddd:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:b9be4cebe2364ed3b93c1468eeeaf5f2:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:12b961194dc242c8b4e9d136988f3395:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:8f8115293545462c978dc9562f822b5f:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:4116f22db06f42ce90894fb36150e2a7:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:24da8be2eb0e4d35b0552c4a86484735:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:015aaeb7c28b4bcc8bd4192af00a35fa:3’)
worker_1_f1b53494de26 | 02:49:13 [INFO] sentry.mail: mail.sent (size=5966 message_id=u’20201013160903.24.85725@bankopen.com’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:ffb70246777043e2acbb5344cc5dd4e0:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:87540c465d714267bb30ffeb477fc021:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:099fa15ae7764b3fa08ece3770f0bc23:3’)
worker_1_f1b53494de26 | 02:49:13 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u’e:848277163b9748ef8fed82fa06d5f2be:3’)
Can you please help us here?
Thanks
Vishwanath
Hi @BYK
Also i tried restarting the containers, after restarting the events i can see for 2 - 3 minutes, after that again the events are stopped on the UI.
Thanks
Vishwanath
Those errors would mean you are not getting your events processed but it doesn’t explain why this happens. Just share the full logs (or better, investigate them yourself) and when sharing please put them in code blocks or use gist.github.com or something.
Hi @BYK
Apologies for sending logs in that way. Will share in file.
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6951c70522ba42d9a82a3b075b96933a:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:ae1151cbab3f4ca7b474b914a7cd1d1e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:d3662d577b0b4e80ab39dc4cea7e9111:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:a4a3669820924796b80c38bf7118fd0e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:198946dbb1f44bd49ae68480739a57e0:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:ce3f17ec1aa045488d34efcf1270f319:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:a4442633e6f547dfa8ab9a0d7c784197:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:64ba3eb5f99b47d284f87dbcbcddfc2f:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:c7373b24bd9f4b06bf158cacb051550f:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:5ff25ef657934155911048f6d80ae7e7:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:768d245ba367486aa6abd8e722c7f0e0:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:607074f07c96459480eb23b2b21be383:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:614be5779a964fd387a8ef49a03540d0:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:09a0f2cf718c4522abf18dae48403bd1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:2572ff17ff424fddb046cde9e2044d87:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3d42a54ec7f347a391a2dd2f1430d76b:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6144622ffb984dc5aa821780dda9dd9c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:0c2913d6dae44062b61899df4bfed1e3:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6e814152332740bab0dc637e8369d334:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3a6be0ecb73c4b3c9a64fd8f6dc409c8:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:088c2edc76db4e0a9bdfffee7d14ebef:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6d7781f1190840a891b75136aadc2b3d:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:bf027c4e84224fa0896516ea9c302b85:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:f6ca9b00cb434845b9451bac9bd2f97c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:11eb91ca8f8d485c96bfa41386d80a10:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:cb05179892be442786104027431120a4:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3356ec0e6837448da550e2affba5e72b:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:d97de7e8859f4bf494512eb6b34c6de1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6f9c5a2ae247451ca819ba04917e774d:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:18e857b404b7459fab1935b8e269bf8f:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:0efd9f8c024b42b2a1247579d1c43039:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:9576901c16094a0d95e309e2957ae72d:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:b0d368fd5d724eeda886cfcb6649f6e8:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7cc53cafe4514375b04917764371f5dc:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7763b59fa6a7405a9aafa932caa9967e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:b3bdf443ae7e48d980d7aeb463ee58b7:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:af839870097c4fe8ba1211851df5bf33:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:b88508ebb95947788cb33dd8e4e7510c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:fef17c3a3ad54f728fccf63c1f8aabaf:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3ceb0e592eb440ca910b8fc8d804f7bf:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:0191ab6a60434885a0e1e2c7dcb41594:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:d4be5ae8b351414d8e4e725cd9fadc61:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:5961d169eb6f48a8a7f1c24b0d503152:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6148fa0c6a6f4968ae98306efe6c769e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:23c73f405f924d8db3eb6e823d745003:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:92392205d4d34339b28b9c5fae2414d9:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7c3c672f948945c28cd6d0a050ce375c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:cc0ceac9f5d24431a4d54584e2d695a1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7adc51dadea243edb856b94182078ebc:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:6ded314f066d4354951fc6c5a087cea5:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:11821ba50a96490b95ad5cfcae5f90c1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:61364670cce44d138ef33b3bf66e3a2f:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3481b31854f0419d992493ce2ce411dd:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:e42ccfa8c3754c28ac493d1e315508a5:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:30dab59f08684ee0bb8d698500b45975:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:030cca7e125240758165df91b183e78c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:8cf7f1c04e5748ff86efb928a7e479e1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:fd28a61adb544ffe9536472565034044:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:138faf1288444067961cdde4314628e3:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:88bbe855214942edaa61bea0f59ef7ef:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:636fa56d6a7e4cff893f0027feb9282d:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:066faf1392214268a8f546cbe97cd889:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:94a8fdca9d9c4fb4b027084119625ebf:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:3379c0b5362b4ae9b56466246028f51e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:2a42fd092ea34952bd791aef0600fce0:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:5680e7fdb7034f55a19f0e386e54b640:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:16381ec1fd40434186bc5b58e32feeab:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:c14a54cec7f74a0b965463911d694f89:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:9f6025e96f434bbebb9c3540e240be0e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7be12fdd81724bf6a65bee9d8e976b7e:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:a371d28fe3d9457392fa37810b8a15d4:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:0bb5b4ab3cbe40a19cd929f1b9a4d3e6:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:294d11cc4a5946c8a13344c852eb544b:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:55753672662945bd937bb28e9b3f0666:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7a842ba570b2481083f57988277003e7:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:12c5cf5336254e64bdaaf9a43c9aa4f1:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:4d90bde7356647369ce9dd13d61a2c83:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:721c428a8d4649818de1f5420a61efd5:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:236afa449cc14775933911ccd5d46379:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:891b5cd6f9c14bc3a433ef75211e7706:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:c677d6c536d141808eee7a15c277397c:3')
worker_1_d7e71b2f2bb9 | 08:37:45 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:830a3163ecd142988d1ab41becfa5cfb:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:82e387efda6d4fa99c2227962ae42cb4:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:977ac76dd1df4ae1a32a905a7c1d7fb2:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:943fed232ce24aaf809804fa09299613:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:e385cfea65ee44c790d19c61ed51c880:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:54ec57a6306d48caa99e035ead270e6b:3')
worker_1_d7e71b2f2bb9 | 08:37:46 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:d9b6d6f8102b47058f2bd2c9cf71225b:3')
worker_1_d7e71b2f2bb9 | Traceback (most recent call last):
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 316, in start
worker_1_d7e71b2f2bb9 | blueprint.start(self)
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
worker_1_d7e71b2f2bb9 | step.start(parent)
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 592, in start
worker_1_d7e71b2f2bb9 | c.loop(*c.loop_args())
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 91, in asynloop
worker_1_d7e71b2f2bb9 | next(loop)
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/kombu/asynchronous/hub.py", line 276, in create_loop
worker_1_d7e71b2f2bb9 | tick_callback()
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py", line 1040, in on_poll_start
worker_1_d7e71b2f2bb9 | cycle_poll_start()
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py", line 322, in on_poll_start
worker_1_d7e71b2f2bb9 | self._register_BRPOP(channel)
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py", line 308, in _register_BRPOP
worker_1_d7e71b2f2bb9 | channel._brpop_start()
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/kombu/transport/redis.py", line 714, in _brpop_start
worker_1_d7e71b2f2bb9 | self.client.connection.send_command('BRPOP', *keys)
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/redis/connection.py", line 610, in send_command
worker_1_d7e71b2f2bb9 | self.send_packed_command(self.pack_command(*args))
worker_1_d7e71b2f2bb9 | File "/usr/local/lib/python2.7/site-packages/redis/connection.py", line 603, in send_packed_command
worker_1_d7e71b2f2bb9 | (errno, errmsg))
worker_1_d7e71b2f2bb9 | ConnectionError: Error 32 while writing to socket. Broken pipe.
worker_1_d7e71b2f2bb9 | 08:37:46 [WARNING] celery.worker.consumer.consumer: consumer: Connection to broker lost. Trying to re-establish the connection...
worker_1_d7e71b2f2bb9 | Restoring 4 unacknowledged message(s)
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:90fe978924b14bc3888ff7321ad55aee:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:d9b6d6f8102b47058f2bd2c9cf71225b:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:e385cfea65ee44c790d19c61ed51c880:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:54ec57a6306d48caa99e035ead270e6b:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:55c642cb05ae4d019faa36afd04a19c5:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:4db7c296121f4df398f28249b519a40b:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:239a90de4bd04cd08b68d4a3d64a63bf:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:ba41a999c1c74233b511447c46869b47:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:eca5f180f3e847e180b3b6a70b6c7eb1:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:a6abbf97ae1d40c4bb91c4da47853f0f:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:923dad6f22cc4f79a65cdd05119068e4:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:33a8bb155ded488ca0e05bba73ab358a:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:fead7eb29cc145acbed2ba16cc11c7e6:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:12979de671d244ca82acefe6341be470:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:834162cba8df46fba8d8e16a13b72156:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:2f714d592aee4f33926d54ebed07752a:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:597dbe23b41d4a47932d6b23eb1837a1:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:96227257c1b54ccdbd8d132638e56c3c:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:5bf217c78c8346d181490caf86fa5946:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:cdf0d17aaf944eec92bde2da6af970f3:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:c83051430cf34434b5338b13ea8523ea:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:449e976dea4c4a44967c82b4310947e6:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:7d9e8735c0da4b5eaa2ae8426e751279:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:f636725e761048969a50e9e679dda193:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:43e9042cdd6144d68e2db32d061cb402:3')
worker_1_d7e71b2f2bb9 | 08:37:47 [ERROR] sentry.errors.events: process.failed.empty (cache_key=u'e:e9da9fc2376e433dbc05775fef2d1ead:3')No worries, still waiting for the full logs btw from all the other services.