Currently we receive repeatedly some messages from specific worker nodes which says “Temporary failure in name resolution”, There seems to be a problem with converting the dns address to ip, and the problem is being repeated on a specific node. When we try to stop the docker container, the following error message appears “An Http request took too long to complete”
And this is a more detailed error
“worker MaxRetryError - Max retires exceeded with url: /api/1/store/ (caused by newConnectionError) failed to establish a new connection [Errno-3] Temporary failure in name resolution”
When this happens, the worker stops working like a zombie, cpu usage drops, and memory usage rises rapidly.
And it is not known exactly whether it is related to this, but the redis key increases rapidly, increasing the memory usage by close to 100%. (By the way, these two phenomena do not coincide in time)