- When Celery came back up, it spun manically, going through hundreds of old jobs.
However, neither Celery nor RabbitMQ showed signs of falling over. They were just busy.
It's a separate bug, but we should ensure that Celerybeat jobs have a very short lifetime. Maybe RabbitMQ or Celery have a mechanism for ignoring old jobs. If not, we can add a timestamp check to the job implementations themselves.
I've recently seen a somewhat similar situation:
- Celery went down for a while.
- Celerybeat jobs accumulated in RabbitMQ.
- When Celery came back up, it spun manically, going through hundreds of old jobs.
However, neither Celery nor RabbitMQ showed signs of falling over. They were just busy.
It's a separate bug, but we should ensure that Celerybeat jobs have a very short lifetime. Maybe RabbitMQ or Celery have a mechanism for ignoring old jobs. If not, we can add a timestamp check to the job implementations themselves.