[Python] Celery + Redis + Python: WorkerLostError(billiard.exceptions.WorkerLostError: Worker...

While trying to use Celery with Redis as the broker for my FastAPI application, where each endpoint makes use of memory-intensive functions with diffusers models, now I am facing an error saying

WorkerLostError(billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 6 (SIGABRT) Job: 0.)
​

whenever I use --pool=prefork.

This does not appear when I use --pool=solo, but with that there is no use of --concurrency argument, but I need to make sure that the --concurrency=3 is achieved as its a production-grade application and many people will be accessing it at a time. So, I need to have it process 3 requests at a time.

Here's the error trace:

2024-07-08T12:45:12.751006109Z malloc(): unaligned tcache chunk detected
2024-07-08T12:45:13.000818714Z [2024-07-08 12:45:13,000: ERROR/MainProcess] Process 'ForkPoolWorker-1' pid:1010 exited with 'signal 6 (SIGABRT)'
2024-07-08T12:45:13.017095419Z [2024-07-08 12:45:13,014: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 6 (SIGABRT) Job: 0.')
2024-07-08T12:45:13.017111918Z Traceback (most recent call last):
2024-07-08T12:45:13.017116005Z File "/usr/local/lib/python3.10/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
2024-07-08T12:45:13.017119242Z raise WorkerLostError(
2024-07-08T12:45:13.017122614Z billiard.einfo.ExceptionWithTraceback:
2024-07-08T12:45:13.017125314Z """
2024-07-08T12:45:13.017128427Z Traceback (most recent call last):
2024-07-08T12:45:13.017130962Z File "/usr/local/lib/python3.10/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
2024-07-08T12:45:13.017133636Z raise WorkerLostError(
2024-07-08T12:45:13.017136084Z billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 6 (SIGABRT) Job: 0.
2024-07-08T12:45:13.017139686Z """


PS: I have sufficient memory and processing power on my server with powerful GPUs and multi-core CPUs, running this in a docker with supervisor as the CMD command to trigger celery and uvicorn/gunicorn together.

Continue reading...