Dec 11 02:23:48 localhost celery-worker[2326154]:     self.write_pid()
Dec 11 02:23:48 localhost celery-worker[2326154]:     ~~~~~~~~~~~~~~^^
Dec 11 02:23:48 localhost celery-worker[2326154]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:23:48 localhost celery-worker[2326154]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:23:48 localhost celery-worker[2326154]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:23:51 localhost celery-worker[2326158]: Error: No nodes replied within time constraint
Dec 11 02:23:51 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:23:51 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:24:01 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 11.
Dec 11 02:24:01 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:24:02 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:24:02 localhost celery-worker[2326202]: Traceback (most recent call last):
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:02 localhost celery-worker[2326202]:     self.write_pid()
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:02 localhost celery-worker[2326202]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:02 localhost celery-worker[2326202]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:02 localhost celery-worker[2326202]: During handling of the above exception, another exception occurred:
Dec 11 02:24:02 localhost celery-worker[2326202]: Traceback (most recent call last):
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:24:02 localhost celery-worker[2326202]:     sys.exit(main())
Dec 11 02:24:02 localhost celery-worker[2326202]:              ~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:24:02 localhost celery-worker[2326202]:     sys.exit(_main())
Dec 11 02:24:02 localhost celery-worker[2326202]:              ~~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:24:02 localhost celery-worker[2326202]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:24:02 localhost celery-worker[2326202]:     return self.main(*args, **kwargs)
Dec 11 02:24:02 localhost celery-worker[2326202]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:24:02 localhost celery-worker[2326202]:     rv = self.invoke(ctx)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:24:02 localhost celery-worker[2326202]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:24:02 localhost celery-worker[2326202]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:24:02 localhost celery-worker[2326202]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:24:02 localhost celery-worker[2326202]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:24:02 localhost celery-worker[2326202]:     return callback(*args, **kwargs)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:24:02 localhost celery-worker[2326202]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:24:02 localhost celery-worker[2326202]:     return f(ctx, *args, **kwargs)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:24:02 localhost celery-worker[2326202]:     return detach(sys.executable,
Dec 11 02:24:02 localhost celery-worker[2326202]:                   argv,
Dec 11 02:24:02 localhost celery-worker[2326202]:     ...<6 lines>...
Dec 11 02:24:02 localhost celery-worker[2326202]:                   executable=kwargs.get('executable', None),
Dec 11 02:24:02 localhost celery-worker[2326202]:                   hostname=hostname)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:24:02 localhost celery-worker[2326202]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:24:02 localhost celery-worker[2326202]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:                   after_forkers=False):
Dec 11 02:24:02 localhost celery-worker[2326202]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:24:02 localhost celery-worker[2326202]:     self.after_chdir()
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:24:02 localhost celery-worker[2326202]:     _create_pidlock(pidfile).release()
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:24:02 localhost celery-worker[2326202]:     pidlock.acquire()
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:24:02 localhost celery-worker[2326202]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:24:02 localhost celery-worker[2326202]:     raise value.with_traceback(tb)
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:02 localhost celery-worker[2326202]:     self.write_pid()
Dec 11 02:24:02 localhost celery-worker[2326202]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:02 localhost celery-worker[2326202]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:02 localhost celery-worker[2326202]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:02 localhost celery-worker[2326202]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:05 localhost celery-worker[2326204]: Error: No nodes replied within time constraint
Dec 11 02:24:05 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:24:05 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:24:15 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 12.
Dec 11 02:24:15 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:24:17 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:24:17 localhost celery-worker[2326232]: Traceback (most recent call last):
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:17 localhost celery-worker[2326232]:     self.write_pid()
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:17 localhost celery-worker[2326232]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:17 localhost celery-worker[2326232]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:17 localhost celery-worker[2326232]: During handling of the above exception, another exception occurred:
Dec 11 02:24:17 localhost celery-worker[2326232]: Traceback (most recent call last):
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:24:17 localhost celery-worker[2326232]:     sys.exit(main())
Dec 11 02:24:17 localhost celery-worker[2326232]:              ~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:24:17 localhost celery-worker[2326232]:     sys.exit(_main())
Dec 11 02:24:17 localhost celery-worker[2326232]:              ~~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:24:17 localhost celery-worker[2326232]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:24:17 localhost celery-worker[2326232]:     return self.main(*args, **kwargs)
Dec 11 02:24:17 localhost celery-worker[2326232]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:24:17 localhost celery-worker[2326232]:     rv = self.invoke(ctx)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:24:17 localhost celery-worker[2326232]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:24:17 localhost celery-worker[2326232]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:24:17 localhost celery-worker[2326232]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:24:17 localhost celery-worker[2326232]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:24:17 localhost celery-worker[2326232]:     return callback(*args, **kwargs)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:24:17 localhost celery-worker[2326232]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:24:17 localhost celery-worker[2326232]:     return f(ctx, *args, **kwargs)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:24:17 localhost celery-worker[2326232]:     return detach(sys.executable,
Dec 11 02:24:17 localhost celery-worker[2326232]:                   argv,
Dec 11 02:24:17 localhost celery-worker[2326232]:     ...<6 lines>...
Dec 11 02:24:17 localhost celery-worker[2326232]:                   executable=kwargs.get('executable', None),
Dec 11 02:24:17 localhost celery-worker[2326232]:                   hostname=hostname)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:24:17 localhost celery-worker[2326232]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:24:17 localhost celery-worker[2326232]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:                   after_forkers=False):
Dec 11 02:24:17 localhost celery-worker[2326232]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:24:17 localhost celery-worker[2326232]:     self.after_chdir()
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:24:17 localhost celery-worker[2326232]:     _create_pidlock(pidfile).release()
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:24:17 localhost celery-worker[2326232]:     pidlock.acquire()
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:24:17 localhost celery-worker[2326232]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:24:17 localhost celery-worker[2326232]:     raise value.with_traceback(tb)
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:17 localhost celery-worker[2326232]:     self.write_pid()
Dec 11 02:24:17 localhost celery-worker[2326232]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:17 localhost celery-worker[2326232]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:17 localhost celery-worker[2326232]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:17 localhost celery-worker[2326232]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:20 localhost celery-worker[2326234]: Error: No nodes replied within time constraint
Dec 11 02:24:20 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:24:20 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:24:30 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 13.
Dec 11 02:24:30 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:24:31 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:24:31 localhost celery-worker[2326266]: Traceback (most recent call last):
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:31 localhost celery-worker[2326266]:     self.write_pid()
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:31 localhost celery-worker[2326266]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:31 localhost celery-worker[2326266]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:31 localhost celery-worker[2326266]: During handling of the above exception, another exception occurred:
Dec 11 02:24:31 localhost celery-worker[2326266]: Traceback (most recent call last):
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:24:31 localhost celery-worker[2326266]:     sys.exit(main())
Dec 11 02:24:31 localhost celery-worker[2326266]:              ~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:24:31 localhost celery-worker[2326266]:     sys.exit(_main())
Dec 11 02:24:31 localhost celery-worker[2326266]:              ~~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:24:31 localhost celery-worker[2326266]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:24:31 localhost celery-worker[2326266]:     return self.main(*args, **kwargs)
Dec 11 02:24:31 localhost celery-worker[2326266]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:24:31 localhost celery-worker[2326266]:     rv = self.invoke(ctx)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:24:31 localhost celery-worker[2326266]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:24:31 localhost celery-worker[2326266]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:24:31 localhost celery-worker[2326266]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:24:31 localhost celery-worker[2326266]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:24:31 localhost celery-worker[2326266]:     return callback(*args, **kwargs)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:24:31 localhost celery-worker[2326266]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:24:31 localhost celery-worker[2326266]:     return f(ctx, *args, **kwargs)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:24:31 localhost celery-worker[2326266]:     return detach(sys.executable,
Dec 11 02:24:31 localhost celery-worker[2326266]:                   argv,
Dec 11 02:24:31 localhost celery-worker[2326266]:     ...<6 lines>...
Dec 11 02:24:31 localhost celery-worker[2326266]:                   executable=kwargs.get('executable', None),
Dec 11 02:24:31 localhost celery-worker[2326266]:                   hostname=hostname)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:24:31 localhost celery-worker[2326266]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:24:31 localhost celery-worker[2326266]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:                   after_forkers=False):
Dec 11 02:24:31 localhost celery-worker[2326266]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:24:31 localhost celery-worker[2326266]:     self.after_chdir()
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:24:31 localhost celery-worker[2326266]:     _create_pidlock(pidfile).release()
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:24:31 localhost celery-worker[2326266]:     pidlock.acquire()
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:24:31 localhost celery-worker[2326266]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:24:31 localhost celery-worker[2326266]:     raise value.with_traceback(tb)
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:31 localhost celery-worker[2326266]:     self.write_pid()
Dec 11 02:24:31 localhost celery-worker[2326266]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:31 localhost celery-worker[2326266]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:31 localhost celery-worker[2326266]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:31 localhost celery-worker[2326266]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:34 localhost celery-worker[2326269]: Error: No nodes replied within time constraint
Dec 11 02:24:34 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:24:34 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:24:44 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 14.
Dec 11 02:24:44 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:24:45 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:24:45 localhost celery-worker[2326297]: Traceback (most recent call last):
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:45 localhost celery-worker[2326297]:     self.write_pid()
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:45 localhost celery-worker[2326297]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:45 localhost celery-worker[2326297]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:45 localhost celery-worker[2326297]: During handling of the above exception, another exception occurred:
Dec 11 02:24:45 localhost celery-worker[2326297]: Traceback (most recent call last):
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:24:45 localhost celery-worker[2326297]:     sys.exit(main())
Dec 11 02:24:45 localhost celery-worker[2326297]:              ~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:24:45 localhost celery-worker[2326297]:     sys.exit(_main())
Dec 11 02:24:45 localhost celery-worker[2326297]:              ~~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:24:45 localhost celery-worker[2326297]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:24:45 localhost celery-worker[2326297]:     return self.main(*args, **kwargs)
Dec 11 02:24:45 localhost celery-worker[2326297]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:24:45 localhost celery-worker[2326297]:     rv = self.invoke(ctx)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:24:45 localhost celery-worker[2326297]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:24:45 localhost celery-worker[2326297]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:24:45 localhost celery-worker[2326297]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:24:45 localhost celery-worker[2326297]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:24:45 localhost celery-worker[2326297]:     return callback(*args, **kwargs)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:24:45 localhost celery-worker[2326297]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:24:45 localhost celery-worker[2326297]:     return f(ctx, *args, **kwargs)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:24:45 localhost celery-worker[2326297]:     return detach(sys.executable,
Dec 11 02:24:45 localhost celery-worker[2326297]:                   argv,
Dec 11 02:24:45 localhost celery-worker[2326297]:     ...<6 lines>...
Dec 11 02:24:45 localhost celery-worker[2326297]:                   executable=kwargs.get('executable', None),
Dec 11 02:24:45 localhost celery-worker[2326297]:                   hostname=hostname)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:24:45 localhost celery-worker[2326297]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:24:45 localhost celery-worker[2326297]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:                   after_forkers=False):
Dec 11 02:24:45 localhost celery-worker[2326297]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:24:45 localhost celery-worker[2326297]:     self.after_chdir()
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:24:45 localhost celery-worker[2326297]:     _create_pidlock(pidfile).release()
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:24:45 localhost celery-worker[2326297]:     pidlock.acquire()
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:24:45 localhost celery-worker[2326297]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:24:45 localhost celery-worker[2326297]:     raise value.with_traceback(tb)
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:24:45 localhost celery-worker[2326297]:     self.write_pid()
Dec 11 02:24:45 localhost celery-worker[2326297]:     ~~~~~~~~~~~~~~^^
Dec 11 02:24:45 localhost celery-worker[2326297]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:24:45 localhost celery-worker[2326297]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:24:45 localhost celery-worker[2326297]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:24:48 localhost celery-worker[2326301]: Error: No nodes replied within time constraint
Dec 11 02:24:48 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:24:48 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:24:58 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 15.
Dec 11 02:24:58 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:25:00 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:25:00 localhost celery-worker[2326351]: Traceback (most recent call last):
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:25:00 localhost celery-worker[2326351]:     self.write_pid()
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~~~~~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:25:00 localhost celery-worker[2326351]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:25:00 localhost celery-worker[2326351]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:25:00 localhost celery-worker[2326351]: During handling of the above exception, another exception occurred:
Dec 11 02:25:00 localhost celery-worker[2326351]: Traceback (most recent call last):
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:25:00 localhost celery-worker[2326351]:     sys.exit(main())
Dec 11 02:25:00 localhost celery-worker[2326351]:              ~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:25:00 localhost celery-worker[2326351]:     sys.exit(_main())
Dec 11 02:25:00 localhost celery-worker[2326351]:              ~~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:25:00 localhost celery-worker[2326351]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:25:00 localhost celery-worker[2326351]:     return self.main(*args, **kwargs)
Dec 11 02:25:00 localhost celery-worker[2326351]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:25:00 localhost celery-worker[2326351]:     rv = self.invoke(ctx)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:25:00 localhost celery-worker[2326351]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:25:00 localhost celery-worker[2326351]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:25:00 localhost celery-worker[2326351]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:25:00 localhost celery-worker[2326351]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:25:00 localhost celery-worker[2326351]:     return callback(*args, **kwargs)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:25:00 localhost celery-worker[2326351]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:25:00 localhost celery-worker[2326351]:     return f(ctx, *args, **kwargs)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:25:00 localhost celery-worker[2326351]:     return detach(sys.executable,
Dec 11 02:25:00 localhost celery-worker[2326351]:                   argv,
Dec 11 02:25:00 localhost celery-worker[2326351]:     ...<6 lines>...
Dec 11 02:25:00 localhost celery-worker[2326351]:                   executable=kwargs.get('executable', None),
Dec 11 02:25:00 localhost celery-worker[2326351]:                   hostname=hostname)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:25:00 localhost celery-worker[2326351]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:25:00 localhost celery-worker[2326351]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:                   after_forkers=False):
Dec 11 02:25:00 localhost celery-worker[2326351]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:25:00 localhost celery-worker[2326351]:     self.after_chdir()
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:25:00 localhost celery-worker[2326351]:     _create_pidlock(pidfile).release()
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:25:00 localhost celery-worker[2326351]:     pidlock.acquire()
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:25:00 localhost celery-worker[2326351]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:25:00 localhost celery-worker[2326351]:     raise value.with_traceback(tb)
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:25:00 localhost celery-worker[2326351]:     self.write_pid()
Dec 11 02:25:00 localhost celery-worker[2326351]:     ~~~~~~~~~~~~~~^^
Dec 11 02:25:00 localhost celery-worker[2326351]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:25:00 localhost celery-worker[2326351]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:25:00 localhost celery-worker[2326351]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:25:01 localhost celery-worker[2326352]: Traceback (most recent call last):
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     sys.exit(main())
Dec 11 02:25:01 localhost celery-worker[2326352]:              ~~~~^^
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 14, in main
Dec 11 02:25:01 localhost celery-worker[2326352]:     from celery.bin.celery import main as _main
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 19, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from celery.app.utils import find_app
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/app/__init__.py", line 2, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from celery import _state
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/_state.py", line 15, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from celery.utils.threads import LocalStack
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/utils/__init__.py", line 6, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from kombu.utils.objects import cached_property
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/kombu/utils/__init__.py", line 6, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/kombu/utils/compat.py", line 12, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/kombu/exceptions.py", line 9, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/amqp/__init__.py", line 29, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from .basic_message import Message  # noqa
Dec 11 02:25:01 localhost celery-worker[2326352]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/amqp/basic_message.py", line 3, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from .serialization import GenericContent
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/amqp/serialization.py", line 16, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from .utils import bytes_to_str as pstr_t
Dec 11 02:25:01 localhost celery-worker[2326352]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/amqp/utils.py", line 6, in <module>
Dec 11 02:25:01 localhost celery-worker[2326352]:     from vine import promise  # noqa
Dec 11 02:25:01 localhost celery-worker[2326352]:     ^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:01 localhost celery-worker[2326352]: ModuleNotFoundError: No module named 'vine'
Dec 11 02:25:02 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=1/FAILURE
Dec 11 02:25:02 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:25:12 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 16.
Dec 11 02:25:12 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:25:12 localhost (celery)[2326507]: celery-worker.service: Unable to locate executable '/opt/goraids/server-py/venv/bin/celery': No such file or directory
Dec 11 02:25:12 localhost (celery)[2326507]: celery-worker.service: Failed at step EXEC spawning /opt/goraids/server-py/venv/bin/celery: No such file or directory
Dec 11 02:25:12 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=203/EXEC
Dec 11 02:25:12 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:25:12 localhost systemd[1]: Failed to start celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:25:22 localhost systemd[1]: celery-worker.service: Scheduled restart job, restart counter is at 17.
Dec 11 02:25:22 localhost systemd[1]: Starting celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics)...
Dec 11 02:25:23 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:25:24 localhost celery-worker[2326707]: Traceback (most recent call last):
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:25:24 localhost celery-worker[2326707]:     self.write_pid()
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~~~~~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:25:24 localhost celery-worker[2326707]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:25:24 localhost celery-worker[2326707]: FileNotFoundError: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:25:24 localhost celery-worker[2326707]: During handling of the above exception, another exception occurred:
Dec 11 02:25:24 localhost celery-worker[2326707]: Traceback (most recent call last):
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/bin/celery", line 7, in <module>
Dec 11 02:25:24 localhost celery-worker[2326707]:     sys.exit(main())
Dec 11 02:25:24 localhost celery-worker[2326707]:              ~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/__main__.py", line 15, in main
Dec 11 02:25:24 localhost celery-worker[2326707]:     sys.exit(_main())
Dec 11 02:25:24 localhost celery-worker[2326707]:              ~~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/celery.py", line 236, in main
Dec 11 02:25:24 localhost celery-worker[2326707]:     return celery(auto_envvar_prefix="CELERY")
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
Dec 11 02:25:24 localhost celery-worker[2326707]:     return self.main(*args, **kwargs)
Dec 11 02:25:24 localhost celery-worker[2326707]:            ~~~~~~~~~^^^^^^^^^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
Dec 11 02:25:24 localhost celery-worker[2326707]:     rv = self.invoke(ctx)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
Dec 11 02:25:24 localhost celery-worker[2326707]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
Dec 11 02:25:24 localhost celery-worker[2326707]:                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
Dec 11 02:25:24 localhost celery-worker[2326707]:     return ctx.invoke(self.callback, **ctx.params)
Dec 11 02:25:24 localhost celery-worker[2326707]:            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
Dec 11 02:25:24 localhost celery-worker[2326707]:     return callback(*args, **kwargs)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/click/decorators.py", line 34, in new_func
Dec 11 02:25:24 localhost celery-worker[2326707]:     return f(get_current_context(), *args, **kwargs)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/base.py", line 135, in caller
Dec 11 02:25:24 localhost celery-worker[2326707]:     return f(ctx, *args, **kwargs)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 336, in worker
Dec 11 02:25:24 localhost celery-worker[2326707]:     return detach(sys.executable,
Dec 11 02:25:24 localhost celery-worker[2326707]:                   argv,
Dec 11 02:25:24 localhost celery-worker[2326707]:     ...<6 lines>...
Dec 11 02:25:24 localhost celery-worker[2326707]:                   executable=kwargs.get('executable', None),
Dec 11 02:25:24 localhost celery-worker[2326707]:                   hostname=hostname)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/bin/worker.py", line 118, in detach
Dec 11 02:25:24 localhost celery-worker[2326707]:     with detached(logfile, pidfile, uid, gid, umask, workdir, fake,
Dec 11 02:25:24 localhost celery-worker[2326707]:          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:                   after_forkers=False):
Dec 11 02:25:24 localhost celery-worker[2326707]:                   ^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 331, in open
Dec 11 02:25:24 localhost celery-worker[2326707]:     self.after_chdir()
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~~~~~~~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 413, in after_chdir_do
Dec 11 02:25:24 localhost celery-worker[2326707]:     _create_pidlock(pidfile).release()
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~~~~~~~~~^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 259, in _create_pidlock
Dec 11 02:25:24 localhost celery-worker[2326707]:     pidlock.acquire()
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~~~~~~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 140, in acquire
Dec 11 02:25:24 localhost celery-worker[2326707]:     reraise(LockFailed, LockFailed(str(exc)), sys.exc_info()[2])
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/exceptions.py", line 109, in reraise
Dec 11 02:25:24 localhost celery-worker[2326707]:     raise value.with_traceback(tb)
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 138, in acquire
Dec 11 02:25:24 localhost celery-worker[2326707]:     self.write_pid()
Dec 11 02:25:24 localhost celery-worker[2326707]:     ~~~~~~~~~~~~~~^^
Dec 11 02:25:24 localhost celery-worker[2326707]:   File "/opt/goraids/server-py/venv/lib/python3.13/site-packages/celery/platforms.py", line 207, in write_pid
Dec 11 02:25:24 localhost celery-worker[2326707]:     pidfile_fd = os.open(self.path, PIDFILE_FLAGS, PIDFILE_MODE)
Dec 11 02:25:24 localhost celery-worker[2326707]: celery.platforms.LockFailed: [Errno 2] No such file or directory: '/var/run/goraids/celery-worker.pid'
Dec 11 02:25:27 localhost celery-worker[2326709]: Error: No nodes replied within time constraint
Dec 11 02:25:27 localhost systemd[1]: celery-worker.service: Control process exited, code=exited, status=69/UNAVAILABLE
Dec 11 02:25:27 localhost systemd[1]: celery-worker.service: Failed with result 'exit-code'.
Dec 11 02:25:28 localhost systemd[1]: celery-worker.service: Scheduled restart job immediately on client request, restart counter is at 18.
Dec 11 02:25:28 localhost systemd[1]: Started celery-worker.service - GoRaids Celery Worker - General Tasks (Emails, Processing, Analytics).
Dec 11 02:25:31 localhost celery-worker[2326847]:  
Dec 11 02:25:31 localhost celery-worker[2326847]:  -------------- worker@localhost v5.4.0 (opalescent)
Dec 11 02:25:31 localhost celery-worker[2326847]: --- ***** -----
Dec 11 02:25:31 localhost celery-worker[2326847]: -- ******* ---- Linux-6.12.57+deb13-amd64-x86_64-with-glibc2.41 2025-12-11 02:25:31
Dec 11 02:25:31 localhost celery-worker[2326847]: - *** --- * ---
Dec 11 02:25:31 localhost celery-worker[2326847]: - ** ---------- [config]
Dec 11 02:25:31 localhost celery-worker[2326847]: - ** ---------- .> app:         goraids:0x7fadb27a5fd0
Dec 11 02:25:31 localhost celery-worker[2326847]: - ** ---------- .> transport:   redis://server_user:**@localhost:6379//
Dec 11 02:25:31 localhost celery-worker[2326847]: - ** ---------- .> results:     redis://server_user:**@localhost:6379/
Dec 11 02:25:31 localhost celery-worker[2326847]: - *** --- * --- .> concurrency: 2 (prefork)
Dec 11 02:25:31 localhost celery-worker[2326847]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
Dec 11 02:25:31 localhost celery-worker[2326847]: --- ***** -----
Dec 11 02:25:31 localhost celery-worker[2326847]:  -------------- [queues]
Dec 11 02:25:31 localhost celery-worker[2326847]:                 .> analytics        exchange=analytics(direct) key=analytics
Dec 11 02:25:31 localhost celery-worker[2326847]:                 .> emails           exchange=emails(direct) key=emails
Dec 11 02:25:31 localhost celery-worker[2326847]:                 .> maintenance      exchange=maintenance(direct) key=maintenance
Dec 11 02:25:31 localhost celery-worker[2326847]:                 .> processing       exchange=processing(direct) key=processing
Dec 11 02:25:31 localhost celery-worker[2326847]: [tasks]
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.analytics.cleanup_expired_rooms
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.analytics.cleanup_stale_queue_entries
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.analytics.log_analytics_event
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.emails.send_password_reset_email
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.emails.send_verification_email
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.matching.process_leave_room
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.matching.run_matchmaking
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.notifications.send_push_notification
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.notifications.send_push_notification_batch
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.rooms.process_room_completion
Dec 11 02:25:31 localhost celery-worker[2326847]:   . api.tasks.sync.periodic_sync
Dec 11 02:25:31 localhost celery-worker[2326847]:   . config.celery.debug_task