You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How can I configure celery to use a FilesystemBackend where I can set the "open" parameter in the constructor?
Currently the project is using config_from_object with a module of variables. Is there a way to set this "open" from a variable? Is they way to configure with a mix of this and setting constructed objects?
The following does not seem like the correct way: celery.backend.open = open_with_efs_recache_decorator( celery.backend.open )
Why is this change needed?
2 ) Using EFS files system on AWS ECS between containers there appears to be some caching which delays the detection of task results. That is the open function fails for 30-90 seconds after the file has been created. However doing a "ls" on the directory seemed to refresh the cache enabling detection of the task results sooner.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Currently the project is using
config_from_object
with a module of variables. Is there a way to set this "open" from a variable? Is they way to configure with a mix of this and setting constructed objects?The following does not seem like the correct way:
celery.backend.open = open_with_efs_recache_decorator( celery.backend.open )
Why is this change needed?
2 ) Using EFS files system on AWS ECS between containers there appears to be some caching which delays the detection of task results. That is the
open
function fails for 30-90 seconds after the file has been created. However doing a "ls" on the directory seemed to refresh the cache enabling detection of the task results sooner.So the following decorator addressed this issue:
Has anyone else had this issue with shared files system? Are there other work arounds?
Thanks,
Brad
Beta Was this translation helpful? Give feedback.
All reactions