Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#8456

Possible Duplicates

• None

Environment & Settings

Celery version: 5.3.6 (emerald-rush)

software -> celery:5.3.6 (emerald-rush) kombu:5.3.4 py:3.8.10
            billiard:4.2.0 redis:5.0.1
platform -> system:Darwin arch:64bit
            kernel version:23.1.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://0.0.0.0:6479/

broker_url: 'redis://0.0.0.0:6479//'
result_backend: 'redis://0.0.0.0:6479/'
deprecated_settings: None
task_allow_error_cb_on_chord_header: True

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: 5.3.1
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

aiohttp==3.8.5
aiosignal==1.3.1
amqp==5.2.0
asgiref==3.6.0
astroid==2.15.2
async-timeout==4.0.3
attrs==22.2.0
autopep8==1.5.5
backports.zoneinfo==0.2.1
bandit==1.7.5
billiard==4.2.0
black==23.7.0
boto3-stubs==1.19.12.post1
botocore-stubs==1.29.107
celery==5.3.6
celery-stubs==0.1.3
certifi==2022.12.7
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
coreapi==2.3.3
coreschema==0.0.4
data-science-types==0.2.23
dill==0.3.6
distlib==0.3.6
Django==4.2
django-filter-stubs==0.1.3
django-stubs==1.15.0
django-stubs-ext==0.8.0
djangorestframework==3.14.0
djangorestframework-stubs==1.9.1
drf-yasg==1.20.3
factory-boy==3.2.1
Faker==18.3.4
filelock==3.10.7
flake8==3.8.4
frozenlist==1.4.0
fuzzywuzzy-stubs==0.0.1
gitdb==4.0.10
GitPython==3.1.31
graphene-stubs==0.15
identify==2.5.22
idna==3.4
inflection==0.5.1
isort==5.12.0
itypes==1.2.0
jedi==0.17.2
Jinja2==3.1.2
kombu==5.3.4
lazy-object-proxy==1.9.0
markdown-it-py==2.2.0
MarkupSafe==2.1.2
mccabe==0.6.1
mdurl==0.1.2
multidict==6.0.4
mypy==1.1.1
mypy-extensions==1.0.0
nodeenv==1.7.0
packaging==23.0
parso==0.7.1
pathspec==0.11.1
pbr==5.11.1
pip-licenses==3.5.5
platformdirs==3.2.0
pluggy==1.0.0
pre-commit==2.7.1
prompt-toolkit==3.0.41
PTable==0.9.2
pycodestyle==2.6.0
pydocstyle==6.3.0
pyflakes==2.2.0
Pygments==2.14.0
pylint==2.17.2
python-dateutil==2.8.2
python-jsonrpc-server==0.4.0
python-language-server==0.36.2
pytoolconfig==1.2.5
pytz==2023.3
PyYAML==6.0
ratelimit-stubs==2.2.1
redis==5.0.1
regex==2019.11.1
requests==2.28.2
rich==13.3.3
rope==1.7.0
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.7
six==1.16.0
smmap==5.0.0
snowballstemmer==2.2.0
sqlparse==0.4.3
stevedore==5.0.0
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.7
types-awscrt==0.16.13.post1
types-beautifulsoup4==4.10.20
types-docutils==0.19.1.7
types-pytz==2023.3.0.0
types-PyYAML==5.4.12
types-requests==2.25.12
types-setuptools==67.3.0.2
types-six==0.1.9
typing_extensions==4.5.0
tzdata==2023.3
ujson==5.7.0
uritemplate==4.1.1
urllib3==1.26.15
vine==5.1.0
virtualenv==20.21.0
wcwidth==0.2.12
wrapt==1.15.0
yapf==0.32.0
yarl==1.8.2

Other Dependencies

Minimally Reproducible Test Case

from celery import Celery, chain, chord, signature, group

app = Celery(
    "celery_bug", backend="redis://0.0.0.0:6479/", broker="redis://0.0.0.0:6479/"
)

app.conf.task_allow_error_cb_on_chord_header = True


@app.task(bind=True)
def orig_task(self, arg):
    chord_headers = group([chord_header.s(arg=f"arg{i}") for i in range(5)])
    replacement_chord = chord(chord_headers, chord_body.s())
    return self.replace(replacement_chord)


@app.task
def chord_header(arg):
    return f"header: {arg}"


@app.task
def chord_body(arg):
    return f"body: {arg}]"


@app.task
def handle_error(*args, **kwargs):
    print(f"handle error called with args {args} kwargs {kwargs}")


def main():
    print(f"hello world")
    res = orig_task.apply_async(args=["spam"], link_error=handle_error.s())
    print(f"RESULT: {res.get()}")


if __name__ == "__main__":
    main()

Expected Behavior

I would expect the orig_task to be replaced with replacement_chord and give expected output.

This is the expected output that I do see if task_allow_error_cb_on_chord_header is False, or if the orig_task is called without the link_error= callback:

$ poetry run python celery_bug.py 
hello world
RESULT: body: ['header: arg0', 'header: arg1', 'header: arg2', 'header: arg3', 'header: arg4']]

Actual Behavior

Instead, I get this AttributeError:

$ poetry run python celery_bug.py 
hello world
Traceback (most recent call last):
  File "celery_bug.py", line 39, in <module>
    main()
  File "celery_bug.py", line 35, in main
    print(f"RESULT: {res.get()}")
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/celery/backends/asynchronous.py", line 223, in wait_for_pending
    return result.maybe_throw(callback=callback, propagate=propagate)
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/Users/robertgalloway/Library/Caches/pypoetry/virtualenvs/rpg-play-aJQQ1jqR-py3.8/lib/python3.8/site-packages/vine/utils.py", line 27, in reraise
    raise value
AttributeError: 'dict' object has no attribute 'clone'

I suspect the error is related to this code in the _chord.link_error() method:

        if self.app.conf.task_allow_error_cb_on_chord_header:
            for task in maybe_list(self.tasks) or []:
                task.link_error(errback.clone(immutable=True))