template validation/submit fail if dynamic templateRef is used in nested template

[tczhao]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When using dynamic template in nested workflow template calls, the validation failed and couldn't submit workflow.
calling wftmpl1 -> wftmpl2:dynamic template -> wftmpl3 has issue
calling wftmpl2:dynamic template -> wftmpl3 is fine
The issue applies to both clusterworkflowtemplate and workflowtemplate

To reproduce the error, run

argo submit --from clusterworkflowtemplate/cwt1  --dry-run -o yaml

This would produce error

FATA[2024-10-17T12:09:45.670Z] Failed to submit workflow: rpc error: code = InvalidArgument desc = templates.main.tasks.cwt1-s1 templates.main2.steps[0].cwt2-s1 template reference cwt3.placeholder-11 not found

Version(s)

latest, e18570c

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: cwt1
spec:
  entrypoint: main
  templates:
  - name: main
    dag:
      tasks:
      - name: cwt1-s1
        templateRef:
          name: cwt2
          template: main2
          clusterScope: true
---
apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: cwt2
spec:
  entrypoint: main2
  templates:
  - name: main2
    inputs:
      parameters:
      - name: api-template
        value: "abc-abc"
    outputs:
      parameters:
      - name: output-foo
        valueFrom:
          parameter: "{{steps.cwt2-s1.outputs.parameters.foo}}"
    steps:
    - - name: cwt2-s1
        templateRef:
          name: cwt3
          template: "{{inputs.parameters.api-template}}"
          clusterScope: true
---
apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: cwt3
spec:
  templates:
  - name: abc-abc
    container:
      image: busybox
      command: [sleep]
      args: ["1"]
    outputs:
      parameters:
        - name: foo
          valueFrom:
            path: /tmp/abc

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded