[Bug] Joblib multirun not working when specified within the experiment file

[brfi3983]

🐛 Bug

Description

So basically, I followed the hydra guide () and was able to run a multirun experiment from the command line as expected. However, as soon as I try to specify it in my .yaml file within my experiments folder, I end up getting some namespace issues that prevent me from overriding the hydra launcher for the multirun to Joblib.

To reproduce

/experiment/my_experiment.yaml

# @package _global_

defaults:
  - override hydra/launcher: joblib # as shown in hydra example

Here is the root config
/conf/config.yaml

defaults:
  - _self_

hydra:
  searchpath:
    - file:///${cwd}

** Stack trace/error message **

Now, since running my_app.py +experiment=my_experiment,my_experiment hydra/launcher=joblib -m works, I was expecting to be able to run my_app.py +experiment=my_experiment,my_experiment -m the same way if I simply included hydra/launcher=joblib in my my_experiment.yaml file. However, I get the following error:

In 'experiment/my_experiment': Could not override 'experiment/hydra/[email protected]'. No match in the defaults list.

I have thus tried to change my my_experiment.yaml namespace in various ways (using the @ package operator) but can not seem to get it right. Does anyone have a suggestion?

System information

• Hydra Version :
  hydra-core: 1.3.2
  hydra-joblib-launcher: 1.2.0
• Python version : 3.10.13
• Virtual environment type and version : conda 24.1.2
• Operating system : ubuntu

Additional context

Add any other context about the problem here.

[omry]

Default list entries are relative to the containing config.
try to prefix the hydra override with / :

# @package _global_

defaults:
  - override /hydra/launcher: joblib # as shown in hydra example

[brfi3983]

Hi @omry,

So, I tried that and it got rid of the error, but it still seems to use the basic launcher even though the joblib launcher is chosen. Specifically, this is what it shows inside the terminal:

[2024-09-11 10:53:07,313][HYDRA] Launching 2 jobs locally
[2024-09-11 10:53:07,313][HYDRA]        #0 : +experiment=params

And if I were to override the launcher by command line (instead of inside the file), it seems to properly use the joblib launcher:

[2024-09-11 10:54:46,095][HYDRA] Joblib.Parallel(n_jobs=-1,backend=loky,prefer=processes,require=None,verbose=0,timeout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=None,mmap_mode=r) is launching 2 jobs
[2024-09-11 10:54:46,096][HYDRA] Launching jobs, sweep output dir : multirun/2024-09-11/10-54-45
[2024-09-11 10:54:46,096][HYDRA]        #0 : +experiment=params
[2024-09-11 10:54:46,096][HYDRA]        #1 : +experiment=params

In the former case, we can confirm if joblib was chosen correctly by using the --cfg hydra and see its output:

    choices:
      hydra/env: default
      hydra/callbacks: null
      hydra/job_logging: default
      hydra/hydra_logging: default
      hydra/hydra_help: default
      hydra/help: default
      hydra/sweeper: basic
      hydra/launcher: basic
      hydra/output: default

As you can see, it is still using the basic launcher instead of joblib despite me specifying override.

What is interesting is that if we do the same exact thing (where we override it inside the file), but we do not do a multirun, the launcher is correctly overridden:

    choices:
      experiment: params
      hydra/env: default
      hydra/callbacks: null
      hydra/job_logging: default
      hydra/hydra_logging: default
      hydra/hydra_help: default
      hydra/help: default
      hydra/sweeper: basic
      hydra/launcher: joblib
      hydra/output: default

For reference, if we look at the logged hydra configuration setup for the working multirun (the command line case), we can see the ideal setup:

    choices:
      hydra/env: default
      hydra/callbacks: null
      hydra/job_logging: default
      hydra/hydra_logging: default
      hydra/hydra_help: default
      hydra/help: default
      hydra/sweeper: basic
      hydra/launcher: joblib
      hydra/output: default

For completeness, the overrides picked up by hydra in the command line case seem to be

  overrides:
    hydra:
    - hydra/launcher=joblib
    - hydra.mode=MULTIRUN
    task:
    - +experiment=params,params

while the file case seem to be

  overrides:
    hydra:
    - hydra.mode=MULTIRUN
    task:
    - +experiment=params,params

Is this expected behavior, or is somewhere hydra's multirun loading the basic launcher despite having it overridden with joblib?

[omry]

I don't think this is the expected behavior.
If you don't override hydra.mode and use -m from the command it's working as expected?

[brfi3983]

So, I never explicitly override hydra.mode in the experiment file.

The only difference is one of them I use hydra/launcher=joblib in the command line (which works), while the other option is I use

defaults:
- override /hydra/launcher: joblib

in the experiment file (which does not work).

For both of the setups I specify myapp.py +experiment=my_experiment,my_experiment -m. The only thing that changes is the specification of the hydra launcher.