Issues with vak predict with the latest version of vak

Hi! I am having problems during the vak predict predict.toml step. It always gives this error:
Traceback (most recent call last):
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/bin/vak”, line 10, in
sys.exit(main())
^^^^^^
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/main.py”, line 48, in main
cli.cli(command=args.command, config_file=args.configfile)
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/cli.py”, line 49, in cli
COMMAND_FUNCTION_MAPcommand
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/cli.py”, line 18, in predict
predict(toml_path=toml_path)
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/predict.py”, line 51, in predict
core.predict(
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/core/predict.py”, line 231, in predict
y_pred = pred_dict[spect_path]

KeyError: ‘/gpfs01/veit/data/thomas/avanitesting2/audio/tweetynet_testdata/spectrograms_generated_240415_152302/bu01bk01_240118_43355031_120323.wav.spect.npz’
0%| | 0/16 [00:00<?, ?it/s]
Traceback (most recent call last):
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/bin/vak”, line 10, in
sys.exit(main())
^^^^^^
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/**main**.py”, line 48, in main
cli.cli(command=args.command, config_file=args.configfile)
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/cli.py”, line 49, in cli
COMMAND_FUNCTION_MAPcommand
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/cli.py”, line 18, in predict
predict(toml_path=toml_path)
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/cli/predict.py”, line 51, in predict
core.predict(
File “/gpfs01/veit/user/akoparkar/miniconda3/envs/vakenv/lib/python3.12/site-packages/vak/core/predict.py”, line 231, in predict
y_pred = pred_dict[spect_path]
~~~~~~~~~^^^^^^^^^^^^
KeyError: ‘/gpfs01/veit/data/thomas/avanitesting2/audio/tweetynet_testdata/spectrograms_generated_240415_152302/bu01bk01_240118_43310843_120220.wav.spect.npz’

It is not file specific - I changed the dataset and got the same result

Hi @koparkanya sorry you’re having this issue, thank you for posting here.

I’m wondering if it might be related to the way you’re running vak predict for the analysis you’re working on. I’ll follow up by email.

It could still be a bug, but it’s not immediately obvious to me without more info what the source of the bug would be. I am able to run vak predict using the test configs + data built in to the package without getting an error like this–but that might just mean there’s some case I’m not testing for :confused:

If we do identify a bug, then we should raise an issue on the GitHub repo for vak. I will try to remember to reply here linking to that if we do.

Updating here:
Thank you @koparkanya for helping me get to the root of the issue.

The bug report is here:

Basically this is because our “backend”, lightning, runs with a “distributed” strategy because of the defaults we are using.

A workaround for now is to run export CUDA_VISIBLE_DEVICES=0

See more detail in this comment (preview won’t show comment :confused: ):

This also explains a related bug with learncurve: BUG: Running lightning with default strategy 'DDP' breaks learncurve function · Issue #742 · vocalpy/vak · GitHub

Hi @koparkanya thank you again for catching this, just letting you know it should now be fixed in the main development branch, after merging this pull request: ENH: Add lightning.Trainer config, fix #691 #687 #742 #745 by NickleDave · Pull Request #752 · vocalpy/vak · GitHub

I’m in the middle of some other changes–like, notice I just updated the configuration file format–but as soon as I get those added I will release a new version. (Because I need it for experiments I should already be running :sweat_smile: and we should honestly just fully release a version 1.0 already :confused: )

If you could test once I do that it would be very much appreciated! :pray: :bowing_man: