Slow training with large batch size

[favyen2]

pytorch default dataloader has each worker load one batch, so with a large batch size like batch_size=64, every worker is loading 64 windows sequentially, and we have to wait for the first worker to finish an entire batch before the training can start.

Maybe it'd be better to form a batch by combining items across workers, given that loading from GCS can be slow?

Or maybe we should switch to Weka and find a good way to sync between GCS and Weka, that could solve the problem.