-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
adding expert to hp ignore due to pickeling issue #197
base: master
Are you sure you want to change the base?
Conversation
Signed-off-by: Bonham79 <[email protected]>
Multiprocess + pickle has given me so many issues recently across interdependent libraries... One edge case: how about continued training from a checkpoint? Then we need the expert right, which I assume has some relevant state? |
@Adamits I think this would pose exactly the same issue as the multi-GPU case. |
@kylebgorman Right, I meant an edge case where this solution would break something. If I understand correctly, Travis wants to just not save the expert state to avoid the pickle issue with TQDM. But if we want to continue form a checkpoint, i think we need the expert state. |
you'd have to train the expert again. thankfully your np fix makes it less annoying. but yeah, that's why i was curious if there were better ideas. fwiw, the entire point behind the expert transducer is for single gpu setup. (Actually it works better on CPU for most cases due to the expert queries). You don't get really parallelism speed ups since there's so many gpu-> cpu transfers. |
Yeah I also find the transducer is better on CPU. (The same was true with the original code in DyNet.) |
Yeah it's the main motivation for its use so not surprised. |
Fixes: #192
TQDM in the transducer's expert module doesn't pickle currently. So when doing multi-gpu training, pytorch gets annoyed. Since it's not really needed beyond training, I'm dropping it from the model.
Note, when running prediction with the model, the lack of the hyperparameter means a new expert will have to be created. This is dumb, so need to add a dummy class to the transducer model to avoid this. This portion I'm leaving in draft atm.