Fix interruptGenerate causing the engine to get stuck when using synchronous completion calls #596
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…chatCompletion. It was doing this in effect within asyncGenerate, and kind of in generate, but the completion code would never call generate if interruptGenerate was ever called in the past because it would be stuck true. Also do it in a way that allows users to still call interruptGenerate during a context switch of acquiring the model lock.
mitchellw
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey, first time contributor to Web-LLM, so let me know if there's anything else you need from me. Just fixing a bug that was causing us issues!
I noticed that asyncGenerate is not private, so I didn't want to necessarily change other parts beyond what I had tested to work, but I think we could/should probably get rid of the acquisition of the lock and reseting of the interruptSignal in there too.
Issue / Motivation
Previously if you are using synchronous completion or chatCompletion calls, then when you call interruptGenerate, the engine will be forever stuck. This was not the case when using streaming (asyncGenerate) as far as I can tell.
Fix
Reset the interruptSignal after acquiring the lock in completion and chatCompletion.
It was doing this in effect within asyncGenerate, and kind of in generate, but the completion code would never call generate if interruptGenerate was ever called in the past because it would be stuck true.
Also do it in a way that allows users to still call interruptGenerate during a context switch of acquiring the model lock, which I think was a potential race condition.