Update _wait_for_cluster_to_start func to check for libraries statuses
#822
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Enhance `_wait_for_cluster_to_start` to wait for library installation
Previously, the `_wait_for_cluster_to_start` function only checked if the cluster state was 'running'. This update adds a check to ensure all cluster libraries have been successfully installed (or skipped or restored) before proceeding. This ensures the cluster is fully ready for operations that depend on these libraries.
- Added `get_cluster_libraries_status` to fetch the status of cluster libraries.
- Updated `_wait_for_cluster_to_start` to wait until all libraries are in valid statuses ('INSTALLED', 'RESTORED', 'SKIPPED').
psivash
requested review from
andrefurlan-db,
benc-db and
rcypher-databricks
as code owners
psivash
changed the title
_wait_for_cluster_to_start func to check for libraries statuses
benc-db
reviewed
Oct 10, 2024
| return | ||
| else: | ||
| time.sleep(5) | ||
| libraries_status_response = self.get_cluster_libraries_status() |
There was a problem hiding this comment.
I'm surprised this isn't done as part of 'STARTING'. Does every run need to wait for library installs, or just those that specify additional libraries?
There was a problem hiding this comment.
Good question. I assume that basic default libraries are a part of 'STARTING' indeed and only the libraries that are added explicitly (the ones exposed via Libraries API) are not and need extra time so that python models do work correctly.
There was a problem hiding this comment.
If that's the case, let's take this branch only if additional libraries have been specified.
benc-db
requested changes
Oct 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #821
Description
Enhance
_wait_for_cluster_to_startto wait for library installationPreviously, the
_wait_for_cluster_to_startfunction only checked if the cluster state was 'running'. This update adds a check to ensure all cluster libraries have been successfully installed (or skipped or restored) before proceeding. This ensures the cluster is fully ready for operations that depend on these libraries.get_cluster_libraries_statusto fetch the status of cluster libraries._wait_for_cluster_to_startto wait until all libraries are in valid statuses ('INSTALLED', 'RESTORED', 'SKIPPED').Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.