Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/falcon #71

Open
wants to merge 40 commits into
base: develop
Choose a base branch
from
Open

Feature/falcon #71

wants to merge 40 commits into from

Conversation

xeon27
Copy link
Collaborator

@xeon27 xeon27 commented Sep 15, 2023

PR Type ([Feature | Fix | Documentation | Test])

Feature

Short Description

  1. Enabled Accelerate DeepSpeed for multi-GPU
  2. Bumped up GPUs for both falcon versions (7B-2, 40B-4) enabling longer output sequences

xeon27 and others added 30 commits August 1, 2023 16:06
@xeon27
use 4 GPUs for falcon 40B
16e47ec
@MChoi-git
changed connection timeout for inference client
9f6279c
@xeon27
Increased Triton inference timeout
58a1dc1
@xeon27
fixed qos
6a30ac4
@xeon27
fixed bug
a21cec9
@xeon27
Merge branch 'develop' into feature/falcon
8736481
@xeon27
added inference timeout to config
12f902a
@xeon27
Configured accelerate deepspeed
417c539
@xeon27
Configured accelerate deepspeed
2ea851e
@xeon27
Merge branch 'feature/falcon' of https://github.com/VectorInstitute/k…
12e8938 
…aleidoscope into feature/falcon
@xeon27
Removed commented lines
da25888
@xeon27
Merge branch 'develop' into feature/falcon
defe7dd
@xeon27
Fixed qos
9f85f0a
@xeon27
Updated
0ab5141
@xeon27
Removed PYTHONPATH and memory allocation debug command
63576c5
@xeon27
[DEBUG] Launching accelerate with python
cc5b56a
@xeon27
[DEBUG] Added PYTHONPATH back
cf59052
@xeon27
[DEBUG] Added -s
be04248
@xeon27
[DEBUG] Updated launch file
95c263c
@xeon27
[DEBUG] Updated launch command
8754b44
@xeon27
[DEBUG] Updated launch command
cbef188
@xeon27
[DEBUG] Updated launch command
bec84e4
@xeon27
[DEBUG] Using torchrun
c762b26
@xeon27
[DEBUG] Using torchrun - added master
07c253f
@xeon27
[DEBUG] Using torch launch
a13d5a1
@xeon27
[DEBUG] Switch back to accelerate launch
460794d
@xeon27
[DEBUG] using torch launch
8746cfb
@xeon27
[DEBUG] Added local_rank argument for torch launch
c8a88b1
@xeon27
[DEBUG] Changed default for local_rank
7f887e4
@xeon27
[DEBUG] Back to accelerate launch
0af4885
markcoatsworth
markcoatsworth reviewed Sep 27, 2023
View reviewed changes
model_service/models/falcon/launch_falcon-40b.slurm
echo "SLURM_GPUS_ON_NODE"=$SLURM_GPUS_ON_NODE
echo "SLURM_CPUS_ON_NODE"=$SLURM_CPUS_ON_NODE
echo "SLURM_PROCID"=$SLURM_PROCID

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommend you just run env instead so you get all the Slurm variables, along with other environment variables tha tmight be helpful

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am just printing these for debug purposes, can you elaborate further on using env to print them?

markcoatsworth
markcoatsworth requested changes Sep 27, 2023
View reviewed changes
Copy link
Collaborator

@markcoatsworth markcoatsworth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a bunch of comments. Couple points of confusion, nothing too serious. Also there is lots of commented-out code here that creates clutter, this should be deleted.

markcoatsworth
markcoatsworth reviewed Sep 29, 2023
View reviewed changes
web/utils/triton.py Outdated Show resolved Hide resolved
markcoatsworth
markcoatsworth reviewed Sep 29, 2023
View reviewed changes
web/utils/triton.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@markcoatsworth markcoatsworth markcoatsworth requested changes

@jwilles jwilles Awaiting requested review from jwilles

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xeon27 @markcoatsworth @MChoi-git