Note regarding mega nodes #3823
Conversation
jtravee
force-pushed
the
Tejeev-patch-4
branch
from
5d56068
to
f674d89
Compare
cbron
requested review from
bmdepesa and
cbron
and removed request for
rmweir
jtravee
requested review from
dnoland1 and
rmweir
and removed request for
rmweir
content/rke/latest/en/os/_index.md
Outdated
|
|
||
| - If you must use servers larger than 24 CPU, use virtualization tooling, such as what [Harvester](https://docs.harvesterhci.io/v1.0/rancher/virtualization-management/) provides, to subdivide the servers. | ||
|
|
||
| - Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a minimum of roughly 24 cores per server and a maximum of the recommended 110 pods per server. |
There was a problem hiding this comment.
| - Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a minimum of roughly 24 cores per server and a maximum of the recommended 110 pods per server. | |
| - Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a maximum of roughly 24 cores per server and a maximum of the recommended 110 pods per server. |
There was a problem hiding this comment.
@Tejeev, checking that minimum should be maximum here?
There was a problem hiding this comment.
yes, though these are somewhat softer than I think I make it sound. Folks that know what they are doing may up the pod count to the real limit of 251 (IIRC. we should double check that) and there are use cases that might use more cores, per pod, though at the moment they are rare. Not sure how to convey this a bit better or if we leave it as is to prevent over committing by folks that do not have real need of increased limits
There was a problem hiding this comment.
Perhaps:
"It is recommended to maintain a maximum of roughly 24 cores and 110 pods per server"
Co-authored-by: Jen Travinski <[email protected]>
There was a problem hiding this comment.
Given questions the scaling team has gotten in the past about our docs, we need to understand where any information comes from and how we got those numbers. That way we can understand and relay that information as necessary, as well as re-test in the future. More specifically if this is the average case, or a single case, what distro it references, if it's an upstream recommendation etc...
|
|
||
| - Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a maximum of roughly 24 cores per server and a maximum of the recommended 110 pods per server. | ||
|
|
||
| - If you are deploying an application or system that specifically requires a large number of pods, you may increase the pod limit. Please note, however, that going above 254 pods per server is not supported by default pod CIDR settings unless the pods are using host networking. |
There was a problem hiding this comment.
This goes against upstream docs best practice: "No more than 110 pods per node": https://kubernetes.io/docs/setup/best-practices/cluster-large/
Also saying "you may increase the pod limi" implies that this is supported, and we have not tested this in our scaling tests, we do not want to give this impression.
| For hardware recommendations for large Kubernetes clusters, refer to the official Kubernetes documentation on [building large clusters](https://kubernetes.io/docs/setup/best-practices/cluster-large/). | ||
| Kubernetes is engineered around the concept of horizontal scaling for redundancy, so scaling vertically with large servers can be problematic if the proper minimum/maximum requirements aren’t followed. The following are tips and recommendations for large Kubernetes clusters: | ||
|
|
||
| - If you must use servers larger than 24 CPU, use virtualization tooling, such as what [Harvester](https://docs.harvesterhci.io/v1.0/rancher/virtualization-management/) provides, to subdivide the servers. |
There was a problem hiding this comment.
If you must use servers larger than 24 CPU
Where did that 24 CPU number come from ?
|
|
||
| - If you must use servers larger than 24 CPU, use virtualization tooling, such as what [Harvester](https://docs.harvesterhci.io/v1.0/rancher/virtualization-management/) provides, to subdivide the servers. | ||
|
|
||
| - Kubernetes, kernel, and network limitations prevent having too many pods per server. You should maintain a maximum of roughly 24 cores per server and a maximum of the recommended 110 pods per server. |
There was a problem hiding this comment.
roughly 24 cores per server
Where did this data come from ?
| @@ -297,7 +297,17 @@ Regarding CPU and memory, it is recommended that the different planes of Kuberne | |||
|
|
|||
There was a problem hiding this comment.
Is there a reason we put this in the rke docs ? Wouldn't this apply to any large cluster, including rke2.
We need to make a scaling section, but for now most of that data lives over here: https://rancher.com/docs/rancher/v2.5/en/installation/requirements/
For Rancher (product) docs only
When contributing to docs, please update the versioned docs. For example, the docs in the v2.6 folder of the
rancherfolder.Doc versions older than the latest minor version should only be updated to fix inaccuracies or make minor updates as necessary. The majority of new content should be added to the folder for the latest minor version.