mirror of
https://github.com/wassname/ray.git
synced 2026-07-19 11:27:32 +08:00
Ray Kubernetes Operator Part 1: readme, structure, config and CRD realted file (#6332)
* Ray-Operator first PR 1.RayCluster CRD and CR, structure code in golang 2.config file in Kubernetes * Delete go.sum * Ray-Operator first PR 1.add directory structure 2.add guide for submitting RayCluster * Delete ray_v1_raycluster.bk.yaml * Ray-Operator first PR 1.delete file bk 2.add more description about kubernetes and ray-operator features * Ray-Operator first PR: adjust grammar * Ray-Operator first PR: add More Information about proposal * Ray-Operator first PR: 1.add heterogeneous version of CR 2.add reference ot key words, and reference links to the props in yaml 3.file structure to yaml level and function description * Ray-Operator first PR: add ray operator proposal doc * Ray-Operator first PR: add More Information about proposal * Ray-Operator first PR: add command to start * Ray-Operator first PR: add More Information about proposal * Update deploy/ray-operator/README.md Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com> * Update deploy/ray-operator/api/v1/raycluster_types.go Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com> * Update deploy/ray-operator/api/v1/raycluster_types.go Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com> * Ray-Operator first PR: add More Information about proposal * Ray-Operator first PR: remove License * Ray-Operator first PR: rename version from v1 to v1alpha1 * Ray-Operator first PR: use replicas instead of numNodes * Ray-Operator first PR: update replicas in CR yaml file * Ray-Operator first PR: add More Information about proposal
This commit is contained in:
Qstar
@@ -0,0 +1,129 @@
|
||||
# Ray Kubernetes Operator
|
||||
|
||||
The Ray Operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of Ray clusters deployed to Kubernetes.
|
||||
|
||||
Some of the main features of Ray-Operator are:
|
||||
- user management via CRD
|
||||
- heterogeneous pods in one Ray cluster with specific affinity, toleration and other pre-defined settings
|
||||
- monitoring via Prometheus
|
||||
- HA for Ray Kubernetes Operator, there will be a lead election if lead crashes
|
||||
|
||||
## File structure:
|
||||
> ```
|
||||
> ray/deploy/ray-operator
|
||||
> ├── api/v1alpha1 // Package v1alpha1 contains API Schema definitions for the ray v1alpha1 API group
|
||||
> │ ├── groupversion_info.go // contains common metadata about the group-version
|
||||
> │ ├── raycluster_types.go // RayCluster field definitions, user should focus
|
||||
> │ └── zz_generated.deepcopy.go // contains the autogenerated implementation of the aforementioned runtime.Object interface, which marks all of our root types as representing Kinds.
|
||||
> │
|
||||
> └── config // contains Kustomize YAML definitions required to launch our controller on a cluster,hold our CustomResourceDefinitions, RBAC configuration, and WebhookConfigurations.
|
||||
> ├── certmanager
|
||||
> │ ├── certificate.yaml // The following manifests contain a self-signed issuer CR and a certificate CR.
|
||||
> │ ├── kustomization.yaml
|
||||
> │ └── kustomizeconfig.yaml
|
||||
> │
|
||||
> ├── crd
|
||||
> │ └── bases
|
||||
> │ │ └── ray.io_rayclusters.yaml // RayCluster CRD yaml file
|
||||
> │ └── patches
|
||||
> │ │ ├── cainjection_in_rayclusters.yaml // adds a directive for certmanager to inject CA into the CRD
|
||||
> │ │ └── webhook_in_rayclusters.yaml // enables conversion webhook for CRD
|
||||
> │ │── kustomization.yaml
|
||||
> │ └── kustomizeconfig.yaml
|
||||
> │
|
||||
> ├── default // contains a Kustomize base for launching the controller in a standard configuration.
|
||||
> │ ├── kustomization.yaml
|
||||
> │ ├── manager_auth_proxy_patch.yaml // inject a sidecar container which is a HTTP proxy for the controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
|
||||
> │ ├── manager_webhook_patch.yaml // webhook yaml file
|
||||
> │ └── webhookcainjection_patch.yaml // add annotation to admission webhook config
|
||||
> │
|
||||
> ├── manager // launch your controllers as pods in the cluster.
|
||||
> │ ├── kustomization.yaml
|
||||
> │ └── manager.yaml // manager yaml to create controller deployment, user should focus
|
||||
> │
|
||||
> ├── prometheus
|
||||
> │ ├── kustomization.yaml
|
||||
> │ └── monitor.yaml // Prometheus Monitor Service, user should focus
|
||||
> │
|
||||
> ├── rbac // permissions required to run your controllers under their own service account.
|
||||
> │ ├── auth_proxy_role.yaml
|
||||
> │ ├── auth_proxy_role_binding.yaml
|
||||
> │ ├── auth_proxy_service.yaml
|
||||
> │ ├── kustomization.yaml
|
||||
> │ ├── leader_election_role.yaml // permissions to do leader election.
|
||||
> │ ├── leader_election_role_binding.yaml
|
||||
> │ └── role_binding.yaml
|
||||
> │
|
||||
> ├── samples // sample RayCluster yaml, user should focus
|
||||
> │ ├── ray_v1_raycluster.complete.yaml
|
||||
> │ ├── ray_v1_raycluster.heterogeneous.yaml
|
||||
> │ └── ray_v1_raycluster.mini.yaml
|
||||
> │
|
||||
> └── webhook
|
||||
> ├── kustomization.yaml
|
||||
> ├── kustomizeconfig.yaml
|
||||
> ├── manifests.yaml
|
||||
> └── service.yaml // webhook-service
|
||||
> ```
|
||||
|
||||
## RayCluster sample CR
|
||||
|
||||
To introduce the Ray-Operator, give 3 samples of RayCluster CR.
|
||||
|
||||
Sample | desc
|
||||
------------- | -------------
|
||||
[RayCluster.mini.yaml](config/samples/ray_v1_raycluster.mini.yaml) | 2 pods in this sample, 1 for head and 1 for workers.The least information to start ray cluster, run in local test.
|
||||
[RayCluster.heterogeneous.yaml](config/samples/ray_v1_raycluster.heterogeneous.yaml) | 3 pods in this sample, 1 for head and 2 for workers but with different specifications. Different quota(like CPU/MEMORY) compares to mini version, run in local test.
|
||||
[RayCluster.complete.yaml](config/samples/ray_v1_raycluster.complete.yaml) | a complete version CR for customized requirement, show how to set Customized props. More props to set compares to heterogeneous version, run in production.
|
||||
|
||||
## RayCluster CRD
|
||||
|
||||
Refers to file [raycluster_types.go](api/v1alpha1/raycluster_types.go) for code details.
|
||||
|
||||
If interested in CRD, refer to file [CRD](config/crd/bases/ray.io_rayclusters.yaml) for more details.
|
||||
|
||||
## Software requirement
|
||||
Take care some software have dependency.
|
||||
|
||||
software | version | memo
|
||||
:------------- | :---------------:| -------------:
|
||||
kustomize | v3.1.0+ | [download](https://github.com/kubernetes-sigs/kustomize)
|
||||
kubectl | v1.11.3+ | [download](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
|
||||
Kubernetes Cluster | Access to a Kubernetes v1.11.3+ cluster| [Minikube](https://github.com/kubernetes/minikube) for local test
|
||||
go | v1.13+|[download](https://golang.org/dl/)
|
||||
docker | 17.03+|[download](https://docs.docker.com/install/)
|
||||
|
||||
Also you will need kubeconfig in ~/.kube/config, so you can access to Kubernetes Cluster.
|
||||
|
||||
## Get started
|
||||
Below gives a guide for user to submit RayCluster step by step:
|
||||
|
||||
### Install CRDs into a cluster
|
||||
|
||||
```shell script
|
||||
kustomize build config/crd | kubectl apply -f -
|
||||
```
|
||||
|
||||
### Deploy controller in the configured Kubernetes cluster in ~/.kube/config
|
||||
* For this version controller will run in system namespace, which maybe can't be tolerated in production.
|
||||
* We will add more detailed RBAC file to control the namespace used in production, and the controller will run in that namespace to control the permission.
|
||||
* Also we will provide the more detailed guide for user to run in a controlled way.
|
||||
```shell script
|
||||
cd config/manager
|
||||
kustomize build config/default | kubectl apply -f -
|
||||
```
|
||||
|
||||
### Submit RayCluster to Kubernetes
|
||||
```shell script
|
||||
kubectl create -f config/samples/ray_v1_raycluster.mini.yaml
|
||||
```
|
||||
|
||||
### Apply RayCluster to Kubernetes
|
||||
```shell script
|
||||
kubectl apply -f config/samples/ray_v1_raycluster.mini.yaml
|
||||
```
|
||||
|
||||
### Delete RayCluster to Kubernetes
|
||||
```shell script
|
||||
kubectl delete -f config/samples/ray_v1_raycluster.mini.yaml
|
||||
```
|
||||
@@ -0,0 +1,20 @@
|
||||
// Package v1alpha1 contains API Schema definitions for the ray v1alpha1 API group
|
||||
// +kubebuilder:object:generate=true
|
||||
// +groupName=ray.io
|
||||
package v1alpha1
|
||||
|
||||
import (
|
||||
"k8s.io/apimachinery/pkg/runtime/schema"
|
||||
"sigs.k8s.io/controller-runtime/pkg/scheme"
|
||||
)
|
||||
|
||||
var (
|
||||
// GroupVersion is group version used to register these objects
|
||||
GroupVersion = schema.GroupVersion{Group: "ray.io", Version: "v1alpha1"}
|
||||
|
||||
// SchemeBuilder is used to add go types to the GroupVersionKind scheme
|
||||
SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
|
||||
|
||||
// AddToScheme adds the types in this group-version to the given scheme.
|
||||
AddToScheme = SchemeBuilder.AddToScheme
|
||||
)
|
||||
@@ -0,0 +1,137 @@
|
||||
package v1alpha1
|
||||
|
||||
import (
|
||||
v1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
|
||||
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
|
||||
|
||||
// RayClusterSpec defines the desired state of RayCluster
|
||||
type RayClusterSpec struct {
|
||||
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
|
||||
// Important: Run "make" to regenerate code after modifying this file
|
||||
// ClusterName is unique identifier for RayCluster in one namespace.
|
||||
ClusterName string `json:"clusterName"`
|
||||
// Docker image.
|
||||
Images RayClusterImage `json:"images"`
|
||||
// Image pull policy.
|
||||
// One of Always, Never, IfNotPresent.
|
||||
// Defaults to Always if :latest tag is specified, or IfNotPresent otherwise.
|
||||
// Cannot be updated.
|
||||
ImagePullPolicy string `json:"imagePullPolicy,omitempty"`
|
||||
// Specification of the desired behavior of pod group
|
||||
Extensions []Extension `json:"extensions,omitempty"`
|
||||
}
|
||||
|
||||
// RayCluster pod type
|
||||
type ReplicaType string
|
||||
|
||||
const (
|
||||
// ReplicaTypeHead means that this pod will be ray cluster head
|
||||
ReplicaTypeHead ReplicaType = "head"
|
||||
// ReplicaTypeWorker means that this pod will be ray cluster worker
|
||||
ReplicaTypeWorker ReplicaType = "worker"
|
||||
)
|
||||
|
||||
// RayClusterStatus defines the observed state of RayCluster
|
||||
type RayClusterStatus struct {
|
||||
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
|
||||
// Important: Run "make" to regenerate code after modifying this file
|
||||
}
|
||||
|
||||
// +genclient
|
||||
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
||||
|
||||
// +kubebuilder:object:root=true
|
||||
|
||||
// RayCluster is the Schema for the RayClusters API
|
||||
type RayCluster struct {
|
||||
// Standard object metadata.
|
||||
metav1.TypeMeta `json:",inline"`
|
||||
metav1.ObjectMeta `json:"metadata,omitempty"`
|
||||
|
||||
// Specification of the desired behavior of the RayCluster.
|
||||
Spec RayClusterSpec `json:"spec,omitempty"`
|
||||
Status RayClusterStatus `json:"status,omitempty"`
|
||||
}
|
||||
|
||||
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
||||
|
||||
// RayClusterList contains a list of RayCluster
|
||||
type RayClusterList struct {
|
||||
metav1.TypeMeta `json:",inline"`
|
||||
metav1.ListMeta `json:"metadata,omitempty"`
|
||||
Items []RayCluster `json:"items"`
|
||||
}
|
||||
|
||||
func init() {
|
||||
SchemeBuilder.Register(&RayCluster{}, &RayClusterList{})
|
||||
}
|
||||
|
||||
type RayClusterImage struct {
|
||||
// Default docker image
|
||||
DefaultImage string `json:"defaultImage,omitempty"`
|
||||
}
|
||||
|
||||
type Extension struct {
|
||||
// Number of desired pods in this pod group. This is a pointer to distinguish between explicit
|
||||
// zero and not specified. Defaults to 1.
|
||||
Replicas *int32 `json:"replicas"`
|
||||
|
||||
// The pod type in this group worker/head.
|
||||
Type ReplicaType `json:"type,omitempty"`
|
||||
|
||||
// Pod docker image
|
||||
Image string `json:"image,omitempty"`
|
||||
|
||||
// Logical groupName for worker in same group, it's used for heterogeneous feature to distinguish different groups.
|
||||
// GroupName is the unique identifier for the pods with the same configuration in one Ray Cluster
|
||||
GroupName string `json:"groupName,omitempty"`
|
||||
|
||||
// Command to start ray
|
||||
Command string `json:"command,omitempty"`
|
||||
|
||||
// Labels for pod, raycluster.component and rayclusters.ray.io/component-name are default labels, do not overwrite them.
|
||||
// Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users,
|
||||
// but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects.
|
||||
Labels map[string]string `json:"labels,omitempty"`
|
||||
|
||||
// NodeSelector specifies a map of key-value pairs. For the pod to be eligible
|
||||
// to run on a node, the node must have each of the indicated key-value pairs as
|
||||
// labels. Optional.
|
||||
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
|
||||
|
||||
// Affinity is a group of affinity scheduling rules. Optional.
|
||||
// It allows you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node.
|
||||
Affinity *v1.Affinity `json:"affinity,omitempty"`
|
||||
|
||||
// ResourceRequirements describes the compute resource requirements.
|
||||
Resources v1.ResourceRequirements `json:"resources,omitempty"`
|
||||
|
||||
// The pod this Toleration is attached to tolerates any taint that matches
|
||||
// the triple <key,value,effect> using the matching operator <operator>. Optional.
|
||||
// Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.
|
||||
Tolerations []v1.Toleration `json:"tolerations,omitempty"`
|
||||
|
||||
// EnvVar represents an environment variable present in a Container. Optional.
|
||||
// Define environment variables for a container. This can be helpful in defining variables before setting up the container.
|
||||
ContainerEnv []v1.EnvVar `json:"containerEnv,omitempty"`
|
||||
|
||||
// Head service suffix. So head can be accessed by domain name: {namespace}.svc , follows Kubernetes standard.
|
||||
HeadServiceSuffix string `json:"headServiceSuffix,omitempty"`
|
||||
|
||||
// Annotations is an unstructured key value map stored with a resource that may be
|
||||
// set by external tools to store and retrieve arbitrary metadata. They are not
|
||||
// queryable and should be preserved when modifying objects. Optional.
|
||||
// Usage refers to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
|
||||
Annotations map[string]string `json:"annotations,omitempty"`
|
||||
|
||||
// Volume represents a named volume in a pod that may be accessed by any container in the pod. Optional.
|
||||
// At its core, a volume is just a directory, possibly with some data in it, which is accessible to the Containers in a Pod.
|
||||
Volumes []v1.Volume `json:"volumes,omitempty"`
|
||||
|
||||
// VolumeMount describes a mounting of a Volume within a container. Optional.
|
||||
VolumeMounts []v1.VolumeMount `json:"volumeMounts,omitempty"`
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
// +build !ignore_autogenerated
|
||||
|
||||
// Code generated by controller-gen. DO NOT EDIT.
|
||||
|
||||
package v1alpha1
|
||||
|
||||
import (
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
runtime "k8s.io/apimachinery/pkg/runtime"
|
||||
)
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *Extension) DeepCopyInto(out *Extension) {
|
||||
*out = *in
|
||||
if in.Labels != nil {
|
||||
in, out := &in.Labels, &out.Labels
|
||||
*out = make(map[string]string, len(*in))
|
||||
for key, val := range *in {
|
||||
(*out)[key] = val
|
||||
}
|
||||
}
|
||||
if in.NodeSelector != nil {
|
||||
in, out := &in.NodeSelector, &out.NodeSelector
|
||||
*out = make(map[string]string, len(*in))
|
||||
for key, val := range *in {
|
||||
(*out)[key] = val
|
||||
}
|
||||
}
|
||||
if in.Affinity != nil {
|
||||
in, out := &in.Affinity, &out.Affinity
|
||||
*out = new(corev1.Affinity)
|
||||
(*in).DeepCopyInto(*out)
|
||||
}
|
||||
in.Resources.DeepCopyInto(&out.Resources)
|
||||
if in.Tolerations != nil {
|
||||
in, out := &in.Tolerations, &out.Tolerations
|
||||
*out = make([]corev1.Toleration, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
if in.ContainerEnv != nil {
|
||||
in, out := &in.ContainerEnv, &out.ContainerEnv
|
||||
*out = make([]corev1.EnvVar, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
if in.Annotations != nil {
|
||||
in, out := &in.Annotations, &out.Annotations
|
||||
*out = make(map[string]string, len(*in))
|
||||
for key, val := range *in {
|
||||
(*out)[key] = val
|
||||
}
|
||||
}
|
||||
if in.Volumes != nil {
|
||||
in, out := &in.Volumes, &out.Volumes
|
||||
*out = make([]corev1.Volume, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
if in.VolumeMounts != nil {
|
||||
in, out := &in.VolumeMounts, &out.VolumeMounts
|
||||
*out = make([]corev1.VolumeMount, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Extension.
|
||||
func (in *Extension) DeepCopy() *Extension {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(Extension)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *RayCluster) DeepCopyInto(out *RayCluster) {
|
||||
*out = *in
|
||||
out.TypeMeta = in.TypeMeta
|
||||
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
|
||||
in.Spec.DeepCopyInto(&out.Spec)
|
||||
out.Status = in.Status
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayCluster.
|
||||
func (in *RayCluster) DeepCopy() *RayCluster {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(RayCluster)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
|
||||
func (in *RayCluster) DeepCopyObject() runtime.Object {
|
||||
if c := in.DeepCopy(); c != nil {
|
||||
return c
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *RayClusterImage) DeepCopyInto(out *RayClusterImage) {
|
||||
*out = *in
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterImage.
|
||||
func (in *RayClusterImage) DeepCopy() *RayClusterImage {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(RayClusterImage)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *RayClusterList) DeepCopyInto(out *RayClusterList) {
|
||||
*out = *in
|
||||
out.TypeMeta = in.TypeMeta
|
||||
in.ListMeta.DeepCopyInto(&out.ListMeta)
|
||||
if in.Items != nil {
|
||||
in, out := &in.Items, &out.Items
|
||||
*out = make([]RayCluster, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterList.
|
||||
func (in *RayClusterList) DeepCopy() *RayClusterList {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(RayClusterList)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
|
||||
func (in *RayClusterList) DeepCopyObject() runtime.Object {
|
||||
if c := in.DeepCopy(); c != nil {
|
||||
return c
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *RayClusterSpec) DeepCopyInto(out *RayClusterSpec) {
|
||||
*out = *in
|
||||
out.Images = in.Images
|
||||
if in.Extensions != nil {
|
||||
in, out := &in.Extensions, &out.Extensions
|
||||
*out = make([]Extension, len(*in))
|
||||
for i := range *in {
|
||||
(*in)[i].DeepCopyInto(&(*out)[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterSpec.
|
||||
func (in *RayClusterSpec) DeepCopy() *RayClusterSpec {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(RayClusterSpec)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||
func (in *RayClusterStatus) DeepCopyInto(out *RayClusterStatus) {
|
||||
*out = *in
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RayClusterStatus.
|
||||
func (in *RayClusterStatus) DeepCopy() *RayClusterStatus {
|
||||
if in == nil {
|
||||
return nil
|
||||
}
|
||||
out := new(RayClusterStatus)
|
||||
in.DeepCopyInto(out)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# The following manifests contain a self-signed issuer CR and a certificate CR.
|
||||
# More document can be found at https://docs.cert-manager.io
|
||||
# WARNING: Targets CertManager 0.11 check https://docs.cert-manager.io/en/latest/tasks/upgrading/index.html for breaking changes
|
||||
apiVersion: cert-manager.io/v1alpha2
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: selfsigned-issuer
|
||||
namespace: system
|
||||
spec:
|
||||
selfSigned: {}
|
||||
---
|
||||
apiVersion: cert-manager.io/v1alpha2
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: serving-cert # this name should match the one appeared in kustomizeconfig.yaml
|
||||
namespace: system
|
||||
spec:
|
||||
# $(SERVICE_NAME) and $(SERVICE_NAMESPACE) will be substituted by kustomize
|
||||
dnsNames:
|
||||
- $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc
|
||||
- $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc.cluster.local
|
||||
issuerRef:
|
||||
kind: Issuer
|
||||
name: selfsigned-issuer
|
||||
secretName: webhook-server-cert # this secret will not be prefixed, since it's not managed by kustomize
|
||||
@@ -0,0 +1,5 @@
|
||||
resources:
|
||||
- certificate.yaml
|
||||
|
||||
configurations:
|
||||
- kustomizeconfig.yaml
|
||||
@@ -0,0 +1,16 @@
|
||||
# This configuration is for teaching kustomize how to update name ref and var substitution
|
||||
nameReference:
|
||||
- kind: Issuer
|
||||
group: cert-manager.io
|
||||
fieldSpecs:
|
||||
- kind: Certificate
|
||||
group: cert-manager.io
|
||||
path: spec/issuerRef/name
|
||||
|
||||
varReference:
|
||||
- kind: Certificate
|
||||
group: cert-manager.io
|
||||
path: spec/commonName
|
||||
- kind: Certificate
|
||||
group: cert-manager.io
|
||||
path: spec/dnsNames
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,21 @@
|
||||
# This kustomization.yaml is not intended to be run by itself,
|
||||
# since it depends on service name and namespace that are out of this kustomize package.
|
||||
# It should be run by config/default
|
||||
resources:
|
||||
- bases/ray.io_rayclusters.yaml
|
||||
# +kubebuilder:scaffold:crdkustomizeresource
|
||||
|
||||
patchesStrategicMerge:
|
||||
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix.
|
||||
# patches here are for enabling the conversion webhook for each CRD
|
||||
#- patches/webhook_in_rayclusters.yaml
|
||||
# +kubebuilder:scaffold:crdkustomizewebhookpatch
|
||||
|
||||
# [CERTMANAGER] To enable webhook, uncomment all the sections with [CERTMANAGER] prefix.
|
||||
# patches here are for enabling the CA injection for each CRD
|
||||
#- patches/cainjection_in_rayclusters.yaml
|
||||
# +kubebuilder:scaffold:crdkustomizecainjectionpatch
|
||||
|
||||
# the following config is for teaching kustomize how to do kustomization for CRDs.
|
||||
configurations:
|
||||
- kustomizeconfig.yaml
|
||||
@@ -0,0 +1,17 @@
|
||||
# This file is for teaching kustomize how to substitute name and namespace reference in CRD
|
||||
nameReference:
|
||||
- kind: Service
|
||||
version: v1
|
||||
fieldSpecs:
|
||||
- kind: CustomResourceDefinition
|
||||
group: apiextensions.k8s.io
|
||||
path: spec/conversion/webhookClientConfig/service/name
|
||||
|
||||
namespace:
|
||||
- kind: CustomResourceDefinition
|
||||
group: apiextensions.k8s.io
|
||||
path: spec/conversion/webhookClientConfig/service/namespace
|
||||
create: false
|
||||
|
||||
varReference:
|
||||
- path: metadata/annotations
|
||||
@@ -0,0 +1,8 @@
|
||||
# The following patch adds a directive for certmanager to inject CA into the CRD
|
||||
# CRD conversion requires k8s 1.13 or later.
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
annotations:
|
||||
cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
|
||||
name: rayclusters.ray.io
|
||||
@@ -0,0 +1,17 @@
|
||||
# The following patch enables conversion webhook for CRD
|
||||
# CRD conversion requires k8s 1.13 or later.
|
||||
apiVersion: apiextensions.k8s.io/v1beta1
|
||||
kind: CustomResourceDefinition
|
||||
metadata:
|
||||
name: rayclusters.ray.io
|
||||
spec:
|
||||
conversion:
|
||||
strategy: Webhook
|
||||
webhookClientConfig:
|
||||
# this is "\n" used as a placeholder, otherwise it will be rejected by the apiserver for being blank,
|
||||
# but we're going to set it later using the cert-manager (or potentially a patch if not using cert-manager)
|
||||
caBundle: Cg==
|
||||
service:
|
||||
namespace: system
|
||||
name: webhook-service
|
||||
path: /convert
|
||||
@@ -0,0 +1,74 @@
|
||||
# Adds namespace to all resources.
|
||||
namespace: ray-operator-system
|
||||
|
||||
# Value of this field is prepended to the
|
||||
# names of all resources, e.g. a deployment named
|
||||
# "wordpress" becomes "alices-wordpress".
|
||||
# Note that it should also match with the prefix (text before '-') of the namespace
|
||||
# field above.
|
||||
namePrefix: ray-operator-
|
||||
|
||||
# Labels to add to all resources and selectors.
|
||||
#commonLabels:
|
||||
# someName: someValue
|
||||
|
||||
bases:
|
||||
- ../crd
|
||||
- ../rbac
|
||||
- ../manager
|
||||
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
|
||||
#- ../webhook
|
||||
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
|
||||
#- ../certmanager
|
||||
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
|
||||
#- ../prometheus
|
||||
|
||||
patchesStrategicMerge:
|
||||
# Protect the /metrics endpoint by putting it behind auth.
|
||||
# Only one of manager_auth_proxy_patch.yaml and
|
||||
# manager_prometheus_metrics_patch.yaml should be enabled.
|
||||
- manager_auth_proxy_patch.yaml
|
||||
# If you want your controller-manager to expose the /metrics
|
||||
# endpoint w/o any authn/z, uncomment the following line and
|
||||
# comment manager_auth_proxy_patch.yaml.
|
||||
# Only one of manager_auth_proxy_patch.yaml and
|
||||
# manager_prometheus_metrics_patch.yaml should be enabled.
|
||||
#- manager_prometheus_metrics_patch.yaml
|
||||
|
||||
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
|
||||
#- manager_webhook_patch.yaml
|
||||
|
||||
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
|
||||
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
|
||||
# 'CERTMANAGER' needs to be enabled to use ca injection
|
||||
#- webhookcainjection_patch.yaml
|
||||
|
||||
# the following config is for teaching kustomize how to do var substitution
|
||||
vars:
|
||||
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
|
||||
#- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
|
||||
# objref:
|
||||
# kind: Certificate
|
||||
# group: cert-manager.io
|
||||
# version: v1alpha2
|
||||
# name: serving-cert # this name should match the one in certificate.yaml
|
||||
# fieldref:
|
||||
# fieldpath: metadata.namespace
|
||||
#- name: CERTIFICATE_NAME
|
||||
# objref:
|
||||
# kind: Certificate
|
||||
# group: cert-manager.io
|
||||
# version: v1alpha2
|
||||
# name: serving-cert # this name should match the one in certificate.yaml
|
||||
#- name: SERVICE_NAMESPACE # namespace of the service
|
||||
# objref:
|
||||
# kind: Service
|
||||
# version: v1
|
||||
# name: webhook-service
|
||||
# fieldref:
|
||||
# fieldpath: metadata.namespace
|
||||
#- name: SERVICE_NAME
|
||||
# objref:
|
||||
# kind: Service
|
||||
# version: v1
|
||||
# name: webhook-service
|
||||
@@ -0,0 +1,25 @@
|
||||
# This patch inject a sidecar container which is a HTTP proxy for the controller manager,
|
||||
# it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: controller-manager
|
||||
namespace: system
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: kube-rbac-proxy
|
||||
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.1
|
||||
args:
|
||||
- "--secure-listen-address=0.0.0.0:8443"
|
||||
- "--upstream=http://127.0.0.1:8080/"
|
||||
- "--logtostderr=true"
|
||||
- "--v=10"
|
||||
ports:
|
||||
- containerPort: 8443
|
||||
name: https
|
||||
- name: manager
|
||||
args:
|
||||
- "--metrics-addr=127.0.0.1:8080"
|
||||
- "--enable-leader-election"
|
||||
@@ -0,0 +1,23 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: controller-manager
|
||||
namespace: system
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: manager
|
||||
ports:
|
||||
- containerPort: 9443
|
||||
name: webhook-server
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- mountPath: /tmp/k8s-webhook-server/serving-certs
|
||||
name: cert
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: cert
|
||||
secret:
|
||||
defaultMode: 420
|
||||
secretName: webhook-server-cert
|
||||
@@ -0,0 +1,15 @@
|
||||
# This patch add annotation to admission webhook config and
|
||||
# the variables $(CERTIFICATE_NAMESPACE) and $(CERTIFICATE_NAME) will be substituted by kustomize.
|
||||
apiVersion: admissionregistration.k8s.io/v1beta1
|
||||
kind: MutatingWebhookConfiguration
|
||||
metadata:
|
||||
name: mutating-webhook-configuration
|
||||
annotations:
|
||||
cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
|
||||
---
|
||||
apiVersion: admissionregistration.k8s.io/v1beta1
|
||||
kind: ValidatingWebhookConfiguration
|
||||
metadata:
|
||||
name: validating-webhook-configuration
|
||||
annotations:
|
||||
cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
|
||||
@@ -0,0 +1,2 @@
|
||||
resources:
|
||||
- manager.yaml
|
||||
@@ -0,0 +1,39 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
labels:
|
||||
control-plane: controller-manager
|
||||
name: system
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: controller-manager
|
||||
namespace: system
|
||||
labels:
|
||||
control-plane: controller-manager
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
control-plane: controller-manager
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
control-plane: controller-manager
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- /manager
|
||||
args:
|
||||
- --enable-leader-election
|
||||
image: controller:latest
|
||||
name: manager
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 30Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 20Mi
|
||||
terminationGracePeriodSeconds: 10
|
||||
@@ -0,0 +1,2 @@
|
||||
resources:
|
||||
- monitor.yaml
|
||||
@@ -0,0 +1,15 @@
|
||||
|
||||
# Prometheus Monitor Service (Metrics)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
labels:
|
||||
control-plane: controller-manager
|
||||
name: controller-manager-metrics-monitor
|
||||
namespace: system
|
||||
spec:
|
||||
endpoints:
|
||||
- path: /metrics
|
||||
port: https
|
||||
selector:
|
||||
control-plane: controller-manager
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: proxy-role
|
||||
rules:
|
||||
- apiGroups: ["authentication.k8s.io"]
|
||||
resources:
|
||||
- tokenreviews
|
||||
verbs: ["create"]
|
||||
- apiGroups: ["authorization.k8s.io"]
|
||||
resources:
|
||||
- subjectaccessreviews
|
||||
verbs: ["create"]
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: proxy-rolebinding
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: proxy-role
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: default
|
||||
namespace: system
|
||||
@@ -0,0 +1,14 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
labels:
|
||||
control-plane: controller-manager
|
||||
name: controller-manager-metrics-service
|
||||
namespace: system
|
||||
spec:
|
||||
ports:
|
||||
- name: https
|
||||
port: 8443
|
||||
targetPort: https
|
||||
selector:
|
||||
control-plane: controller-manager
|
||||
@@ -0,0 +1,11 @@
|
||||
resources:
|
||||
- role.yaml
|
||||
- role_binding.yaml
|
||||
- leader_election_role.yaml
|
||||
- leader_election_role_binding.yaml
|
||||
# Comment the following 3 lines if you want to disable
|
||||
# the auth proxy (https://github.com/brancz/kube-rbac-proxy)
|
||||
# which protects your /metrics endpoint.
|
||||
- auth_proxy_service.yaml
|
||||
- auth_proxy_role.yaml
|
||||
- auth_proxy_role_binding.yaml
|
||||
@@ -0,0 +1,32 @@
|
||||
# permissions to do leader election.
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: leader-election-role
|
||||
rules:
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- configmaps
|
||||
verbs:
|
||||
- get
|
||||
- list
|
||||
- watch
|
||||
- create
|
||||
- update
|
||||
- patch
|
||||
- delete
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- configmaps/status
|
||||
verbs:
|
||||
- get
|
||||
- update
|
||||
- patch
|
||||
- apiGroups:
|
||||
- ""
|
||||
resources:
|
||||
- events
|
||||
verbs:
|
||||
- create
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: leader-election-rolebinding
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: leader-election-role
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: default
|
||||
namespace: system
|
||||
@@ -0,0 +1,12 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: manager-rolebinding
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: manager-role
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: default
|
||||
namespace: system
|
||||
@@ -0,0 +1,260 @@
|
||||
apiVersion: ray.io/v1alpha1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
labels:
|
||||
controller-tools.k8s.io: "1.0"
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
name: raycluster-sample
|
||||
spec:
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
clusterName: raycluster-sample
|
||||
images:
|
||||
defaultImage: "docker-image"
|
||||
imagePullPolicy: "Always"
|
||||
|
||||
extensions:
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
# logical group name, for this called small-group, also can be functional
|
||||
groupName: small-group
|
||||
# pod type
|
||||
type: worker
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: small-group
|
||||
|
||||
# annotations for pod
|
||||
annotations:
|
||||
key: value
|
||||
|
||||
# use affinity to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: key1
|
||||
operator: In
|
||||
values: ["true"]
|
||||
- key: key2
|
||||
operator: In
|
||||
values: ["true"]
|
||||
|
||||
# use tolerations to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "true"
|
||||
effect: "NoSchedule"
|
||||
|
||||
# resource requirements.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
|
||||
# environment variables to set in the container.Optional.
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: APPNAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# use volumes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumes:
|
||||
- name: log-volume
|
||||
emptyDir: {}
|
||||
|
||||
# use volumeMounts.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumeMounts:
|
||||
- mountPath: /path/to/log
|
||||
name: log-volume
|
||||
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
# logical group name
|
||||
groupName: medium-group
|
||||
# pod type
|
||||
type: worker
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: medium-group
|
||||
|
||||
# annotations for pod
|
||||
annotations:
|
||||
key: value
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# use affinity to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: key1
|
||||
operator: In
|
||||
values: ["true"]
|
||||
- key: key2
|
||||
operator: In
|
||||
values: ["true"]
|
||||
|
||||
# use tolerations to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "true"
|
||||
effect: "NoSchedule"
|
||||
|
||||
# resource requirements.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
requests:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
|
||||
# environment variables to set in the container.Optional.
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: APPNAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# use volumes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumes:
|
||||
- name: log-volume
|
||||
emptyDir: {}
|
||||
|
||||
# use volumeMounts.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumeMounts:
|
||||
- mountPath: /path/to/log
|
||||
name: log-volume
|
||||
|
||||
# the pod replicas in this group typed head
|
||||
- replicas: 1
|
||||
# logical group name
|
||||
groupName: headgroup
|
||||
# pod type
|
||||
type: head
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: headgroup
|
||||
|
||||
# annotations for pod
|
||||
annotations:
|
||||
key: value
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# use affinity to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: key1
|
||||
operator: In
|
||||
values: ["true"]
|
||||
- key: key2
|
||||
operator: In
|
||||
values: ["true"]
|
||||
|
||||
# use tolerations to select nodes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
|
||||
tolerations:
|
||||
- key: "key1"
|
||||
operator: "Equal"
|
||||
value: "true"
|
||||
effect: "NoSchedule"
|
||||
|
||||
# resource requirements.
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 1Gi
|
||||
ephemeral-storage: 4Gi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 1Gi
|
||||
ephemeral-storage: 4Gi
|
||||
|
||||
# environment variables to set in the container.Optional.
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: APPNAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# use volumes.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumes:
|
||||
- name: log-volume
|
||||
emptyDir: {}
|
||||
|
||||
# use volumeMounts.Optional.
|
||||
# Refer to https://kubernetes.io/docs/concepts/storage/volumes/
|
||||
volumeMounts:
|
||||
- mountPath: /path/to/log
|
||||
name: log-volume
|
||||
@@ -0,0 +1,146 @@
|
||||
apiVersion: ray.io/v1alpha1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
labels:
|
||||
controller-tools.k8s.io: "1.0"
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
name: raycluster-sample
|
||||
spec:
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
clusterName: raycluster-sample
|
||||
images:
|
||||
defaultImage: "docker-image"
|
||||
imagePullPolicy: "Always"
|
||||
|
||||
extensions:
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
# logical group name, for this called small-group, also can be functional
|
||||
groupName: small-group
|
||||
# pod type
|
||||
type: worker
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: small-group
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# resource requirements
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
|
||||
# environment variables to set in the container.Optional
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: CLUSTER_NAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
# logical group name, for this called medium-group, also can be functional
|
||||
groupName: medium-group
|
||||
# pod type
|
||||
type: worker
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: medium-group
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# resource requirements
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
requests:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
|
||||
# environment variables to set in the container.Optional
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: CLUSTER_NAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# the pod replicas in this group typed head
|
||||
- replicas: 1
|
||||
# logical group name, also can be functional
|
||||
groupName: head-group
|
||||
# pod type
|
||||
type: head
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: head-group
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# resource requirements
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
requests:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
|
||||
# environment variables to set in the container.Optional
|
||||
# Refer to https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
|
||||
containerEnv:
|
||||
- name: CLUSTER_NAME
|
||||
value: raycluster-sample
|
||||
- name: MY_POD_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: metadata.name
|
||||
- name: MY_POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
@@ -0,0 +1,74 @@
|
||||
apiVersion: ray.io/v1alpha1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
labels:
|
||||
controller-tools.k8s.io: "1.0"
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
name: raycluster-sample
|
||||
spec:
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
clusterName: raycluster-sample
|
||||
images:
|
||||
defaultImage: "docker-image"
|
||||
imagePullPolicy: "Always"
|
||||
|
||||
extensions:
|
||||
# the pod replicas in this group typed worker
|
||||
- replicas: 1
|
||||
# logical group name, for this called small-group, also can be functional like hello-group
|
||||
groupName: small-group
|
||||
# pod type
|
||||
type: worker
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: small-group
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# resource requirements
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
requests:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
ephemeral-storage: 2Gi
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
|
||||
# the pod replicas in this group typed head
|
||||
- replicas: 1
|
||||
# logical group name, also can be functional
|
||||
groupName: head-group
|
||||
# pod type
|
||||
type: head
|
||||
|
||||
# custom labels. NOTE: do not define custom labels start with `raycluster.`, they may be used in controller.
|
||||
# Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
|
||||
labels:
|
||||
raycluster.group.name: head-group
|
||||
|
||||
# Command to start ray
|
||||
command: ray stop; ulimit -n 65536; ray start --object-manager-port=8076
|
||||
|
||||
# resource requirements
|
||||
# Refer to https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
requests:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
ephemeral-storage: 4Gi
|
||||
|
||||
# head service suffix: {namespace}.svc , follows Kubernetes standard
|
||||
headServiceSuffix: "ray-operator.svc"
|
||||
@@ -0,0 +1,6 @@
|
||||
resources:
|
||||
- manifests.yaml
|
||||
- service.yaml
|
||||
|
||||
configurations:
|
||||
- kustomizeconfig.yaml
|
||||
@@ -0,0 +1,25 @@
|
||||
# the following config is for teaching kustomize where to look at when substituting vars.
|
||||
# It requires kustomize v2.1.0 or newer to work properly.
|
||||
nameReference:
|
||||
- kind: Service
|
||||
version: v1
|
||||
fieldSpecs:
|
||||
- kind: MutatingWebhookConfiguration
|
||||
group: admissionregistration.k8s.io
|
||||
path: webhooks/clientConfig/service/name
|
||||
- kind: ValidatingWebhookConfiguration
|
||||
group: admissionregistration.k8s.io
|
||||
path: webhooks/clientConfig/service/name
|
||||
|
||||
namespace:
|
||||
- kind: MutatingWebhookConfiguration
|
||||
group: admissionregistration.k8s.io
|
||||
path: webhooks/clientConfig/service/namespace
|
||||
create: true
|
||||
- kind: ValidatingWebhookConfiguration
|
||||
group: admissionregistration.k8s.io
|
||||
path: webhooks/clientConfig/service/namespace
|
||||
create: true
|
||||
|
||||
varReference:
|
||||
- path: metadata/annotations
|
||||
@@ -0,0 +1,12 @@
|
||||
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: webhook-service
|
||||
namespace: system
|
||||
spec:
|
||||
ports:
|
||||
- port: 443
|
||||
targetPort: 9443
|
||||
selector:
|
||||
control-plane: controller-manager
|
||||
Reference in New Issue
Block a user