| ================== |
| Kubernetes Formula |
| ================== |
| |
| Kubernetes is an open-source system for automating deployment, scaling, and |
| management of containerized applications. This formula deploys production |
| ready Kubernetes and generate Kubernetes manifests as well. |
| |
| You can download `kubectl` configuration and connect to your cluster. However, |
| keep in mind `kubernetes_control_address` needs to be accessible from your computer: |
| |
| .. code-block:: yaml |
| |
| mkdir -p ~/.kube |
| [ -f ~/.kube/config ] && cp -v ~/.kube/config ~/.kube/config-backup |
| ssh cfg01 "sudo ssh ctl01 /etc/kubernetes/kubeconfig.sh" > ~/.kube/config |
| kubectl get no |
| |
| |
| `cfg01` is Salt master node and `ctl01` is one of Kubernetes masters |
| |
| Sample Pillars |
| ============== |
| |
| **REQUIRED:** Define image to use for hyperkube, CNIs and calicoctl image |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| hyperkube: |
| image: gcr.io/google_containers/hyperkube:v1.6.5 |
| pool: |
| network: |
| calicoctl: |
| image: calico/ctl |
| cni: |
| image: calico/cni |
| |
| Enable helm-tiller addon |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| addons: |
| helm: |
| enabled: true |
| |
| Enable calico-policy addon |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| addons: |
| calico_policy: |
| enabled: true |
| |
| Enable virtlet addon |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| addons: |
| virtlet: |
| enabled: true |
| namespace: kube-system |
| image: mirantis/virtlet:v0.7.0 |
| hosts: |
| - cmp01 |
| - cmp02 |
| |
| Enable netchecker addon |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| addons: |
| netchecker: |
| enabled: true |
| master: |
| namespace: |
| netchecker: |
| enabled: true |
| |
| Enable Kubenetes Federation control plane |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| master: |
| federation: |
| enabled: True |
| name: federation |
| namespace: federation-system |
| source: https://dl.k8s.io/v1.6.6/kubernetes-client-linux-amd64.tar.gz |
| hash: 94b2c9cd29981a8e150c187193bab0d8c0b6e906260f837367feff99860a6376 |
| service_type: NodePort |
| dns_provider: coredns |
| childclusters: |
| - secondcluster.mydomain |
| - thirdcluster.mydomain |
| |
| Configure service verbosity |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| master: |
| verbosity: 2 |
| pool: |
| verbosity: 2 |
| |
| Set cluster name and domain |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| common: |
| kubernetes_cluster_domain: mycluster.domain |
| cluster_name : mycluster |
| |
| Enable autoscaler for dns addon. Poll period can be skipped. |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| common: |
| addons: |
| dns: |
| domain: cluster.local |
| enabled: true |
| replicas: 1 |
| server: 10.254.0.10 |
| autoscaler: |
| enabled: true |
| poll-period-seconds: 60 |
| |
| |
| Pass aditional parameters to daemons: |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| master: |
| apiserver: |
| daemon_opts: |
| storage-backend: pigeon |
| controller_manager: |
| daemon_opts: |
| log-dir: /dev/nulL |
| pool: |
| kubelet: |
| daemon_opts: |
| max-pods: "6" |
| |
| |
| Containers on pool definitions in pool.service.local |
| |
| .. code-block:: yaml |
| |
| parameters: |
| kubernetes: |
| pool: |
| service: |
| local: |
| enabled: False |
| service: libvirt |
| cluster: openstack-compute |
| namespace: default |
| role: ${linux:system:name} |
| type: LoadBalancer |
| kind: Deployment |
| apiVersion: extensions/v1beta1 |
| replicas: 1 |
| host_pid: True |
| nodeSelector: |
| - key: openstack |
| value: ${linux:system:name} |
| hostNetwork: True |
| container: |
| libvirt-compute: |
| privileged: True |
| image: ${_param:docker_repository}/libvirt-compute |
| tag: ${_param:openstack_container_tag} |
| |
| Master definition |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| common: |
| cluster_name: cluster |
| addons: |
| dns: |
| domain: cluster.local |
| enabled: true |
| replicas: 1 |
| server: 10.254.0.10 |
| master: |
| admin: |
| password: password |
| username: admin |
| apiserver: |
| address: 10.0.175.100 |
| secure_port: 443 |
| insecure_address: 127.0.0.1 |
| insecure_port: 8080 |
| ca: kubernetes |
| enabled: true |
| etcd: |
| host: 127.0.0.1 |
| members: |
| - host: 10.0.175.100 |
| name: node040 |
| name: node040 |
| token: ca939ec9c2a17b0786f6d411fe019e9b |
| kubelet: |
| allow_privileged: true |
| network: |
| engine: calico |
| mtu: 1500 |
| hash: fb5e30ebe6154911a66ec3fb5f1195b2 |
| private_ip_range: 10.150.0.0/16 |
| version: v0.19.0 |
| service_addresses: 10.254.0.0/16 |
| storage: |
| engine: glusterfs |
| members: |
| - host: 10.0.175.101 |
| port: 24007 |
| - host: 10.0.175.102 |
| port: 24007 |
| - host: 10.0.175.103 |
| port: 24007 |
| port: 24007 |
| token: |
| admin: DFvQ8GJ9JD4fKNfuyEddw3rjnFTkUKsv |
| controller_manager: EreGh6AnWf8DxH8cYavB2zS029PUi7vx |
| dns: RAFeVSE4UvsCz4gk3KYReuOI5jsZ1Xt3 |
| kube_proxy: DFvQ8GelB7afH3wClC9romaMPhquyyEe |
| kubelet: 7bN5hJ9JD4fKjnFTkUKsvVNfuyEddw3r |
| logging: MJkXKdbgqRmTHSa2ykTaOaMykgO6KcEf |
| monitoring: hnsj0XqABgrSww7Nqo7UVTSZLJUt2XRd |
| scheduler: HY1UUxEPpmjW4a1dDLGIANYQp1nZkLDk |
| version: v1.2.4 |
| |
| |
| kubernetes: |
| pool: |
| address: 0.0.0.0 |
| allow_privileged: true |
| ca: kubernetes |
| cluster_dns: 10.254.0.10 |
| cluster_domain: cluster.local |
| enabled: true |
| kubelet: |
| allow_privileged: true |
| config: /etc/kubernetes/manifests |
| frequency: 5s |
| master: |
| apiserver: |
| members: |
| - host: 10.0.175.100 |
| etcd: |
| members: |
| - host: 10.0.175.100 |
| host: 10.0.175.100 |
| network: |
| engine: calico |
| mtu: 1500 |
| hash: fb5e30ebe6154911a66ec3fb5f1195b2 |
| version: v0.19.0 |
| token: |
| kube_proxy: DFvQ8GelB7afH3wClC9romaMPhquyyEe |
| kubelet: 7bN5hJ9JD4fKjnFTkUKsvVNfuyEddw3r |
| version: v1.2.4 |
| |
| |
| Kubernetes with OpenContrail network plugin |
| ------------------------------------------------ |
| |
| On Master: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| common: |
| addons: |
| contrail_network_controller: |
| enabled: true |
| namespace: kube-system |
| image: yashulyak/contrail-controller:latest |
| master: |
| network: |
| engine: opencontrail |
| default_domain: default-domain |
| default_project: default-domain:default-project |
| public_network: default-domain:default-project:Public |
| public_ip_range: 185.22.97.128/26 |
| private_ip_range: 10.150.0.0/16 |
| service_cluster_ip_range: 10.254.0.0/16 |
| network_label: name |
| service_label: uses |
| cluster_service: kube-system/default |
| config: |
| api: |
| host: 10.0.170.70 |
| On pools: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| engine: opencontrail |
| |
| |
| Dashboard public IP must be configured when Contrail network is used: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| common: |
| addons: |
| public_ip: 1.1.1.1 |
| |
| Kubernetes control plane running in systemd |
| ------------------------------------------- |
| |
| By default kube-apiserver, kube-scheduler, kube-controllermanager, kube-proxy, etcd running in docker containers through manifests. For stable production environment this should be run in systemd. |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| container: false |
| |
| kubernetes: |
| pool: |
| container: false |
| |
| Because k8s services run under kube user without root privileges, there is need to change secure port for apiserver. |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| apiserver: |
| secure_port: 8081 |
| |
| Kubernetes with Flannel |
| ----------------------- |
| |
| On Master: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| network: |
| engine: flannel |
| # If you don't register master as node: |
| etcd: |
| members: |
| - host: 10.0.175.101 |
| port: 4001 |
| - host: 10.0.175.102 |
| port: 4001 |
| - host: 10.0.175.103 |
| port: 4001 |
| common: |
| network: |
| engine: flannel |
| |
| On pools: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| engine: flannel |
| etcd: |
| members: |
| - host: 10.0.175.101 |
| port: 4001 |
| - host: 10.0.175.102 |
| port: 4001 |
| - host: 10.0.175.103 |
| port: 4001 |
| common: |
| network: |
| engine: flannel |
| |
| Kubernetes with Calico |
| ----------------------- |
| |
| On Master: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| network: |
| engine: calico |
| mtu: 1500 |
| # If you don't register master as node: |
| etcd: |
| members: |
| - host: 10.0.175.101 |
| port: 4001 |
| - host: 10.0.175.102 |
| port: 4001 |
| - host: 10.0.175.103 |
| port: 4001 |
| |
| On pools: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| engine: calico |
| mtu: 1500 |
| etcd: |
| members: |
| - host: 10.0.175.101 |
| port: 4001 |
| - host: 10.0.175.102 |
| port: 4001 |
| - host: 10.0.175.103 |
| port: 4001 |
| |
| Running with secured etcd: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| engine: calico |
| mtu: 1500 |
| etcd: |
| ssl: |
| enabled: true |
| master: |
| network: |
| engine: calico |
| etcd: |
| ssl: |
| enabled: true |
| |
| Running with calico-policy controller: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| engine: calico |
| mtu: 1500 |
| addons: |
| calico_policy: |
| enabled: true |
| |
| master: |
| network: |
| engine: calico |
| mtu: 1500 |
| addons: |
| calico_policy: |
| enabled: true |
| |
| |
| |
| Enable Prometheus metrics in Felix |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| pool: |
| network: |
| prometheus: |
| enabled: true |
| master: |
| network: |
| prometheus: |
| enabled: true |
| |
| Post deployment configuration |
| |
| .. code-block:: bash |
| |
| # set ETCD |
| export ETCD_AUTHORITY=10.0.111.201:4001 |
| |
| # Set NAT for pods subnet |
| calicoctl pool add 192.168.0.0/16 --nat-outgoing |
| |
| # Status commands |
| calicoctl status |
| calicoctl node show |
| |
| Kubernetes with GlusterFS for storage |
| --------------------------------------------- |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| ... |
| storage: |
| engine: glusterfs |
| port: 24007 |
| members: |
| - host: 10.0.175.101 |
| port: 24007 |
| - host: 10.0.175.102 |
| port: 24007 |
| - host: 10.0.175.103 |
| port: 24007 |
| ... |
| |
| Kubernetes namespaces |
| --------------------- |
| |
| Create namespace: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| ... |
| namespace: |
| kube-system: |
| enabled: True |
| namespace2: |
| enabled: True |
| namespace3: |
| enabled: False |
| ... |
| |
| Kubernetes labels |
| ----------------- |
| |
| Label node: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| label: |
| label01: |
| value: value01 |
| node: node01 |
| enabled: true |
| key: key01 |
| ... |
| |
| Pull images from private registries |
| ----------------------------------- |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| master: |
| ... |
| registry: |
| secret: |
| registry01: |
| enabled: True |
| key: (get from `cat /root/.docker/config.json | base64`) |
| namespace: default |
| ... |
| control: |
| ... |
| service: |
| service01: |
| ... |
| image_pull_secretes: registry01 |
| ... |
| |
| Kubernetes Service Definitions in pillars |
| ========================================== |
| |
| Following samples show how to generate kubernetes manifest as well and provide single tool for complete infrastructure management. |
| |
| Deployment manifest |
| --------------------- |
| |
| .. code-block:: yaml |
| |
| salt: |
| control: |
| enabled: True |
| hostNetwork: True |
| service: |
| memcached: |
| privileged: True |
| service: memcached |
| role: server |
| type: LoadBalancer |
| replicas: 3 |
| kind: Deployment |
| apiVersion: extensions/v1beta1 |
| ports: |
| - port: 8774 |
| name: nova-api |
| - port: 8775 |
| name: nova-metadata |
| volume: |
| volume_name: |
| type: hostPath |
| mount: /certs |
| path: /etc/certs |
| container: |
| memcached: |
| image: memcached |
| tag:2 |
| ports: |
| - port: 8774 |
| name: nova-api |
| - port: 8775 |
| name: nova-metadata |
| variables: |
| - name: HTTP_TLS_CERTIFICATE: |
| value: /certs/domain.crt |
| - name: HTTP_TLS_KEY |
| value: /certs/domain.key |
| volumes: |
| - name: /etc/certs |
| type: hostPath |
| mount: /certs |
| path: /etc/certs |
| |
| PetSet manifest |
| --------------------- |
| |
| .. code-block:: yaml |
| |
| service: |
| memcached: |
| apiVersion: apps/v1alpha1 |
| kind: PetSet |
| service_name: 'memcached' |
| container: |
| memcached: |
| ... |
| |
| |
| Configmap |
| --------- |
| |
| You are able to create configmaps using support layer between formulas. |
| It works simple, eg. in nova formula there's file ``meta/config.yml`` which |
| defines config files used by that service and roles. |
| |
| Kubernetes formula is able to generate these files using custom pillar and |
| grains structure. This way you are able to run docker images built by any way |
| while still re-using your configuration management. |
| |
| Example pillar: |
| |
| .. code-block:: bash |
| |
| kubernetes: |
| control: |
| config_type: default|kubernetes # Output is yaml k8s or default single files |
| configmap: |
| nova-control: |
| grains: |
| # Alternate grains as OS running in container may differ from |
| # salt minion OS. Needed only if grains matters for config |
| # generation. |
| os_family: Debian |
| pillar: |
| # Generic pillar for nova controller |
| nova: |
| controller: |
| enabled: true |
| versionn: liberty |
| ... |
| |
| To tell which services supports config generation, you need to ensure pillar |
| structure like this to determine support: |
| |
| .. code-block:: yaml |
| |
| nova: |
| _support: |
| config: |
| enabled: true |
| |
| initContainers |
| -------------- |
| |
| Example pillar: |
| |
| .. code-block:: bash |
| |
| kubernetes: |
| control: |
| service: |
| memcached: |
| init_containers: |
| - name: test-mysql |
| image: busybox |
| command: |
| - sleep |
| - 3600 |
| volumes: |
| - name: config |
| mount: /test |
| - name: test-memcached |
| image: busybox |
| command: |
| - sleep |
| - 3600 |
| volumes: |
| - name: config |
| mount: /test |
| |
| Affinity |
| -------- |
| |
| podAffinity |
| =========== |
| |
| Example pillar: |
| |
| .. code-block:: bash |
| |
| kubernetes: |
| control: |
| service: |
| memcached: |
| affinity: |
| pod_affinity: |
| name: podAffinity |
| expression: |
| label_selector: |
| name: labelSelector |
| selectors: |
| - key: app |
| value: memcached |
| topology_key: kubernetes.io/hostname |
| |
| podAntiAffinity |
| =============== |
| |
| Example pillar: |
| |
| .. code-block:: bash |
| |
| kubernetes: |
| control: |
| service: |
| memcached: |
| affinity: |
| anti_affinity: |
| name: podAntiAffinity |
| expression: |
| label_selector: |
| name: labelSelector |
| selectors: |
| - key: app |
| value: opencontrail-control |
| topology_key: kubernetes.io/hostname |
| |
| nodeAffinity |
| =============== |
| |
| Example pillar: |
| |
| .. code-block:: bash |
| |
| kubernetes: |
| control: |
| service: |
| memcached: |
| affinity: |
| node_affinity: |
| name: nodeAffinity |
| expression: |
| match_expressions: |
| name: matchExpressions |
| selectors: |
| - key: key |
| operator: In |
| values: |
| - value1 |
| - value2 |
| |
| Volumes |
| ------- |
| |
| hostPath |
| ========== |
| |
| .. code-block:: yaml |
| |
| service: |
| memcached: |
| container: |
| memcached: |
| volumes: |
| - name: volume1 |
| mountPath: /volume |
| readOnly: True |
| ... |
| volume: |
| volume1: |
| name: /etc/certs |
| type: hostPath |
| path: /etc/certs |
| |
| emptyDir |
| ======== |
| |
| .. code-block:: yaml |
| |
| service: |
| memcached: |
| container: |
| memcached: |
| volumes: |
| - name: volume1 |
| mountPath: /volume |
| readOnly: True |
| ... |
| volume: |
| volume1: |
| name: /etc/certs |
| type: emptyDir |
| |
| configMap |
| ========= |
| |
| .. code-block:: yaml |
| |
| service: |
| memcached: |
| container: |
| memcached: |
| volumes: |
| - name: volume1 |
| mountPath: /volume |
| readOnly: True |
| ... |
| volume: |
| volume1: |
| type: config_map |
| item: |
| configMap1: |
| key: config.conf |
| path: config.conf |
| configMap2: |
| key: policy.json |
| path: policy.json |
| |
| To mount single configuration file instead of whole directory: |
| |
| .. code-block:: yaml |
| |
| service: |
| memcached: |
| container: |
| memcached: |
| volumes: |
| - name: volume1 |
| mountPath: /volume/config.conf |
| sub_path: config.conf |
| |
| Generating Jobs |
| =============== |
| |
| Example pillar: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| control: |
| job: |
| sleep: |
| job: sleep |
| restart_policy: Never |
| container: |
| sleep: |
| image: busybox |
| tag: latest |
| command: |
| - sleep |
| - "3600" |
| |
| Volumes and Variables can be used as the same way as during Deployment generation. |
| |
| Custom params: |
| |
| .. code-block:: yaml |
| |
| kubernetes: |
| control: |
| job: |
| host_network: True |
| host_pid: True |
| container: |
| sleep: |
| privileged: True |
| node_selector: |
| key: node |
| value: one |
| image_pull_secretes: password |
| |
| |
| More Information |
| ================ |
| |
| * https://github.com/Juniper/kubernetes/blob |
| /opencontrail-integration/docs /getting-started-guides/opencontrail.md |
| * https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase |
| |
| |
| Documentation and Bugs |
| ====================== |
| |
| To learn how to install and update salt-formulas, consult the documentation |
| available online at: |
| |
| http://salt-formulas.readthedocs.io/ |
| |
| In the unfortunate event that bugs are discovered, they should be reported to |
| the appropriate issue tracker. Use Github issue tracker for specific salt |
| formula: |
| |
| https://github.com/salt-formulas/salt-formula-kubernetes/issues |
| |
| For feature requests, bug reports or blueprints affecting entire ecosystem, |
| use Launchpad salt-formulas project: |
| |
| https://launchpad.net/salt-formulas |
| |
| You can also join salt-formulas-users team and subscribe to mailing list: |
| |
| https://launchpad.net/~salt-formulas-users |
| |
| Developers wishing to work on the salt-formulas projects should always base |
| their work on master branch and submit pull request against specific formula. |
| |
| https://github.com/salt-formulas/salt-formula-kubernetes |
| |
| Any questions or feedback is always welcome so feel free to join our IRC |
| channel: |
| |
| #salt-formulas @ irc.freenode.net |