Install Manually

SuperEdge officially supports Kubernetes 1.16 and 1.18. This document walk you through how to bootstrap SuperEdge on your Kubernetes cluster.

1. Install Tunnel

1.1 Deploy Tunnel’s CoreDNS

On master nodes,

$ kubectl apply -f deployment/tunnel-coredns.yaml

1.2 Configure tunnel-cloud

Generate and set the following parameters in the deployment/tunnel-cloud.yaml

  TunnelCloudEdgeToken:  #Used for authentication between tunnel cloud and tunnel edge，No less than 32 characters
  TunnelPersistentConnectionServerKey:  #Tunnel cloud server private key(BASE64 encoded), accessed by Tunnel edge
  TunnelPersistentConnectionServerCrt:  #Tunnel cloud server certificate, X.509 BASE64 encoding (PEM format). It can be generated by OpenSSL，signed tunnel-cloud's service name: "tunnelcloud.io".
  TunnelProxyServerKey:  #Tunnel proxy server private key(BASE64 encoding), accessed by kube-apiserver
  TunnelProxyServerCrt:  #Tunnel proxy server certificate, X.509 BASE64 encoding (PEM format)

How to create TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt?

Certifications for authentication between tunnel-cloud and tunnel-edge.

Generate tunnel-cloud’s CA (You can choose to reuse the Kubernetes cluster’s CA)

# Generate CA private key
openssl genrsa -out tunnel_ca.key 2048

# Generate CSR
openssl req -new -key tunnel_ca.key -out tunnel_ca.csr

# Add DNS and IP
echo "subjectAltName=DNS:superedge.io,IP:127.0.0.1" > tunnel_ca_cert_extensions

# Generate Self Signed certificate
openssl x509 -req -days 365 -in tunnel_ca.csr -signkey tunnel_ca.key -extfile tunnel_ca_cert_extensions -out tunnel_ca.crt

Generate TunnelPersistentConnectionServerKey and TunnelPersistentConnectionServerCrt

# private key
openssl genrsa -des3 -out tunnel_persistent_connectiong_server.key 2048

# generate csr
openssl req -new -key tunnel_persistent_connectiong_server.key -subj "/CN=tunnel-cloud" -out tunnel_persistent_connectiong_server.csr

# Add DNS and IP
echo "subjectAltName=DNS:tunnelcloud.io,IP:127.0.0.1" > tunnel_cloud_cert_extensions

# Generate Self Signed certificate
openssl x509 -req -days 365 -in tunnel_persistent_connectiong_server.csr -CA tunnel-cloud-ca.crt -CAkey tunnel_ca.key -CAcreateserial  -extfile tunnel_cloud_cert_extensions -out tunnel_persistent_connectiong_server.crt

Get base64 encoded certifications

# generate TunnelPersistentConnectionServerKey
cat tunnel_persistent_connectiong_server.key | base64 --wrap=0
#generate TunnelPersistentConnectionServerCrt
cat tunnel_persistent_connectiong_server.crt | base64 --wrap=0

<details>
<summary>How to create TunnelProxyServerKey and TunnelProxyServerCrt?</summary>
<br>

Certifications for authentication between kube-apiserver and tunnel-cloud.

- Generate TunnelProxyServerKey and TunnelProxyServerCrt

```bash
# private key
openssl genrsa -des3 -out tunnel_proxy_server.key 2048

# generate csr
openssl req -new -key tunnel_proxy_server.key -subj "/CN=tunnel-cloud" -out tunnel_proxy_server.csr

# Add DNS and IP
echo "subjectAltName=DNS:superedge.io,IP:127.0.0.1" > cert_extensions

# Generate Self Signed certificate（Notice: It is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki）
openssl x509 -req -days 365 -in tunnel_proxy_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial  -extfile cert_extensions -out tunnel_proxy_server.crt
```

- BASE64 encoding tunnel_proxy_server.key and tunnel_proxy_server.crt, just like encoding tunnel_persistent_connectiong_server.key and tunnel_persistent_connectiong_server.crt above
</details>

1.3 Deploy tunnel-cloud

On master nodes,

$ kubectl apply -f deployment/tunnel-cloud.yaml

1.4 Let kube-apiserver using Tunnel

Point the DNS resolution of kube-apiserver to tunnel-CoreDNS. Through DNS hijacking, tunnel proxies the traffic from kube-apiserver to edge nodes. This solves the problem that kube-apiserver ususally can’t connect to edge nodes directly.

#Get tunnel-coredns's Cluster IP
$ kubectl get service tunnel-coredns -n edge-system
NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
tunnel-coredns   ClusterIP   10.10.47.74   <none>        53/UDP,53/TCP,9153/TCP   140m
#Replace kube-apierver's DNS nameservers with tunnel-coredns's Cluster IP
...
dnsConfig:
    nameservers:
    - 10.10.47.74 #tunnel-cloud's CLUSTER IP；  
...

Notice: Avoid using IP address as the name of the edge node to avoid DNS hijacking failure.

1.5 Configure tunnel-edge

Set the following parameters in the deployment/tunnel-edge.yaml

MasterIP:  #Normal Kubernetes master node's IP or domain(currently, only one IP address or domain is supported)
TunnelCloudEdgeToken:  #Fill in the same token as "TunnelCloudEdgeToken" in Tunnel-cloud
TunnelPersistentConnectionPort:  #Tunnel-cloud's Persistent connection server Port
KubernetesCaCert:  #kube-apiserver's ca.crt(base64 encoded)
KubeletClientKey:  #Kubelet client key for Tunnel-edge to access Kubelet
KubeletClientCrt:  #Kubelet client cert for Tunnel-edge to access Kubelet

How to create KubeletClientKey and KubeletClientCrt?

Certifications for anthentication between tunnel-edge and Kubelet.

# private key
openssl genrsa -des3 -out kubelet_client.key 1024
# generate csr
openssl req -new -key kubelet_client.key -out kubelet_client.csr

# Generate Self Signed certificate（Notice: it is Kubernetes cluster's ca.crt and ca.key, In Kubeadm install method，ca.crt and ca.key path at /etc/kubernetes/pki）
openssl ca -in kubelet-client.csr -out kubelet-client.crt -cert ca.crt -keyfile ca.key

BASE64 encoding KubeletClientKey and KubeletClientCrt

1.6 Deploy tunnel-edge

On edge worker nodes,

$ kubectl apply -f deployment/tunnel-edge.yaml

2. Install lite-apiserver

2.1 Deploy lite-apiserver

Use Kubernetes cluster’s ca.crt and ca.key to generate lite_apiserver key and certificate(lite-apiserver.key and lite-apiserver.crt). If the cluster is created via Kubeadm，ca.crt and ca.key can be found at /etc/kubernetes/pki

#get service 'kubernetes' ClusterIP
$ kubectl get service kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.10.0.1    <none>        443/TCP   23d

#Generate lite-apiserver.key
$ openssl genrsa -out lite-apiserver.key 2048

#create lite-apiserver.csr
$ cat << EOF >lite_apiserver.conf
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
CN = lite-apiserver
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
IP.1 = 127.0.0.1
IP.2 = 10.10.0.1 # please change the value to Kubernetes's Cluster IP
EOF

$ openssl req -new -key lite-apiserver.key -subj "/CN=lite-apiserver" -config lite-apiserver.conf -out lite-apiserver.csr

#generating lite-apiserver.crt
openssl x509 -req -in lite-apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 5000 -extensions v3_req -extfile lite-apiserver.conf -out lite-apiserver.crt

Copy lite-apiserver.crt and lite-apiserver.key into edge worker node, path at /etc/kubernetes/pki/
Modify deployment/lite-apiserver.yaml, set –kube-apiserver-url and –kube-apiserver-port to apiserver’s host and port
Set –tls-config-file=/etc/kubernetes/edge/tls.json, create /etc/kubernetes/edge/tls.json in edge worker node, and write：
We need use lite-apiserver to proxy request from kubelet to kube-apiserver, so need let lite-apiserver known kubelet-client key and certificate
Because of kube-apiserver client key and kube-apiserver client certificate are in kubelet-client-current.pem, so value of “key” and “cert” are “/var/lib/kubelet/pki/kubelet-client-current.pem”.
```
[
    {
        "key":"/var/lib/kubelet/pki/kubelet-client-current.pem",
        "cert":"/var/lib/kubelet/pki/kubelet-client-current.pem"
    }
]
```
Notice: kubelet-client-current.pem generated by kubeadm, please do not modify any data of kubelet-client-current.pem
Use Static Pod to deploy lite-apiserver in Edge Worker Node, copy deployment/lite-apiserver.yaml to Edge Worker Node’s kubelet manifests directory (the directory of Kubernetes cluster builded by kubeadm usually locate at /etc/kubernetes/manifests/）。

2.2 Configure Kubelet to use lite-apiserver

lite-apiserver listen on port 51003 by default (use parameter –port to assign port, in deployment/lite-apiserver.yaml)，please replace kube-apiserver by https://127.0.0.1:51003

kubelet: modify cluster.server=https://127.0.0.1:51003, in kubelet.conf, and restart kubelet。

3. Install application grid

3.1 Deploy Application Grid Controller

On master nodes,

$ kubectl apply -f deployment/application-grid-controller.yaml

3.2 Add annotate endpoint Kubernetes

Configure Kubernetes endspints to point to lite-apiserver, all traffic from pod to kube-apiserver would be proxyed by lite-apiserver.

kubectl annotate endpoints kubernetes superedge.io/local-endpoint=127.0.0.1
kubectl annotate endpoints kubernetes superedge.io/local-port=51003

3.3 Deploy application grid wrapper

On edge worker nodes,

$ kubectl apply -f deployment/application-grid-wrapper.yaml

Application-grid-wrapper will access kube-apiserver proxyed by lite-apiserver

3.4 Configure kube-proxy to Use Application Grid Wrapper

Modify kube-proxy’s cluster.server to http://127.0.0.1:51006 (kube-proxy’s configuration file is a configmap resource named kube-proxy in kube-system namespace)

application-grid-wrapper listen on port 51006 by default

4. Install edge-health

4.1 Deploy edge-health admission and webhook

On master nodes,

$ kubectl apply -f deployment/edge-health-admission.yaml
$ kubectl apply -f deployment/edge-health-webhook.yaml

Currently the certificates in the webhook is pre-populated, you can replace them with your certificates.
The caBundle in deployment/edge-health-webhook.yaml can be replaced with your CA certificate.
The server.crt and server.key in validate-admission-control-server-certs Secret of deployment/edge-health-admission.yaml can be replaced with your signed certificate and key.

4.3 Configure edge-health

Set the following parameters in the deployment/edge-health.yaml

  HmacKey:  #Hmackey is used in communication between edge-healths, no less than 16 characters

4.3 Deploy edge-health

On edge worker nodes,

$ kubectl apply -f deployment/edge-health.yaml

Feedback

Was this page helpful?

Glad to hear from you! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified June 15, 2021 : initial commit (974355a)