验证kubernetes的pod安全策略
最近有客户提出需要对业务集群的pod做安全限制，不允许使用pod拥有privileged的权限，研究一番，刚好k8s的Pod Security Policies可以实现该需求。
什么是pod安全策略
Pod Security Policy(简称psp)是集群级别的资源，该资源控制pod的spec中安全相关的方面，具体的方面参考下表：
| Control Aspect | Field Names |
|---|---|
| Running of privileged containers | privileged |
| Usage of host namespaces | hostPID, hostIPC |
| Usage of host networking and ports | hostNetwork, hostPorts |
| Usage of volume types | volumes |
| Usage of the host filesystem | allowedHostPaths |
| White list of Flexvolume drivers | allowedFlexVolumes |
| Allocating an FSGroup that owns the pod’s volumes | fsGroup |
| Requiring the use of a read only root file system | readOnlyRootFilesystem |
| The user and group IDs of the container | runAsUser, runAsGroup, supplementalGroups |
| The SELinux context of the container | seLinux |
如何开启Pod Security Policy
- Enable API extensions
For Kubernetes < 1.6.0, the API Server must enable the extensions/v1beta1/podsecuritypolicy API extensions group (–runtime-config=extensions/v1beta1/podsecuritypolicy=true).
- Enable PodSecurityPolicy admission control policy
The following parameter needs to be added to the API server startup argument: –admission-control=PodSecurityPolicy
默认psp是不开启的，若要开启，需要配置上述apiserver启动参数，默认的apiserver的yaml文件为/etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
containers:
- command:
- kube-apiserver
- --authorization-mode=Node,RBAC
- --enable-admission-plugins=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota,PodSecurityPolicy
- --runtime-config=extensions/v1beta1/podsecuritypolicy=true
...配置默认的psp
由于已有打开了pod创建的安全策略,但此时还未创建任何的policy，所以任何pod都无法被创建，此时如果尝试去创建一个pod，会发现pod无法进行调度，
[root@build-master apiserver]# kubectl run --image=nginx yxli
deployment.apps/yxli created
[root@build-master apiserver]# kubectl get deploy yxli
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
yxli 1 0 0 0 4m查看controller的日志会发现报错forbidden: no providers available to validate pod request
[root@build-master ~]# kubectl logs --tail=10 -f kube-controller-manager-build-master -n kube-system
E0318 03:01:17.321184 1 replica_set.go:450] Sync "default/yxli-6978dddc9d" failed with pods "yxli-6978dddc9d-" is forbidden: no providers available to validate pod request
I0318 03:01:17.321302 1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"yxli-6978dddc9d", UID:"0d11050f-492a-11e9-8ef0-00163e004fd2", APIVersion:"apps/v1", ResourceVersion:"764927", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "yxli-6978dddc9d-" is forbidden: no providers available to validate pod request
E0318 03:01:37.807060 1 replica_set.go:450] Sync "default/yxli-6978dddc9d" failed with pods "yxli-6978dddc9d-" is forbidden: no providers available to validate pod request
I0318 03:01:37.807046 1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"yxli-6978dddc9d", UID:"0d11050f-492a-11e9-8ef0-00163e004fd2", APIVersion:"apps/v1", ResourceVersion:"764927", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "yxli-6978dddc9d-" is forbidden: no providers available to validate pod request
E0318 03:02:18.773092 1 replica_set.go:450] Sync "default/yxli-6978dddc9d" failed with pods "yxli-6978dddc9d-" is forbidden: no providers available to validate pod request而由于我们修改了apiserver的参数，pod受kubelet管理，自动触发了重建，此时apiserver的pod也是因为缺少权限没法创建出来
[root@build-master apiserver]# journalctl -fu kubelet
-- Logs begin at Tue 2019-03-12 20:33:59 CST. --
Mar 18 11:25:29 build-master kubelet[4013]: E0318 11:25:29.016416 4013 kubelet.go:1594] Failed creating a mirror pod for "kube-apiserver-build-master_kube-system(77a7d13c654e7233306d4e2948aaaa78)": pods "kube-apiserver-build-master" is forbidden: no providers available to validate pod request
Mar 18 11:25:30 build-master kubelet[4013]: W0318 11:25:30.113587 4013 kubelet.go:1579] Deleting mirror pod "kube-apiserver-build-master_kube-system(e48d84cd-46f0-11e9-8ef0-00163e004fd2)" because it is outdated
Mar 18 11:25:30 build-master kubelet[4013]: E0318 11:25:30.117242 4013 kubelet.go:1594] Failed creating a mirror pod for "kube-apiserver-build-master_kube-system(77a7d13c654e7233306d4e2948aaaa78)": pods "kube-apiserver-build-master" is forbidden: no providers available to validate pod request所以我们需要创建一个policy，为我们需要的受kubelet管理的静态pod以及kube-system命名空间下的pod提供权限，否则pod一旦发生重建，都将因为缺少权限导致无法正常创建出来。
首先新建文件 privileged.policy.yaml，权限不受限制，
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: privileged
spec:
allowedCapabilities:
- '*'
allowPrivilegeEscalation: true
fsGroup:
rule: 'RunAsAny'
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- min: 0
max: 65535
privileged: true
readOnlyRootFilesystem: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
volumes:
- '*'创建并查看该policy
[root@build-master apiserver]# kubectl create -f privileged.policy.yaml
podsecuritypolicy.policy/privileged created
[root@build-master apiserver]# kubectl get psp
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
privileged true * RunAsAny RunAsAny RunAsAny RunAsAny false *然后还需要创建一个clusterrole，并且赋予该role对上述psp对使用权
# Cluster role which grants access to the privileged pod security policy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: privileged-psp
rules:
- apiGroups:
- policy
resourceNames:
- privileged
resources:
- podsecuritypolicies
verbs:
- use然后把clusterrole赋予serviceaccount或者对应的user、group，针对kube-system命名空间下的pod来说，都使用了kube-system下的serviceaccount或者由kubelet管理，kubelet是使用system:nodes这个组来管理的pod，所以只需要做如下binding即可为kube-system下的所有pod提供权限，
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kube-system-psp
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
namespace: kube-system
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts:kube-system都创建好之后，查看kube-system下的pod，发现apiserver已经创建成功
[root@build-master apiserver]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-68f5b48ccb-9hjvr 1/1 Running 0 5d
coredns-68f5b48ccb-qrjnr 1/1 Running 0 5d
etcd-build-master 1/1 Running 0 5d
kube-apiserver-build-master 1/1 Running 0 18s上面的binding只是为kube-system空间赋予了权限，若想要为别的命名空间赋予权限，可以使用ClusterRoleBinding的方式为多个namespace绑定privileged-psp的clusterrole，或者像如下方式，为所有合法的serviceaccounts和user绑定权限，当然前提是pod使用了namespace下的serviceAccount
# Authorize all service accounts in a namespace:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccounts
# Or equivalently, all authenticated users in a namespace:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:authenticated验证psp的privileged限制
该章节会创建一个名为psp-namespace的namespace做测试，来验证psp中对privileged的pod的限制
[root@build-master apiserver]# kubectl create namespace psp-example
namespace/psp-example created
[root@build-master apiserver]# kubectl create serviceaccount fake-user -n psp-example
serviceaccount/fake-user created
[root@build-master apiserver]# kubectl run --image=nginx psp-pod-nginx-1 -n psp-example --serviceaccount=fake-user
deployment.apps/psp-pod-nginx-1 created
[root@build-master apiserver]# kubectl get po -n psp-example
No resources found.
[root@build-master apiserver]# kubectl get deploy -n psp-example
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
psp-pod-nginx-1 1 0 0 0 23s如上所示，新的psp-example命名空间由于没有任何权限，所以无法创建新pod，接下来我们创建一个policy并给psp-example做authorize，但是policy中会限制无法创建privileged的pod
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: false # Don't allow privileged pods!
# The rest fills in some required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
volumes:
- '*'然后创建role和rolebinding，并验证fake-user是否有权限使用example的policy
[root@build-master psp-example]# kubectl -n psp-example create role psp:unprivileged --verb=use --resource=podsecuritypolicy --resource-name=example
role.rbac.authorization.k8s.io/psp:unprivileged created
[root@build-master psp-example]# kubectl -n psp-example create rolebinding fake-user:psp:unprivileged --role=psp:unprivileged --serviceaccount=psp-example:fake-user
rolebinding.rbac.authorization.k8s.io/fake-user:psp:unprivileged created
[root@build-master psp-example]# kubectl --as=system:serviceaccount:psp-example:fake-user -n psp-example auth can-i use podsecuritypolicy/example
yes等待片刻，再次查看刚才创建的deploy，发现已经调度成功
[root@build-master psp-example]# kubectl get deploy -n psp-example psp-pod-nginx-1
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
psp-pod-nginx-1 1 1 1 1 22m此时我们尝试在psp-example下创建一个privileged的特权容器，
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
run: psp-pod-nginx-privileged
name: psp-pod-nginx-privileged
namespace: psp-example
spec:
selector:
matchLabels:
run: psp-pod-nginx-privileged
template:
metadata:
labels:
run: psp-pod-nginx-privileged
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: psp-pod-nginx-privileged
securityContext:
privileged: true
serviceAccount: fake-user创建并查看结果
[root@build-master psp-example]# kubectl create -f privileged-pod.yaml
deployment.extensions/psp-pod-nginx-privileged created
[root@build-master psp-example]# kubectl get deploy -n psp-example psp-pod-nginx-privileged
psp-pod-nginx-privileged 1 0 0 0 24s
[root@build-master ~]# kubectl logs --tail=1 -f kube-controller-manager-build-master -n kube-system
I0318 04:32:42.777200 1 event.go:221] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"psp-example", Name:"psp-pod-nginx-privileged-d7f8dbd75", UID:"c66fb10f-4936-11e9-9c58-00163e004fd2", APIVersion:"apps/v1", ResourceVersion:"773292", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "psp-pod-nginx-privileged-d7f8dbd75-" is forbidden: unable to validate against any pod security policy: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]查看controller-manager的日志，发现Privileged containers are not allowed