K8S集群中部署 MinIO 以支持对象存储

前置条件

MinIO 是一个兼容 s3 标准的对象存储。创建 NFS 存储作为容器存储插件

部署文档：https://github.com/minio/minio/tree/master/helm/minio

部署准备

创建命名空间

$ kubectl create ns minio

添加 helm 仓库

$ helm repo add minio https://charts.min.io/
$ helm repo update
$ helm search repo minio/minio
NAME            CHART VERSION   APP VERSION                     DESCRIPTION
minio/minio     5.1.0           RELEASE.2024-03-03T17-50-39Z    High Performance Object Storage

拉取文件，并修改 values.yaml 文件

$ helm pull minio/minio
$ tar -xf minio-5.1.0.tgz

#values.yaml
...
# Number of MinIO containers running
replicas: 2
...
## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  enabled: true
  annotations: {}
  storageClass: qiqios-nfs-storage
  volumeName: ""
  accessMode: ReadWriteOnce
  size: 10Gi
...
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  requests:
    memory: 1024Mi
...
## Expose the MinIO service to be accessed from outside the cluster (LoadBalancer service)
service:
  type: ClusterIP
  clusterIP: ~
  port: "9000"
  #nodePort: 32000

部署并检查

部署 MinIO

$ cd /apps/helm_chart/minio
$ helm install minio -n minio -f values.yaml .
NAME: minio
LAST DEPLOYED: Fri Mar 15 16:25:28 2024
NAMESPACE: minio
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MinIO can be accessed via port 9000 on the following DNS name from within your cluster:
minio.minio.svc.cluster.local

To access MinIO from localhost, run the below commands:

  1. export POD_NAME=$(kubectl get pods --namespace minio -l "release=minio" -o jsonpath="{.items[0].metadata.name}")

  2. kubectl port-forward $POD_NAME 9000 --namespace minio

Read more about port forwarding here: http://kubernetes.io/docs/user-guide/kubectl/kubectl_port-forward/

You can now access MinIO server on http://localhost:9000. Follow the below steps to connect to MinIO server with mc client:

  1. Download the MinIO mc client - https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart

  2. export MC_HOST_minio-local=http://$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootUser}" | base64 --decode):$(kubectl get secret --namespace minio minio -o jsonpath="{.data.rootPassword}" | base64 --decode)@localhost:9000

  3. mc ls minio-local

检查相关组件

$ kubectl get all -n minio
NAME          READY   STATUS    RESTARTS   AGE
pod/minio-0   1/1     Running   0          4m41s
pod/minio-1   1/1     Running   0          4m41s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/minio           ClusterIP   10.68.24.47    <none>        9000/TCP   4m41s
service/minio-console   ClusterIP   10.68.20.233   <none>        9001/TCP   4m41s
service/minio-svc       ClusterIP   None           <none>        9000/TCP   4m41s

NAME                     READY   AGE
statefulset.apps/minio   2/2     4m41s

# 查看 pvc
$ kubectl get pvc -n minio
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
export-minio-0   Bound    pvc-62d685b1-7137-491a-9f67-a21ddca65f14   10Gi       RWO            qiqios-nfs-storage   116s
export-minio-1   Bound    pvc-75b5da77-c9a3-4c40-b542-b69e1354eab4   10Gi       RWO            qiqios-nfs-storage   116s

配置证书和域名

创建 Ingress 并通过cert-manager组件 自动签发 https 证书

# minio-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio
  namespace: minio
  annotations:
    cert-manager.io/cluster-issuer: cert-manager-webhook-dnspod-cluster-issuer # 配置自动生成 https 证书
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - 'minio.qiqios.com'
      secretName: minio-letsencrypt-tls
  rules:
    - host: 'minio.qiqios.com'
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: minio-console
                port:
                  number: 9001

等待证书签发完成

$ kubectl get certificate -n minio
NAME                    READY   SECRET                  AGE
minio-letsencrypt-tls   True    minio-letsencrypt-tls   116s
$ kubectl get ingress -n minio
NAME    CLASS   HOSTS              ADDRESS   PORTS     AGE
minio   nginx   minio.qiqios.com             80, 443   2m9s

访问测试

$ curl -I https://minio.qiqios.com --resolve minio.qiqios.com:443:192.168.1.91
HTTP/2 200
date: Fri, 15 Mar 2024 08:50:10 GMT
content-type: text/html
content-length: 1310
accept-ranges: bytes
content-security-policy: default-src 'self' 'unsafe-eval' 'unsafe-inline';
last-modified: Fri, 15 Mar 2024 08:50:10 GMT
referrer-policy: strict-origin-when-cross-origin
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block
strict-transport-security: max-age=15724800; includeSubDomains

用户名密码就是在清单文件中配置的 accessKey 和 secretKey