Kubernetes的NameSpace无法删除

前言

在日常运维中，有时候我们操作不规范，删除资源的先后顺序不正确，或某项关键服务没有启动，导致 Kubernetes 经常会出现无法删除 NameSpace 的情况

规范删除流程

很多时候出现这种问题，主要是因为我们的删除操作不规范，典型情况如下：

• 删除的先后顺序有问题，如：
  • 先删除了 Traefik 的关键组件，再尝试删除包含 Traefik Ingress 或 EdgeIngress 的 CRD
• 某项关键服务没有启动，如：
  • 对于安装了 Prometheus Operator + custom adapter 的 Kubernetes 集群，在 Prometheus 的一些关键组件 scale down 的情况下，删除包含这些监控 CRD 或 HPA custom metric 的 NameSpace

…

为了避免此类错误，推荐搭建删除按照如下流程：

1. 保证所有基础服务组件都是正常运行状态；
2. 检查要删除的 NameSpace 下的所有资源，特别是 CRD ，这里推荐使用 Krew - Kubernetes 的 CLI 插件管理器 安装 get-all 来 真正 地获取该 NameSpace 下的所有资源，如后面的代码块所示。
3. 针对其中的一些 CRD 或特殊资源，最好先明确指定删除并确保可以成功删除；
4. 最后再删除该 NameSpace。

$  kubectl get-all -n kubeapps
NAME                                NAMESPACE  AGE
apprepository.kubeapps.com/bitnami  kubeapps   13d
# 说明这个namespace 是基于helm 仓库安装的

尝试强制删除

如果 NameSpace 已经处于 terminating 的状态，且久久无法删除，可以试试加上这 2 个参数强制删除：

• --force
• --grace-period=0

$ kubectl delete ns kubeapps --force --grace-period=0

调用 Kubernetes API 删除

获取删除 NameSpace 的 JSON 文件

$ NAMESPACE=kubeapps
$ kubectl get ns ${NAMESPACE} -o json > namespace.json

编辑 namespace.json

编辑 namespace.json, 从 finalizers 字段中删除 kubernetes的值并保存

$ vim namespace.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2024-03-20T01:40:19Z",
        "deletionTimestamp": "2024-03-20T05:45:59Z",
        "labels": {
            "kubernetes.io/metadata.name": "kubeapps"
        },
        "name": "kubeapps",
        "resourceVersion": "3115786",
        "uid": "165f7ead-4d8b-4281-b442-ff191c2a30b8"
    },
    "spec": {
        "finalizers": []  # 原来的 finalizers里面有值
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2024-03-20T06:41:41Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2024-03-20T05:46:05Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2024-03-20T05:46:05Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2024-03-20T05:46:05Z",
                "message": "Some resources are remaining: apprepositories.kubeapps.com has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2024-03-20T05:46:05Z",
                "message": "Some content in the namespace has finalizers remaining: apprepositories.kubeapps.com/apprepo-cleanup-finalizer in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

通过 kubectl proxy 设置 APIServer 的临时 IP 和端口

$ kubectl proxy --port=6880 &

进行 API 调用来强制删除：

$ export NAMESPACE=kubeapps
$ curl -k -H "Content-Type: application/json" -X  PUT --data-binary @namespace.json http://127.0.0.1:6880/api/v1/namespaces/${NAMESPACE}/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "kubeapps",
    "uid": "165f7ead-4d8b-4281-b442-ff191c2a30b8",
    "resourceVersion": "3115786",
    "creationTimestamp": "2024-03-20T01:40:19Z",
    "deletionTimestamp": "2024-03-20T05:45:59Z",
    "labels": {
      "kubernetes.io/metadata.name": "kubeapps"
    },
    "managedFields": [
      {
        "manager": "kubectl-create",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2024-03-20T01:40:19Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:metadata": {
            "f:labels": {
              ".": {},
              "f:kubernetes.io/metadata.name": {}
            }
          }
        }
      },
      {
        "manager": "kube-controller-manager",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2024-03-20T06:41:41Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:status": {
            "f:conditions": {
              ".": {},
              "k:{\"type\":\"NamespaceContentRemaining\"}": {
                ".": {},
                "f:lastTransitionTime": {},
                "f:message": {},
                "f:reason": {},
                "f:status": {},
                "f:type": {}
              },
              "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
                ".": {},
                "f:lastTransitionTime": {},
                "f:message": {},
                "f:reason": {},
                "f:status": {},
                "f:type": {}
              },
              "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
                ".": {},
                "f:lastTransitionTime": {},
                "f:message": {},
                "f:reason": {},
                "f:status": {},
                "f:type": {}
              },
              "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
                ".": {},
                "f:lastTransitionTime": {},
                "f:message": {},
                "f:reason": {},
                "f:status": {},
                "f:type": {}
              },
              "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
                ".": {},
                "f:lastTransitionTime": {},
                "f:message": {},
                "f:reason": {},
                "f:status": {},
                "f:type": {}
              }
            }
          }
        },
        "subresource": "status"
      }
    ]
  },
  "spec": {},
  "status": {
    "phase": "Terminating",
    "conditions": [
      {
        "type": "NamespaceDeletionDiscoveryFailure",
        "status": "False",
        "lastTransitionTime": "2024-03-20T06:41:41Z",
        "reason": "ResourcesDiscovered",
        "message": "All resources successfully discovered"
      },
      {
        "type": "NamespaceDeletionGroupVersionParsingFailure",
        "status": "False",
        "lastTransitionTime": "2024-03-20T05:46:05Z",
        "reason": "ParsedGroupVersions",
        "message": "All legacy kube types successfully parsed"
      },
      {
        "type": "NamespaceDeletionContentFailure",
        "status": "False",
        "lastTransitionTime": "2024-03-20T05:46:05Z",
        "reason": "ContentDeleted",
        "message": "All content successfully deleted, may be waiting on finalization"
      },
      {
        "type": "NamespaceContentRemaining",
        "status": "True",
        "lastTransitionTime": "2024-03-20T05:46:05Z",
        "reason": "SomeResourcesRemain",
        "message": "Some resources are remaining: apprepositories.kubeapps.com has 1 resource instances"
      },
      {
        "type": "NamespaceFinalizersRemaining",
        "status": "True",
        "lastTransitionTime": "2024-03-20T05:46:05Z",
        "reason": "SomeFinalizersRemain",
        "message": "Some content in the namespace has finalizers remaining: apprepositories.kubeapps.com/apprepo-cleanup-finalizer in 1 resource instances"
      }
    ]
  }
}

查询结果

$ kubectl get ns ${NAMESPACE}
Error from server (NotFound): namespaces "kubeapps" not found  # 已经删除成功

编排脚本

依赖组件： kubectl、jq、curl

$ vim force-delete-ns.sh
#!/bin/bash
set -ex
PATH=$PATH:.
NAMESPACE=$1    # 读取命令行第一个参数
kill -9  $(ps -ef|grep proxy|grep -v grep |awk '{print $2}')
kubectl proxy --port=6880 &
kubectl get namespace ${NAMESPACE} -o json |jq '.spec = {"finalizers":[]}' > namespace.json
curl -k -H  "Content-Type: application/json"  -X  PUT --data-binary @namespace.json 127.0.0.1:6880/api/v1/namespaces/${NAMESPACE}/finalize