Guides · PRACTICAL GUIDE
Kubernetes PVC stuck Terminating? Audit every storage reference before removing a finalizer
A tested read-only collector joins the PVC, PV, Pods, StorageClass and VolumeAttachments into one evidence ledger before any destructive storage action.
A stuck PVC is a relationship problem until evidence proves otherwise. Preserve the API graph before changing the object.
This guide supplies a tested collector that runs kubectl get only and converts missing permissions or malformed responses into unresolved findings.
A finalizer is a contract, not the root cause
When you delete a PVC, Kubernetes sets deletionTimestamp and leaves the object visible while listed finalizers finish their purpose. The object can therefore show Terminating even though the delete request itself succeeded. Kubernetes explicitly warns against removing a finalizer manually unless you understand why it exists and have completed that purpose.
For storage, protection can reflect a real reference. A PVC is considered in active use when a Pod object still names it, and a bound PV has its own protection and reclaim-policy lifecycle. CSI migration also changes which controller finalizer may appear. Start by preserving relationships, not by editing metadata.
Capture five API views before interpreting the state
The supplied audit reads the target PVC, all Pods in its namespace, its bound PV when present, the StorageClass and cluster-wide VolumeAttachments. It records UIDs, resource versions, deletion timestamps, finalizers, claimRef, reclaim policy, CSI driver and volume handle without printing secret data.
A failed read is unresolved evidence. RBAC denial on VolumeAttachments must not be converted to an empty list, and malformed JSON must not be treated as zero references. The program stops short of a destructive conclusion whenever any read is incomplete.
Save and run the no-write collector
Save the complete program below as audit_kubernetes_pvc.mjs. Run node --check audit_kubernetes_pvc.mjs, then run node audit_kubernetes_pvc.mjs NAMESPACE PVC_NAME > pvc-before.json. Use exact lowercase Kubernetes names; the program rejects values that could be parsed as command options.
#!/usr/bin/env node import { spawnSync } from 'node:child_process'; const SAFE_NAME = /^(?=.{1,253}$)[a-z0-9](?:[-a-z0-9.]*[a-z0-9])?$/; export function validateName(value, label) { if (typeof value !== 'string' || !SAFE_NAME.test(value) || value.startsWith('-')) { throw new Error(`${label} must be a safe lowercase Kubernetes name`); } } function summarizeObject(object) { if (!object) return null; return { apiVersion: object.apiVersion ?? null, kind: object.kind ?? null, name: object.metadata?.name ?? null, namespace: object.metadata?.namespace ?? null, uid: object.metadata?.uid ?? null, resourceVersion: object.metadata?.resourceVersion ?? null, deletionTimestamp: object.metadata?.deletionTimestamp ?? null, finalizers: object.metadata?.finalizers ?? [], }; } export function spawnKubectl(args) { const result = spawnSync('kubectl', args, { encoding: 'utf8', shell: false, timeout: 20_000, maxBuffer: 8 * 1024 * 1024, windowsHide: true, }); return { status: result.status, stdout: result.stdout ?? '', stderr: result.stderr ?? '', error: result.error?.message ?? null, }; } function readJson(run, args, label, unresolved, { required = false } = {}) { const result = run(args); if (result.error || result.status !== 0) { unresolved.push({ check: label, reason: result.error ?? `kubectl exited ${result.status}`, stderr: String(result.stderr ?? '').trim().slice(0, 1000), }); if (required) throw new Error(`${label} could not be read`); return null; } try { return JSON.parse(result.stdout); } catch (error) { unresolved.push({ check: label, reason: `malformed JSON: ${error.message}` }); if (required) throw new Error(`${label} returned malformed JSON`); return null; } } export function auditPVC(namespace, pvcName, run = spawnKubectl) { validateName(namespace, 'namespace'); validateName(pvcName, 'PVC name'); const unresolved = []; const commands = []; const read = (args, label, options) => { commands.push(['kubectl', ...args]); return readJson(run, args, label, unresolved, options); }; const pvc = read(['get', 'pvc', pvcName, '-n', namespace, '-o', 'json'], 'PVC', { required: true }); const pods = read(['get', 'pods', '-n', namespace, '-o', 'json'], 'Pods'); const pvName = pvc.spec?.volumeName || null; const pv = pvName ? read(['get', 'pv', pvName, '-o', 'json'], 'PV') : null; const storageClassName = pvc.spec?.storageClassName || pv?.spec?.storageClassName || null; const storageClass = storageClassName ? read(['get', 'storageclass', storageClassName, '-o', 'json'], 'StorageClass') : null; const volumeAttachments = read( ['get', 'volumeattachments.storage.k8s.io', '-o', 'json'], 'VolumeAttachments', ); const podReferences = (pods?.items ?? []).flatMap((pod) => { const matchingVolumes = (pod.spec?.volumes ?? []) .filter((volume) => volume.persistentVolumeClaim?.claimName === pvcName) .map((volume) => ({ volumeName: volume.name, readOnly: Boolean(volume.persistentVolumeClaim?.readOnly) })); return matchingVolumes.length ? [{ ...summarizeObject(pod), phase: pod.status?.phase ?? null, nodeName: pod.spec?.nodeName ?? null, matchingVolumes, }] : []; }); const matchingAttachments = (volumeAttachments?.items ?? []) .filter((attachment) => attachment.spec?.source?.persistentVolumeName === pvName) .map((attachment) => ({ ...summarizeObject(attachment), attacher: attachment.spec?.attacher ?? null, nodeName: attachment.spec?.nodeName ?? null, attached: attachment.status?.attached ?? null, attachError: attachment.status?.attachError?.message ?? null, detachError: attachment.status?.detachError?.message ?? null, })); return { schemaVersion: 1, generatedAt: new Date().toISOString(), target: { namespace, pvcName, pvName, storageClassName }, safety: { mode: 'read-only', mutatingCommandsRun: false, notRun: ['patch', 'delete', 'replace', 'edit', 'scale', 'drain', 'detach', 'finalizer removal'], backendContacted: false, }, commands, pvc: { ...summarizeObject(pvc), phase: pvc.status?.phase ?? null, accessModes: pvc.spec?.accessModes ?? [], requestedStorage: pvc.spec?.resources?.requests?.storage ?? null, volumeName: pvName, storageClassName, }, pv: pv ? { ...summarizeObject(pv), phase: pv.status?.phase ?? null, reclaimPolicy: pv.spec?.persistentVolumeReclaimPolicy ?? null, storageClassName: pv.spec?.storageClassName ?? null, claimRef: pv.spec?.claimRef ? { namespace: pv.spec.claimRef.namespace ?? null, name: pv.spec.claimRef.name ?? null, uid: pv.spec.claimRef.uid ?? null, } : null, csi: pv.spec?.csi ? { driver: pv.spec.csi.driver ?? null, volumeHandle: pv.spec.csi.volumeHandle ?? null, fsType: pv.spec.csi.fsType ?? null, } : null, } : null, storageClass: storageClass ? { ...summarizeObject(storageClass), provisioner: storageClass.provisioner ?? null, reclaimPolicy: storageClass.reclaimPolicy ?? null, volumeBindingMode: storageClass.volumeBindingMode ?? null, allowVolumeExpansion: storageClass.allowVolumeExpansion ?? null, } : null, podReferences, volumeAttachments: matchingAttachments, unresolved, decisionBoundary: { stopBeforeMutation: true, reasons: [ ...(podReferences.length ? ['one or more Pods still reference the PVC'] : []), ...(matchingAttachments.length ? ['one or more VolumeAttachments still reference the PV'] : []), ...(unresolved.length ? ['one or more required API reads are unresolved'] : []), ...(pvc.metadata?.finalizers?.length ? ['PVC finalizers require controller-purpose investigation'] : []), ...(pv?.metadata?.finalizers?.length ? ['PV finalizers require controller-purpose investigation'] : []), ], }, }; } if (import.meta.url === `file://${process.argv[1]}`) { const [namespace, pvcName] = process.argv.slice(2); try { const report = auditPVC(namespace, pvcName); process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); if (report.unresolved.length) process.exitCode = 2; } catch (error) { process.stderr.write(`${error.message}\n`); process.exitCode = 1; } }
Program SHA-256 is 397076a9d736bceca9fb90438d83f55e55b6b4d7ce7b3a35da6609c2fcc0adf1. Eight synthetic scenarios passed on Node v24.19.0: a bound claim with a Pod and VolumeAttachment, deletion timestamp, no PV, RBAC denial, malformed JSON, unsafe names and a scan proving that no mutating verb is invoked. No real cluster or storage backend was contacted.
Read Pod references before touching the claim
The report lists only Pods whose spec.volumes entries name the target claim. A completed higher-level workflow is not proof that every retained object stopped referring to the PVC; an Argo report shows how a workflow relationship could keep deletion stuck until that owner was addressed.
Do not delete a Pod merely because it appears in the ledger. Identify the workload controller, confirm the application shutdown and data policy, and decide with the workload owner. The collector intentionally does not scale, delete or evict anything.
Trace the PV, reclaim policy and CSI attachment separately
A Bound PV should be compared with claimRef and the PVC UID. Record the StorageClass provisioner and both reclaim-policy values. Delete and Retain describe what should happen after release; neither proves that a provider volume has already detached or disappeared.
A matching VolumeAttachment can show the CSI attacher, node and reported attached state, but it is still an API object rather than direct backend evidence. If the API is unavailable or an attach/detach error appears, preserve it and move the investigation to the CSI controller and storage-provider runbook.
Use a stop matrix, not a force-delete recipe
Stop when a Pod still references the claim, a VolumeAttachment still references the PV, any required read is unresolved, or a finalizer purpose is unidentified. Review means choose the next responsible owner; it never means automatically strip metadata.
Community answers demonstrate why this boundary matters: different clusters may be waiting on workloads, provisioners, attachments or stale objects. Copying one finalizer patch from another environment can hide the only evidence that identifies the failed controller.
Preserve before and after ledgers around one approved change
Once the responsible controller or workload owner approves one narrow action, save the old JSON, record that action and run the same collector again. Compare UIDs, resource versions, references, finalizers and attachment state rather than relying on a screenshot of Terminating disappearing.
An object disappearing from the API is not proof that data is safe, detached or recoverable. Validate application data and the storage backend through their separate supported procedures. This guide provides an evidence boundary, not a recovery guarantee.
