Kubernetes Guardrails at Scale with Kyverno and ECR Pull Through Cache

When you start working with a few Kubernetes clusters, pulling public images directly from Docker Hub, GHCR, Quay, or registry.k8s.io feels normal.

When you start working with many clusters, many teams, and many Helm charts, it becomes something else entirely.

You suddenly have a lot of workloads depending on external registries that you do not control. You also have development teams copying image references from upstream documentation, vendors, blog posts, and old examples. Some are pinned properly. Some are not. Some come from Docker Hub official images. Some come from registries that need authentication. Some are buried deep inside third-party charts.

This is where I found Kyverno to be a really good fit.

The project was simple in principle: use Amazon ECR pull through cache as the approved route for public image pulls, then use Kyverno to mutate Kubernetes workloads so those image pulls went through ECR automatically.

The bigger lesson was more interesting: Kyverno is not just a security tool. It is a way to ringfence a Kubernetes environment without making every application team become experts in every platform decision.

The Problem: Public Image Pulls Everywhere

The easy version of Kubernetes is one team, one cluster, and manifests you mostly control.

The real version is:

Multiple clusters across environments and regions.
Multiple development teams with different levels of Kubernetes experience.
Third-party Helm charts with their own defaults.
Public image references coming from Docker Hub, GHCR, Quay, registry.k8s.io, and other upstream registries.
Platform teams trying to keep the whole thing reliable and secure without becoming a bottleneck.

Direct public image pulls create a few problems.

Availability: if an upstream registry has issues, your workloads can be affected.
Rate limits: Docker Hub rate limits are an obvious example, but the general problem is depending on someone else’s limits.
Visibility: it is harder to understand which public dependencies are actually being pulled by your clusters.
Consistency: different teams solve the same problem in different ways.
Control: it is harder to apply scanning, lifecycle policy, permissions, and audit around public images.

You can ask every team to rewrite every image reference. That works once or twice. It does not work well at scale.

It also creates a bad platform experience. Developers just want to deploy a workload. They do not want to learn the exact private registry path for every upstream source before they can install a chart.

ECR Pull Through Cache

Amazon ECR pull through cache lets you sync images from an upstream registry into your private ECR registry.

Instead of pulling directly from Docker Hub, for example, a workload can pull from your ECR registry under a cache prefix. ECR then retrieves the upstream image when needed and serves it from your private registry on later pulls.

That gives you a better control point:

Public images are pulled through an AWS-owned registry in your account.
Cached repositories can use normal ECR features.
Repository creation templates can apply defaults to repositories created by the pull through cache flow.
Teams no longer need direct runtime dependency on every public registry.

It does not solve everything on its own though.

The Kubernetes manifests still say things like this:

image: nginx:1.25
image: docker.io/library/redis:7
image: ghcr.io/example/app:v1.0.0
image: registry.k8s.io/pause:3.9

For ECR pull through cache to be used, those references need to become ECR references.

You can do that manually in every chart and every manifest, but that quickly turns into toil. It also means application teams need to care about registry routing, which is really a platform concern.

This is where Kyverno fits nicely.

What Kyverno Is

Kyverno is a Kubernetes-native policy engine.

It runs in the cluster as a dynamic admission controller. When a resource is created or updated, the Kubernetes API server sends an admission review request to Kyverno. Kyverno checks the request against matching policies and can then allow it, reject it, mutate it, or report on it.

The important bit is that Kyverno policies are Kubernetes resources themselves.

You write YAML. You store it in Git. You apply it to the cluster. You manage it the same way you manage the rest of your platform configuration.

Kyverno can be used for:

Validation: block or report resources that do not meet your standards.
Mutation: change resources before they are stored in the cluster.
Generation: create supporting resources automatically.
Image verification: enforce signatures, attestations, or trusted image rules.
Policy reports: understand what is passing, failing, or being skipped.

That makes it very approachable if your platform is already built around Kubernetes and GitOps.

There are other policy engines out there, and some are extremely powerful. What I like about Kyverno is that it feels close to Kubernetes. You are not asking every platform engineer to learn a separate policy language before they can solve a practical admission control problem.

Why It Works Well as a Guardrail

Good guardrails do not exist to slow teams down.

They exist to make the correct path the easiest path.

That is the part I think Kyverno is especially good at. You can take a platform decision and move it closer to the API server, where the workload is actually admitted.

For example:

“All public image pulls should go through ECR.”
“All Pods should have resource requests and limits.”
“Privileged containers should not be allowed by default.”
“Ingresses should use approved classes.”
“Namespaces should have required labels.”
“Images should come from trusted registries.”

Without a policy engine, these rules often live in documentation, Terraform modules, Helm helpers, or Slack messages.

That is fragile.

With Kyverno, the rule becomes executable. It can be applied consistently across clusters. It can be rolled out in stages. It can be audited. It can be changed centrally when the platform changes.

This is what I mean by ringfencing a Kubernetes environment. Not locking teams out, but defining the boundaries clearly enough that teams can move quickly inside them.

The Mutation Pattern

The policy pattern for this project was:

Match Pods on create and update.
Look at the image references on each container.
Detect images from public upstream registries.
Rewrite those image references to the matching ECR pull through cache path.
Exclude system namespaces and anything that should not be touched.

At a high level, a Docker Hub image like this:

image: nginx:1.25

Can become something like this:

image: 111122223333.dkr.ecr.eu-west-1.amazonaws.com/docker-hub/library/nginx:1.25

And a Kubernetes registry image like this:

image: registry.k8s.io/pause:3.9

Can become:

image: 111122223333.dkr.ecr.eu-west-1.amazonaws.com/kubernetes/pause:3.9

The exact path depends on the ECR pull through cache rules you create. That mapping is the bit you need to design carefully.

A Simplified Kyverno Policy

This is a simplified example of the shape of the policy.

It is not something I would blindly copy into production, but it shows the important idea: inspect the image registry, then rewrite the image to the ECR cache path.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: rewrite-dockerhub-images-to-ecr-cache
  annotations:
    policies.kyverno.io/title: Rewrite Docker Hub Images to ECR Cache
    policies.kyverno.io/category: Supply Chain
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
spec:
  background: false
  rules:
    - name: rewrite-dockerhub-containers
      match:
        any:
          - resources:
              kinds:
                - Pod
              operations:
                - CREATE
                - UPDATE
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
      mutate:
        foreach:
          - list: request.object.spec.containers[]
            context:
              - name: imageData
                imageRegistry:
                  reference: "{{ element.image }}"
            preconditions:
              all:
                - key: "{{ imageData.registry }}"
                  operator: Equals
                  value: index.docker.io
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    image: 111122223333.dkr.ecr.eu-west-1.amazonaws.com/docker-hub/{{ imageData.repository }}:{{ imageData.identifier }}

In a real policy, you would normally handle more than just containers.

You would also think about:

initContainers.
ephemeralContainers, if you use them.
Docker Hub official images and the library/ prefix.
Images referenced by digest.
Which namespaces should be excluded.
Which registries should be rewritten.
Which registries should be rejected completely.

I like splitting this into a few focused policies rather than making one giant policy that handles every registry in one place. It is easier to test, easier to roll back, and easier for someone else to read six months later.

Mutate, Then Validate

Mutation is useful because it removes work from application teams.

Validation is useful because it gives you a hard boundary.

For this kind of setup, I like the idea of using both.

First, mutate known public image references to their ECR pull through cache equivalent. This keeps the developer experience clean. Teams can still install a chart that references nginx, redis, or registry.k8s.io/pause.

Then, add validation policies that reject image references you do not want in the cluster.

That gives you a flow like this:

Known public registry: mutate to ECR cache.
Approved private registry: allow.
Unknown external registry: reject or report.
Sensitive namespace: exclude or handle separately.

This is where Kyverno starts to feel like a proper platform control plane rather than just an admission webhook.

Rollout Matters

The dangerous way to introduce this is to go straight to enforcement across every cluster.

The better way is slower:

Start in a sandbox cluster.
Test with plain Pods, Deployments, Jobs, CronJobs, and Helm-installed workloads.
Use policy reports to see what would be touched.
Exclude the namespaces that should not be mutated.
Roll out to non-production clusters.
Watch image pull failures closely.
Move to production once the mapping is boring.

Boring is the goal here.

A policy like this should eventually disappear into the platform. Developers should not need to think about it unless something is genuinely wrong.

Sharp Edges

There are a few details worth calling out.

Docker Hub Official Images

Docker Hub official images are a special case.

People write:

image: nginx:1.25

But the full Docker Hub repository path is effectively:

image: docker.io/library/nginx:1.25

When pulling through ECR, that library/ path matters. If your mutation does not account for it, you can end up with image pull errors that look confusing at first.

Tags Are Still Tags

ECR pull through cache does not magically make mutable tags immutable.

If teams use latest, they still get the operational problems that come with latest. ECR can cache and refresh, but it does not turn a weak image reference into a strong one.

For production workloads, pinning by digest is still the cleaner approach.

First Pull Behaviour

The first pull may still need ECR to reach the upstream registry, depending on the upstream and your network setup.

After the image is cached, the behaviour is much nicer, but the first pull path is still something to test. This is especially important in private networking setups.

Debugging Can Be Surprising

If Kyverno mutates the Pod, the Pod image can differ from the image declared in the original Deployment or Helm values.

That is expected, but it is worth documenting for teams.

When someone runs:

kubectl get pod -o yaml

They may see the ECR image, even though their chart values still reference Docker Hub or another upstream registry.

That can be a good thing, but only if people know why it is happening.

Do Not Mutate Everything

Some namespaces should be left alone.

At minimum, I would be very careful with:

kube-system
kyverno
CNI namespaces
service mesh namespaces
monitoring and security tooling

That does not mean they can never be covered by policy. It means they deserve their own testing and rollout path.

Why This Matters at Scale

At small scale, you can solve this kind of thing with documentation.

At medium scale, you can solve it with templates and shared Helm values.

At larger scale, you need the platform itself to carry more of the burden.

Kyverno gives you a way to say:

This is allowed.
This is not allowed.
This should be changed automatically.
This should be reported before we enforce it.

That is extremely useful when you have many teams deploying to many clusters.

It also changes the relationship between platform teams and application teams. Instead of reviewing every manifest by hand or asking teams to remember every rule, the platform can apply the boring defaults automatically.

The best version of this is not heavy-handed. It is quiet and predictable.

Developers keep shipping. The platform keeps the boundary in place.

Recommendation

I would use this pattern when:

You run many Kubernetes clusters.
You have many teams deploying workloads.
You depend on public container registries.
You already use ECR heavily.
You want a central place to control registry routing.
You want guardrails without forcing every team to rewrite every chart.

I would be more cautious if:

You only have one or two clusters.
Your image sources are already tightly controlled.
You do not have a good test environment for admission policies.
You cannot tolerate any surprise during Pod admission.

For me, the main takeaway is that Kyverno is strongest when it is used to make good platform decisions automatic.

This project started as an image pull problem. It ended up being a good example of why Kubernetes-native policy matters.

ECR pull through cache gave us the registry control point.

Kyverno gave us the admission control point.

Together, they made the right behaviour the default.