Troubles with Istio

Istio would not inject the side-car into a namespace labeled for the injection, for example:

kubectl label namespace default istio-injection=enabled

a redeployment would fail when forced:

for i in $(kubectl get deployments.apps -n argocd \
     | tail -n +2 | awk '{print $1}'); \
      do kubectl -n argocd patch deployment $i \
       -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": { \"redeploy\": \"$(date +%s)\"}}}}}" ; done

the istio-init would fail, as seen in the logs of the pod:

Status:           Pending
...
IPs:
...
Controlled By:  ReplicaSet/nginx-deployment-6d878f6fc7
Init Containers:
  istio-init:
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
...

with a really sparse Error output:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  57s                default-scheduler  Successfully assigned default/nginx-deployment-6d878f6fc7-qmc96 to polyxena
  Normal   Pulled     14s (x4 over 56s)  kubelet            Container image "docker.io/istio/proxyv2:1.16.1" already present on machine
  Normal   Created    14s (x4 over 56s)  kubelet            Created container istio-init
  Normal   Started    14s (x4 over 56s)  kubelet            Started container istio-init
  Warning  BackOff    1s (x6 over 55s)   kubelet            Back-off restarting failed container istio-init in pod nginx-deployment-6d878f6fc7-qmc96_default(a0679447-ced1-4669-b9d9-0d310c26da2b)

Fortunately, the very container info is a bit more verbose:

kubectl logs -f pods/nginx-deployment-6d833f6fc7-qmc96 -c istio-init

2023-01-24T15:37:52.475392Z	info	Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=

2023-01-24T15:37:52.475441Z	info	Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
OUTPUT_PATH=
NETWORK_NAMESPACE=
CNI_MODE=false
HOST_NSENTER_EXEC=false
EXCLUDE_INTERFACES=

2023-01-24T15:37:52.475703Z	info	Writing following contents to rules file: /tmp/iptables-rules-1674574672475491790.txt2488634600
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2023-01-24T15:37:52.475767Z	info	Running command: iptables-restore --noflush /tmp/iptables-rules-1674574672475491790.txt2488634600
2023-01-24T15:37:52.477933Z	error	Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2023-01-24T15:37:52.477972Z	error	Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1674574672475491790.txt2488634600, exit status 2

So, an IPTables-related issue, after all?

Yes, RockyLinux, used for this BareMetal Kubernetes cluster on worker nodes, changed a few details in 9.0 .

User zackzhangkai noticed it well on the GitHub and a simple loading of the required modules sorts the problem even after a reboot of the worker nodes:

cat files/99-istio-modules.conf 
# These modules need to be loaded on boot so that Istio (as required b
# Kubeflow) runs properly.
#
# See also: https://github.com/istio/istio/issues/23009

br_netfilter
nf_nat
xt_REDIRECT
xt_owner
iptable_nat
iptable_mangle
iptable_filter


ansible all -i inventory --become --limit='newkubes' -m copy -a 'src=99-istio-modules.conf dest=/etc/modules-load.d/99-istio-modules.conf owner=root group=root mode=0640'

After a reboot, and a redeployment, the side-car gets injected as expected!