{"id":167,"date":"2023-01-24T17:10:03","date_gmt":"2023-01-24T17:10:03","guid":{"rendered":"https:\/\/blog.simulakrum.vpndns.org\/?p=167"},"modified":"2023-01-24T17:14:54","modified_gmt":"2023-01-24T17:14:54","slug":"troubles-with-istio","status":"publish","type":"post","link":"https:\/\/blog.simulakrum.vpndns.org\/?p=167","title":{"rendered":"Troubles with Istio"},"content":{"rendered":"\n<p>Istio would not inject the side-car into a namespace labeled for the injection, for example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl label namespace default istio-injection=enabled<\/code><\/pre>\n\n\n\n<p>a redeployment would fail when forced:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>for i in $(kubectl get deployments.apps -n argocd \\\n     | tail -n +2 | awk '{print $1}'); \\\n      do kubectl -n argocd patch deployment $i \\\n       -p \"{\\\"spec\\\": {\\\"template\\\": {\\\"metadata\\\": { \\\"labels\\\": { \\\"redeploy\\\": \\\"$(date +%s)\\\"}}}}}\" ; done<\/code><\/pre>\n\n\n\n<p>the istio-init would fail, as seen in the logs of the pod:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Status:           Pending\n...\nIPs:\n...\nControlled By:  ReplicaSet\/nginx-deployment-6d878f6fc7\nInit Containers:\n  istio-init:\n...\n    State:          Waiting\n      Reason:       CrashLoopBackOff\n    Last State:     Terminated\n      Reason:       Error\n      Exit Code:    255\n...<\/code><\/pre>\n\n\n\n<p>with a really sparse Error output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Events:\n  Type     Reason     Age                From               Message\n  ----     ------     ----               ----               -------\n  Normal   Scheduled  57s                default-scheduler  Successfully assigned default\/nginx-deployment-6d878f6fc7-qmc96 to polyxena\n  Normal   Pulled     14s (x4 over 56s)  kubelet            Container image \"docker.io\/istio\/proxyv2:1.16.1\" already present on machine\n  Normal   Created    14s (x4 over 56s)  kubelet            Created container istio-init\n  Normal   Started    14s (x4 over 56s)  kubelet            Started container istio-init\n  Warning  BackOff    1s (x6 over 55s)   kubelet            Back-off restarting failed container istio-init in pod nginx-deployment-6d878f6fc7-qmc96_default(a0679447-ced1-4669-b9d9-0d310c26da2b)<\/code><\/pre>\n\n\n\n<p>Fortunately, the very container info is a bit more verbose:<br> <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>kubectl logs -f pods\/nginx-deployment-6d833f6fc7-qmc96 -c istio-init\n\n2023-01-24T15:37:52.475392Z\tinfo\tIstio iptables environment:\nENVOY_PORT=\nINBOUND_CAPTURE_PORT=\nISTIO_INBOUND_INTERCEPTION_MODE=\nISTIO_INBOUND_TPROXY_ROUTE_TABLE=\nISTIO_INBOUND_PORTS=\nISTIO_OUTBOUND_PORTS=\nISTIO_LOCAL_EXCLUDE_PORTS=\nISTIO_EXCLUDE_INTERFACES=\nISTIO_SERVICE_CIDR=\nISTIO_SERVICE_EXCLUDE_CIDR=\nISTIO_META_DNS_CAPTURE=\nINVALID_DROP=\n\n2023-01-24T15:37:52.475441Z\tinfo\tIstio iptables variables:\nPROXY_PORT=15001\nPROXY_INBOUND_CAPTURE_PORT=15006\nPROXY_TUNNEL_PORT=15008\nPROXY_UID=1337\nPROXY_GID=1337\nINBOUND_INTERCEPTION_MODE=REDIRECT\nINBOUND_TPROXY_MARK=1337\nINBOUND_TPROXY_ROUTE_TABLE=133\nINBOUND_PORTS_INCLUDE=*\nINBOUND_PORTS_EXCLUDE=15090,15021,15020\nOUTBOUND_OWNER_GROUPS_INCLUDE=*\nOUTBOUND_OWNER_GROUPS_EXCLUDE=\nOUTBOUND_IP_RANGES_INCLUDE=*\nOUTBOUND_IP_RANGES_EXCLUDE=\nOUTBOUND_PORTS_INCLUDE=\nOUTBOUND_PORTS_EXCLUDE=\nKUBE_VIRT_INTERFACES=\nENABLE_INBOUND_IPV6=false\nDNS_CAPTURE=false\nDROP_INVALID=false\nCAPTURE_ALL_DNS=false\nDNS_SERVERS=&#91;],&#91;]\nOUTPUT_PATH=\nNETWORK_NAMESPACE=\nCNI_MODE=false\nHOST_NSENTER_EXEC=false\nEXCLUDE_INTERFACES=\n\n2023-01-24T15:37:52.475703Z\tinfo\tWriting following contents to rules file: \/tmp\/iptables-rules-1674574672475491790.txt2488634600\n* nat\n-N ISTIO_INBOUND\n-N ISTIO_REDIRECT\n-N ISTIO_IN_REDIRECT\n-N ISTIO_OUTPUT\n-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN\n-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001\n-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006\n-A PREROUTING -p tcp -j ISTIO_INBOUND\n-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN\n-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN\n-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN\n-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT\n-A OUTPUT -p tcp -j ISTIO_OUTPUT\n-A ISTIO_OUTPUT -o lo -s 127.0.0.6\/32 -j RETURN\n-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1\/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT\n-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN\n-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN\n-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1\/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT\n-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN\n-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN\n-A ISTIO_OUTPUT -d 127.0.0.1\/32 -j RETURN\n-A ISTIO_OUTPUT -j ISTIO_REDIRECT\nCOMMIT\n2023-01-24T15:37:52.475767Z\tinfo\tRunning command: iptables-restore --noflush \/tmp\/iptables-rules-1674574672475491790.txt2488634600\n2023-01-24T15:37:52.477933Z\terror\tCommand error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'\n\nError occurred at line: 1\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n2023-01-24T15:37:52.477972Z\terror\tFailed to execute: iptables-restore --noflush \/tmp\/iptables-rules-1674574672475491790.txt2488634600, exit status 2\n<\/code><\/pre>\n\n\n\n<p>So, an IPTables-related issue, after all? <br><br>Yes, RockyLinux, used for this BareMetal Kubernetes cluster on worker nodes, changed a few details in 9.0 .<br><br>User zackzhangkai noticed it well on the <a rel=\"noreferrer noopener\" href=\"https:\/\/github.com\/istio\/istio\/issues\/23009#issuecomment-980864913\" target=\"_blank\">GitHub<\/a> and a simple loading of the required modules sorts the problem even after a reboot of the worker nodes:<br><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cat files\/99-istio-modules.conf \n# These modules need to be loaded on boot so that Istio (as required b\n# Kubeflow) runs properly.\n#\n# See also: https:\/\/github.com\/istio\/istio\/issues\/23009\n\nbr_netfilter\nnf_nat\nxt_REDIRECT\nxt_owner\niptable_nat\niptable_mangle\niptable_filter\n\n\nansible all -i inventory --become --limit='newkubes' -m copy -a 'src=99-istio-modules.conf dest=\/etc\/modules-load.d\/99-istio-modules.conf owner=root group=root mode=0640'<\/code><\/pre>\n\n\n\n<p>After a reboot, and a redeployment, the side-car gets injected as expected!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Istio would not inject the side-car into a namespace labeled for the injection, for example: a redeployment would fail when forced: the istio-init would fail, as seen in the logs of the pod: with a really sparse Error output: Fortunately, the very container info is a bit more verbose: So, an IPTables-related issue, after all? Yes, RockyLinux, used for this &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-167","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/posts\/167","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=167"}],"version-history":[{"count":3,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/posts\/167\/revisions"}],"predecessor-version":[{"id":170,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=\/wp\/v2\/posts\/167\/revisions\/170"}],"wp:attachment":[{"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=167"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=167"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.simulakrum.vpndns.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=167"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}