MetalLB in a Raspberry PI cluster

After a recent installation of a Multi-master Istio mesh on two clusters, I decided to add the third cluster in the mesh, one completely based on RaspberryPIs. All worked well until I added the MetalLB (L2 mode) for the LoadBalancers, that, unlike the first two clusters, wasn’t working properly: all was there – the IPs were issued straight from the assigned pool, Istio could see all of the endpoints, and LoadBalancing apparently worked, and the pods in the newest cluster could communicate with the services in other two clusters via their internal “east-west” Istio Gateways, while still letting nothing in through the “ingressgateway”.

Nmap showed filtered ports of the Gateways and LoadBalancers.
I could only figure out using tcpdump that there were no ARP answers when Istio GW was queried.
Everything else remained cryptic to me, like messages:

Error from server InternalError error when creating metallb-pool.yaml Internal error occurred failed calling webhook ipaddresspoolvalidationwebhook.metallb.io failed to call webhook Post https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s dial tcp 10.101.81.237:443: connect: no route to host

and

Error from server InternalError metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool connect: no route to host

After a long bug-hunt, changing to legacy iptables and arptables, changings of CNIs (Flannel, Calico, Canal…) attempts to see if there was a difference with cri-socket (docker, docker+containerd, cri-o…) installations via Helm, tryouts of FFR with BGP server, and what not, I finally tried 0.13.12 instead of MetalLB 0.15.3, and it worked!
Apparently, somewhere in between, MetalLB started strictly respecting the exclusion labels of the nodes, that at the moment held the dreadful label:

node.kubernetes.io/exclude-from-external-load-balancers

After the label removal from the nodes:

for i in $(kubectl get nodes --no-headers | awk '{print $1}') ; do kubectl label node $i node.kubernetes.io/exclude-from-external-load-balancers=- ; done

the latest (at the moment of writing) MetalLB finally started working, and Multi-mesh automagically became fully operational:

I hope this hint saves you a day or two, I know now that it would do so for me…