[bug] Issues mounting extended disk in ppc64

[adrianriobo]

mapt issue: ibm-power: fix podman networking for nested containers in GitLab runner builds

Background

The install-glrunner.sh script configures podman networking for the IBM Power (ppc64le)
GitLab runners. Several configuration bugs cause DNS failures inside build containers,
particularly in jobs that run nested podman run calls from within the gitlab-runner build
container (e.g. bootstrap_and_onboard.sh pattern).

---

Issues found

1. executor-containers.conf — wrong default_subnet causes routing conflict

File: whatever generates executor-containers.conf (mounted into build containers as
/etc/containers/containers.conf:ro)

The file currently sets:

[network]
default_subnet = "192.168.100.0/24"

With network_mode = "podman" in config.toml (see issue 3 below), all gitlab-runner build
containers are attached to the host's podman network at 192.168.100.0/24. When a build
container's job script runs a nested podman run, podman inside the build container reads
this containers.conf and tries to create a bridge at 192.168.100.1/24 inside the build
container's network namespace — the same subnet as the build container's own eth0.
This creates a conflicting route inside the build container, breaking all outbound traffic
from nested containers including DNS.

Fix: use a subnet that does not conflict with the host's podman network:

[network]
default_subnet = "172.20.0.0/24"

---

2. executor-containers.conf — no public DNS fallback

The file only lists IT-internal DNS resolvers:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69"]
dns_options = ["timeout:2", "attempts:5", "single-request"]

If the internal resolvers are temporarily unreachable (VPN/TGW hiccup), DNS fails
completely in nested containers with no fallback. attempts:5 also makes the worst-case
timeout 30 seconds before failure.

Fix:

dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

---

3. config.toml — missing network_mode causes concurrent bridge race condition

With concurrent = 2, gitlab-runner creates two per-job bridge networks simultaneously.
The second bridge's nftables masquerade rules may not be applied before its containers
start sending traffic, causing intermittent DNS timeouts under concurrent load.

Fix: add to the [runners.docker] section:

network_mode = "podman"

This makes all job containers share the existing host podman network instead of creating
per-job bridges, eliminating the race condition entirely.

---

4. config.toml — missing explicit DNS for build containers

Without dns = in config.toml, gitlab-runner passes an empty DNS list via the Docker API,
which may override the host's containers.conf dns_servers setting for build containers.

Fix: add to [runners.docker]:

dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]

---

5. config.toml — missing MTU setting

PowerVS network interfaces have an MTU of 1450 (lower than the default 1500). Without
setting this, large packets (e.g. during image pulls) may be silently dropped.

Fix: add to [runners.docker]:

network_mtu = 1450

---

Required final configuration

executor-containers.conf

[containers]
log_driver = "journald"
pids_limit = -1
dns_servers = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
dns_options = ["timeout:2", "attempts:3", "single-request"]

[network]
default_subnet = "172.20.0.0/24"

config.toml — [runners.docker] section

[runners.docker]
  host = "unix:///run/podman/podman.sock"
  privileged = true
  network_mode = "podman"
  network_mtu = 1450
  dns = ["10.130.98.132", "10.130.98.5", "10.130.98.69", "8.8.8.8"]
  volumes = ["/etc/containers/executor-containers.conf:/etc/containers/containers.conf:ro", "/cache"]

---

Notes

• The IT DNS resolvers (10.130.98.x) above are for us-south. Other regions use
  different addresses — the install script should parameterise these per region:

  Region	DNS resolvers
  us-east	10.130.64.4, 10.130.64.70, 10.130.64.134
  us-south	10.130.98.5, 10.130.98.69, 10.130.98.132
  eu-de	10.130.128.5, 10.130.128.69, 10.130.128.133
  lon	10.130.129.4, 10.130.129.68, 10.130.129.132

• The 172.20.0.0/24 subnet for executor-containers.conf must not conflict with the
  host's podman network (192.168.100.0/24), the PowerVS subnet, or the 10.0.0.0/8
  internal route.

• All fixes were validated on live dal14 ppc64le runners (aipcc-pro-gl-ppc64le-1..4).
  Post-fix jobs ran successfully including nested podman run workloads
  (bootstrap-and-onboard jobs across all 4 runners, 2 confirmed successe