Added IMDS emulator container.

docs/storage-transfer.md

@@ -27,7 +27,7 @@ A transfer from a filesystem is agent based. It involves three identities:
 | Identity | Principal | Purpose | Roles |
 |---|---|---|---|
 | Job creator | `sa-test` through the `wif-test` configuration | Creates and runs transfer jobs | `roles/storagetransfer.user`, `roles/iam.roleViewer` |
-| Transfer agent | `sa-test` through Application Default Credentials (ADC) with `/etc/sts-agent/cred.json` in the agent container | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
+| Transfer agent | `sa-test` through tokens of the IMDS emulator, which authenticates with Application Default Credentials (ADC) and `/etc/sts-agent/cred.json` | Reads the files and writes the objects | `roles/storagetransfer.transferAgent`, `roles/storage.objectAdmin` on the bucket |
 | Service agent | `project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com` | Manages the Pub/Sub topics of the pool, checks the transferred objects and writes the transfer logs | `roles/storagetransfer.serviceAgent`, `roles/storage.legacyBucketReader`, `roles/storage.objectViewer` and `roles/storage.objectCreator` on the bucket |
 
 The service agent is managed by Google and exists once per project. It cannot be replaced with a user-managed service account in agent based transfers.
@@ -95,31 +95,37 @@ The agent reads the certificate and key from the mounted directory, so certifica
 
 Role: **Server administrator**
 
-The agent container runs with the following settings:
+The agent cannot use the X.509 credentials itself. Its launcher is a statically linked binary that cannot load the `cryptography` package, which certificate-based authentication requires, and the `--creds-file` option accepts only a service account key or user credentials. The agent therefore gets its tokens from the IMDS emulator, a Compute Engine metadata server emulator in a second container. The emulator authenticates with ADC and the X.509 credentials in `/etc/sts-agent`, and serves the access tokens of `sa-test` on the metadata token endpoint. Without `GOOGLE_APPLICATION_CREDENTIALS`, ADC in the agent falls back to the metadata server, which `GCE_METADATA_HOST` and `GCE_METADATA_IP` point to the emulator.
 
-| Setting | Value |
-|---|---|
-| Image | built from `sts-agent/Dockerfile` on `gcr.io/cloud-ingest/tsop-agent:latest` |
-| Mount | `/mnt/ext-storage`, read only |
-| Mount | `/etc/sts-agent`, read only |
-| Environment | `GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json` |
-| Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json` |
-| Locked memory limit | `64000000` bytes |
-| Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
+The containers run with the following settings:
 
-`sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env` into the agent arguments; they are not passed to the container as environment variables. Create the file with the values of `env.sh` and the host name printed by `hostname`:
+| Container | Setting | Value |
+|---|---|---|
+| `imds-emulator` | Image | built from `imds-emulator/Dockerfile` |
+| | Mount | `/etc/sts-agent`, read only |
+| | Environment | `GOOGLE_APPLICATION_CREDENTIALS=/etc/sts-agent/cred.json` |
+| | Environment | `GOOGLE_API_CERTIFICATE_CONFIG=/etc/sts-agent/certificate_config.json` |
+| | Environment | `PROJECT_ID`, `PROJECT_NUMBER` |
+| `sts-agent` | Image | `gcr.io/cloud-ingest/tsop-agent:latest` |
+| | Mount | `/mnt/ext-storage`, read only |
+| | Environment | `GCE_METADATA_HOST=imds-emulator`, `GCE_METADATA_IP=imds-emulator` |
+| | Locked memory limit | `64000000` bytes |
+| | Arguments | `--project-id=${PROJECT_ID} --hostname=$(hostname) --agent-pool=${STS_AGENT_POOL}` |
+
+The emulator publishes no port; only the containers of the Compose project reach it. The agent container must not set `GOOGLE_API_CERTIFICATE_CONFIG`, because the metadata token code of the agent parses the certificate it points to with `cryptography`.
+
+`sts-agent/compose.yaml` holds these settings for Docker Compose. Compose substitutes `PROJECT_ID`, `PROJECT_NUMBER`, `STS_AGENT_POOL` and `HOSTNAME` from `sts-agent/.env`. Create the file with the values of `env.sh` and the host name printed by `hostname`:
 
 `sts-agent/.env`:
 
 ```bash
 PROJECT_ID=my-project
+PROJECT_NUMBER=123456789012
 STS_AGENT_POOL=my-agents
 HOSTNAME=agent-host
 ```
 
-The image of Google lacks the shared glibc libraries that the `cryptography` package needs for certificate-based authentication. The agent launcher is statically linked with GRTE v5, the Google Run Time Environment based on glibc 2.27, which loads shared libraries from `/usr/grte/v5/lib64`. The libraries must come from the same GRTE build, because GRTE changes internal structures of the dynamic loader. The first stage of `sts-agent/Dockerfile` builds them from the GRTE v5 source at `https://third-party-mirror.googlesource.com/GRTEv5` on `ubuntu:18.04`, and the second stage adds them to the image of Google.
-
-Build the image and start the agent:
+Build the emulator and start both containers:
 
 ```bash
 cd sts-agent
@@ -127,9 +133,24 @@ docker compose build
 docker compose up -d
 ```
 
-The directories are mounted at the same paths as on the host. The transfer job refers to the source directory by that path, and `certificate_config.json` holds the host paths of the certificate and key.
+The emulator logs one line per request with the path and the status, never the token:
 
-The agent authenticates with ADC, not with `--creds-file`. The `--creds-file` option accepts only a service account key or user credentials, not a Workload Identity Federation credential configuration.
+```bash
+docker compose logs imds-emulator
+```
+
+Requests for paths that the emulator does not serve are logged with the status 404.
+
+`imds-test` in the emulator container tests the emulator. It checks the metadata endpoints and the rejection of requests without the `Metadata-Flavor` header or with `X-Forwarded-For`. It then authenticates with the Compute Engine credentials of `google-auth` against the emulator, the same way as the agent, and checks the token with the Google token information endpoint. With `--bucket`, it also lists the objects of the bucket with the token:
+
+```bash
+source ../env.sh
+docker compose exec imds-emulator imds-test --bucket "${BUCKET}"
+```
+
+It prints `PASS` or `FAIL` for each check and exits with status 1 when a check fails.
+
+The source directory is mounted at the same path as on the host, because the transfer job refers to it by that path.
 
 The connected agents of the pool are listed on the Agent pools page of the Google Cloud console. An agent appears there only after it has authenticated.
 

imds-emulator/Dockerfile

@@ -0,0 +1,8 @@
+FROM python:3.13-slim
+
+RUN pip install --no-cache-dir "google-auth[requests,cryptography]>=2.39.0"
+
+COPY imds_emulator.py /usr/local/bin/imds-emulator
+COPY imds_test.py /usr/local/bin/imds-test
+
+CMD ["/usr/local/bin/imds-emulator"]

imds-emulator/imds_emulator.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python3
+"""
+Compute Engine metadata server (IMDS) emulator.
+
+Serves access tokens of the credentials found by Application Default
+Credentials (ADC) on the IMDS token endpoint, together with the metadata
+that Google client libraries read when they authenticate on Compute Engine.
+"""
+import datetime
+import http.server
+import json
+import os
+import socket
+import sys
+import threading
+import urllib.parse
+
+import google.auth
+import google.auth.credentials
+import google.auth.transport.requests
+
+DEFAULT_SCOPE = "https://www.googleapis.com/auth/cloud-platform"
+ROOT = "/computeMetadata/v1/"
+
+port = int(os.environ.get("IMDS_PORT", "80"))
+hostname = os.environ.get("IMDS_HOSTNAME", socket.gethostname())
+project_number = os.environ.get("PROJECT_NUMBER", "")
+
+base_credentials, adc_project_id = google.auth.default(scopes=[DEFAULT_SCOPE])
+project_id = os.environ.get("PROJECT_ID") or adc_project_id or ""
+email = getattr(base_credentials, "service_account_email", None) or os.environ.get("SERVICE_ACCOUNT_EMAIL", "")
+if not email:
+    sys.exit("The service account email is unknown, set SERVICE_ACCOUNT_EMAIL")
+
+credentials_by_scopes = {}
+lock = threading.Lock()
+
+
+def token(scopes):
+    """
+    Returns the access token and its remaining lifetime in seconds for the
+    scopes, refreshing the cached credentials of the scopes when needed.
+    """
+    key = tuple(sorted(scopes))
+    with lock:
+        credentials = credentials_by_scopes.get(key)
+        if credentials is None:
+            credentials = google.auth.credentials.with_scopes_if_required(base_credentials, list(key))
+            credentials_by_scopes[key] = credentials
+        if not credentials.valid:
+            credentials.refresh(google.auth.transport.requests.Request())
+        expires_in = int((credentials.expiry - datetime.datetime.now(datetime.timezone.utc).replace(tzinfo=None)).total_seconds())
+        return credentials.token, expires_in
+
+
+class Handler(http.server.BaseHTTPRequestHandler):
+    """
+    Answers the IMDS paths used by the Google client libraries, and 404 for
+    any other path.
+    """
+
+    server_version = "Metadata Server for VM"
+    sys_version = ""
+
+    def reply(self, status, body, content_type="application/text"):
+        data = body.encode() if isinstance(body, str) else json.dumps(body).encode()
+        self.send_response(status)
+        self.send_header("Metadata-Flavor", "Google")
+        self.send_header("Content-Type", content_type)
+        self.send_header("Content-Length", str(len(data)))
+        self.end_headers()
+        self.wfile.write(data)
+
+    def do_GET(self):
+        url = urllib.parse.urlsplit(self.path)
+        query = urllib.parse.parse_qs(url.query)
+
+        # The real IMDS rejects requests without the flavor header and proxied requests.
+        if self.headers.get("Metadata-Flavor") != "Google" or self.headers.get("X-Forwarded-For"):
+            return self.reply(403, "Forbidden")
+
+        if url.path in ("/", ROOT):
+            return self.reply(200, "")
+
+        if not url.path.startswith(ROOT):
+            return self.reply(404, "Not Found")
+        path = url.path[len(ROOT):]
+
+        # Both "default" and the email address name the service account.
+        for alias in ("default", email):
+            prefix = f"instance/service-accounts/{alias}/"
+            if path.startswith(prefix):
+                path = "instance/service-accounts/default/" + path[len(prefix):]
+                break
+
+        account = {"aliases": ["default"], "email": email, "scopes": [DEFAULT_SCOPE]}
+        if path == "instance/service-accounts/default/token":
+            scopes = query.get("scopes", [DEFAULT_SCOPE])[0].split(",")
+            access_token, expires_in = token(scopes)
+            return self.reply(200, {"access_token": access_token, "expires_in": expires_in, "token_type": "Bearer"}, "application/json")
+        if path == "instance/service-accounts/default/email":
+            return self.reply(200, email)
+        if path == "instance/service-accounts/default/scopes":
+            return self.reply(200, DEFAULT_SCOPE + "\n")
+        if path == "instance/service-accounts/default/" and query.get("recursive") == ["true"]:
+            return self.reply(200, account, "application/json")
+        if path == "instance/service-accounts/" and query.get("recursive") == ["true"]:
+            return self.reply(200, {"default": account, email: account}, "application/json")
+        if path == "instance/service-accounts/":
+            return self.reply(200, f"default/\n{email}/\n")
+        if path == "project/project-id" and project_id:
+            return self.reply(200, project_id)
+        if path == "project/numeric-project-id" and project_number:
+            return self.reply(200, project_number)
+        if path == "instance/hostname":
+            return self.reply(200, hostname)
+        if path == "universe/universe-domain":
+            return self.reply(200, "googleapis.com")
+
+        return self.reply(404, "Not Found")
+
+    def log_request(self, code="-", size="-"):
+        print(f"{self.command} {urllib.parse.urlsplit(self.path).path} {code}", flush=True)
+
+
+print(f"Serving tokens of {email} for project {project_id or '-'} on port {port}", flush=True)
+http.server.ThreadingHTTPServer(("", port), Handler).serve_forever()

imds-emulator/imds_test.py

@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+"""
+Tests the IMDS emulator: the metadata endpoints, the request checks, the
+Compute Engine credentials of google-auth on top of the emulator, and the
+validity of the issued access token at Google.
+"""
+import argparse
+import json
+import os
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+parser = argparse.ArgumentParser(description="Test the IMDS emulator.")
+parser.add_argument("--host", default="localhost", help="host and optional port of the emulator")
+parser.add_argument("--bucket", help="bucket to list with the token, optional")
+args = parser.parse_args()
+
+root = f"http://{args.host}/computeMetadata/v1/"
+failures = 0
+
+
+def check(name, condition, detail=""):
+    """
+    Prints the result of a check and counts the failures.
+    """
+    global failures
+    print(f"{'PASS' if condition else 'FAIL'} {name}{': ' + detail if detail else ''}")
+    if not condition:
+        failures += 1
+
+
+def get(url, headers=None):
+    """
+    Returns the status, the headers and the body of a GET request.
+    """
+    request = urllib.request.Request(url, headers=headers or {})
+    try:
+        with urllib.request.urlopen(request, timeout=10) as response:
+            return response.status, response.headers, response.read().decode()
+    except urllib.error.HTTPError as error:
+        return error.code, error.headers, error.read().decode()
+
+
+flavor = {"Metadata-Flavor": "Google"}
+
+# Detection ping and request checks
+status, headers, _ = get(f"http://{args.host}/", flavor)
+check("ping", status == 200 and headers.get("Metadata-Flavor") == "Google", f"status {status}")
+status, _, _ = get(root + "project/project-id")
+check("request without Metadata-Flavor is rejected", status == 403, f"status {status}")
+status, _, _ = get(root + "project/project-id", {**flavor, "X-Forwarded-For": "192.0.2.1"})
+check("request with X-Forwarded-For is rejected", status == 403, f"status {status}")
+
+# Metadata
+status, _, project_id = get(root + "project/project-id", flavor)
+check("project ID", status == 200 and project_id != "", project_id)
+status, _, email = get(root + "instance/service-accounts/default/email", flavor)
+check("service account email", status == 200 and "@" in email, email)
+status, _, body = get(root + "instance/service-accounts/default/?recursive=true", flavor)
+check("service account info", status == 200 and json.loads(body).get("email") == email, body)
+status, _, body = get(root + f"instance/service-accounts/{email}/email", flavor)
+check("service account by email", status == 200 and body == email, body)
+
+# Token
+status, _, body = get(root + "instance/service-accounts/default/token", flavor)
+token = json.loads(body) if status == 200 else {}
+check("token", status == 200 and token.get("access_token") and token.get("expires_in", 0) > 0, f"expires in {token.get('expires_in')} s")
+
+# Compute Engine credentials of google-auth, the path used by the STS agent
+for name in ("GOOGLE_APPLICATION_CREDENTIALS", "GOOGLE_API_CERTIFICATE_CONFIG"):
+    os.environ.pop(name, None)
+os.environ["GCE_METADATA_HOST"] = args.host
+os.environ["GCE_METADATA_IP"] = args.host
+
+import google.auth
+import google.auth.transport.requests
+
+credentials, adc_project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
+credentials.refresh(google.auth.transport.requests.Request())
+check("google-auth uses Compute Engine credentials", type(credentials).__module__ == "google.auth.compute_engine.credentials", type(credentials).__module__)
+check("google-auth project ID", adc_project_id == project_id, str(adc_project_id))
+check("google-auth service account", credentials.service_account_email == email, credentials.service_account_email)
+check("google-auth token", credentials.valid, f"expires {credentials.expiry} UTC")
+
+# Validity of the token at Google
+status, _, body = get("https://oauth2.googleapis.com/tokeninfo?" + urllib.parse.urlencode({"access_token": credentials.token}))
+info = json.loads(body) if status == 200 else {}
+check("token accepted by Google", status == 200, f"scope {info.get('scope')}, expires in {info.get('expires_in')} s")
+
+# Use of the token with Cloud Storage
+if args.bucket:
+    status, _, body = get(
+        f"https://storage.googleapis.com/storage/v1/b/{urllib.parse.quote(args.bucket)}/o?maxResults=1",
+        {"Authorization": f"Bearer {credentials.token}"},
+    )
+    check(f"list objects of {args.bucket}", status == 200, f"status {status}")
+
+print(f"{failures} check(s) failed" if failures else "All checks passed")
+sys.exit(1 if failures else 0)

sts-agent/Dockerfile

@@ -1,25 +0,0 @@
-FROM ubuntu:18.04 AS grte
-
-ARG DEBIAN_FRONTEND=noninteractive
-RUN apt-get update \
-    && apt-get install -y --no-install-recommends build-essential gawk bison python3 git texinfo ca-certificates \
-    && rm -rf /var/lib/apt/lists/*
-
-# GRTE v5 source, Google's glibc 2.27
-RUN git clone --depth 1 https://third-party-mirror.googlesource.com/GRTEv5 /src
-
-# Build the libraries into the library directory searched by the GRTE v5 loader
-WORKDIR /build
-RUN printf 'slibdir=/usr/grte/v5/lib64\nrtlddir=/usr/grte/v5/lib64\n' > configparms \
-    && /src/google3/third_party/grte/v5_src/glibc-2.27/configure \
-        --prefix=/usr/grte/v5 \
-        --libdir=/usr/grte/v5/lib64 \
-        --disable-werror \
-        --enable-kernel=3.2 \
-    && make -j"$(nproc)" \
-    && make install DESTDIR=/out
-
-FROM gcr.io/cloud-ingest/tsop-agent:latest
-
-# GRTE v5 shared libraries matching the GRTE v5 glibc the launcher is statically linked with.
-COPY --from=grte /out/usr/grte/v5/lib64/ /usr/grte/v5/lib64/

sts-agent/compose.yaml

@@ -1,14 +1,25 @@
 services:
-  sts-agent:
-    build: .
-    ulimits:
-      memlock: 64000000
+  imds-emulator:
+    build: ../imds-emulator
     volumes:
-      - /mnt/ext-storage:/mnt/ext-storage:ro
       - /etc/sts-agent:/etc/sts-agent:ro
     environment:
       GOOGLE_APPLICATION_CREDENTIALS: /etc/sts-agent/cred.json
       GOOGLE_API_CERTIFICATE_CONFIG: /etc/sts-agent/certificate_config.json
+      PROJECT_ID: ${PROJECT_ID}
+      PROJECT_NUMBER: ${PROJECT_NUMBER}
+
+  sts-agent:
+    image: gcr.io/cloud-ingest/tsop-agent:latest
+    depends_on:
+      - imds-emulator
+    ulimits:
+      memlock: 64000000
+    volumes:
+      - /mnt/ext-storage:/mnt/ext-storage:ro
+    environment:
+      GCE_METADATA_HOST: imds-emulator
+      GCE_METADATA_IP: imds-emulator
     command:
       - --project-id=${PROJECT_ID}
       - --hostname=${HOSTNAME}