Kubernetes

Try to deploy Argo on GCP Autopilot

Creating an Autopilot cluster in GCP K8S is quite easy. But after deploying Argo and launching our pipeline, the Argo report errors:

Failed to pull image "eu-docker.pkg.dev/project-name/mytag:123456789"
b85d23bf513ba037f4b2fbd5e": rpc error: code = Unknown desc = failed to pull and unpack image eu-docker.pkg.dev/project-name/mytag:123456789": failed to resolve reference "eu-docker.pkg.dev/project-name/mytag:123456789": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Console
​x
 
Failed to pull image "eu-docker.pkg.dev/project-name/mytag:123456789"
b85d23bf513ba037f4b2fbd5e": rpc error: code = Unknown desc = failed to pull and unpack image eu-docker.pkg.dev/project-name/mytag:123456789": failed to resolve reference "eu-docker.pkg.dev/project-name/mytag:123456789": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

The solution is (give k8s cluster the permission to pull docker image from our docker repository):

kubectl create secret docker-registry gcr-json-key  --docker-server=eu-docker.pkg.dev  --docker-username=_json_key  --docker-password="$(cat our_service_account.json)"  --docker-email=your@email.address -n argo
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}' -n argo

Console
 
kubectl create secret docker-registry gcr-json-key  --docker-server=eu-docker.pkg.dev  --docker-username=_json_key  --docker-password="$(cat our_service_account.json)"  --docker-email=your@email.address -n argo
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}' -n argo

Then the second problem jumped out:

admission webhook "validation.gatekeeper.sh" deni
ed the request: [denied by autogke-no-write-mode-hostpath] hostPath volume docker-sock used in container wait
uses path /var/run/docker.sock which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes a
re: ["/var/log/"]. Requesting user: <system:serviceaccount:argo:argo> and groups: <["system:serviceaccounts",
"system:serviceaccounts:argo", "system:authenticated"]>

Console
 
admission webhook "validation.gatekeeper.sh" deni
ed the request: [denied by autogke-no-write-mode-hostpath] hostPath volume docker-sock used in container wait
uses path /var/run/docker.sock which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes a
re: ["/var/log/"]. Requesting user: <system:serviceaccount:argo:argo> and groups: <["system:serviceaccounts",
"system:serviceaccounts:argo", "system:authenticated"]>

The solution is to set emissary as containerRuntimeExecutor by modifying the file of Argo’s install.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: emissary
    containerRuntimeExecutors:
      - name: emissary
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: emissary
      - name: pns
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: pns
      - name: k8sapi
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: k8sapi

Python
 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: emissary
    containerRuntimeExecutors:
      - name: emissary
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: emissary
      - name: pns
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: pns
      - name: k8sapi
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: k8sapi

Finally, seems all problems have been solved. My colleague Tianchu find out that Autopilot couldn’t support a pod with memory larger than 80GB:

Since many of our applications need memory more than 80 GB, Autopilot can’t be our choice in recent limitations.

How to mount multiple Kubernetes secrets into one directory

As the title, Kubernetes already has a new component called `projected volume` that support the mounting of multiple secrets into one directory.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: my-app
  name: my-app
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine
    name: alpine-secret
    volumeMounts:
    - name: my_secrets
      mountPath: "/var/secrets/"
      readOnly: true
  volumes:
  - name: my_secrets
    projected:
      sources:
      - secret:
          name: my-secret-one
      - secret:
          name: my-secret-two

YAML
 
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: my-app
  name: my-app
spec:
  containers:
  - command:
    - sleep
    - "3600"
    image: alpine
    name: alpine-secret
    volumeMounts:
    - name: my_secrets
      mountPath: "/var/secrets/"
      readOnly: true
  volumes:
  - name: my_secrets
    projected:
      sources:
      - secret:
          name: my-secret-one
      - secret:
          name: my-secret-two

Upgrade GKE cluster

Normally, to upgrade a cluster of Google Kubernetes Engine, we need to upgrade the master at first, and then node_pools. For convenience, I just click the button “UPGRADE AVAILABLE” in the “Release Channel” section under the “DETAILS” tab of the cluster GUI.

After about 5-10 mins, I started to use command to upgrade all our node_pools in this cluster

for pool in highcpu-2 highcpu-4 highcpu-8 ; do
  echo y|gcloud container clusters upgrade my-cluster --node-pool="${pool}" --zone=my-zone --project=my-project
done

Bash
 
for pool in highcpu-2 highcpu-4 highcpu-8 ; do
  echo y|gcloud container clusters upgrade my-cluster --node-pool="${pool}" --zone=my-zone --project=my-project
done

Since we have quite a bunch of node_pools, this upgrade process takes about half an hour to finish. Even the script has ended, there was still two node_pools showed “Error” status. And the detail of the error is:

Insufficient quota to satisfy the request: waiting on IG: instance https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a/instances/my-zone-highcpu-2-e435c7e2-xelk is still CREATING. Last attempt error: [QUOTA_EXCEEDED] Instance ‘my-zone-highcpu-2-e435c7e2-xelk’ creation failed: Quota ‘CPUS’ exceeded. Limit: 1200.0 in region …

Seems the upgrade process need to create extra new nodes with a new version first and then delete the old nodes hence would require extra CPUs and cause the exceeding of CPU quota.

Don’t worry. We just need to rerun the upgrade for these two node_pools with “Error” status and eventually all cluster upgraded to 1.19.9-gke.1900

Strange time output in a container of Kubernetes cluster

After running a workflow in Argo, I found out the output of the “date” command is totally wrong:

# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# TZ='America/New_York' date
Wed Mar 3 00:41:38 2021
# TZ='Australia/Sydney' date
Wed Mar 3 00:42:01 2021

Console
 
# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# TZ='America/New_York' date
Wed Mar 3 00:41:38 2021
# TZ='Australia/Sydney' date
Wed Mar 3 00:42:01 2021

No matter what timezones I used, the time from the “date” command looks didn’t change at all.

This phenomenon only existed in the container of our k8s cluster. In my personal VM, it works just fine.

Finally, my colleague Tianchu found out the reason: the docker image used by the container of the k8s cluster didn’t even install the timezone file! And the solution is simple: just install the timezone data

DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata

Console
 
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata

Run cron jobs in Kubernetes

In the old age of a single machine, we use /etc/crontab to schedule the job. But in the new age of the distributed system, especially Kubernetes, what should we use?

The first choice in my mind is CronJob, like this one:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

YAML
 
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

Then my colleague recommends the CronWorkflow (need at least v2.5 of Argo) to me. Its configuration seems not very different:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron-wf
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: whalesay
    templates:
    - name: whalesay
      container:
        image: alpine:3.6
        command: [sh, -c]
        args: ["date; sleep 90"]

YAML
 
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron-wf
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: whalesay
    templates:
    - name: whalesay
      container:
        image: alpine:3.6
        command: [sh, -c]
        args: ["date; sleep 90"]

But CronWorkflow have a advantage over CronJob: the UI. We can use Web UI to access and manage different types of workflows:

And the UI also makes the accessing of pod log very easy.

Submit Argo workflow to different clusters

A couple of days ago I am looking for a tool to manage different Kubernetes Clusters in my only laptop. But after a while, I realized that kubectl actually support multi-clusters by itself (link).

Then I opened my .kube/config file and saw

clusters:
- cluster:
  name: ClusterA
- cluster:
  name: ClusterB
users:
- user:
  name: UserA
- user:
  name: UserB
contexts:
- context:
    cluster: ClusterA
    user: UserA
  name: ContextA
- context:
    cluster: ClusterB
    user: UserB
  name: ContextB
current-context: ContextA

Seems that we could directly use argo command to submit Argo workflow to different k8s clusters:

argo submit job_a.xml --context ContextA
argo submit job_b.xml --context ContextB

Since the context names are automatically generated by gcloud container clusters get-credentials <gke_cluster_name>, what should I do if I want to make the name more intuitive?

The answer is here. We can just use:

kubectl config rename-context <old_context_name> <new_context_name>

Failed to establish pod watch in Argo

After creating a brand new Kubernetes cluster in GKE, I launched an Argo workflow but saw these errors:

Argo will create two containers for a step: ‘main’ container and ‘wait’ container. But why the creation of ‘wait’ failed? The reason is the default service account for Argo haven’t permission for `get pod`, as this comment said.
So I just run

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

to give default service account ‘admin’ privilege. And everything goes successfully now.

A problem about running Argo

After I launched an Argo workflow, it just hanged on ContainerCreating stage. After waiting for more than 10 minutes, it hasn’t changed at all. Then I found this article.
After using

kubectl describe pods

I finally found out the problem in Message column:

MountVolume.SetUp failed for volume "gcp-key" : secret "service-key" not found

That’s because I forgot to add secret to this Kubernetes cluster…

Problems about using treelite in Kubernetes

treelite is an easy-to-use tool to accelerate prediction speed of XGBoost/LightGBM models. Three days ago, I tested it in my 4-CPU-cores virtual machine and found out that it could reduce the running time half. But after I deployed it into our Kubernetes cluster, it ran even slower than LightGBM library!
There are some strange phenomenons I noticed in the Kubernetes environment:

If I run only one pod, it will run as fastly as previous test in my virtual machine. But if I run 100 pods simutaneously, they will all run very slow.
After login to a pod for profiling, I found out that CPU usage of the application would start from 400% (for I set the cpu request and limit to "4") and gradually descend to less than 10%
There was no memory-swapping happend in any node
I used sleep 3600 to hang all the pods, login to one pod with kubectl exec -ti <pod_name> /bin/bash and run my application manually. It could still run very fast.

This weird problem haunted me, dejected me, and frustrated me for two days and almost ruined one night of my sleeping. Only until I turned to look into the code of treelite, the answer jumped out:

# src/thread_pool/thread_pool.h
    for (int i = 0; i < num_worker_; ++i) {
      thread_[i] = std::thread(task_, incoming_queue_[i].get(),
                                      outgoing_queue_[i].get(),
                                      context_);
    }
    /* bind threads to cores */
    SetAffinity();

The treelite wil pin its processes to each CPU-core by default!
This explained all the thing: when we start 100 pods, they were all trying to ping their 4 processes to CPU-core 0-3. But in the container environment, every pod could see all the CPU-cores of the node (Unlike a virtual machine, in which every VM could only see 4 CPU-cores of itself). Hence they all pin their processes to first 4 CPU-cores of that node. Using only 4 CPU-cores to run all 100 pods? That’s a terrible disaster!

The conclusion is, some acceleration tricks that work very well in classic-computer-era couldn’t be used in container-era.

Use both ‘withParam’ and ‘when’ in Argo Workflows (on Kubernetes)

In Argo, we can use ‘withParam’ to create loop logic:

    - - name: generate
        template: gen-number-list
    # Iterate over the list of numbers generated by the generate step above
    - - name: sleep
        template: sleep-n-sec
        arguments:
          parameters:
          - name: seconds
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"

But in my YAML, it also use when in Argo:

    - - name: generate
        template: gen-number-list
        when: "{{workflow.parameters.NEED_RUN}} == 1"
    # Iterate over the list of numbers generated by the generate step above
    - - name: sleep
        template: sleep-n-sec
        arguments:
          parameters:
          - name: seconds
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"
        when: "{{workflow.parameters.NEED_RUN}} == 1"

When the NEED_RUN is 0, the Argo will report error since it can’t find the {{steps.generate.outputs.result}}. Seems the YAML parser of Argo will try to parse withParam before when phrase.
Fortunately we don’t need to modify Argo or Kubernetes to solve this problem — we just need to let template gen-number-list generate a fake output (empty array):

      script:
        image: "{{workflow.parameters.BASEIMAGE}}"
        command: [bash]
        source: |
            if [ $NEED_RUN -eq 1 ]; then
                python3 -u output.py
            else
                echo "[]"
            fi
        env:
          - name: NEED_RUN
            value: "{{workflow.parameters.NEED_RUN}}"