Argo

Use a specific service account in the Argo job

I created a simple Argo job to pull messages from a Google Cloud Pub/Sub topic. Permission has been given to the service account of GKE’s workload identity. But the Argo job failed with errors:

argo submit example.json -n argoproj

hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 72, in error_remapped_callable
hello-world-pqbm5:     return callable_(*args, **kwargs)
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 1030, in __call__
hello-world-pqbm5:     return _end_unary_response_blocking(state, call, False, None)
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
hello-world-pqbm5:     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
hello-world-pqbm5: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
hello-world-pqbm5:      status = StatusCode.PERMISSION_DENIED
hello-world-pqbm5:      details = "User not authorized to perform this action."
hello-world-pqbm5:      debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.69.95:443 {grpc_message:"User not authorized to perform this action.", grpc_status:7, created_time:"2023-05-15T01:10:43.128528579+00:00"}"
hello-world-pqbm5: >
hello-world-pqbm5: 
hello-world-pqbm5: The above exception was the direct cause of the following exception:
hello-world-pqbm5: 
hello-world-pqbm5: Traceback (most recent call last):
hello-world-pqbm5:   File "<string>", line 26, in <module>
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/pubsub_v1/services/subscriber/client.py", line 1495, in pull
hello-world-pqbm5:     response = rpc(
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
hello-world-pqbm5:     return wrapped_func(*args, **kwargs)
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
hello-world-pqbm5:     return retry_target(
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/retry.py", line 191, in retry_target
hello-world-pqbm5:     return target()
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/timeout.py", line 120, in func_with_timeout
hello-world-pqbm5:     return func(*args, **kwargs)
hello-world-pqbm5:   File "/usr/local/lib/python3.9/dist-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
hello-world-pqbm5:     raise exceptions.from_grpc_error(exc) from exc
hello-world-pqbm5: google.api_core.exceptions.PermissionDenied: 403 User not authorized to perform this action.

Thanks to my colleagues. They remind me that an Argo job needs to specify a service account when running in the workload identity namespace.

argo submit example.json -n argoproj --serviceaccount argo-workflow

Or, I can add this service account to the YAML file:

apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
spec:
  entrypoint: whalesay          # invoke the whalesay template
  serviceAccountName: argo-workflow

Resolve dependencies in Argo workflow

My configuration DAG of Argo workflow was like this:

{
  name: "my-workflow",
  dag: {
    tasks: [
      {
        name: "step1",
        template: "template1",
      },
      {
        name: "manual-check",
        template: "template-manual-check",
        depends: "step1.Failed",
      },
      {
        name: "step2",
        template: "template2",
        depends: "manual-check",
      },
    ],
  },
}

If “step1” failed, the “manual-check” will suspend the whole pipeline and let users (or customers) decide whether this pipeline could continue. But I met a funny situation when “step1” is successful: the “manual-check” step was omitted by workflow and “step2” would never be executed because it depends on “manual-check”!

The correct solution should let “step2” run when “step1” is successful. Therefore I need to change the configuration to this:

{
  ......
      {
        name: "step2",
        template: "template2",
        depends: "manual-check || step1.Succeeded",
      },
    ],
  },
}

Thanks to this reference: “Enhanced Depends Logic“.

Conditional checking in Argo workflow

My company has been using Argo for executing workflow for more than three years. I knew every step in the Argo workflow could be controlled by when expression, like this:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: example-
spec:
  entrypoint: first-step
  arguments:
  - parameters:
    - - name: 'COLOR',
        value: std.extVar('COLOR'),
  templates:
  - name: first-stage
    steps:
    # flip a coin
    - - name: first-step
        template: step-first
        when: '{{workflow.parameters.COLOR}} == red',

What if I want to check more than one condition? Just use “&&” as “and”, “||” as “or”:

...
  - name: first-stage
    steps:
    # flip a coin
    - - name: first-step
        template: step-first
        when: '{{workflow.parameters.COLOR}} == red || {{workflow.parameters.COLOR}} == blue',

Reference

Try to deploy Argo on GCP Autopilot

Creating an Autopilot cluster in GCP K8S is quite easy. But after deploying Argo and launching our pipeline, the Argo report errors:

Failed to pull image "eu-docker.pkg.dev/project-name/mytag:123456789"
b85d23bf513ba037f4b2fbd5e": rpc error: code = Unknown desc = failed to pull and unpack image eu-docker.pkg.dev/project-name/mytag:123456789": failed to resolve reference "eu-docker.pkg.dev/project-name/mytag:123456789": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

The solution is (give k8s cluster the permission to pull docker image from our docker repository):

kubectl create secret docker-registry gcr-json-key  --docker-server=eu-docker.pkg.dev  --docker-username=_json_key  --docker-password="$(cat our_service_account.json)"  --docker-email=your@email.address -n argo
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}' -n argo

Then the second problem jumped out:

admission webhook "validation.gatekeeper.sh" deni
ed the request: [denied by autogke-no-write-mode-hostpath] hostPath volume docker-sock used in container wait
uses path /var/run/docker.sock which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes a
re: ["/var/log/"]. Requesting user: <system:serviceaccount:argo:argo> and groups: <["system:serviceaccounts",
"system:serviceaccounts:argo", "system:authenticated"]>

The solution is to set emissary as containerRuntimeExecutor by modifying the file of Argo’s install.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: emissary
    containerRuntimeExecutors:
      - name: emissary
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: emissary
      - name: pns
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: pns
      - name: k8sapi
        selector:
          matchLabels:
            workflows.argoproj.io/container-runtime-executor: k8sapi

Finally, seems all problems have been solved. My colleague Tianchu find out that Autopilot couldn’t support a pod with memory larger than 80GB:

Since many of our applications need memory more than 80 GB, Autopilot can’t be our choice in recent limitations.

Strange time output in a container of Kubernetes cluster

After running a workflow in Argo, I found out the output of the “date” command is totally wrong:

# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# TZ='America/New_York' date
Wed Mar 3 00:41:38 2021
# TZ='Australia/Sydney' date
Wed Mar 3 00:42:01 2021

No matter what timezones I used, the time from the “date” command looks didn’t change at all.

This phenomenon only existed in the container of our k8s cluster. In my personal VM, it works just fine.

Finally, my colleague Tianchu found out the reason: the docker image used by the container of the k8s cluster didn’t even install the timezone file! And the solution is simple: just install the timezone data

DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata

Run cron jobs in Kubernetes

In the old age of a single machine, we use /etc/crontab to schedule the job. But in the new age of the distributed system, especially Kubernetes, what should we use?

The first choice in my mind is CronJob, like this one:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

Then my colleague recommends the CronWorkflow (need at least v2.5 of Argo) to me. Its configuration seems not very different:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron-wf
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: whalesay
    templates:
    - name: whalesay
      container:
        image: alpine:3.6
        command: [sh, -c]
        args: ["date; sleep 90"]

But CronWorkflow have a advantage over CronJob: the UI. We can use Web UI to access and manage different types of workflows:

And the UI also makes the accessing of pod log very easy.

Some tips for using bash and Jsonnet

To build an integration test by using Argo, I used bash to run scripts and Jsonnet to orchestrate Argo workflow. As expected, I met a bunch of problems with them.

problem about base64

[default]
aws_access_key_id = apple
aws_secret_access_key = banana

for this text file, we can use base64 to encode it and assign to an environment variable.

export MY=`cat my.ini |base64`
echo $MY|base64 -d

But it will report error ‘invalid input’:

[default]
aws_access_key_id = apple
aws_secret_access_keybase64: invalid input

The correct way is using -i, --ignore-garbage :

echo $MY|base64 -di

2. problem about exit status of Bash

If a step (or a pod) in Argo workflow failed, the whole workflow will fail. That’s the behaviour we want. But if we use diff command many times in one step, the fail (for diff, it means two files are not the same) of one diff will not cause the fail of the step. So I need a way to check the status of diff command. Fortunately, it’s very easy:

diff a b || exit -1
diff c d || exit -1

3. writes multiple lines of string in Jsonnet

local test_script = |||
  diff a b || exit -1
  diff c d || exit -1
|||;
...
  args = [test_script]

Submit Argo workflow to different clusters

A couple of days ago I am looking for a tool to manage different Kubernetes Clusters in my only laptop. But after a while, I realized that kubectl actually support multi-clusters by itself (link).

Then I opened my .kube/config file and saw

clusters:
- cluster:
  name: ClusterA
- cluster:
  name: ClusterB
users:
- user:
  name: UserA
- user:
  name: UserB
contexts:
- context:
    cluster: ClusterA
    user: UserA
  name: ContextA
- context:
    cluster: ClusterB
    user: UserB
  name: ContextB
current-context: ContextA

Seems that we could directly use argo command to submit Argo workflow to different k8s clusters:

argo submit job_a.xml --context ContextA
argo submit job_b.xml --context ContextB

Since the context names are automatically generated by gcloud container clusters get-credentials <gke_cluster_name>, what should I do if I want to make the name more intuitive?

The answer is here. We can just use:

kubectl config rename-context <old_context_name> <new_context_name>

Failed to establish pod watch in Argo

After creating a brand new Kubernetes cluster in GKE, I launched an Argo workflow but saw these errors:

Argo will create two containers for a step: ‘main’ container and ‘wait’ container. But why the creation of ‘wait’ failed? The reason is the default service account for Argo haven’t permission for `get pod`, as this comment said.
So I just run

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

to give default service account ‘admin’ privilege. And everything goes successfully now.

A problem about running Argo

After I launched an Argo workflow, it just hanged on ContainerCreating stage. After waiting for more than 10 minutes, it hasn’t changed at all. Then I found this article.
After using

kubectl describe pods

I finally found out the problem in Message column:

MountVolume.SetUp failed for volume "gcp-key" : secret "service-key" not found

That’s because I forgot to add secret to this Kubernetes cluster…