docker

Running Django in docker

I am trying to learn Django (a python framework for developing web) in docker container recently. After running the docker with port redirect

sudo docker run -p 8000 -t -i centos/django /bin/bash
#in the docker now
cd mysite
python manage.py runserver

The output of Django server is

System check identified no issues (0 silenced).
September 24, 2014 - 08:52:05
Django version 1.7, using settings 'oddjobs.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Then I use command sudo docker ps to find out the port number for host machine:

CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS              PORTS                     NAMES
08d01206d676        centos/oddjobs:run   "/bin/bash"         About an hour ago   Up About an hour    0.0.0.0:49198->8000/tcp   kickass_pare

but when using curl 127.0.0.1:49198 in host machine it just report “Connection Refused”
After searching in google, I only find one article which seems useful for me. But my problem is still there after I follow its steps for solution. With no choice, I have to read the documents of docker carefully and do my experiment step by step.
First, I run a nc-server in docker:

sudo docker run -p 8000 -t -i centos/django /bin/bash
#in the docker now
nc -l 8000
#listening on port 8000

Then using nc 127.0.0.1 8000 in host. It failed too. Why can’t nc client connect to server in docker even I followed the document of docker?After running netstat in docker, I find out the answer: my centos image is centos7 and the ‘nc’ in it will listen on ipv6 address by default. If anyone want to listen on ipv4 address, it should type in

nc -l 8000 -4

Now, the nc client could connect to server now.
But how to run Django server in ipv4 address? This article tells me the way. Now, it seems everything is ok. I start Django again with python manage.py runserver 127.0.0.1:8000 but it still could not be connected with nc client in host. Oh, the ip “127.0.0.1” and “0.0.0.0” is very different so I should run Django like:

python manage.py 0.0.0.0:8000

The browser in host could access the Django example site now.

Strange time output in a container of Kubernetes cluster

After running a workflow in Argo, I found out the output of the “date” command is totally wrong:

# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# TZ='America/New_York' date
Wed Mar 3 00:41:38 2021
# TZ='Australia/Sydney' date
Wed Mar 3 00:42:01 2021

No matter what timezones I used, the time from the “date” command looks didn’t change at all.

This phenomenon only existed in the container of our k8s cluster. In my personal VM, it works just fine.

Finally, my colleague Tianchu found out the reason: the docker image used by the container of the k8s cluster didn’t even install the timezone file! And the solution is simple: just install the timezone data

DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata

Some problems when using GCP

After I launched a compute engine with container, it report error:

gcr.io/xx/xx-xx/feature:yy
Feb 03 00:12:28 xx-d19b201 konlet-startup[4664]: {“errorDetail”:{“message”:”failed to register layer: Error processing tar file(exit status 1): write /xxx/2020-01-16/base_cmd/part-00191-2e99af0e-1615-42af-9c60-910f9a9e6a17-c000.snappy.parquet: no space left on device”},”error”:”failed to register layer: Error processing tar file(exit status 1): write /xxx/2020-01-16/base_cmd/part-00191-2e99af0e-1615-42af-9c60-910f9a9e6a17-c000.snappy.parquet: no space left on device”}

The key is in the no space left on device. Then I use df to see the disk space:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.2G  879M  343M  72% /
devtmpfs         26G     0   26G   0% /dev
tmpfs            26G     0   26G   0% /dev/shm
tmpfs            26G  436K   26G   1% /run
tmpfs            26G     0   26G   0% /sys/fs/cgroup
tmpfs           1.0M  108K  916K  11% /etc/machine-id
tmpfs            26G     0   26G   0% /tmp
tmpfs           256K     0  256K   0% /mnt/disks
overlayfs       1.0M  108K  916K  11% /etc
/dev/sda8        12M   28K   12M   1% /usr/share/oem
/dev/sda1       5.7G  5.7G     0 100% /mnt/stateful_partition
tmpfs           1.0M  4.0K 1020K   1% /var/lib/cloud

Obviously the space on /mnt/stateful_partition has been used out. The solution is simple: add new argument for gcloud command

gcloud compute instances create-with-container [INSTANCE_NAME] \
     --container-image [DOCKER_IMAGE] \
     --boot-disk-size=30GB

Another problem occurred when I trying to launch an instance of Cloud Run. It reported a mess:

Traceback (most recent call last): File “/usr/local/lib/python3.6/site-packages/google/auth/compute_engine/credentials.py”, line 98, in refresh request, service_account=self._service_account_email File “/usr/local/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py”, line 241, in get_service_account_token request, “instance/service-accounts/{0}/token”.format(service_account) File “/usr/local/lib/python3.6/site-packages/google/auth/compute_engine/_metadata.py”, line 172, in get response, google.auth.exceptions.TransportError: (“Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/564585695625-compute@developer.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 500 Response:\nb’Could not fetch URI /computeMetadata/v1/instance/service-accounts/564585695625-compute@developer.gserviceaccount.com/token\\n'”, )

Actually, the reason is quite simple: I haven’t realized that Cloud Run need its instance to listen on PORT. Otherwise, the service will not be launched successfully.

Use docker as normal user

I have used docker for more than 4 years, although not in product environment. Until last week, my colleague told that docker can be used as non-root user.
The document is here.
I just need to

sudo usermod -aG docker $USER
sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R

So easy.

Network problem of installing docker-engine on Centos-7

After installing docker-engine on Centos-7, it failed to start by

systemctl start docker.service

After I use

systemctl status docker.service

it shows:

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2016-04-01 19:07:00 CST; 3s ago
     Docs: https://docs.docker.com
  Process: 8136 ExecStart=/usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/direct-lvm/data --storage-opt dm.metadatadev=/dev/direct-lvm/metadata -H fd:// --insecure-registry reg.docker.alibaba-inc.com (code=exited, status=1/FAILURE)
 Main PID: 8136 (code=exited, status=1/FAILURE)

To see more details, I use

sudo journalctl -xe

There were much more informations now:

...
Apr 01 19:07:00 XXX docker[8136]: time="2016-04-01T19:07:00.187369563+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Apr 01 19:07:00 XXX docker[8136]: time="2016-04-01T19:07:00.511949325+08:00" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: Failed to setup IP tables, cannot acquire Interface address: Interface docker0 has no IPv4 addresses"
...

The ethernet interface of docker0 can not be assigned IPv4 addresse. It seems many people meet this problem after I searching the google, but none of the solutions works for me. Consequently, I find out this solution in my environment (totally remove the network bridge):

sudo ifconfig docker0 down
sudo brctl delbr docker0

and my docker service startup now!
The key is: docker-engine will not assign IP to docker0 interface which is already exists.

Run docker on centos6

Docker use thin-provision of device mapper as its default storage, therefore if we wan’t run docker on centos6, we should update kernel first. I use linux kernel 4.11 and notice these kernel options should be set:

CONFIG_DM_THIN_PROVISIONING=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_MASQUERADE_IPV4=m
CONFIG_MEMCG=y
CONFIG_IP_NF_TARGET_MASQUERADE

After build and reboot the kernel, I still can’t launch docker service, and finally find out the solution:

sudo route del -net 172.16.0.0 netmask 255.240.0.0