Robin on Linux – Page 33 – All about technology

China Linux Storage & Filesystem 2015 workshop (second day)

Zheng Liu from Alibaba lead the topic about ext4. The most important change in EXT-series filesystem this year is: ext3 has gone, people could only use ext3 by mount ext4 with special arguments in latest kernel (actually, in CentOS 7.0). Encrypt feature has complete in ext4.
Robin Dong (Yes, it’s me) from Alibaba give a presentation about cold storage (Slide is here). We develop distributed storage system based on a small open-source software called “sheepdog“, and modified it heavily to improve data recovery performance and make sure it could run in low-end but high-density storage servers.

Discussion in tea break

Yanhai Zhu from Alibaba (We have done so much works on storage) lead a topic about cache in virtual machines environment. Alibaba choose Bcache as code base to develop a new cache software.
Robin: Why Bcache? Why not flashcache?
Yanhai: I started my work on flashcache first, but flashcache is not profit to the product environment. First, flashcache is unfriendly to sequential-write. Second, it use hash data structure to distributed IO requests at beginning, which will split the cache data in multi-tenant environment. Bcache use B-tree instead of hash-table to store data, it’s better for our requirements.
They use radical write-back strategy on cache. It works very well because the cache sequentialize the write IOs and make backend easy to absorb the pressure peak.
The last topic is lead by Zhongjie Wu from Memblaze, a famous startup company in China on flash storage technology. It’s about NVDIMM, the most hot hardware technology in recent years. A NVDIMM is not expensive, it is only a DDR DIMM with a capacitance. Memblaze has develop a new 1U storage server with a NVDIMM and many flash cards. It contain their own developed OS and could use Fabric-Channel/Ethernet to connect to client. The main purpose of NVDIMM is to reduce latency, and they use write-back strategy(Surely).
The big problem they face with NVDIMM is CPU can’t flush data in its L1 cache to NVDIMM when whole server powers down. To solve this problem, Memblaze use write-combining in CPU multi-cores, it hurts the performance a little but avoid the data missing finally.

clsf2015

All the staff in this CLSF 2015

Articles from other attenders:
https://blogs.oracle.com/linuxkernel/entry/china_linux_storage_and_file1

China Linux Storage & Filesystem 2015 workshop (first day)

The first topic is lead by Haomai Wang from XSKY. He first introduce some basic concepts about Ceph and I did catch this opportunity to ask some questions.
Robin(from Alibaba): Dose Ceph cache all meta-data information (may called “cluster map”) on monitor-nodes so client could fetch data by just one jump in network?
Haomai: Yes. One jump, and comes to the OSD.
Robin: If I use Cephfs, is it still one jump?
Haomai: Still one jump. Although we add MDS in Cephfs, the MDS does not store data or meta-data of filesystem but only to store the context of distributed lock.
Ceph also support samba2.0/3.0 now. In linux, it is recommend to use iSCSI to access ceph storage cluster because it will have to update kernel in clients if we use rbd/libceph kernel modules. Ceph use pipeline model in message processing therefore it is good to Hard Disk but not SSD. In the future, developers will use async-framework (such as Seastar) to refactor the ceph.
Robin: If I use three replications in ceph, will the client write three copies concurrently?
Haomai: No. Firstly, the IO will come to the primary OSD, and then the primary OSD will issue two other replicated IOs to other two OSDs, waiting until the two IOs back, and return to client “the IO is success”.
Robin: Ok, now we still have two jumps….Is it difficult to change OSD to write at the same time so we can make the latency of ceph low?
Haomai: That will not be easy. Ceph use primary OSD to make sure the consistent of writing transaction.
The future developing plan for ceph is de-duplication on pool level. Coly Li(from Suse) said that de-duplication is better to be made on business level instead of block level because the duplicated information has be split in block level. But the developers in ceph community looks still want make ceph to be omnipotent.

Discussion about ceph

Jiaju Zhang from Redhat lead the topic about use cases of ceph in enterprises. Ceph has become the most famous open source storage software around the world and also be used in Redhat/Intel/Sandisk(Low-end Storage Array)/Samsung/Suse.
Next topic is about ScyllaDB and Seastar. Asias He from OSv lead this topic. ScyllaDB is a distributed Key/Value store engine which is written in C++14 code and completely compatible to Cassandra. It could also run CQL (Cassandra Query Language). In the graph, ScyllaDB is 40 times more faster than Cassandra. The asynchronous developing framework in ScyllaDB is called Seastar.
Robin: What’s the magic in ScyllaDB?
Asias: We shard requests to every CPU core, and run with no locks/no threads. Data is zero-copy and use bi-direction queue to transfer messages between cores. The test result is base on kernel TCP/IP network stack but we will use our own network stack in our future.
Yanhai Zhu(from Alibaba): I think the test you do is not fair enough: ScyllaDB is designed to be run in multi-cores but Cassandra is not. You guys should run 24 Cassandra instances to compare with ScyllaDB, not just one.
Asias: May be you are right. But ScyllaDB use message queues to transfer messages between CPU cores, so it avoid atomic-operation and lock-operation cost. And, Cassandra is written by Java, which means the performance will be low when the JVM do garbage- collection. ScyllaDB is written completely by c++ so its performance is much steady.
Last topic today is lead by Xu Wang, the CTO of Hyper ( A startup company in china, works on how to run container like VM).
Hyper means “hypervisor” addd “docker image”. Customers could run docker image on Xen/KVM/Virtualbox now.

clsf2015

The guy on the right side is Xu Wang

Run docker on centos6

Docker use thin-provision of device mapper as its default storage, therefore if we wan’t run docker on centos6, we should update kernel first. I use linux kernel 4.11 and notice these kernel options should be set:

CONFIG_DM_THIN_PROVISIONING=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_MASQUERADE_IPV4=m
CONFIG_MEMCG=y
CONFIG_IP_NF_TARGET_MASQUERADE

After build and reboot the kernel, I still can’t launch docker service, and finally find out the solution:

sudo route del -net 172.16.0.0 netmask 255.240.0.0

A example of Mesos Python Framework to calculate Pi

I have written an example of Mesos Framework by python. It simply calculate “Pi” by using Mento-Carlo algorithm
The whole source code is at https://github.com/RobinDong/mesos-python-examples/tree/master/calculate_pi
At beginning, I use python threading in “launchTask()”:

class PiExecutor(mesos.interface.Executor):
    def launchTask(self, driver, task):
        def run_thread():
        ......
        thread = threading.Thread(target=run_thread)
        thread.start()

But I found out that the executors only spend a small part of CPU resource in slave machines (about 100%~150%, which is too low in a 8-cores server). First, I thought it may be limited by cgroup, but later, the answer is revealed: multi-threaded python application is actually single thread because of GIL.
With no choice, I have to change my code: the thread will launch a new process, then the new process will calculate the result and finally return the result to thread. It works fine in my Mesos cluster.

Override the init of BaseHTTPRequestHandler in python

Currently, I am writing some python program for the front of our storage system. By using ‘BaseHTTPRequestHandler’ and ‘ThreadedHTTPServer’, we could implement a simple multi-thread http server quickly. But after add ‘__init__()’ for our ‘MyHandler’, it doesn’t work correctly now:

class MyHandler(BaseHTTPRequestHandler):
    def __init__(self):
        self.request_id = ''

Then I found this statement in python docs:

Subclasses should not need to override or extend the __init__() method.

But thanks for the code example, we could override ‘__init__’ of BaseHTTPRequestHandler’ this way:

class MyHandler(BaseHTTPRequestHandler):
    def __init__(self, request, client_address, server):
        BaseHTTPRequestHandler.__init__(self, request, client_address, server)
        self.request_id = ''

Solve a USB network card problem

I am doing some source code porting works on linux kernel recently.
After I reboot my server (ubuntu 14.04) into new kernel version of 3.19.8, it can’t be connected by using ssh but only by using serial port.
The server is using a USB network card, so firstly I suspect some kernel driver for USB NIC has missing in .config file. Therefore I boot back into the old version kernel and try to find some useful information in ‘dmesg’:

asix 3-1.4:1.0 eth5: register 'asix' at usb-0000:00:11.0-1.4, ASIX AX88772B USB 2.0 Ethernet, 00:24:9c:04:44:28
......
eth3: link up, 100Mbps, full-duplex, lpa 0x45E1

The eth3 is using MII port, so when I try to grep “link up …lpa” in 3.19.8 kernel source code, I find out it must be printed by these codes in drivers/net/mii.c:

unsigned int mii_check_media (struct mii_if_info *mii,
                  unsigned int ok_to_print,
                  unsigned int init_media)
{
......
    if (ok_to_print)
        netdev_info(mii->dev, "link up, %uMbps, %s-duplex, lpa 0x%04X\n",
                lpa2 & (LPA_1000FULL | LPA_1000HALF) ? 1000 :
                media & (ADVERTISE_100FULL | ADVERTISE_100HALF) ?
                100 : 10,
                duplex ? "full" : "half",
                lpa);
......

The only place “ASIX AX88772B” driver call mii_check_media is in drivers/net/usb/asix_devices.c:

static int ax88772_link_reset(struct usbnet *dev)
{
    u16 mode;
    struct ethtool_cmd ecmd = { .cmd = ETHTOOL_GSET };
    mii_check_media(&dev->mii, 1, 1);
    mii_ethtool_gset(&dev->mii, &ecmd);
    mode = AX88772_MEDIUM_DEFAULT;
......
static const struct driver_info ax88772b_info = {
    .description = "ASIX AX88772B USB 2.0 Ethernet",
    .bind = ax88772_bind,
    .unbind = ax88772_unbind,
    .status = asix_status,
    .link_reset = ax88772_link_reset,
    .reset = ax88772_reset,
    .flags = FLAG_ETHER | FLAG_FRAMING_AX | FLAG_LINK_INTR |
             FLAG_MULTI_PACKET,
    .rx_fixup = asix_rx_fixup_common,
    .tx_fixup = asix_tx_fixup,
    .data = FLAG_EEPROM_MAC,
};

So far, the reason is that: the system does not “reset” the USB network card after booting up. But why it only forget to reset USB NIC in 3.19.8 kernel? After checking the /etc/network/interfaces:

auto eth0
iface eth0 inet dhcp
auto eth3
iface eth3 inet dhcp
auto usbnet0

the answer is: the device name of the USB NIC has been changed to “eth5” by udevd in 3.9.18 kernel (new version kernel recognise new network port so eth0/eth1/eth2/eth3/eth4 all has been occupied by system) so the network scripts can’t start it up.
Solution:
Fix the name of USB NIC by adding below content into /etc/udev/rules.d/70-persistent-net.rules:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:9c:04:44:28", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="usbnet0"

and add configurations into /etc/network/interfaces to start “usbnet0” up automatically:

auto usbnet0
iface usbnet0 inet dhcp

puppet 3 certification problem on centos 7

I configure the puppet master and agent followed by this step. But when I run “puppet agent -t”, it report error:

[root@iZ25s7c99ynZ ~]# puppet agent --test
Info: Creating a new SSL key for iz25s7c99ynz
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for iz25s7c99ynz
Info: Certificate Request fingerprint (SHA256): E0:17:BA:8F:0A:97:FC:90:84:75:96:75:CC:DB:B8:07:C8:48:AD:CA:26:5C:E5:5B:01:4B:9E:71:6A:52:C9:AE
Info: Caching certificate for iz25s7c99ynz
Info: Caching certificate_revocation_list for ca
Error: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate revoked for /CN=puppet.builder1.com]

My OS version is “Centos 7” and puppet version is “3.7.5”. After I have tried the way as this page answered, the problem still exists. Therefore, I write down the final correct operations here for future reference:

#In master
rm -rf /var/lib/puppet/ssl/*
service httpd stop
## use Ctrl + c to break when you see "Notice: Starting Puppet master version..."
puppet master --no-daemonize --verbose
service httpd start
#In agent
rm -rf /var/lib/puppet/ssl/*
puppet agent -t

How to set the value of “$releasever” permanently for yum

In a test server I typed “sudo yum update”, it reported errors like:

$sudo yum update
Loaded plugins: branch, fastestmirror, security
Setting up Update Process
Bad id for repo: base.$releasever.x86_64, byte = $ 8
Bad id for repo: base.$releasever.noarch, byte = $ 8
Bad id for repo: ops.$releasever.x86_64, byte = $ 4
Bad id for repo: ops.$releasever.noarch, byte = $ 4
Loading mirror speeds from cached hostfile
http://mirrors.xxx.com/centos/%24releasever/os/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: base. Please verify its path and try again<

Then I found this web in google for introducing how to get the value of "$releasever", but it does not tell us how to set "$releasever" permanently. Therefore, I have to search word like "releasever" in the source code of 'yum':

rpm -ql yum|xargs sudo grep releasever * -rn

Finally the source code in '/usr/lib/python2.6/site-packages/yum/__init__.py' comes out:

releasever=self.conf.yumvar['releasever']

But where does 'self.conf.yumvar' get its values? The answer is '/etc/yum/vars/'. After

sudo bash -c "echo 6 > /etc/yum/vars/releasever"

My 'sudo yum update' works correctly now.

Vault Conference 2015, in Boston

Vault Conference 2015

Registration
vault

First talk, and the sponsors
vault

John Spray and his talk about development updates of Ceph
vault

I and Coly also give a presentation about our Cold Storage project, the slide is here: Lambert: Achieve High Durability, Low Cost & Flexibility at Same Time — open source cold storage engine for ExaBytes data in Alibaba
vault

I am appreciating the help from open source community, seems my oral English is not bad 🙂

Trip to Boston

Last week, I flight to Boston, MA to attend the Vault Conference 2015. This is my first time for coming to Boston.

The Boston Common
Boston

The Charles River, from MIT side.
Boston

The Hall of MIT
Boston

In the Havard Square
Boston

The well-known Havard Book Store
Boston

The Old Corner Book Store, it used to be a real book store about 100 years ago 🙂