I have written an example of Mesos Framework by python. It simply calculate “Pi” by using Mento-Carlo algorithm
The whole source code is at https://github.com/RobinDong/mesos-python-examples/tree/master/calculate_pi
At beginning, I use python threading in “launchTask()”:

class PiExecutor(mesos.interface.Executor):
    def launchTask(self, driver, task):
        def run_thread():
        ......
        thread = threading.Thread(target=run_thread)
        thread.start()

But I found out that the executors only spend a small part of CPU resource in slave machines (about 100%~150%, which is too low in a 8-cores server). First, I thought it may be limited by cgroup, but later, the answer is revealed: multi-threaded python application is actually single thread because of GIL.
With no choice, I have to change my code: the thread will launch a new process, then the new process will calculate the result and finally return the result to thread. It works fine in my Mesos cluster.

Related Posts