has been learning Python recently, and a small example of the distributed process is interesting. Here's a record. The multiprocessing module of

distributed process

Python not only supports multi processes, but managers sub module also supports the distribution of multiple processes to multiple machines. A service process can be used as a dispatcher to distribute tasks to many other processes and rely on network communication. Because the managers module is very well encapsulated, it is easy to write distributed multi process programs without having to understand the details of network communication.

master: the Queue server principle exposed out through the network through the managers module, the other machine process can access the Queue
service process responsible for starting the Queue, the Queue is registered to the network, and then to the Queue code written inside the task, as follows:

 #task_master.py #coding=utf-8 # multi process distributed # server from multiprocessing.managers import example BaseManager from multiprocessing import freeze_support #server boot Baocuo, suggesting the need to refer to this package import random, time, queue # send task queue (task_queue = queue.Queue) # receives the results of queue result_queue = queue.Queue (#) inherited from BaseManager QueueManager class QueueManager (BaseManager): pass #win7 64 does not support the callable under anonymous call lambda here, Def (return_task_queue) package: Global task_queue return task_queue def (return_result_queue): Global result_queue return result_queue def (test): # to register two Queue to the network, the callable Queue object QueueManager.register # parameter correlation ('get_task_queue', callable=lambda:task_queue) #QueueManager.register ('get_result_queue', callable=lambda:result_queue) QueueManager.register ('get_task_queue','get_result_queue', QueueManager.register (callable=return_task_queue) callable=return_result_queue) # binding port 5000, set the verification code "abc'Manager = QueueManager (address= ('', 5000), authkey=b'abc') # here must be added to the default IP address # local start Queue manager.start (#server) = (manager.get_server) (#server.serve_forever) print ('Start server Master') # obtained through the network access to the Queue object task (result = manager.get_task_queue) = manager.get_result_queue (for) # put the task into several I in range (10): n = random.randint (010000) print ('put task%d...'% n) task.put (n) print # read results from the result cohort ('Try get results for I in... ') range (10): r = result.get (timeout=10) print ('result:%s'% R) manager.shutdown (print) # closed ('master Exit') if __name__ (test): freeze_support ='__main__' (


task process is as follows:

 #task_worker.py #coding=utf-8 code # multi process distributed server: worker import example # non time, Sys, queue from multiprocessing.managers import BaseManager QueueManager class QueueManager # create similar (BaseManager): pass # because this QueueManager only get Queue from the network, so the registration only provide names (QueueManager.register'get_task_queue') QueueManager.register ('get_result_queue') # connect to the server, is a machine running task_master.py server_addr =''print ('connect to server%s server_addr...') # port and verification code to remain completely consistent with M = QueueManager (address= (server_addr, 5000), authkey=b'abc'(m.connect) # from network connection) # gets Queue as for task = m.get _task_queue (result) = m.get_result_queue (task) # acquisition tasks from the queue, and writes the results to the result for I in range cohort (10): try: n = task.get (timeout=1) print ('run task%d *%d... '(n, n)) r ='%d *%d =%d' (n, n,% n*n) time.sleep (1) result.put (R) except queue.Empty: print ('task queue is empty'print ('worker) # processing results of Exit' 

) run as follows:

above is the article, I hope all of you help, I hope you will support a script.

This paper fixed link:http://www.script-home.com/a-small-and-distributed-example-of-python3-learning-notes.html | Script Home | +Copy Link

Article reprint please specify:A small and distributed example of python3 learning notes | Script Home

You may also be interested in these articles!