[Greater-commits] r3689 - in branches/3.0.0-all-models/GREAT-ER: . Greater/Modules

[scm-commit@wald.intevation.org]

Author: aheinecke
Date: 2011-07-26 11:55:03 +0200 (Tue, 26 Jul 2011)
New Revision: 3689

Modified:
   branches/3.0.0-all-models/GREAT-ER/
   branches/3.0.0-all-models/GREAT-ER/ChangeLog
   branches/3.0.0-all-models/GREAT-ER/Greater/Modules/model.py
Log:
Not Merged revisions 3638-3639
Changes in SciParam
svn+ssh://wald.intevation.org/greater/trunk/GREAT-ER

........
  r3638 | aheinecke | 2011-07-22 11:38:12 +0200 (Fri, 22 Jul 2011) | 2 lines
  
  Add encoding line
........
  r3639 | aheinecke | 2011-07-22 12:04:57 +0200 (Fri, 22 Jul 2011) | 2 lines
  
  Make sciparam work with unicode
........
  r3640 | aheinecke | 2011-07-22 17:27:51 +0200 (Fri, 22 Jul 2011) | 4 lines
  
  Quote paths in scheduler/worker calling and run multiple workers
  The path quoting fixes the problem that greater was unable to be executed
  from a path with a whitespace in it,..
........



Property changes on: branches/3.0.0-all-models/GREAT-ER
___________________________________________________________________
Name: svnmerge-integrated
   - /trunk/GREAT-ER:1-232,310,312-315,319,322,327,333,335,340-344,346-347,368,375,397-398,400,403,410,412-431,434-448,3577-3578,3604-3606,3611,3613-3618,3620,3623,3633-3634
   + /trunk/GREAT-ER:1-232,310,312-315,319,322,327,333,335,340-344,346-347,368,375,397-398,400,403,410,412-431,434-448,3577-3578,3604-3606,3611,3613-3618,3620,3623,3633-3634,3638-3640

Modified: branches/3.0.0-all-models/GREAT-ER/ChangeLog
===================================================================
--- branches/3.0.0-all-models/GREAT-ER/ChangeLog	2011-07-26 09:50:35 UTC (rev 3688)
+++ branches/3.0.0-all-models/GREAT-ER/ChangeLog	2011-07-26 09:55:03 UTC (rev 3689)
@@ -19,6 +19,9 @@
 	* Greater/Modules/model.py:
 		- Introducing new dependency to winpaths on windows and
 		  start the schedu
+		- Make the scheduler/worker run even when there is a space
+		  in the path.
+		- Start multiple workers when multiple cpus are available
 
 2011-07-25  Andre Heinecke <aheinecke at intevation.de>
 

Modified: branches/3.0.0-all-models/GREAT-ER/Greater/Modules/model.py
===================================================================
--- branches/3.0.0-all-models/GREAT-ER/Greater/Modules/model.py	2011-07-26 09:50:35 UTC (rev 3688)
+++ branches/3.0.0-all-models/GREAT-ER/Greater/Modules/model.py	2011-07-26 09:55:03 UTC (rev 3689)
@@ -53,6 +53,7 @@
 import os
 import re
 import xmlrpclib
+from multiprocessing import cpu_count
 
 class SchedulerProcess:
 
@@ -90,13 +91,14 @@
         """
         python_exe = sys.executable
         path = os.path.join(GreaterModel.__path__[0], '..')
-        cmdline = [python_exe, os.path.join(path, 'start_scheduler.py'),
+        cmdline = ['"%s"' % python_exe,
+                   '"%s"' % os.path.join(path, 'start_scheduler.py'),
                    '--client-port', str(self.client_port),
                    '--worker-port', str(self.worker_port)]
         if self.port_scan_range is not None:
             cmdline.extend(["--port-scan-range", str(self.port_scan_range)])
         if self.log_file is not None:
-            cmdline.extend(['--log-file', self.log_file])
+            cmdline.extend(['--log-file', '"%s"' % self.log_file])
         if self.db_api_module is not None:
             cmdline.extend(['--db-api-module', self.db_api_module])
         if self.allow_shutdown:
@@ -242,10 +244,11 @@
         """Start the worker"""
         python_exe = sys.executable
         path = os.path.join(GreaterModel.__path__[0], '..')
-        cmdline = [python_exe, os.path.join(path, 'start_worker.py'),
+        cmdline = ['"%s"' % python_exe,
+                   '"%s"' % os.path.join(path, 'start_worker.py'),
                    '--port', str(self.scheduler_port)]
         if (self.log_file is not None):
-            cmdline.extend(['--log-file', self.log_file])
+            cmdline.extend(['--log-file', '"%s"' % self.log_file])
         if (self.db_api_module is not None):
             cmdline.extend(['--db-api-module', self.db_api_module])
 
@@ -337,10 +340,13 @@
                                           db_api_module = self.db_api_module,
                                           log_file = os.path.join(logdir, "scheduler.log"))
                 server.start()
-                worker = WorkerProcess(scheduler_port = server.real_worker_port,
-                                       db_api_module = self.db_api_module,
-                                       log_file = os.path.join(logdir, "worker%s.log" % "1"))
-                worker.start()
+                # Start multiple workers for the CPU's available
+                for i in range(0, cpu_count()):
+                    worker = WorkerProcess(scheduler_port = server.real_worker_port,
+                                           test_port = 10103 + i,
+                                           db_api_module = self.db_api_module,
+                                           log_file = os.path.join(logdir, "worker%s.log" % i))
+                    worker.start()
                 server.wait_for_worker()
                 self.rpc = xmlrpclib.Server('http://localhost:%d' % \
                                         server.real_client_port)