pygrass.modules.interface package¶
Submodules¶
pygrass.modules.interface.docstring module¶
-
class
pygrass.modules.interface.docstring.
DocstringProperty
(class_doc, fget)[source]¶ Bases:
object
Property for the __doc__ attribute.
Different than property in the following two ways:
- When the attribute is accessed from the main class, it returns the value of class_doc, not the property itself. This is necessary so Sphinx and other documentation tools can access the class docstring.
- Only supports getting the attribute; setting and deleting raise an AttributeError.
-
pygrass.modules.interface.docstring.
docstring_property
(class_doc)[source]¶ Property attribute for docstrings. Took from: https://gist.github.com/bfroehle/4041015
>>> class A(object): ... '''Main docstring''' ... def __init__(self, x): ... self.x = x ... @docstring_property(__doc__) ... def __doc__(self): ... return "My value of x is %s." % self.x
>>> A.__doc__ 'Main docstring'
>>> a = A(10) >>> a.__doc__ 'My value of x is 10.'
pygrass.modules.interface.env module¶
Created on Thu May 28 17:41:32 2015
@author: pietro
pygrass.modules.interface.flag module¶
-
class
pygrass.modules.interface.flag.
Flag
(xflag=None, diz=None)[source]¶ Bases:
object
The Flag object store all information about a flag of module.
It is possible to set flags of command using this object.
>>> flag = Flag(diz=dict(name='a', description='Flag description', ... default=True)) >>> flag.name 'a' >>> flag.special False >>> flag.description 'Flag description' >>> flag = Flag(diz=dict(name='overwrite')) >>> flag.name 'overwrite' >>> flag.special True
-
get_bash
()[source]¶ Return the BASH representation of a flag.
>>> flag = Flag(diz=dict(name='a', description='Flag description', ... default=True)) >>> flag.get_bash() '' >>> flag.value = True >>> flag.get_bash() '-a' >>> flag = Flag(diz=dict(name='overwrite')) >>> flag.get_bash() '' >>> flag.value = True >>> flag.get_bash() '--o'
-
get_python
()[source]¶ Return the python representation of a flag.
>>> flag = Flag(diz=dict(name='a', description='Flag description', ... default=True)) >>> flag.get_python() '' >>> flag.value = True >>> flag.get_python() 'a' >>> flag = Flag(diz=dict(name='overwrite')) >>> flag.get_python() '' >>> flag.value = True >>> flag.get_python() 'overwrite=True'
-
pygrass.modules.interface.module module¶
-
class
pygrass.modules.interface.module.
Module
(cmd, *args, **kargs)[source]¶ Bases:
object
This class is design to wrap/run/interact with the GRASS modules.
The class during the init phase read the XML description generate using the
--interface-description
in order to understand which parameters are required which optionals.>>> from grass.pygrass.modules import Module >>> from subprocess import PIPE >>> import copy
>>> region = Module("g.region") >>> region.flags.p = True # set flags >>> region.flags.u = True >>> region.flags["3"].value = True # set numeric flags >>> region.get_bash() 'g.region -p -3 -u' >>> new_region = copy.deepcopy(region) >>> new_region.inputs.res = "10" >>> new_region.get_bash() 'g.region res=10 -p -3 -u'
>>> neighbors = Module("r.neighbors") >>> neighbors.inputs.input = "mapA" >>> neighbors.outputs.output = "mapB" >>> neighbors.inputs.size = 5 >>> neighbors.inputs.quantile = 0.5 >>> neighbors.get_bash() 'r.neighbors input=mapA method=average size=5 quantile=0.5 output=mapB'
>>> new_neighbors1 = copy.deepcopy(neighbors) >>> new_neighbors1.inputs.input = "mapD" >>> new_neighbors1.inputs.size = 3 >>> new_neighbors1.inputs.quantile = 0.5 >>> new_neighbors1.get_bash() 'r.neighbors input=mapD method=average size=3 quantile=0.5 output=mapB'
>>> new_neighbors2 = copy.deepcopy(neighbors) >>> new_neighbors2(input="mapD", size=3, run_=False) Module('r.neighbors') >>> new_neighbors2.get_bash() 'r.neighbors input=mapD method=average size=3 quantile=0.5 output=mapB'
>>> neighbors = Module("r.neighbors") >>> neighbors.get_bash() 'r.neighbors method=average size=3'
>>> new_neighbors3 = copy.deepcopy(neighbors) >>> new_neighbors3(input="mapA", size=3, output="mapB", run_=False) Module('r.neighbors') >>> new_neighbors3.get_bash() 'r.neighbors input=mapA method=average size=3 output=mapB'
>>> mapcalc = Module("r.mapcalc", expression="test_a = 1", ... overwrite=True, run_=False) >>> mapcalc.run() Module('r.mapcalc') >>> mapcalc.returncode 0
>>> mapcalc = Module("r.mapcalc", expression="test_a = 1", ... overwrite=True, run_=False, finish_=False) >>> mapcalc.run() Module('r.mapcalc') >>> p = mapcalc.wait() >>> p.returncode 0 >>> mapcalc.run() Module('r.mapcalc') >>> p = mapcalc.wait() >>> p.returncode 0
>>> colors = Module("r.colors", map="test_a", rules="-", ... run_=False, stdout_=PIPE, ... stderr_=PIPE, stdin_="1 red") >>> colors.run() Module('r.colors') >>> p = mapcalc.wait() >>> p.returncode 0 >>> colors.inputs["stdin"].value '1 red' >>> colors.outputs["stdout"].value '' >>> colors.outputs["stderr"].value.strip() "Color table for raster map <test_a> set to 'rules'"
>>> colors = Module("r.colors", map="test_a", rules="-", ... run_=False, finish_=False, stdin_=PIPE) >>> colors.inputs["stdin"].value = "1 red" >>> colors.run() Module('r.colors') >>> colors.wait() Module('r.colors') >>> colors.returncode 0
>>> colors = Module("r.colors", map="test_a", rules="-", ... run_=False, finish_=False, ... stdin_=PIPE, stderr_=PIPE) >>> colors.inputs["stdin"].value = "1 red" >>> colors.run() Module('r.colors') >>> colors.wait() Module('r.colors') >>> colors.outputs["stderr"].value.strip() "Color table for raster map <test_a> set to 'rules'"
>>> colors.returncode 0
Run a second time
>>> colors.inputs["stdin"].value = "1 red" >>> colors.run() Module('r.colors') >>> colors.wait() Module('r.colors') >>> colors.outputs["stderr"].value.strip() "Color table for raster map <test_a> set to 'rules'"
>>> colors.returncode 0
Multiple run test
>>> colors = Module("r.colors", map="test_a", ... color="ryb", run_=False) >>> colors.get_bash() 'r.colors map=test_a color=ryb offset=0.0 scale=1.0' >>> colors.run() Module('r.colors') >>> colors(color="gyr") Module('r.colors') >>> colors.run() Module('r.colors') >>> colors(color="ryg") Module('r.colors') >>> colors(stderr_=PIPE) Module('r.colors') >>> colors.run() Module('r.colors') >>> print(colors.outputs["stderr"].value.strip()) Color table for raster map <test_a> set to 'ryg' >>> colors(color="byg") Module('r.colors') >>> colors(stdout_=PIPE) Module('r.colors') >>> colors.run() Module('r.colors') >>> print(colors.outputs["stderr"].value.strip()) Color table for raster map <test_a> set to 'byg'
Often in the Module class you can find
*args
andkwargs
annotation in methods, like in the __call__ method. Python allow developers to not specify all the arguments and keyword arguments of a method or function.def f(*args): for arg in args: print arg
therefore if we call the function like:
>>> f('grass', 'gis', 'modules') # doctest: +SKIP grass gis modules
or we can define a new list:
>>> words = ['grass', 'gis', 'modules'] # doctest: +SKIP >>> f(*words) # doctest: +SKIP grass gis modules
we can do the same with keyword arguments, rewrite the above function:
def f(*args, **kargs): for arg in args: print arg for key, value in kargs.items(): print "%s = %r" % (key, value)
now we can use the new function, with:
>>> f('grass', 'gis', 'modules', os = 'linux', language = 'python') ... # doctest: +SKIP grass gis modules os = 'linux' language = 'python'
or, as before we can, define a dictionary and give the dictionary to the function, like:
>>> keywords = {'os' : 'linux', 'language' : 'python'} # doctest: +SKIP >>> f(*words, **keywords) # doctest: +SKIP grass gis modules os = 'linux' language = 'python'
In the Module class we heavily use this language feature to pass arguments and keyword arguments to the grass module.
-
make_cmd
()[source]¶ Create the command string that can be executed in a shell
Returns: the command string
-
run
()[source]¶ Run the module This function will wait for the process to terminate in case finish_==True and sets up stdout and stderr. If finish_==False this function will return after starting the process. Use wait() to wait for the started process
Returns: A reference to this object
-
wait
()[source]¶ Wait for the module to finish. Call this method if the run() call was performed with self.false_ = False.
Returns: A reference to this object
-
-
class
pygrass.modules.interface.module.
MultiModule
(module_list, sync=True, set_temp_region=False)[source]¶ Bases:
object
This class is designed to run a list of modules in serial in the provided order within a temporary region environment.
Module can be run in serial synchronously or asynchronously.
- Synchronously: When calling run() all modules will run in serial order
until they are finished, The run() method will return until all modules finished. The modules objects can be accessed by calling get_modules() to check their return values.
- Asynchronously: When calling run() all modules will run in serial order in a background process.
Method run() will return after starting the modules without waiting for them to finish. The user must call the wait() method to wait for the modules to finish. Asynchronously called module can be optionally run in a temporary region environment, hence invokeing g.region will not alter the current region or the region of other MultiModule runs.
Note:
Modules run in asynchronous mode can only be accessed via the wait() method. The wait() method will return all finished module objects as list.
Objects of this class can be passed to the ParallelModuleQueue to run serial stacks of modules in parallel. This is meaningful if region settings must be applied to each parallel module run.
>>> from grass.pygrass.modules import Module >>> from grass.pygrass.modules import MultiModule >>> from multiprocessing import Process >>> import copy
Synchronous module run
>>> region_1 = Module("g.region", run_=False) >>> region_1.flags.p = True >>> region_2 = copy.deepcopy(region_1) >>> region_2.flags.p = True >>> mm = MultiModule(module_list=[region_1, region_2]) >>> mm.run() >>> m_list = mm.get_modules() >>> m_list[0].returncode 0 >>> m_list[1].returncode 0
Asynchronous module run, setting finish = False
>>> region_1 = Module("g.region", run_=False) >>> region_1.flags.p = True >>> region_2 = copy.deepcopy(region_1) >>> region_2.flags.p = True >>> region_3 = copy.deepcopy(region_1) >>> region_3.flags.p = True >>> region_4 = copy.deepcopy(region_1) >>> region_4.flags.p = True >>> region_5 = copy.deepcopy(region_1) >>> region_5.flags.p = True >>> mm = MultiModule(module_list=[region_1, region_2, region_3, region_4, region_5], ... sync=False) >>> t = mm.run() >>> isinstance(t, Process) True >>> m_list = mm.wait() >>> m_list[0].returncode 0 >>> m_list[1].returncode 0 >>> m_list[2].returncode 0 >>> m_list[3].returncode 0 >>> m_list[4].returncode 0
Asynchronous module run, setting finish = False and using temporary region
>>> mm = MultiModule(module_list=[region_1, region_2, region_3, region_4, region_5], ... sync=False, set_temp_region=True) >>> str(mm) 'g.region -p ; g.region -p ; g.region -p ; g.region -p ; g.region -p' >>> t = mm.run() >>> isinstance(t, Process) True >>> m_list = mm.wait() >>> m_list[0].returncode 0 >>> m_list[1].returncode 0 >>> m_list[2].returncode 0 >>> m_list[3].returncode 0 >>> m_list[4].returncode 0
-
get_modules
()[source]¶ Return the list of modules that have been run in synchronous mode
Note: Asynchronously run module can only be accessed via the wait() method.
Returns: The list of modules
-
run
()[source]¶ Start the modules in the list. If self.finished_ is set True this method will return after all processes finished.
If self.finish_ is set False, this method will return after the process list was started for execution. In a background process, the processes in the list will be run one after the another.
Returns: None in case of self.finish_ is True, otherwise a multiprocessing.Process object that invokes the modules
-
class
pygrass.modules.interface.module.
ParallelModuleQueue
(nprocs=1)[source]¶ Bases:
object
This class is designed to run an arbitrary number of pygrass Module or MultiModule processes in parallel.
Objects of type grass.pygrass.modules.Module or grass.pygrass.modules.MultiModule can be put into the queue using put() method. When the queue is full with the maximum number of parallel processes it will wait for all processes to finish, sets the stdout and stderr of the Module object and removes it from the queue when its finished.
To finish the queue before the maximum number of parallel processes was reached call wait() .
This class will raise a GrassError in case a Module process exits with a return code other than 0.
Processes that were run asynchronously with the MultiModule class will not raise a GrassError in case of failure. This must be manually checked by accessing finished modules by calling get_finished_modules().
Usage:
Check with a queue size of 3 and 5 processes
>>> import copy >>> from grass.pygrass.modules import Module, MultiModule, ParallelModuleQueue >>> mapcalc_list = []
Setting run_ to False is important, otherwise a parallel processing is not possible
>>> mapcalc = Module("r.mapcalc", overwrite=True, run_=False) >>> queue = ParallelModuleQueue(nprocs=3) >>> for i in range(5): ... new_mapcalc = copy.deepcopy(mapcalc) ... mapcalc_list.append(new_mapcalc) ... m = new_mapcalc(expression="test_pygrass_%i = %i"%(i, i)) ... queue.put(m) >>> queue.wait() >>> mapcalc_list = queue.get_finished_modules() >>> queue.get_num_run_procs() 0 >>> queue.get_max_num_procs() 3 >>> for mapcalc in mapcalc_list: ... print(mapcalc.returncode) 0 0 0 0 0
Check with a queue size of 8 and 5 processes
>>> queue = ParallelModuleQueue(nprocs=8) >>> mapcalc_list = [] >>> for i in range(5): ... new_mapcalc = copy.deepcopy(mapcalc) ... mapcalc_list.append(new_mapcalc) ... m = new_mapcalc(expression="test_pygrass_%i = %i"%(i, i)) ... queue.put(m) >>> queue.wait() >>> mapcalc_list = queue.get_finished_modules() >>> queue.get_num_run_procs() 0 >>> queue.get_max_num_procs() 8 >>> for mapcalc in mapcalc_list: ... print(mapcalc.returncode) 0 0 0 0 0
Check MultiModule approach with three by two processes running in a background process
>>> gregion = Module("g.region", flags="p", run_=False) >>> queue = ParallelModuleQueue(nprocs=3) >>> proc_list = [] >>> for i in range(3): ... new_gregion = copy.deepcopy(gregion) ... proc_list.append(new_gregion) ... new_mapcalc = copy.deepcopy(mapcalc) ... m = new_mapcalc(expression="test_pygrass_%i = %i"%(i, i)) ... proc_list.append(new_mapcalc) ... mm = MultiModule(module_list=[new_gregion, new_mapcalc], sync=False, set_temp_region=True) ... queue.put(mm) >>> queue.wait() >>> proc_list = queue.get_finished_modules() >>> queue.get_num_run_procs() 0 >>> queue.get_max_num_procs() 3 >>> for proc in proc_list: ... print(proc.returncode) 0 0 0 0 0 0
Check with a queue size of 8 and 4 processes
>>> queue = ParallelModuleQueue(nprocs=8) >>> mapcalc_list = [] >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_1 =1") >>> queue.put(m) >>> queue.get_num_run_procs() 1 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_2 =2") >>> queue.put(m) >>> queue.get_num_run_procs() 2 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_3 =3") >>> queue.put(m) >>> queue.get_num_run_procs() 3 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_4 =4") >>> queue.put(m) >>> queue.get_num_run_procs() 4 >>> queue.wait() >>> mapcalc_list = queue.get_finished_modules() >>> queue.get_num_run_procs() 0 >>> queue.get_max_num_procs() 8 >>> for mapcalc in mapcalc_list: ... print(mapcalc.returncode) 0 0 0 0
Check with a queue size of 3 and 4 processes
>>> queue = ParallelModuleQueue(nprocs=3) >>> mapcalc_list = [] >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_1 =1") >>> queue.put(m) >>> queue.get_num_run_procs() 1 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_2 =2") >>> queue.put(m) >>> queue.get_num_run_procs() 2 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_3 =3") >>> queue.put(m) # Now it will wait until all procs finish and set the counter back to 0 >>> queue.get_num_run_procs() 0 >>> new_mapcalc = copy.deepcopy(mapcalc) >>> mapcalc_list.append(new_mapcalc) >>> m = new_mapcalc(expression="test_pygrass_%i = %i"%(i, i)) >>> queue.put(m) >>> queue.get_num_run_procs() 1 >>> queue.wait() >>> mapcalc_list = queue.get_finished_modules() >>> queue.get_num_run_procs() 0 >>> queue.get_max_num_procs() 3 >>> for mapcalc in mapcalc_list: ... print(mapcalc.returncode) 0 0 0 0
-
get
(num)[source]¶ Get a Module object or list of Module objects from the queue
Parameters: num (int) – the number of the object in queue Returns: the Module object or list of Module objects or None if num is not in the queue
-
get_finished_modules
()[source]¶ Return all finished processes that were run by this queue
Returns: A list of Module objects
-
get_max_num_procs
()[source]¶ Return the maximum number of parallel Module processes
Returns: the maximum number of parallel Module processes
-
get_num_run_procs
()[source]¶ Get the number of Module processes that are in the queue running or finished
Returns: the number fo Module processes running/finished in the queue
-
put
(module)[source]¶ Put the next Module or MultiModule object in the queue
To run the Module objects in parallel the run_ and finish_ options of the Module must be set to False.
Parameters: module (Module or MultiModule object) – a preconfigured Module or MultiModule object that were configured with run_ and finish_ set to False,
-
-
pygrass.modules.interface.module.
run_modules
(module_list, q)[source]¶ Run the modules
This function is the argument for multiprocessing.Process class in the MultiModule asynchronous execution.
Parameters: - module_list – The list of modules to run in serial
- q – The process queue to put the finished process list
-
pygrass.modules.interface.module.
run_modules_in_temp_region
(module_list, q)[source]¶ Run the modules in a temporary region environment
This function is the argument for multiprocessing.Process class in the MultiModule asynchronous execution.
Parameters: - module_list – The list of modules to run in serial
- q – The process queue to put the finished process list
pygrass.modules.interface.parameter module¶
Created on Tue Apr 2 18:31:47 2013
@author: pietro
-
class
pygrass.modules.interface.parameter.
Parameter
(xparameter=None, diz=None)[source]¶ Bases:
object
The Parameter object store all information about a parameter of a GRASS GIS module.
>>> param = Parameter(diz=dict(name='int_number', required='yes', ... multiple='no', type='integer', ... values=[2, 4, 6, 8])) >>> param.value = 2 >>> param.value 2 >>> param.value = 3 Traceback (most recent call last): ... ValueError: The Parameter <int_number>, must be one of the following values: [2, 4, 6, 8]
…
-
get_bash
()[source]¶ Return the BASH representation of the parameter.
>>> param = Parameter(diz=dict(name='int_number', required='yes', ... multiple='no', type='integer', ... values=[2, 4, 6, 8], default=8)) >>> param.get_bash() 'int_number=8'
-
get_python
()[source]¶ Return a string with the Python representation of the parameter.
>>> param = Parameter(diz=dict(name='int_number', required='yes', ... multiple='no', type='integer', ... values=[2, 4, 6, 8], default=8)) >>> param.get_python() 'int_number=8'
-
rawvalue
¶ Parameter value as insert by user without transformation
-
value
¶ Parameter value transformed and validated.
-