g.cloud - GRASS GIS manual

Note: This addon document is for an older version of GRASS GIS that will be discontinued soon. You should upgrade your GRASS GIS installation, and read the current addon manual page.

NAME

g.cloud - Connects a local GRASS GIS session to another one in a cluster system.

KEYWORDS

general, cloud computing

SYNOPSIS

g.cloud

g.cloud --help

g.cloud [-cka] config=string server=name [grass_script=name] [qsub_script=name] [variables=name] [raster=name[,name,...]] [vector=name[,name,...]] [mail=name] [reconnect=name] [path=name] [--help] [--verbose] [--quiet] [--ui]

Flags:

-c: Cycle through all variables
-k: Keep temporal files and mapsets
-a: Use ssh-add for faster access (requires to launch ssh-agent beforehand)
--help: Print usage summary
--verbose: Verbose module output
--quiet: Quiet module output
--ui: Force launching GUI dialog

Parameters:

config=string [required]: Path to ASCII file containing authentication parameters; "-" to pass the parameters interactively
server=name [required]: Name or IP of server to be connected
grass_script=name: Path to the input GRASS script
qsub_script=name: Path to the input qsub script
variables=name: Python dictionary with the variables to pass to the GRASS script via qsub; Example: "{'year':[2010,2011]}" will call 'qsub -v year=2010 ...' and 'qsub -v year=2011 ...'
raster=name[,name,...]: Name of input raster map(s) used by GRASS script
vector=name[,name,...]: Name of input vector map(s) used by GRASS script
mail=name: Mail address to send emails at the end of jobs
reconnect=name: Reconnect with old job to see if it is finished
path=name: Path for temporal for g.cloud operations; The directory must be visible on the network to the blades; Default: $HOME

DESCRIPTION
NOTES
EXAMPLE
REQUIREMENTS
REFERENCES
SEE ALSO
AUTHORS

DESCRIPTION

g.cloud is a module to connect a personal GRASS GIS session to a remote GRASS GIS installation on a much faster cluster facility (high performance computing system). It currently only supports the Grid Engine system. The general aim is to offer the possibility to a user to perform high speed calculations, which are hard to compute on a personal computer by connecting to an external, powerful server or cloud computing system.

The g.cloud environment consists of a Python module. It checks if all program requirements are fullfilled on both the client and the server side. If everything is ok, it connects the current session to the cloud server; it transfers the user and other required scripts as well as the GRASS data to the server and eventually executes the remote GRASS GIS job. Subsequently, it transfers back the resulting maps and data and cleans up the server session.

For a job execution, the script requirements are two: on the client-side the user prepared GRASS GIS script which performs the calculations and on the server-side the qsub control script for Grid Engine to launch the GRASS GIS script job.

NOTES

It is highly recommended to use "ssh-add" in order to avoid the authentication to the cluster via password file.

EXAMPLE

North Carolina example, calculation of texture maps:

g.cloud config=$HOME/.gc_loginfile.txt server=giscluster \
  grass_script=$HOME/g.cloud/test/test_onevariable_raster.sh \
  qsub_script=$HOME/g.cloud/test/launch_SGE_grassjob.sh \
  variables="{'TEXTURE' : ['asm','corr','entr','se','var']}" \
  raster=lsat7_2002_40

North Carolina example, calculation of daily sun radiation:

g.cloud config=$HOME/.gc_loginfile.txt server=giscluster \
  grass_script=$HOME/g.cloud/test/test_onevariable_sun.sh \
  qsub_script=$HOME/test/launch_SGE_grassjob.sh \
  variables="{'DOY' : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325]}" \
  raster=elevation

REQUIREMENTS

The software requirements are GRASS GIS 7 and ssh/scp on the client side; while on the server side the Grid Engine software or compatible software needs to be present to launch the GRASS jobs. Furthermore, python-pexpect is required to be installed on the client system.

REFERENCES

FOSS4G 2011 presentation

AUTHORS

Luca Delucchi, Markus Neteler, Fondazione Edmund Mach, Research and Innovation Centre, Department of Biodiversity and Molecular Ecology, GIS and Remote Sensing Unit

SOURCE CODE

Available at: g.cloud source code (history)

Latest change: Mon Jun 28 07:54:09 2021 in commit: 1cfc0af029a35a5d6c7dae5ca7204d0eb85dbc55