Cloud Compute Resource¶

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the Cloud tab.
7. Select the resource type from the drop down menu. You may have access to more than one type of cloud service, e.g., NeCTAR and Amazon.
8. Enter a unique resource name.
9. Enter credentials such as EC2 access key and EC2 secret key
10. Click Register. The newly registered cloud-based compute resource will be displayed under Cloud - NeCTAR/CSRack/Amazon EC2.

Figure. Registering a cloud-based compute resource

HPC Compute Resource¶

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the HPC tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the HPC cluster head node or the standalone server.
9. Enter credentials, i.e. username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
10. Click Register. The newly registered resource will be displayed under HPC - Cluster or Standalone Server list.

Figure. Registering a HPC compute resource

Analytics Compute Resource¶

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the Analytics tab.
7. Select Hadoop MapReduce as the resource type from the drop down menu.
8. Enter a unique resource name.
9. Enter IP address or hostname of the Hadoop MapReduce resource.
10. Enter username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
11. Click Register. The newly registered resource will be displayed under Analytics - Hadoop MapReduce list.

Figure. Registering an analytics compute resource (Hadoop MapReduce)

Remote File System¶

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Storage Resource from the Settings menu.
5. Click Register Storage Resource
6. Click the Remote File System tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the remote file system.
9. Enter credentials, i.e. username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
10. Click Register. The newly registered resource will be displayed under Remote File System list.

Figure. Registering a remote file system as a storage resource

MyTardis Storage Resource¶

1. Navigate to the Chiminey portal.
2. Log in with credentials
3. Click Settings
4. Click Storage Resource from the Settings menu
5. Click Register Storage Resource
6. Click the MyTardis tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the MyTardis instance.
9. Enter credentials, i.e. username and password. Username and password are stored on the Chiminey platform.
10. Click Register. The newly registered storage resource will be displayed under MyTardis list.

Figure. Registering MyTardis, a data curation service.

1. Presets¶

The end-user can save the set of parameters values of a job as a preset. Each preset must have a unique name. Using the unique preset name, the end-user can retrieve, update and delete saved presets.

2. Compute Resource¶

This section includes the parameters that are needed to utilise the compute resource associated with the given smart connector. For instance, hadoop compute resources need only the name of the registered hadoop cluster (see Analytics Compute Resource), while the cloud compute resource needs the resource name as well as the total VMs that can be used for the computation. Note that the names of all registered compute resources are automatically populated to a dropdown menu on the submission form.

3. Locations¶

These parameters are used to specify either input or output directories on a registered storage resource. Each location consists of two parameters: a storage location and a relative path. Storage location is a drop-down menu that lists the name of all registered storages and, their corresponding root path. A root path is an absolute path to the directory on the storage resource onto which all input and output files will be saved. Relative path is the name of a subdirectory of the root path that contains input and/or output files. In the case of input locations, Chiminey retrieves the input files that are needed to run the smart connector job from this subdirectory. In the case of output location, Chiminey will save the output of the smart connector job to the subdirectory.

4. Optional Parameter Sections¶

Some job submission forms include one or more of the following sections:

The Core Function¶

The core functionality of a smart connector is provided either via a payload or by overriding the run_task method of chiminey.corestages.execute.Execute class. In this example, we use a minimal payload to provide the core functionality of this smart connector. Thus, we will prepare the following payload.

payload_randnum/
|--- process_payload
│    |--- main.sh

Below is the content of main.sh:

#!/bin/sh
OUTPUT_DIR=$1
echo $RANDOM > $OUTPUT_DIR/signed_randnum date > $OUTPUT_DIR/signed_randnum
# --- EOF ---

Notice OUTPUT_DIR. This is the path to the output directory, and thus Chiminey expects all outputs to be redirected to that location. The contents of OUTPUT_DIR will be transferred to the output location at the end of each computation.

Attaching resources and non-functional properties¶

Resources and non-functional properties are attached to a smart connector by overriding get_ui_schema_namespace method of chiminey.initialisation.coreinitial.CoreInitial class. New domain-specific variables can be introduced via get_domain_specific_schemas method. In this example, we will need to attached a unix compute resource for the computation, and a storage resource for the output location. However, we will not add a non-functional property.

Under chiminey/, we create a python package randnum, and add initialise.py with the following content

from chiminey.initialisation import CoreInitial
from django.conf import settings
class RandNumInitial(CoreInitial):
def get_ui_schema_namespace(self):
        schemas = [
                settings.INPUT_FIELDS[’unix’],
                settings.INPUT_FIELDS[’output_location’],
] return schemas
# ---EOF ---

NB: The list of available resources and non-functional properties is given by INPUT_FIELDS parameter in chiminey/settings_changeme.py

Registration¶

The final step is registering the smart connector with the Chiminey platform. The details of this smart connector will be added to the dictionary SMART CONNECTORS in chiminey/settings changeme.py. The details include a unique name (with no spaces), a python path to RandNumInitial class, the description of the smart connector, and the absolute path to the payload.

"randnum": {
           "name": "randnum",
           "init": "chiminey.randnum.initialise.RandNumInitial",
           "description": "Randnum generator, with timestamp",
           "payload": "/opt/chiminey/current/payload_randnum"
},

Finally, restart the Chiminey platform and then activate randnum smart connector. You need to exit the docker container in order to restart:

$ exit
$ sh restart
$ ./activatesc randnum

Food for Thought¶

In the example above, we created a smart connector that generates a random number on a unix-based machines. Even though the random number generator a simple smart connector, the tasks that are involved in creating a smart connector for complex programs is similar. If your program can be executed on a cloud, HPC cluster, hadoop cluster, then this program can be packaged as a smart connector. The huge benefit of using the Chiminey platform to run your program is you don’t need to worry about how to manage the execution of your program on any of the provided compute resources. You can run your program on different types of compute resources with minimal effort. For instance, to generate random on a cloud-based virtual machine, we need to change only one word in get_ui_schema_namespace method. Replace unix by cloud. Then, restart Chiminey, and activate your cloud-based random number generator.

Check the various examples are given in this documentation. These examples discuss the different types of compute and storage resources, and non-functional properties like reliability and parameter sweep.

Imapct of unknown parameters in a sweep map¶

An unknown parameter is a parameter that is not needed during the execution of a smart connector. A known parameter, on the other hand, is a parameter whose value is needed for the correct functioning of a smart connector.

Including an unknown parameter in a sweep map does not have any ill-effect during execution. However, this parameter causes an increase in the number of generated tasks/jobs, provided that the number of the values of the unknown parameter is more than one.

Suppose var1 and var2 are known parameters, and x is an unknown parameter of a specific smart connector; and the sweep map is {"var1": [7,9], "var2": [42], "x": [1,2]}.

• If the sweep map is used for external parameter sweep, the number of jobs doubles due to the inclusion of parameter x with two values. If the sweep map is used for internal sweep, the number of tasks doubles as well. If the value of x is changed to [1,2,3], the number of jobs/tasks triples, and so on.

The additional jobs or tasks waste computing, storage and network resources. However, there are cases where such feature is useful.

• The end-user can use this feature to run jobs with identical inputs, and then compare whether the jobs produce the same output.
• If each task has unpredictable output irrespective of other variables being constant, the developer can use the feature to run many of these tasks per job, each task with different output. For example, generating a random number without fixing the seed almost always guarantees a new number.

Including domain-specific input fields¶

Input field types that are included within the Chiminey platform are generic and are included within the platform. However domain-specific input fields must be defined when needed. A domain-specific input field type is provided by overriding get_domain_specific_schemas(self) of the CoreInitial class. This method will return a list of two elements:

1. The description of the input field type e.g. HRMCLite Smart Connector

2. A dictionary whose keys are the names of domain-specific input fields, their values are dictionaries with the following keys:

   • type: There are three types of input fields: numeric (models.ParameterName.NUMERIC), string (models.ParameterName.STRING), list of strings (models.ParameterName.STRLIST). numeric and string inputs have a text field while a list of strings has a drop-down menu. Enduser inputs are validated against the type of the input field.
   • subtype: Subtypes are used for additional validations: numeric fields can be validated for containing whole and natural numbers.
   • description: The label of the input field.
   • choices: If the type is list of strings, the values of the dropdown menu is provided via choices.
   • ranking: Ranking sets the ordering of input fields when the fields are displayed.
   • initial: The default value of the field.
   • help_text: The text displayed when a mouse hovers over the question mark next to the field.

Below are two examples of domain-specific input field types: wordcount and HRMCLite smart connector.

• WordCount smart connector input field type

def get_domain_specific_schemas(self):
        schema_data =  [u'Word Count Smart Connector',
             {
                 u'word_pattern': {'type': models.ParameterName.STRING,
                                  'subtype': 'string',
                                  'description': 'Word Pattern',
                                  'ranking': 0,
                                  'initial': "'[a-z.]+'",
                                  'help_text': 'Regular expression of filtered words'},
             }
            ]
        return schema_data

• HRMCLite smart connector input field type

def get_domain_specific_schemas(self):
    schema_data =  [u'HRMCLite Smart Connector',
         {
             u'iseed': {'type': models.ParameterName.NUMERIC,
                        'subtype': 'natural',
                        'description': 'Random Number Seed',
                        'ranking': 0,
                        'initial': 42,
                        'help_text': 'Initial seed for random numbers'},
             u'pottype': {'type': models.ParameterName.NUMERIC,
                          'subtype': 'natural',
                          'description': 'Pottype',
                          'ranking': 10,
                          'help_text': '',
                          'initial': 1},
             u'error_threshold': {'type': models.ParameterName.STRING,
                                  'subtype': 'float',
                                  'description': 'Error Threshold',
                                  'ranking': 23,
                                  'initial': '0.03',
                                  'help_text': 'Delta for iteration convergence'},
             u'optimisation_scheme': {'type': models.ParameterName.STRLIST,
                                      'subtype': 'choicefield',
                                      'description': 'No. varying parameters',
                                      'ranking': 45,
                                      'choices': '[("MC","Monte Carlo"), ("MCSA", "Monte Carlo with Simulated Annealing")]',
                                      'initial': 'MC', 'help_text': '',
                                      'hidefield': 'http://rmit.edu.au/schemas/input/hrmc/fanout_per_kept_result',
                                      'hidecondition': '== "MCSA"'},
             u'fanout_per_kept_result': {'type': models.ParameterName.NUMERIC,
                                        'subtype': 'natural',
                                         'description': 'No. fanout kept per result',
                                         'initial': 1,
                                         'ranking': 52,
                                         'help_text': ''},
             u'threshold': {'type': models.ParameterName.STRING,
                            'subtype': 'string',
                            'description': 'No. results kept per iteration',
                            'ranking': 60,
                            'initial': '[1]',
                            'help_text': 'Number of outputs to keep between iterations. eg. [2] would keep the top 2 results.'},
             u'max_iteration': {'type': models.ParameterName.NUMERIC,
                                'subtype': 'whole',
                                'description': 'Maximum no. iterations',
                                'ranking': 72,
                                'initial': 10,
                                'help_text': 'Computation ends when either convergence or maximum iteration reached'},
         }
        ]

    return schema_data