Custom Processing in Agent – Documentation and Support | Knowi

In the uncommon scenario where you need custom processing of data beyond the native query support and Cloud9QL, custom processors can be plugged into datasource and query files to enable custom logic for manipulating/processing data from your datasources.

Custom processors can be plugged as part of a query configuration into the agent to pre/post process data.

Example use cases: send data to a third party API during a report process, perform prediction algorithms using custom algorithms ...

Cloud9Agent Query configuration example:

   [
     {
       "entityName": "Volume Predictions",
       "dsName": "mySQLDS",
       "queryStr": "SELECT Emails, Date FROM emails where Date >= $c9_today-365d and  Date < $c9_today",
       "customProcessors": [
         {
           "className": "com.example.ExampleCustomProcessor",
           "processOrder": "post",
           "params": {
             "someKey": "valA",
             "someKey1": 1
           }
         }
       ],
       "frequencyType": "hourly",
       "frequency": 2
     }
   ]

The above:

Executes a query every two hours to the underlying datasource defined (mySQLDS).
One or more custom processors can be defined for this report, either with a pre query processing or post query processing (example above), where the results of the query is passed into a Custom Processor for further processing.

Notes:

1. Custom processors must be implemented in Java and must extend a BaseProcessor class (or an implementation of ClientCustomProcessor) we provide.
2. Package up the custom processor into a jar file (example: jar cvf TicTacToe.jar TickTackToe.class). Place the jar file into the lib folder of your agent.
3. The agent must be restarted to pick up any changes to the custom processor.

Example Custom Processor:

package example;

import com.cloud9.connector.ConnectorContext;
import com.cloud9.connector.processors.BaseProcessor;
import com.cloud9.connector.server.dto.DataSource;
import com.cloud9.connector.server.dto.Query;
import com.cloud9.core.TabularData;
import com.cloud9.frmwk.c9query.C9Query;
import com.cloud9.frmwk.processors.C9Context;
import org.apache.log4j.Logger;

import java.util.List;
import java.util.Map;


 /**
  * Example Custom Processor Configuration. Custom Processor allows you manipulate the
  * data beyond native query and Cloud9QL to perform custom processing of the data.
  *
  * To Deploy: Package up any custom processors into a jar file and drop it into the 'lib'
  * folder of the Cloud9Agent. Restart the agent.
  *
  * Example custom processor configuration in the query file:
  *         "customProcessors":
  *                [
  *                    {
  *                        "className":"example.ExampleCustomProcessor",
  *                        "processOrder":"postDataLoad",
  *                        "params":{
  *                            "customProperty":"someVal"
  *                        }
  *                    }
  *                ]
  *
  *
  * @author Knowi
  */
public class ExampleCustomProcessor extends BaseProcessor {

    private static final Logger LOGGER = Logger.getLogger(ExampleCustomProcessor.class);


    public void process(C9Context context) {
        try {

            // Processor custom config parameters, if you need it.
            Map<String, Object> params = (Map) context.get(ConnectorContext.IDENTIFIER_CUSTOM_PROCESSOR_PARAMS_OBJ);
            //Datasource object, if you need it
            DataSource ds = (DataSource) context.get(ConnectorContext.IDENTIFIER_DATASOURCE_OBJ);
            //Query object, if you need it
            Query query = (Query) context.get(ConnectorContext.IDENTIFIER_QUERY_OBJ);

            // Get the Data - TabularData is our object representation of the data we've processed thus far
            TabularData data = (TabularData) context.get(ConnectorContext.IDENTIFIER_DATA_OBJ);

            // Do some custom processing. Here, we get a subset of the data we've got so far
            TabularData subset = C9Query.executeQueries("select * where something > 0", data);

            // Get Columns
            List<String> columns = subset.getColumns();

            // Iterate through rows
            while(subset.next()){
                TabularData.TabularRow row = subset.getRow();
                // Each row item for a row
                for(Object obj: row.getRowItemsAsObjArr()){

                }
                // Or, if you want to grab the data for a specific field
                subset.getValAsObject("someCol");
            }
            // Do your custom stuff

            // If you do not want to post the data to Knowi:
            // throw new SkipProcessException();

        } catch (Exception e) {
            LOGGER.error("Unable to execute Processor");
            // Handle any exceptions as you see fit
        }
    }
}

Related articles