Snowflake Reporting
Knowi enables visualization, warehousing and reporting automation from Snowflake along with other structured and unstructured datasources.
Overview
-
Connect, extract and transform data from your Snowflake database, using one of the following options:
- Using our Cloud9Agent. This can securely pull data inside your network. See agent configuration for more details.
- Through our UI to connect directly, if the database is accessible from the cloud.
-
Visualize and Automate your Reporting instantly.
UI
If you are not a current Knowi user, check out our Snowflake Instant Reporting page to get started and sign up.
Connecting
-
Login to Knowi and select Settings -> New Datasources from the left down menu.
-
Click on Snowflake. Either follow the prompts to set up connectivity to your own Snowflake database, or, use Knowi's own simulation demo Snowflake database.
-
Snowflake is a Writable Destination. Meaning it allows you to use this database to write datasets into this schema. When a query is saved, this database will appear as an option to write into for that dataset. To enable this, you can check the Writable Destination checkbox.
If you connecting through an agent, check Internal Datasource to assign it to your agent. The agent (running inside your network) will synchronize it automatically. Alternatively, configure datasource and queries directly into agent.
When connecting from the UI directly to your Snowflake Server database, please follow the connectivity instructions to allow Knowi to access your database.
-
Save the Connection. Click on the "Configure Queries" link on the success bar.
Queries & Reports
-
This section enables you to set up automated queries that can either be run on a schedule or once.
Report Name: Specify a name for the report.
SQL Query: Modify or enter SQL queries directly.
Cloud9QL: Optional SQL-Like post processor for the data returned by the SQL query. See Cloud9QL Docs for more details.
If you're using the Knowi simulation demo Snowflake database, click 'Preview' to see the results.
-
Scheduling: Configure how often this should be run. Select 'None' for a one time operation. The results are stored within Knowi.
-
Overwrite Strategy (for scheduled query runs):
Overwrite Strategies determines how the data is stored in Knowi:
i. If empty, data will be added on to the existing data for this dataset within Knowi. Or,
ii. "All": Any existing data for this dataset will be replaced by this results.
iii. One or More Field Names (Example: "A,B,C"): A new record is created where the values of the combination of the field names do not exist, and, updates current records for the field grouping where it exists. For example, if this is set to say "Date, Type", existing data with the same Date and Type values will be updated with the latest data, and new records created when they do not exist.
-
Click on 'Save' to complete setting up the report. This also sets up this data extraction on a schedule, if configured.
-
Click on 'Dashboards' to access dashboards. You can drag and drop the newly created report from the bottom list into to the dashboard.
Cloud9Agent Configuration
As an alternative to the UI based connectivity above, you can use Cloud9Agent inside your network to pull from SQL Server securely. See Cloud9Agent to download your agent along with instructions to run it.
Highlights:
- Pull data using SQL.
- Execute queries on a schedule, or, one time.
The agent contains a datasource_example_snowflake.json and query_example_snowflake.json under the examples folder of the agent installation to get you started.
- Edit those to point to your database and modify the queries to pull your data.
- Move it into the config directory (datasource_XXX.json files first if the Agent is running).
Datasource Configuration:
Parameter | Comments |
---|---|
name | Unique Datasource Name. |
datasource | Set value to snowflake |
url | URL with host, database, warehouse, and schema to connect to. Example for Snowflake: abc123.snowflakecomputing.com/?warehouse=mywh&db=mydb&schema=public |
userId | User ID to connect |
Password | Password |
Query Configuration:
Query Config Params | Comments |
---|---|
entityName | Dataset Name Identifier |
identifier | A unique identifier for the dataset. Either identifier or entityName must be specified. |
dsName | Name of the datasource name configured in the datasource_XXX.json file to execute the query against. Required. |
queryStr | SQL Snowflake query to execute. Required. |
frequencyType | One of minutes, hours, days,weeks,months. If this is not specified, this is treated as a one time query, executed upon Cloud9Agent startup (or when the query is first saved) |
frequency | Indicates the frequency, if frequencyType is defined. For example, if this value is 10 and the frequencyType is minutes, the query will be executed every 10 minutes |
startTime | Optional, can be used to specify when the query should be run for the first time. If set, the the frequency will be determined from that time onwards. For example, is a weekly run is scheduled to start at 07/01/2014 13:30, the first run will run on 07/01 at 13:30, with the next run at the same time on 07/08/2014. The time is based on the local time of the machine running the Agent. Supported Date Formats: MM/dd/yyyy HH:mm, MM/dd/yy HH:mm, MM/dd/yyyy, MM/dd/yy, HH:mm:ss,HH:mm,mm |
c9QLFilter | Optional post processing of the results using Cloud9QL. Typically uncommon against SQL based datastores. |
overrideVals | This enables data storage strategies to be specified. If this is not defined, the results of the query is added to the existing dataset. To replace all data for this dataset within Knowi, specify {"replaceAll":true}. To upsert data specify "replaceValuesForKey":["fieldA","fieldB"]. This will replace all existing records in Knowi with the same fieldA and fieldB with the the current data and insert records where they are not present. |
Datasource Example:
[
{
"name":"demoSqlServer",
"url":"abc123.snowflakecomputing.com/?warehouse=mywh&db=mydb&schema=public",
"datasource":"snowflake",
"userId":"cloud9demo",
"password":"cloud92014"
}
]
Query Examples:
[
{
"entityName":"Errors",
"dsName":"demoSqlServer",
"queryStr":"select error_condition as 'Error', count 'Count' from errors",
"frequencyType":"minute",
"frequency":10,
"overrideVals":{
"replaceAll":true
}
},
{
"entityName":"Queues",
"dsName":"demoSqlServer",
"queryStr":"select Name, size as 'Queue Size', Type from queue",
"overrideVals":{
"replaceValuesForKey":["Type"]
},
"startTime":"07:20",
"frequencyType":"daily",
"frequency":1
}
]
The first query is run every 10 minutes at the top of the hour and replaces all data for that dataset in Knowi. The second is run once a day at 07:20 AM and updates existing data with the same Type field, or inserts new records otherwise.