Data Source Configuration
Upriver allows you to add and monitor any data source with just a few clicks.
Configuration
Each data source has the following attributes that make up it's configuration.
Basic Settings
Name
This is the name with which the data source is displayed in both the data set catalog as well as any other pages (such as incidents and notifications).
This name can can be changed in order to provide a name that is more meaningful than the name given to the physical storage.
Owners
You can assign users to a data source by selecting from any of the users who have access to the platform. This feature enhances collaboration by clearly identifying who is responsible for each data source.
When a user is assigned as the owner, they will be tagged and notified whenever incidents occur related to their data source, ensuring timely attention and resolution.
Type
Select the data source type from the supported types. For each data source type, a different sub-menu will appear with the relevant details for the data source.
S3
Bucket - The name of the given bucket or the path to it (e.g. s3://bucket-name).
Bucket Prefix - The prefix inside the bucket which needs to be monitored.
Region - The region where the bucket is set.
Format - The format of the data in the given prefix (can be JSON, Parquet, CSV, Iceberg)
BigQuery
Integration - Choose the integration configured (find instructions in BigQuery).
Dataset - Name of the BigQuery dataset.
Table - Name of the table.
For example given the following table upriver-proj.product_analytics.events , integration would be the one set on upriver-proj , dataset would be product_analytics and table would be events.
Kinesis
Stream Name - The name of the stream containing the data.
Region - The region where the stream is set.
GCS
Bucket - The name of the given bucket or the path to it (e.g. s3://bucket-name).
Bucket Prefix - The prefix inside the bucket which needs to be monitored.
Region - The region where the bucket is set.
Format - The format of the data in the given prefix (can be JSON, Parquet, CSV, Iceberg)
Snowflake
Database - The name of the Snowflake database.
Schema - The schema within the database.
Table - The name of the table to be monitored.
Confluent Kafka
Integration - Choose the integration configured (find instructions in Confluent Kafka).
Topic - Stream topic.
Table - Group ID of the consumer.
Redshift
Integration - Choose the integration configured (find instructions in Redshift).
Database - The database the table is in.
Schema - The schema the table is in.
Table - The table to monitor.
Tags
These tags can be created, to allow additional segmentation as they appear inside the data source, in incidents, and in alerts.
Description
This field allows for a detailed description of the data source, to provide context to anyone who needs to review it.
Connection
In this creen, you can configure the access details for your chosen connector (connector types are above).
Note: The button for "create more" will allow, once this setup is complete, to start again, with much of the configuration copied over, in case you are defining more than a single source at once.
Monitor Scheduling
The interval in which the data analysis process will run. Users can select multiple options for when to run a scan on that specific data source.
When choosing Update Time , users can set the exact time and delay they want for that data source, in order to make sure it runs at the exact time they want, and on the right time window.
Set the
Hour&minutein UTC.Set the
Run Delay, This value delays the effective run time of the extraction. For example: setting a delay of 10 hours means the extraction will process data as if it ran 10 hours earlier than the scheduled time.
When choosing Dynamic Update, Upriver's algorithms decides & sets the relevant run interval time by analyzing the update patterns seen for the data source
When choosing Manual Trigger , the source will only start a scan when triggered manually by a user.
Pivot Fields
Here, you can decide if you want to define pivots, which would allow a more granular approach to monitoring, pivoting tables based on fields you decide to be created as child data sources to the parent source. This would allow the platform to detect changes & anomalies for subsets of the data (specific products, specific regions, etc) inside big data sources.
There are a two distinct ways to define pivots:
Nested - Splits the data in layers. The first pivot groups the main table, then each group is split again by the next pivot, and so on. Useful for drilling down into data step by step.
Multiple - Applies each pivot directly to the main table, creating separate groupings side by side. Best for comparing different dimensions independently.
Additional configuration options:
Pause new derived datasources - this means any newly created pivots will be created in "paused" mode, so they will not go into learning mode automatically. This helps when creating a lot of new pivots, letting the user start them gradually or by priority once the right ones are chose.
Set new derived data sources as manual trigger - This will set all newly created pivots to run in manual mode, meaning the user can choose which to run, or change to triggered runs.
Advanced Configuration
Primary Key
A primary key is a column (or set of columns) that uniquely identifies each row in a table. This should be defined per table.
Sampling Rate
The percentage of the data that will be used to create the profile. Users can either decide the percentage of the data (0-1) or select dynamic sampling.
If the user decides to use Dynamic Sampling , Upriver's algorithms decide the sampling rate based on the volume of data and the distribution it sees.
Filter
A filter can be used to only monitor parts of the data where a certain condition holds true. In order to use the filter, the user needs to enter the name of field/column they wish to filter and the desired value.
Cardinality Threshold
The number of distinct values needed to determine if a field is categorical or not.
Fields with less distinct values than the set threshold will be considered categorical.
Default Incident Severity
The user can set the default severity that will be assigned to incidents related to the datasource. See Managing Incidents
Staleness Threshold
The maximum number of days a datasource can go without seeing data before it is considered "stale". Stale data sources are considered those not being updated.
Timestamp Column
Specifies the name of the column to be used as the timestamp for each row in the table.
Nullify Empty Strings (Checkbox)
If enabled, empty strings ("") will be treated as empty fields when calculating the completeness of a field.
Nullify Empty Arrays (Checkbox)
If enabled, empty arrays ([]) will be treated as empty fields when calculating the completeness of a field.
Only analyze updated data (checkbox)
If enabled, the platform will only scan updated data (delta) from the last scan.
For full scans each time, make sure not to check this box.
Notification Configuration
External ID
A user-defined identifier that represents the ID used in the its systems.
Webhook
A URL where incidents will be sent. Enables integration with external systems by forwarding incident data in real time.
Slack
The ID for the channel that will receive Upriver incident alerts. If the Slack app is not used, a single channel can be defined. If it is, multiple channels can be used (per data source).
Last updated