> For the complete documentation index, see [llms.txt](https://docs.upriverdata.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.upriverdata.com/product-usage/data-assets/data-source/data-source-configuration.md).

# Data Source Configuration

Upriver allows you to add and monitor any data source with just a few clicks.

## Configuration

Each data source has the following attributes that make up it's configuration.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FvPETVrjFC6hSFqCPskfa%2Fimage.png?alt=media&amp;token=6f9ba704-6b5d-45d5-bcdd-cd128ffcd179" alt=""><figcaption></figcaption></figure>

### Basic Settings

#### Name

This is the name with which the data source is displayed in both the data set catalog as well as any other pages (such as incidents and notifications).

This name can can be changed in order to provide a name that is more meaningful than the name given to the physical storage.

#### Owners

You can assign users to a data source by selecting from any of the users who have access to the platform. This feature enhances collaboration by clearly identifying who is responsible for each data source.

When a user is assigned as the owner, they will be tagged and notified whenever incidents occur related to their data source, ensuring timely attention and resolution.

#### Type

Select the data source type from the supported types. For each data source type, a different sub-menu will appear with the relevant details for the data source.

<details>

<summary>S3</summary>

**Bucket** - The name of the given bucket or the path to it (e.g. s3://bucket-name).

**Bucket Prefix** - The prefix inside the bucket which needs to be monitored.

**Region** - The region where the bucket is set.

**Format** - The format of the data in the given prefix (can be JSON, Parquet, CSV, Iceberg)

![](https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2F4UGy5cuECedGQLdH3CyR%2Fimage.png?alt=media\&token=4e635dbd-f640-4083-bbf3-21476aca5669)

</details>

<details>

<summary>BigQuery</summary>

**Integration -** Choose the integration configured (find instructions in [BigQuery](/integrations/gcp/bigquery.md)).

**Dataset** - Name of the BigQuery dataset.

**Table** - Name of the table.

For example given the following table `upriver-proj.product_analytics.events` , integration would be the one set on `upriver-proj` , dataset would be `product_analytics` and table would be `events`.

<div align="left"><figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FOmS45kwHrwU2uyZefYVu%2Fimage.png?alt=media&amp;token=daaed9fb-2522-4f6c-ac54-b9506d2baf80" alt="" width="375"><figcaption></figcaption></figure></div>

</details>

<details>

<summary>Kinesis</summary>

**Stream Name** - The name of the stream containing the data.

**Region** - The region where the stream is set.

![](https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FcAPf5ZzX1xGQNUdgYvre%2Fimage.png?alt=media\&token=4e0f3d28-0a9b-4d90-b11a-3bfdc53939e9)

</details>

<details>

<summary>GCS</summary>

**Bucket** - The name of the given bucket or the path to it (e.g. s3://bucket-name).

**Bucket Prefix** - The prefix inside the bucket which needs to be monitored.

**Region** - The region where the bucket is set.

**Format** - The format of the data in the given prefix (can be JSON, Parquet, CSV, Iceberg)

![](https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FATUQ9BXjt7Y5kvdJNLgM%2Fimage.png?alt=media\&token=4c1159b3-9532-4181-b812-0d3ef6a30f5b)

</details>

<details>

<summary>Snowflake</summary>

**Database -** The name of the Snowflake database.

**Schema -** The schema within the database.

**Table -** The name of the table to be monitored.

<img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FvbYFknoSYOMZYJFHXVyl%2Fimage.png?alt=media&amp;token=51047493-bf60-4170-8157-997ef00797d5" alt="" data-size="original">

</details>

<details>

<summary>Confluent Kafka</summary>

**Integration -** Choose the integration configured (find instructions in [Confluent Kafka](/integrations/confluent-kafka.md)).

**Topic -** Stream topic.

**Table -** Group ID of the consumer.

<div align="left"><figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FmMMsWqV8YeZGXiRvionI%2Fimage.png?alt=media&amp;token=e33508ae-d26a-44c6-94af-e3ed35eab1cf" alt="" width="375"><figcaption></figcaption></figure></div>

</details>

<details>

<summary>Redshift</summary>

**Integration -** Choose the integration configured (find instructions in [Redshift](/integrations/aws/redshift.md)).

**Database** - The database the table is in.

**Schema** - The schema the table is in.

**Table** - The table to monitor.

<div align="left"><figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FdCe4bfIm5SrL1lk6R5DF%2Fimage.png?alt=media&amp;token=c8d0bf84-6081-4ecd-9c8b-63998bb497a1" alt="" width="375"><figcaption></figcaption></figure></div>

</details>

#### Tags

These tags can be created, to allow additional segmentation as they appear inside the data source, in incidents, and in alerts.

#### Description

This field allows for a detailed description of the data source, to provide context to anyone who needs to review it.

### Connection

In this creen, you can configure the access details for your chosen connector (connector types are above).

**Note:** The button for "create more" will allow, once this setup is complete, to start again, with much of the configuration copied over, in case you are defining more than a single source at once.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FuUWv0a5YHuWWxE5lQqtn%2Fimage.png?alt=media&amp;token=4b8fb71f-93b9-45ed-a4b3-736ed142239a" alt=""><figcaption></figcaption></figure>

### Monitor Scheduling

The interval in which the data analysis process will run. Users can select multiple options for when to run a scan on that specific data source.

When choosing `Update Time` , users can set the exact time and delay they want for that data source, in order to make sure it runs at the exact time they want, and on the right time window.

1. Set the `Hour` & `minute` in UTC.
2. Set the `Run Delay` , This value delays the effective run time of the extraction. For example: setting a delay of 10 hours means the extraction will process data as if it ran 10 hours earlier than the scheduled time.

When choosing `Dynamic Update`, Upriver's algorithms decides & sets the relevant run interval time by analyzing the update patterns seen for the data source

When choosing `Manual Trigger` , the source will only start a scan when triggered manually by a user.

### Pivot Fields

Here, you can decide if you want to define pivots, which would allow a more granular approach to monitoring, pivoting tables based on fields you decide to be created as child data sources to the parent source. This would allow the platform to detect changes & anomalies for subsets of the data (specific products, specific regions, etc) inside big data sources.

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2F6ilHUMms5C0XJCMYmoTL%2Fimage.png?alt=media&amp;token=a01083e4-05f6-407f-a85c-ff896e0ca596" alt=""><figcaption></figcaption></figure>

There are a two distinct ways to define pivots:

1. **Nested** - Splits the data in layers. The first pivot groups the main table, then each group is split again by the next pivot, and so on. Useful for drilling down into data step by step.
2. **Multiple** - Applies each pivot directly to the main table, creating separate groupings side by side. Best for comparing different dimensions independently.

{% hint style="info" %}
After each allowed value, make sure to click on `Add` or press the `Enter` button on your keyboard.
{% endhint %}

{% hint style="info" %}
Currently only up to two different pivot fields can be defined.
{% endhint %}

Additional configuration options:

Pause new derived datasources - this means any newly created pivots will be created in "paused" mode, so they will not go into learning mode automatically. This helps when creating a lot of new pivots, letting the user start them gradually or by priority once the right ones are chose.

Set new derived data sources as manual trigger - This will set all newly created pivots to run in manual mode, meaning the user can choose which to run, or change to triggered runs.

### Advanced Configuration

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FC3SuROBTE7kQHIdTIbr6%2Fimage.png?alt=media&amp;token=137c0b2c-d9f3-484d-bccf-0e783947cbca" alt=""><figcaption></figcaption></figure>

#### Primary Key

A primary key is a column (or set of columns) that uniquely identifies each row in a table. This should be defined per table.

#### Sampling Rate

The percentage of the data that will be used to create the profile. Users can either decide the percentage of the data (0-1) or select dynamic sampling.

If the user decides to use `Dynamic Sampling` , Upriver's algorithms decide the sampling rate based on the volume of data and the distribution it sees.

#### Filter

A filter can be used to only monitor parts of the data where a certain condition holds true. In order to use the filter, the user needs to enter the name of field/column they wish to filter and the desired value.

{% hint style="info" %}
Currently only a single value filter is available. Support for multiple filters is under development.
{% endhint %}

#### Cardinality Threshold

The number of distinct values needed to determine if a field is categorical or not.

Fields with less distinct values than the set threshold will be considered categorical.

#### Default Incident Severity

The user can set the default severity that will be assigned to incidents related to the datasource. See [Managing Incidents](/product-usage/incidents/managing-incidents.md)

#### Staleness Threshold

The maximum number of days a datasource can go without seeing data before it is considered "stale". Stale data sources are considered those not being updated.

#### Timestamp Column

Specifies the name of the column to be used as the timestamp for each row in the table.

#### Nullify Empty Strings (Checkbox)

If enabled, empty strings (`"")` will be treated as empty fields when calculating the completeness of a field.

#### Nullify Empty Arrays (Checkbox)

If enabled, empty arrays (`[])` will be treated as empty fields when calculating the completeness of a field.

#### Only analyze updated data (checkbox)

If enabled, the platform will only scan updated data (delta) from the last scan.

For full scans each time, make sure not to check this box.

### Notification Configuration

<figure><img src="https://875415170-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FLE0zsJmUpeLjYw085cnO%2Fuploads%2FGj9RP8kppK6Dwe7KWxpI%2Fimage.png?alt=media&amp;token=b155a7cc-dcf9-40e6-8ab9-4b108bd0d084" alt=""><figcaption></figcaption></figure>

#### External ID

A user-defined identifier that represents the ID used in the its systems.

#### Webhook

A URL where incidents will be sent. Enables integration with external systems by forwarding incident data in real time.

#### Slack

The ID for the channel that will receive Upriver incident alerts. If the Slack app is not used, a single channel can be defined. If it is, multiple channels can be used (per data source).

***

* [Troubleshooting Monitoring Configuration](/product-usage/data-assets/data-source/troubleshooting-monitoring-configuration.md)