Data Source

Overview

The data source page offers users a comprehensive overview of all critical information about a specific data source. Here, users can access key details such as the schema, data quality metrics, and recent incidents associated with the data source. This centralized view enables users to quickly grasp the current state and performance of their data, facilitating informed decision-making and enhancing collaboration within the team.

Data source overview page

Available data source views

• Overview - A comprehensive overview of the data source including it's volume, freshness and a brief overview of the schema.

• Fields - A clear overview of each field in the data source including the schema, profile and expectations. See Data Source Fields

• Custom Metrics - In this tab user can define and monitor custom metrics that meet their specific business needs. See Custom Metrics

• Profile Diff - In this tab users can easily compare the profile of the data source at different times and between different datasources. See Profile Diff.

Data source status

Status	Description
Initializing	The data source has been identified and has still not seen a single run.
Learning	The data source has already performed at least one run, however it is still waiting for more update iterations in order to define a more stable baseline.
Running	The data source is continuously monitored by Upriver.
Stale	The data source has not been updated for more than the specified threshold.
Paused	The data source is not being monitored because the user explicitly decided to stop the monitor.

Initiating a profile scan

You can trigger an out-of-schedule scan by clicking the “Scan Now” button located at the top right of the data source page. This feature allows you to initiate monitoring whenever necessary, such as after making changes to the data source. By doing so, the system can immediately assess the data and help you verify that everything is functioning as expected.

Extracting contract to validation rules

Upriver allows users to can export the data contract into both CSV format and a Great Expectations test suite. This can be done by selecting the relevant option under Generate at the top right of the data source page.

The Great Expectations test suite includes all the expectations shown on the “Fields” page (see Data Source Fields), combining both user-defined and model-generated expectations. This allows you to incorporate the expectations created by Upriver directly into your data pipeline, enhancing data validation and ensuring data quality throughout your workflow.

Defining a custom schema

The platform allows users to define a custom schema that they expect for each data source, including pivots, using JSON Schema formats. Once such schema is defined, the platform matches the incoming data against it and ensure its validity. In addition, Upriver will also highlight any field that is seen in the data and not present in the schema when looking at the Data Source Fields page.

Uploading a schema to the platform can be done by selecting the relevant option at the top ... button.

Below is a sample JSON schema accepted by the platform. Notice that fields set as required will also be set with a completeness user expectations of 100%.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "User",
  "type": "object",
  "properties": {
    "id": {
      "type": "integer"
    },
    "name": {
      "type": "string"
    },
    "address": {
      "type": "object",
      "properties": {
        "city": {
          "type": "string"
        },
        "state": {
          "type": "string"
        },
        "country": {
          "type": "object",
          "properties": {
            "name": {
              "type": "string"
            },
            "code": {
              "type": "string"
            }
          },
          "required": [
            "name",
            "code"
          ],
          "additionalProperties": false
        }
      },
      "required": [
        "city",
        "state"
      ],
      "additionalProperties": false
    }
  },
  "required": [
    "id",
    "name",
    "address"
  ],
  "additionalProperties": false
}

Deleting a data source

A data source can be deleted by clicking on the ... button at the top right and then selecting the Delete Data Source option. This will prompt a dialog to verify the action.

Deleting a data source cannot be undone.

Delete data source option