Data Source Fields
Overview
Upriver automatically infers the schema of your data for any data source, including semi-structured data, and creates a comprehensive view which allows you to understand what your data looks like and set expectations on top of it.
Data Source Fields
From within the data source page, click on the Fields tab to get an overview of all the fields in the data source. It is from this tab where you can drill
The overview allows the users to search or filter any field in the data either by any attribute that is relevant to the field. This includes field names, types and even whether or not they have a known semantic format (such as UUID, SSN, IP, etc.).
For each field, if there are unresolved incidents associated with the field detected by Upriver (see Incidents), that will also be displayed in the overview display. Further detail how to dive into those incidents is provided below.
Field Profile
When expending the view for a specific field, more information regarding the profile of the the field is presented to the user.
The profiles provided allow the users to see the key data quality metrics for each field without any need to query the data. Users can change between the different quality metrics shown in the graph to easily see how their data is behaving.
Field Expectations
Upriver transforms complex data profiles into clear, easy-to-understand expectations, giving users a precise view of what to anticipate from their data. This transparency ensures users not only know what “normal” looks like but also understand why an incident is triggered when data deviates from these expectations. By setting intuitive benchmarks, Upriver helps users quickly grasp the reasoning behind alerts and confidently monitor data health.
Automatic Expectations
Upriver converts the profiles it tracks for each of the different data quality metrics into clear expectations for each field. These expectations can be viewed by looking ath the Expectations tab inside each of the fields presented.
The expectations set by Upriver are related to the completeness, uniqueness, value set (i.e. cardinality) and ranges of for the different values. This leaves the guesswork behind and allows the users to derive expectations from the profile.
User Expectations
In addition to the automatic expectations, Upriver also enables users to define data expectations through a straightforward user interface, so there’s no need to write custom queries.
This makes it accessible for both technical and non-technical users to establish assumptions about their data. When the data doesn’t meet these set expectations, Upriver generates an incident, helping users quickly spot and address unexpected changes (see Incidents).
User expectations can be set by clicking on the Edit button from within the expectations view. This opens up a new dialog which allows the users to simply enter their expectations for the data and let Upriver take care of the rest.
Data Examples
To help users understand the structure of their data, Upriver displays examples of the fields, providing a clear view of the data’s contents. These examples are organized by “Semantic Formats,” allowing users to gain deeper insights if there are distinct patterns or formats within the data. This grouping helps users identify and understand any underlying structures, making it easier to interpret and work with their data.
These examples can be seen by clicking on the Examples tab inside the expanded data field view.
Field Incidents
Upriver allows you to easily understand the incidents related to a specific field from within the data source field view. For each field, you can see the unresolved incidents by clicking on the Incidents tab inside the expanded data field view. This opens a list of all the unresolved incidents with a short explanation regarding their date and type. Clicking on an incident will redirect the user to the full incident page (see Incidents).
Excluded Fields
If you want to disable incidents for a specific field, you can simply exclude it. Excluding a field removes it from monitoring, so no incidents will be raised for that field, even if it deviates from expected patterns. This allows you to focus monitoring efforts on only the most relevant fields in your data.
To exclude a field, select it using the check box on the left and click on the "hide" icon (
) at the top.
Once a field has been excluded, it will appear with the hide icon next to the field name.
To restore the incidents for an excluded field, select it again and press the "unhide" icon. This icon replaces the "hide" icon for excluded sources.
Last updated