Metadata Monitoring

Upriver enables comprehensive monitoring of your data pipelines without requiring manual setup. It automatically learns the expected trends in data updates, size fluctuations, and growth trends, and notifies you whenever these patterns are disrupted.

This approach is highly effective for spotting data flow interruptions or unexpected breakages.

Volume

Volume monitoring checks for unexpected changes in the total size of the data or the amount of data being ingested as part of the pipeline.

These checks help detect issues with the pipelines such as breakdowns which lead to the the table not updating or ingesting duplicate rows which can manifest as an unexpected increase in the number of rows.

In addition to the automatic monitors, users can also set volume expectations for each data source. These expectations can include the maximum/minimus values expected in the ingestion process. This is done from within the datasource page by clicking Edit on the volume chart for a specific datasource.

Freshness

Freshness monitoring checks for unexpected changes in the time between data updates and the "age" of the newest available data. This is an indicator of whether the data source has been updated at the expected time.

Similarly to the volume monitors, users can also set freshness expectations for each data source. These expectations specify the maximum allowed "freshness" for data, i.e. the maximum expected time since the last data update when running on Upriver's checks.