Measure Databricks job run outcomes as metric

The Arcus.BackgroundJobs.Databricks library provides a background job to repeatedly query for Databricks finished job runs, and reports them as metrics.

💡 With using our Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights, you can report these Databricks reports as metrics in Application Insights.

Installation

To use these features, you have to install the following package:

PM > Install-Package Arcus.BackgroundJobs.Databricks

Usage

Our background job has to be configured in ConfigureServices method:

using Arcus.Security.Core;using Microsoft.Extensions.DependencyInjection;
public class Startup{    public void ConfigureServices(IServiceCollection services)    {        // An 'ISecretProvider' implementation (see: https://security.arcus-azure.net/) to access the Azure Service Bus Topic resource;        //     this will get the 'tokenSecretKey' string (configured below) and has to retrieve the connection token for the Databricks instance.        services.AddSingleton<ISecretProvider>(serviceProvider => ...);
        // Simplest registration of the scheduler job:        services.AddDatabricksJobMetricsJob(            baseUrl: "https://url.to.databricks.instance/"            // Token secret key to connect to the Databricks token.            tokenSecretKey: "Databricks.Token");
        // Customized registration of the scheduler job:        services.AddDatabricksJobMetricsJob(            baseUrl: "https://url.to.databricks.instance/"            // Token secret key to connect to the Databricks token.            tokenSecretKey: "Databricks.Token",            options =>            {                // Setting the name which will be used when reporting the metric for finished Databricks job runs (default: "Databricks Job Completed").                options.MetricName = "MyDatabricksJobMetric";
                // Settings the time interval (in minutes) in which the scheduler job should run (default: 5 minutes).                options.IntervalInMinutes = 6;            });    }}

← back