Platform Integration

Below are described the patterns currently in-use across the Service for integrating applications with Azure resources.

Managed Identities (MI)

Using a managed identity allows an application to authenticate to any service that supports Azure AD authentication without having credentials. This has several key benefits:

Improved security - credentials don’t exist so are not accessible to anyone
Applications won’t experience expired secrets resulting in:
- Improved development/testing experience
- No management overhead on redeploying applications in order to renew secrets
- No risk of downtime due to expiring secrets
Simplified configuration to reference for applications

Examples where Managed Identity Azure AD authentication should be used:

Acquiring a bearer token for an internal API behind Azure AD authentication
Connecting to a SQL database
Connecting to a Service Bus namepsace

The Azure.Identity (Previously Microsoft.Azure.Services.AppAuthentication) library manages authentication automatically.

Internal API communication

The SFA.DAS.Api.Common NuGet package configured at https://github.com/SkillsFundingAgency/das-shared-packages/tree/master/SFA.DAS.Api.Common is useful for API projects and adds some of the common components required such as Azure authentication via MI and healthcheck response writer etc.

The below examples use the SFA.DAS.Api.Common NuGet package.

Example - Enforcing Azure AD authentication for an internal API:

public class Startup
{
    ...
    public void ConfigureServices(IServiceCollection services)
    {
        if (!ConfigurationIsLocalOrDev())
        {
            var azureAdConfiguration = _configuration
                .GetSection("AzureAd")
                .Get<AzureActiveDirectoryConfiguration>();

            var policies = new Dictionary<string, string>
            {
                {PolicyNames.Default, RoleNames.Default}
            };

            services.AddAuthentication(azureAdConfiguration, policies);
        }

        services
            .AddMvc(o =>
            {
                o.Conventions.Add(new ApiExplorerGroupPerVersionConvention());
            }).SetCompatibilityVersion(CompatibilityVersion.Version_3_0)
            .AddJsonOptions(options =>
            {
                options.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter());
            });
    }
    ...
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IIndexBuilder indexBuilder, ILogger<Startup> logger)
    {
        ...
        app.UseAuthentication();
        ...
    }
    ...
}

As used in: Courses API StartUp.cs

Example - Acquiring a bearer token for an internal API behind Azure AD authentication:

public class InternalApiClient<T> : ApiClient<T>, IInternalApiClient<T> where T : IInternalApiConfiguration
{
    private readonly IAzureClientCredentialHelper _azureClientCredentialHelper;

    public InternalApiClient(
        IHttpClientFactory httpClientFactory,
        T apiConfiguration,
        IAzureClientCredentialHelper azureClientCredentialHelper) : base(httpClientFactory, apiConfiguration)
    {
        _azureClientCredentialHelper = azureClientCredentialHelper;
    }

    protected override async Task AddAuthenticationHeader(HttpRequestMessage httpRequestMessage)
    {
        if (!string.IsNullOrEmpty(Configuration.Identifier))
        {
            var accessToken = await _azureClientCredentialHelper.GetAccessTokenAsync(Configuration.Identifier);
            httpRequestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
        }
    }
    ...
}

As used in Shared Outer API InternalApiClient.cs

Connecting to a SQL database:

To connect to a SQL database you can use a connection string specifying the authentication method as Active Directory Default.

Server={SERVER_NAME};Initial Catalog={DATABASE_NAME};Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Authentication=Active Directory Default;

This will then connect using Managed Identity when deployed

Example - Creating a SqlConnection within a DbContext:

public AppDataContext()
{
}

public AppDataContext(DbContextOptions options) : base(options)
{
}

public AppDataContext(IOptions<AppConfiguration> config, DbContextOptions options) :base(options)
{
    _configuration = config.Value;
}  

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    var connection = new SqlConnection
    {
        ConnectionString = _configuration.ConnectionString,
    };
}

As used in: Candidate API CandidateDataContext.cs

An example of the database extension being added for the AppStart when using the Microsoft ASP.NET Core Built-in IoC Container:

public static class AddDatabaseExtension
{
    public static void AddDatabaseRegistration(this IServiceCollection services, AppConfiguration config, string environmentName)
    {
        if (environmentName.Equals("DEV", StringComparison.CurrentCultureIgnoreCase))
        {
            services.AddDbContext<AppDataContext>(options => options.UseInMemoryDatabase("SFA.DAS.App"), ServiceLifetime.Transient);
        }
        else
        {
            services.AddDbContext<AppDataContext>(ServiceLifetime.Transient);
        }

        services.AddTransient<IAppDataContext, AppDataContext>(provider => provider.GetService<AppDataContext>());
        services.AddTransient(provider => new Lazy<AppDataContext>(provider.GetService<AppDataContext>()));
    }
}

As used in: Courses API AddDatabaseExtension.cs

Example - Passing a SqlConnection into a DbContext:

Sometimes there may be a need to access data outside of a DbContext e.g. when using Dapper, and additionaly data access may also be required in the same solution inside a DbContext; in which case a shared IDbConnection may be configured outside and then passed into the DbContext.

private readonly IDbConnection _connection;

public AppDataContext()
{
}

public AppDataContext(IDbConnection connection, DbContextOptions<AssessorDbContext> options)
    : base(options)
{
    _connection = connection;
}

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder.UseSqlServer(_connection as DbConnection, options =>
         options.EnableRetryOnFailure(3));
}

As used in: Assessor API AssessorDbContext.cs

An example of the database extension being added for the AppStart when using StructureMap for DI:

public static class DatabaseExtensions
{
    private const string AzureResource = "https://database.windows.net/";

    public static void AddDatabaseRegistration(this ConfigurationExpression config, string environment, string sqlConnectionString)
    {
        config.For<IDbConnection>().Use($"Build IDbConnection", c => {
            var azureServiceTokenProvider = new AzureServiceTokenProvider();
            return environment.Equals("LOCAL", StringComparison.CurrentCultureIgnoreCase)
                ? new SqlConnection(sqlConnectionString)
                : new SqlConnection
                {
                    ConnectionString = sqlConnectionString,
                    AccessToken = azureServiceTokenProvider.GetAccessTokenAsync(AzureResource).Result
                };
        });
            
        var option = new DbContextOptionsBuilder<AppDataContext>();
        config.For<AppDataContext>().Use(c => new AppDataContext(c.GetInstance<IDbConnection>(), option.Options));
    }
}

As used in: Assessor API AddDatabaseExtension.cs

References

ASP.NET Core Data Protection

Web applications often need to store security-sensitive data. The ASP.NET Core data protection stack is designed to serve as the long-term replacement for the <machineKey> element in ASP.NET 1.x - 4.x. One common use of Data Protection is the storing of cookies for apps using the standard ASP.NET Core cookie authentication.

By default, apps hosted on app services have data protection keys that are persisted to the %HOME%\ASP.NET\DataProtection-Keys folder. This folder is backed by network storage and is synchronized across all machines hosting the app.

Separate deployment slots (Staging and Production) don’t share a key ring, meaning the standard slot swapping deployment that takes place in releases would result in an app using Data Protection to not be able to decrypt stored data using the key ring inside the previous slot.

Additionally, Data Protection exceptions are seen on apps that do not persist to Redis Cache on Azure instances:

Log property name	Log property value
Exception.type	AntiforgeryValidationException
Exception.innerException.source	Microsoft.AspNetCore.DataProtection
Exception.innerException.message	The key {GUID} was not found in the key ring.

These exceptions correlate with users receiving an error page when signing in.

To maintain zero-downtime releases for Apprenticeship Service applications and prevent other Data Protection exceptions, Data Protection keys should be persisted to the environment’s Redis Cache, where the redis connection string and database number are in the application’s configuration.

Example

using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using SFA.DAS.AppName.Configuration;
using StackExchange.Redis;

namespace SFA.DAS.AppName.Web.Startup
{
    public static class DataProtectionStartupExtensions
    {
        public static IServiceCollection AddDataProtection(this IServiceCollection services, IConfiguration configuration, IHostingEnvironment environment)
        {
            if (!environment.IsDevelopment())
            {
                var redisConfiguration = configuration.GetSection(ConfigurationKeys.ConnectionStrings)
                    .Get<AppNameSettings>();

                if (redisConfiguration != null)
                {
                    var redisConnectionString = redisConfiguration.RedisConnectionString;
                    var dataProtectionKeysDatabase = redisConfiguration.DataProtectionKeysDatabase;

                    var redis = ConnectionMultiplexer
                        .Connect($"{redisConnectionString},{dataProtectionKeysDatabase}");

                    services.AddDataProtection()
                        .SetApplicationName("RepositoryName")
                        .PersistKeysToStackExchangeRedis(redis, "DataProtection-Keys");
                }
            }
            return services;
        }
    }
}

Example of persisting data protection keys in an Apprenticeship Service app.

References

Logging to Redis and not to File

All applications use NLog to write logs to logging Redis Cache on Azure instances.

The following flow is applied:

App Services use NLog to write to Redis Caches.
Each Redis Cache buffers the logs.
Logstash indexes the logs from Redis Cache and performs data transformation and processing.
Kibana is used to visualise the data in Elasticsearch.

Logging to File, as configured in the NLog configuration, results in a storage exception such as There is not enough space on the disk : 'D:\home\site\wwwroot\logs\app-name.YYYY-MM-DD.log' seen in a profiler trace due to a local cache limit of 1GB being reached.

These exceptions correlate with users receiving 504 Gateway Timeout error pages.

Only Redis should be logged to and not File as:

Kibana is reliably available.
Application Insights is the backup solution, Azure’s Application Insights SLA guarantees query availability will not fall below 99.9%.
Checking logs on individual app services is impractical for debugging.
High level Contributor permissions are needed to access the App Services’ files, these permissions should be restricted.
Not logging to File prevents the storage exceptions.

For local development, it is useful to log to File.

Example of NLog configuration, with logging to File for develeopment mode and logging to Redis for non-development mode (running on Azure app services).

References

App Service local cache size limits

ASP.NET Core Health Checks

ASP.NET Core provides built in features to easily configure health check endpoints. All apps should implement a basic health check with a /ping endpoint that establishes whether an app has started. In addition, more sophisticated health checks can be implemented with a /health endpoint.

The /ping endpoint will be used by the Application Gateway to establish whether the application is available, if it does not return a status code in the 200-399 range then the application will be marked as unhealthy and the Application Gateway will not route traffic to it. By default Health Checks will return a 200 if the application is healthy. An alert will be posted to the #das-alerts Slack channel in the event that a staus code outside the acceptable range is returned.

Example

Adding health checks to the Startup class, they can be added to Startup extensions in a similar manner.

public void ConfigureServices(IServiceCollection services)
{
    services.AddHealthChecks()
        //add health checks appropriate to the application here
        .AddDbContextCheck<ApplicationDataContext>();
        .AddCheck<NServiceBusHealthCheck>("Service Bus Health Check")
        .AddCheck<ApiHealthCheck>("Check the health of some API");
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseHealthChecks("/ping", new HealthCheckOptions
    {
        //By returning false to the Predicate option we ensure that none of the health checks registered in ConfigureServices are ran for this endpoint
        Predicate = (_) => false,
        ResponseWriter = (context, report) =>
        {
            context.Response.ContentType = "application/json";
            return context.Response.WriteAsync("");
        }
    });

    app.UseHealthChecks("/health", new HealthCheckOptions
    {
        //configuration for a more complex response
    });
}

References

Health checks in ASP.NET Core