Logging

actionETL includes a comprehensive and highly configurable logging system that automatically logs the progress of the worker system and its workers, and also allows the library user to add custom logging.

For troubleshooting, consider increasing the logging from the default Info level to Debug, which is normally still fine to temporarily enable even on production servers. For even more detailed logging, enable Trace logging, which however is normally too verbose for production servers. Also note that the logging level can be changed on only one or a few workers or ports, leaving all others at the default logging level, see rules.

This article covers where the logging configuration comes from; Logging Configuration File covers the configuration file content, and Logging Messages covers how to log your own messages.

The actionETL.Logging namespace contains the classes etc. that the worker system and workers use to perform logging. A logging factory creates the instances that perform the actual writes to files, databases etc., which by default uses NLog, an excellent logging framework used by thousands of applications.

At a high level, a worker system checks for a logging factory or a logging configuration in this order:

  1. Injected Factory - the logging factory (created by the library user from a configuration file or programmatically) is passed to the WorkerSystem constructor
  2. NLog Singleton Factory - the application wide NLog logging factory (LogManager) is used with:
    • Standard Configuration Location - An NLog configuration file from any of the standard locations
    • Default Configuration - An in-memory configuration automatically created for the LogManager, which logs to a file and the console

These three options will be described in reverse order - Default Configuration is the simplest to use while Injected Factory is the most flexible.

Note

While NLog is the included and recommended logging factory, a good and easy way to log to a different logging framework is to configure NLog to forward log events to one of the many supported other logging frameworks, see NLog Targets for details.

Furthermore, it is also straight forward to replace NLog completely by implementing two adapters using the IALogService and IALogFactory interfaces, and inject the factory.

NLog Singleton Factory

The worker system uses an IALogFactory for logging, which can be accessed as LogFactory.

With no injected logging factory, the worker system gets its factory from GetSingleton(), which can also be used before and outside the worker system to create loggers that still log to the same logging target as the worker system:

private static readonly ALog _logger = 
    NLogFactory.GetSingleton().GetLogger(nameof(MyUnitTest));

// ...

_logger.Info("Custom.ParsingCommandLine");

NLogFactory is a wrapper for in this case an NLog application wide log factory LogManager, which can also be accessed directly, e.g. setting the LogManager global logging threshold:

LogManager.GlobalThreshold = NLog.Warn;

Default Configuration

This example will create an in-memory default configuration for the LogManager application wide logging factory if there are no NLog configuration files in the standard locations (e.g. "nlog.config" in the executable folder):

using actionETL;
using System.Threading.Tasks;

namespace actionetl.console.csharp
{
    static class Program
    {
        static async Task Main()
        {
            // Create worker system: test if file exists
            var workerSystem = new WorkerSystem()
                .Root(ws =>
                {
                    // Example worker: check filename loaded from "actionetl.aconfig.json"
                    _ = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);
                });

            // Run the worker system
            var systemOutcomeStatus = await workerSystem.StartAsync().ConfigureAwait(false);
            
            // Exit with success or failure code
            systemOutcomeStatus.Exit();
        }
    }
}

The default configuration will:

  • Log all information, warning, error, and fatal messages to a file with the same name and location as the executable, but with the ".log" suffix: MyCompany.ETL.FileExists.log
  • Also log the above messages to the console, but with some informational messages filtered out for brevity

The default logging is often sufficient, and even handles multiple worker systems running in parallel in the same application, but there are also many useful ways to customize the logging.

Standard Configuration Location

The LogManager and actionETL logging can be customized by providing a configuration file to NLog, e.g. by creating an "nlog.config" file in the same folder as where the executable resides.

Note

The code example in the previous section can still be used unchanged.

Logging Configuration File describes the syntax of the configuration file. The full set of possible configuration file locations is described on the NLog wiki.

Injected Factory

The NLog logging configuration (or a custom logging factory written by the library user) can also be created and injected into the worker system constructor at runtime, which provides the most flexibility.

The injected logging factory won't be shared, unless you explicitly pass the NLogFactory to additional worker systems as in the below example. In this case it is particularly useful to give each worker system a unique name, to tell them apart in the log output.

This example also shows disposing the custom NLog factory after use, (with a using statement), and loading a non-standard configuration file "Other.config":

using actionETL;
using actionETL.Logging.NLogExternal;
using NLog.Config;
using System.Threading.Tasks;

public async static Task RunExampleAsync()
{
    using (var logFactory = 
        NLogDisposableFactory.Create(new XmlLoggingConfiguration("Other.config")))
    {
        var first = new WorkerSystem("First worker system", logFactory)
            .Root(ws => { /* ... */ })
            .StartAsync();

        var second = new WorkerSystem("Second worker system", logFactory)
            .Root(ws => { /* ... */ })
            .StartAsync();

        // ...

        (await first.ConfigureAwait(false)).ThrowOnFailure();
        (await second.ConfigureAwait(false)).ThrowOnFailure();
    }
}

Discard All Messages

One can discard some or all logging messages by modifying the configuration file, e.g. setting log levels to Off, or to use an empty writeTo attribute. This can be useful to for instance perform unit testing or running a benchmark without logging.

A runtime alternative for this (which can e.g. be used on a per unit test basis) is to inject a NullLogFactory.Instance logging factory that discards all messages, irrespective of any configuration files:

using actionETL;
using actionETL.Logging;

new WorkerSystem(NullLogFactory.Instance)
    .Root(root =>
    {
        // ...
    })
    .Start()
    .ThrowOnFailure();

See Also