AConfig Configuration Facility

AConfig is the primary way to:

  • Store and retrieve arbitrary user created settings (folder and file names, URLs, connection strings, etc.)
  • Set or overwrite predefined actionETL settings (dataflow buffer sizes etc., see AConfigSetting)

Each worker system uses an AConfig instance via the Config property, but it can also be used stand-alone, without any worker system, which can be useful when sharing settings with other applications.

Beyond the usual name-value pairs, this facility also supports (optional) ApplyTo values which control what part of the worker system hierarchy the setting applies to. This allows setting e.g. a port buffer size on a particular port, on a particular worker, to a specific value. If ApplyTo is not specified, the setting applies to the whole worker system hierarchy.

There are several places the configurations can be accessed, which is also the normal order in which they are loaded:

  1. In the configuration store back-end, by default a JSON file
  2. GetValue and SetValue methods on the configuration store class, by default JsonConfigurationService
    • You typically also create an AConfig instance which provides simpler to use methods
  3. SetValue overloads on the worker system class
  4. WorkerSystemBase.Config property inside the worker system

Furthermore, many system provided configurations are automatically mapped to properties, e.g. BufferCapacity, where the actual value (in this case for a particular worker port) can be retrieved, and in some cases overridden.

Note
  • Setting a non-default license location must be done before the worker system is created, i.e. using option 1 or 2 above.
  • If a configuration with a particular key is set multiple times (from files or at runtime) with identical "applyTo" values, the last occurrence is used.
  • A different configuration store can be used by implementing a new IConfigurationService and injecting it when creating the worker system.

1. JSON Configuration Files and Format

By default, the worker system loads the files returned by LoadAnyDefault(), in this order:

  1. "actionetl.aconfig.json" in the same folder as the executable file (or DLL in e.g. test frameworks)
  2. "MyApp.exe.aconfig.json", "MyApp.dll.aconfig.json", or (for .NET Framework web applications) "web.aconfig.json", where "MyApp" is the name of the executable or assembly.

AConfig configuration files, streams, and strings use the JSON file format since it is both well known and quite terse. Here's an example:

{
  "configurations": [
    { "GenerateRecordCount": 2000000 }, 

    // Note use of forward slash, escape any backslash with '\':
    { "SalesFact_ExtractFolder": "C:/Data/Incoming/SalesFact/" }, 

    { "InputPort.BufferCapacity": 512 }, // Applies to all workers by default

    { 
        "InputPort.BufferCapacity": 1024, 
        "applyTo": "/Root/Target.Inputs[Input]" // Applies to specific Locator
    },

    { "KeepChildrenLevels": -1 }, // Retain workers to check result status in tests

    { "SqlServer": "Data Source=(localdb)\\MSSQLLocalDB;Database=actionetl_tester;Integrated Security=True;TrustServerCertificate=True" }
  ]
}
  • The key of the property (e.g. "GenerateRecordCount" above) becomes the name of the setting.
  • The value of the property (e.g. 2000000 above) becomes the value of the setting.
  • "applyTo" is a special optional key that specifies which worker, worker system, or port this setting applies to.
    • If empty or not specified, this setting applies to the worker system, all its workers, and all their ports
    • If non-empty, only applies to a single object (worker system, worker, or port) with a matching Locator string (e.g. a WorkerParent locator. This overrides any setting with the same name, but with a blank or unspecified "applyTo".
  • Use forward slash in paths ("C:/Data/Incoming/SalesFact/"), or escape any backslash ("C:\\Data\\Incoming\\SalesFact\\")
  • Comments are prefixed with //
Important

To get Visual Studio to copy the configuration file to the output folder:

  • Put "actionetl.aconfig.json" in the root project folder
  • Change properties on "actionetl.aconfig.json":
    • Build Action = "Content"
    • Copy To Output Directory = "PreserveNewest"

2. Configuration Store Class

The JsonConfigurationService class loads configuration settings from JSON files, streams, or strings into memory, and allows getting and setting their in-memory values at runtime.

JsonConfigurationService also has TryGetValue and SetValue methods that can retrieve, set, and overwrite configurations. Note however that in this class, these methods lack extensive overloads, so are seldom used directly. Settings are instead more commonly manipulated via the WorkerSystem, as described below, or by wrapping it in an AConfig instance:

// using actionETL;
// using actionETL.Configuration;

var configService = new JsonConfigurationService();
var aConfig = new AConfig(configService);
aConfig["MySetting"] = "MyValue"; // Set value
var myValue = aConfig["MySetting"]; // Get value

new WorkerSystem(configService) // Use configuration service in worker system
.Root(ws =>
{
    // ...
})
.Start()
.ThrowOnFailure();
Note

JsonConfigurationService and the JSON format can also be replaced with a custom configuration store by implementing IConfigurationService and related interfaces.

3. Worker System SetValue()

The worker system SetValue(String, Int64) overloads are very convenient for setting configurations before the worker system has started, both global ones and ones that are targeted to a specific object in the worker hierarchy.

The overloads returns the worker system itself, so you can chain multiple calls together.

4. WorkerSystem.Config

The Config property on WorkerSystem is the main way for getting and setting individual configurations from inside the worker system, while it's running. It's available for use as soon as the WorkerSystem instance has been created.

Furthermore, all workers have a WorkerSystem property, so workers further down in the hierarchy can also easily reach the Config property. To support this safely, its members for setting and getting configuration values are thread-safe.

Examples

Default Behavior

By placing a configuration file named "actionetl.aconfig.json" in a folder, all worker systems running in all executables in this directory will load this file and its configurations. With the above configuration file, the "Target" worker in the example below will use the value 1024 for the "InputPort.BufferCapacity" setting, since both the key name and the "applyTo" specifier match.

If there had been no matching "InputPort.BufferCapacity" setting in the file, the built-in default (256) would have been used instead.

// using actionETL;
// using actionETL.Configuration;

new WorkerSystem()
.Root(ws =>
{
    new EnumerableSource<SmallClass>(ws, "Source"
        , SmallClass.GetRange(0, 1000000))

    .Output.Link.TrashTarget();
})
.Start()
.ThrowOnFailure();

Set and Get Values at Runtime

You can:

  • Modify the default configuration by chaining multiple worker system SetValue(...) calls together, which will override any matching configuration file setting
  • Use the Config[] indexer to get a value (useful for strings without default value)
  • Use GetValueWithDefault() to get a value, with a fall-back default value
// using actionETL;
// using actionETL.Configuration;

var systemOutcomeStatus = new WorkerSystem()
.SetValue(AConfigSetting.InputPortBufferCapacity, 2048)
.SetValue("TriggerFile", @"Trigger.trg")
.Root(ws =>
{
    var few = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);

    var source = new EnumerableSource<SmallClass>(
        ws
        , "Source"
        , () => few.IsSucceeded
        , SmallClass.GetRange(0
            , ws.Config.GetValueWithDefault("GenerateRecordCount", 1000000))
        );

    source.Output.Link.TrashTarget();
})
.Start();

Here is a second example that loads a connection string:

// using actionETL;
// using actionETL.Adb;
// using actionETL.Adb.SqlClientExternal;
// using actionETL.Configuration;

new WorkerSystem()
.Root(root =>
{
    var connectionBuilder = AdbSqlClientProvider.Get()
        .CreateConnectionBuilder(root.Config["SqlServer"]);

    // ...
})
.Start()
.ThrowOnFailure();

Load Other Files

You can load non-default files by providing a JsonConfigurationService configuration store when creating the WorkerSystem. This also stops the default configuration files from being loaded.

Note

Note that multiple calls to the JsonConfigurationService.*Load*() methods can be made, which can be useful e.g. when loading a mix of mandatory and optional files.

Here we override "InputPort.BufferCapacity" to 128 for a specific worker port "/Root/Target.Inputs[Input]":

// using actionETL;
// using actionETL.Configuration;

var sos = new WorkerSystem(
    new JsonConfigurationService().LoadAll(
        "../global-config.json", "local-config.json")
    )
.SetValue(AConfigSetting.InputPortBufferCapacity, "/Root/Target.Inputs[Input]", 128)
.Root(ws =>
{
    var source = new EnumerableSource<SmallClass>(
        ws, "Source", SmallClass.GetRange(0, 1000000));
    var target = source.Output.Link.TrashTarget();
})
.Start();

See Also