AConfig
Configuration Facility
AConfig is the primary way to:
- Store and retrieve arbitrary user created settings (folder and file names, URLs, connection strings, etc.)
- Set or overwrite predefined actionETL settings (dataflow buffer sizes etc., see AConfigSetting)
Each worker system uses an AConfig instance via the Config property, but it can also be used stand-alone, without any worker system, which can be useful when sharing settings with other applications.
Beyond the usual name-value pairs, this facility also supports (optional) ApplyTo
values
which control what part of the worker system hierarchy the setting applies to. This allows
setting e.g. a port buffer size on a particular port, on a particular worker, to a specific
value. If ApplyTo
is not specified, the setting applies to the whole worker system
hierarchy.
There are several places the configurations can be accessed, which is also the normal order in which they are loaded:
- In the configuration store back-end, by default a JSON file
GetValue
andSetValue
methods on the configuration store class, by default JsonConfigurationService- You typically also create an AConfig instance which provides simpler to use methods
SetValue
overloads on the worker system classWorkerSystemBase.Config
property inside the worker system
Furthermore, many system provided configurations are automatically mapped to properties, e.g. BufferCapacity, where the actual value (in this case for a particular worker port) can be retrieved, and in some cases overridden.
Note
- Setting a non-default license location must be done before the worker system is created, i.e. using option 1 or 2 above.
- If a configuration with a particular key is set multiple times (from files or at runtime) with identical "applyTo" values, the last occurrence is used.
- A different configuration store can be used by implementing a new IConfigurationService and injecting it when creating the worker system.
1. JSON Configuration Files and Format
By default, the worker system loads the files returned by LoadAnyDefault(), in this order:
- "actionetl.aconfig.json" in the same folder as the executable file (or DLL in e.g. test frameworks)
- "MyApp.exe.aconfig.json", "MyApp.dll.aconfig.json", or (for .NET Framework web applications) "web.aconfig.json", where "MyApp" is the name of the executable or assembly.
AConfig
configuration files, streams, and strings use the
JSON file format since it is both
well known and quite terse. Here's an example:
{
"configurations": [
{ "GenerateRecordCount": 2000000 },
// Note use of forward slash, escape any backslash with '\':
{ "SalesFact_ExtractFolder": "C:/Data/Incoming/SalesFact/" },
{ "InputPort.BufferCapacity": 512 }, // Applies to all workers by default
{
"InputPort.BufferCapacity": 1024,
"applyTo": "/Root/Target.Inputs[Input]" // Applies to specific Locator
},
{ "KeepChildrenLevels": -1 }, // Retain workers to check result status in tests
{ "SqlServer": "Data Source=(localdb)\\MSSQLLocalDB;Database=actionetl_tester;Integrated Security=True;TrustServerCertificate=True" }
]
}
- The key of the property (e.g. "GenerateRecordCount" above) becomes the name of the setting.
- The value of the property (e.g.
2000000
above) becomes the value of the setting. - "applyTo" is a special optional key that specifies which worker, worker system, or port this
setting applies to.
- If empty or not specified, this setting applies to the worker system, all its workers, and all their ports
- If non-empty, only applies to a single object (worker system, worker, or port) with a
matching
Locator
string (e.g. a WorkerParent locator. This overrides any setting with the same name, but with a blank or unspecified "applyTo".
- Use forward slash in paths ("C:/Data/Incoming/SalesFact/"), or escape any backslash ("C:\\Data\\Incoming\\SalesFact\\")
- Comments are prefixed with
//
Important
To get Visual Studio to copy the configuration file to the output folder:
- Put "actionetl.aconfig.json" in the root project folder
- Change properties on "actionetl.aconfig.json":
- Build Action = "Content"
- Copy To Output Directory = "PreserveNewest"
2. Configuration Store Class
The JsonConfigurationService class loads configuration settings from JSON files, streams, or strings into memory, and allows getting and setting their in-memory values at runtime.
JsonConfigurationService
also has TryGetValue
and SetValue
methods that can retrieve,
set, and overwrite configurations. Note however that in this class, these methods lack
extensive overloads, so are seldom used directly. Settings are instead more commonly
manipulated via the WorkerSystem
, as described below, or by wrapping it in
an AConfig instance:
// using actionETL;
// using actionETL.Configuration;
var configService = new JsonConfigurationService();
var aConfig = new AConfig(configService);
aConfig["MySetting"] = "MyValue"; // Set value
var myValue = aConfig["MySetting"]; // Get value
new WorkerSystem(configService) // Use configuration service in worker system
.Root(ws =>
{
// ...
})
.Start()
.ThrowOnFailure();
Note
JsonConfigurationService
and the JSON format can also be replaced with a custom
configuration store by implementing IConfigurationService
and related interfaces.
3. Worker System SetValue()
The worker system SetValue(String, Int64) overloads are very convenient for setting configurations before the worker system has started, both global ones and ones that are targeted to a specific object in the worker hierarchy.
The overloads returns the worker system itself, so you can chain multiple calls together.
4. WorkerSystem.Config
The Config property on WorkerSystem
is the main way
for getting and setting individual configurations from inside the worker system, while
it's running. It's available for use as soon as the WorkerSystem
instance has been
created.
Furthermore, all workers have a WorkerSystem
property, so workers further down in the
hierarchy can also easily reach the Config
property. To support this safely, its
members for setting and getting configuration values are thread-safe.
Examples
Default Behavior
By placing a configuration file named "actionetl.aconfig.json" in a folder, all worker
systems running in all executables in this directory will load this file and its
configurations. With the above configuration file,
the "Target" worker in the example below will use the value 1024
for the
"InputPort.BufferCapacity" setting, since both the key name and the "applyTo" specifier match.
If there had been no matching "InputPort.BufferCapacity" setting in the file, the built-in
default (256
) would have been used instead.
// using actionETL;
// using actionETL.Configuration;
new WorkerSystem()
.Root(ws =>
{
new EnumerableSource<SmallClass>(ws, "Source"
, SmallClass.GetRange(0, 1000000))
.Output.Link.TrashTarget();
})
.Start()
.ThrowOnFailure();
Set and Get Values at Runtime
You can:
- Modify the default configuration by chaining multiple worker system
SetValue(...)
calls together, which will override any matching configuration file setting - Use the
Config[]
indexer to get a value (useful for strings without default value) - Use
GetValueWithDefault()
to get a value, with a fall-back default value
// using actionETL;
// using actionETL.Configuration;
var systemOutcomeStatus = new WorkerSystem()
.SetValue(AConfigSetting.InputPortBufferCapacity, 2048)
.SetValue("TriggerFile", @"Trigger.trg")
.Root(ws =>
{
var few = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);
var source = new EnumerableSource<SmallClass>(
ws
, "Source"
, () => few.IsSucceeded
, SmallClass.GetRange(0
, ws.Config.GetValueWithDefault("GenerateRecordCount", 1000000))
);
source.Output.Link.TrashTarget();
})
.Start();
Here is a second example that loads a connection string:
// using actionETL;
// using actionETL.Adb;
// using actionETL.Adb.SqlClientExternal;
// using actionETL.Configuration;
new WorkerSystem()
.Root(root =>
{
var connectionBuilder = AdbSqlClientProvider.Get()
.CreateConnectionBuilder(root.Config["SqlServer"]);
// ...
})
.Start()
.ThrowOnFailure();
Load Other Files
You can load non-default files by providing a
JsonConfigurationService configuration store when
creating the WorkerSystem
. This also stops the default configuration files from
being loaded.
Note
Note that multiple calls to the JsonConfigurationService.*Load*()
methods can
be made, which can be useful e.g. when loading a mix of mandatory and optional files.
Here we override "InputPort.BufferCapacity" to 128
for a specific worker
port "/Root/Target.Inputs[Input]":
// using actionETL;
// using actionETL.Configuration;
var sos = new WorkerSystem(
new JsonConfigurationService().LoadAll(
"../global-config.json", "local-config.json")
)
.SetValue(AConfigSetting.InputPortBufferCapacity, "/Root/Target.Inputs[Input]", 128)
.Root(ws =>
{
var source = new EnumerableSource<SmallClass>(
ws, "Source", SmallClass.GetRange(0, 1000000));
var target = source.Output.Link.TrashTarget();
})
.Start();
See Also
- Common Tasks
- Getting Started
- Worker System
- Configuration
- AConfigSetting - Configuration name string constants
the system uses with
AConfig
- WorkerSystemBase.Config
- AConfig
- JsonConfigurationService
- AConfigSetting - Configuration name string constants
the system uses with
- Configuration