Worker System

Workers are created and executed in a parent-child hierarchy. A WorkerSystem instance is always at the root of the hierarchy, with an arbitrary number of descendant workers and levels. The WorkerSystem instance also handles configuration settings, logging, and the running of the worker system.

To create and run a worker system, at a minimum your code:

  1. Creates an instance of the WorkerSystem class, which is always at the root of the worker hierarchy
  2. Specifies what work the worker system should perform when it runs, which usually includes creating child workers
  3. Starts the worker system, in the foreground or the background
  4. When the worker system has completed, acts on the success or failure outcome

The following sections covers these steps in detail.

1. Create WorkerSystem Instance

Creating the WorkerSystem instance loads configuration settings, starts logging etc., but does not 'start' the worker system.

Most actionETL applications will run a single WorkerSystem instance, but running multiple instances, including in parallel, is also supported.

2. Specify Work

There are four ways to specify what the worker system should do, with the first one being the most common one:

a. Pass a callback to Root()

  • Delays the execution of the callback (and the creation of any workers) until the worker system runs

b. Add child worker(s) to the WorkerSystem instance

  • Creates workers before the worker system runs

c. Add callbacks to the WorkerSystem instance

  • Execute logic before any Root() callback runs, or after any Root() callback and children run

d. Create a custom reusable worker system type

It's possible to combine approaches with the same worker system instance.

2a. Pass Callback to Root()

Passing a lambda callback (which the system will later run, see Getting Started) to one of the Root(Action<WorkerSystem>) overloads is often the best choice, since it requires the least amount of code, avoids or delays creating workers as much as possible, and the visual structure of the code (i.e. indentation and blocks within curly braces) automatically reflects the structure of the worker hierarchy.

The callback receives a reference to its parent WorkerSystem instance, which e.g. allows access to configuration settings, and is needed to create child workers. The callback code is regular .NET code, and will run without any particular restrictions.

This example checks if a file exists, and exits with a success or failure code. It uses the (optional) fluent coding style of chaining method calls:

using actionETL;
using System.Threading.Tasks;

namespace MyCompany.ETL.FileExists
{
    static class Program
    {
        static async Task Main()
        {
            // Example worker system: test if file exists
            var outcomeStatus = await new WorkerSystem()
                .Root(ws =>
                {
                    // Example worker: check filename loaded from "actionetl.aconfig.json"
                    _ = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);
                })
                .StartAsync();

            // Exit with success or failure code
            outcomeStatus.Exit();
        }
    }
}

Alternatively, you can pass in an existing method by casting to the appropriate overload:

//using actionETL;
//using System;

private static void CreateWorkers(WorkerSystem workerSystem)
{
    _ = new FileExistsWorker(workerSystem, "File exists", workerSystem.Config["TriggerFile"]);
}

public static void RunExample()
{
    new WorkerSystem()
        .Root((Action<WorkerSystem>)CreateWorkers)
        .Start()
        .ThrowOnFailure();
}

The available overload casts are:

Your method Root() cast
void MyMethod(WorkerSystem workerSystem) Action<WorkerSystem>
OutcomeStatus MyMethod(WorkerSystem workerSystem) Func<WorkerSystem, OutcomeStatus>
async Task MyMethod(WorkerSystem workerSystem) Func<WorkerSystem, Task>
async Task<OutcomeStatus> MyMethod(WorkerSystem workerSystem) Func<WorkerSystem, Task<OutcomeStatus>>

2b. Add Child Workers to WorkerSystem Instance

When workers are created they are always added to a specific parent, which can be the WorkerSystem instance or another worker:

public static void RunExample()
{
    var workerSystem = new WorkerSystem();

    _ = new FileExistsWorker(workerSystem, "File exists"
        , workerSystem.Config["TriggerFile"]);

    workerSystem
        .Start()
        .ThrowOnFailure();
}

While creating workers ahead of time is sometimes preferable, it does use up at least some resources earlier than otherwise. The difference in timing of the resource need is however usually negligible, since best practice when implementing a worker is to delay acquiring large resources until it actually runs.

2c. Add Callbacks to the WorkerSystem Instance

It is sometimes convenient to add initialization and cleanup logic as separate callbacks, rather than embed it with child workers and start constraints:

 var workerSystem = new WorkerSystem()
     .Root(ws =>
     {
         _ = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);

         // Create other child workers...
     });

 bool skipProcessing = false;

 // Other logic...

 workerSystem
.AddStartingCallback(ws =>
     skipProcessing ? ProgressStatus.SucceededTask : ProgressStatus.NotCompletedTask)
 .AddCompletedCallback((ws, os) =>
 {
     // Perform cleanup tasks...

     return OutcomeStatus.SucceededTask;
 })
 .Start().ThrowOnFailure();

This can be particularly useful when adding callbacks from outside the worker system.

See AddStartingCallback(Func<TDerived, Task<ProgressStatus>>) and AddCompletedCallback(Func<TDerived, OutcomeStatus, Task<OutcomeStatus>>) for details.

2d. Create a Custom Reusable Worker System Type

Creating a custom worker system type can be useful to e.g.:

  • Run the same complete worker system logic in different places, including across different applications
  • Add properties and methods to the worker system, e.g. to surface aspects of the contained user logic

Inherit from WorkerSystemBase<TDerived> or (if you want to retain the Root() overloads) WorkerSystem. You must also implement RunAsync() with your custom logic.

3. Start the Worker System

When started, the worker system:

  • Runs the callback passed to Root(), if any, which optionally creates workers
  • Runs the children of WorkerSystem, if any and their start constraints allow it
  • For any worker that runs, recursively also attempt to run its children

The worker system completes when:

  • Root() and any other callbacks has completed, and
  • No workers are running, and no new worker can be started

Worker Execution describes the execution phase in detail.

Foreground

Worker systems in console programs (or more exactly, if SynchronizationContext.Current == null) can be started in the foreground using Start(). When the method returns, the worker system is completed, and the returned SystemOutcomeStatus value describes the success or failure of the worker system:

var workerSystem = new WorkerSystem()
    .Root(ws =>
    {
        // ...
    }
    );
workerSystem.Start().Exit();

Background

All types of programs (including console programs), irrespective of SynchronizationContext.Current, can start the worker system in the background by calling StartAsync(). It returns a Task<SystemOutcomeStatus> (see Task<TResult>), and when this task completes, the worker system is completed, with the task result as an SystemOutcomeStatus that describes the success or failure of the worker system.

In an async method, the worker system task is normally awaited with await (see Asynchronous programming with async and await for background information).

Note

In C#7.1 and later, the Main() method of a console program can also be asynchronous and use async.

In non-async methods, options include:

  • Returning the Task from the method as in this example:
public static Task<SystemOutcomeStatus> RunExampleAsync()
{
    return new WorkerSystem()
        .Root(ws =>
        {
            _ = new FileExistsWorker(ws, "File exists", ws.Config["TriggerFile"]);
        })
        .StartAsync();
}
Warning

The next two options should only be used at the top level of a synchronous program, otherwise dead-locks can occur. Async/Await - Best Practices in Asynchronous Programming covers this in depth.

  • Calling Task.Wait() and checking the WorkerSystem Status property
  • Retrieving Task<SystemOutcomeStatus>.Result, which automatically waits for the task to complete

4. Act on Success or Failure

The worker system catches and logs most exceptions, which means it's imperative to always check and act on the success or failure of the worker system after it has completed. Do this by retrieving and acting on the worker system status from one of the following:

OutcomeStatus and WorkerParentStatus have similar members, e.g.:

  • IsSucceeded, IsError etc.
  • Exit() to immediately exit your program with a success or failure exit code
  • ThrowOnFailure() to throw an exception on any failure status

See Also