Development Guidelines

Solution and Project Structure

For non-trivial ETL applications it is usually desirable to create multiple projects for different needs. Here are some ideas for structuring these projects:

  • Create a naming strategy for projects. A good starting point is to name them <organisation>.<grouping>.<purpose>

    Note

    By default, an executable will be named "<project name>.exe".

  • Create a class library project with custom shared workers and related code that is shared across all actionETL projects, e.g. "MyCompany.actionETL"

  • Group projects into logical groupings, managed by separate solutions, and/or sub-folders within solutions

  • Create unit test projects that tests the shared library, e.g. "MyCompany.actionETL.Test", as well as other projects. Common test frameworks include xUnit, nUnit, and MSTest V2.

  • Consider combining related or all ETL tasks into fewer or a single executable, and use command line options to invoke specific functions (DailyLoad, MonthlyLoad etc). There are many good NuGet packages that can parse command lines, e.g. CommandLineParser or System.CommandLine.

    As one example of this approach, see the dotnet command.

Example Layout

Note

Here we use a dedicated executable (MyCompany.DataWarehouse etc.) for each area. You could equally combine all executables below into a single executable.

MyCompany.ETL                           (solution)
  Src                                   (folder)
    actionETL                           (folder)
      MyCompany.actionETL               (class library project)
      ...
    DataWarehouse                       (folder)
      MyCompany.DataWarehouse           (executable project with tasks
                                         DailyLoad, MonthlyLoad, Purge,
                                         Reconciliation...)
      MyCompany.DataWarehouse.Shared    (class library project)
      ...
    OLTP                                (folder)
      MyCompany.OLTP                    (executable project with tasks
                                         FullLoad, IncrementalLoad etc.)
      MyCompany.OLTP.Shared             (class library project)
      ...
    Utility                             (folder)
      MyCompany.EtlUtility              (executable project with tasks
                                         PurgeLogFiles etc.)
      MyCompany.EtlUtility.Shared       (class library project)
      ...
  Test                                  (folder)
    MyCompany.actionETL.Test            (unit test project)
    MyCompany.DataWarehouse.Test        (unit test project)
    MyCompany.OLTP.Test                 (unit test project)
    MyCompany.Utility.Test              (unit test project)

RuntimeInfo

RuntimeInfo provides runtime information about the current operating system and .NET framework. Use it for logging purposes and to select different platform dependent code paths.

Code Analysis

Using code analysis is a great way to maintain high code quality. For Visual Studio®, beyond the default rules, consider also installing the FxCop analyzers, normally as a NuGet package, see the overview at source code analyzers.

Code Analysis Tips

actionETL works well with code analysis, and only a few rules might need tweaking, depending on your coding style:

  • CA1051 "Do not declare visible instance fields"

    • To avoid the warning, use one of the following:
      • Use auto properties instead of fields
      • Suppress the warning
      • Disable the warning for the whole project
    • For CaptureRowErrors, which must be field, suppress or disable the warning.

  • CA1806 "Creates instance that is not used"

    • This warning is raised when creating workers without storing the instance in a variable. To avoid the warning, use one of the following:
      • Disable the warning for the whole project
      • Use discards: _ = new Worker(...);
      • Suppress the warning

  • CA1812 "Avoid uninstantiated internal classes"

    • This warning can be raised when using internal dataflow row classes with source workers that creates the rows for you. To avoid the warning, use one of the following:
      • Disable the warning for the whole project
      • Suppress the warning
      • Make the row class public

See Also