Release Notes

This article lists the main changes in each actionETL release.

The library and the project template nuget packages are available from nuget.org/profiles/envobi.

To update the templates, simply re-install them:

dotnet new --install actionETL.templates.dotnet

To update a project, execute in the project folder:

dotnet add package actionETL

Version 0.41.0

Breaking Changes

• Dropped support for .NET Framework 4.6.1 (now targeting .NET Framework 4.6.2 and .NET Standard 2.0)
• The newer Microsoft.Data.SqlClient 4.x.x provider enables encryption by default, and throws a SqlException if the server is not trusted. See Adb SqlClient Provider for how to address this.
• Dataflow copy behavior for System.Exception was changed from Shallow to SingleShallow

New and Improved Features

• Support any data type in the dataflow. Some workers have specific requirements on the data types, see Dataflow Columns for details
  • Unknown types (unless enum) default to a SingleShallow copy with an estimated size but with TypeDetail.IsAlwaysColumn set to false
  • Support enum as a Shallow copy column in dataflow
  • Read database integer columns into enum dataflow columns using AdbDataReaderSource<TOutput> with all Adb providers.
  • Write enum dataflow columns to database integer columns using AdbInsertTargetFactory, AdbMySqlConnectorBulkInsertTargetFactory, and AdbSqlClientBulkInsertTargetFactory with all Adb providers except Adb Npgsql Provider .
  • Moved TypeDetail.{IsSchema,SchemaNodeCategory} to SchemaNode and revised their definition
• Add support for column struct schemas in bulk insert targets AdbMySqlConnectorBulkInsertTargetFactory and AdbSqlClientBulkInsertTargetFactory; the Slowly Changing Dimension Example uses this for its composite key
• Improved default dataflow port buffer sizes
  • Renamed BufferCapacity to RowsPerBuffer in InputPort/OutputPort/ErrorOutputPort, and obsoleted the old name
  • Added WorkerParent.BytesPerRowBuffer, which allows setting default buffer sizes for hierarchies of workers
  • Added TypeRowSchema.BytesPerRow
  • BytesPerRowBuffer and BytesPerRow are used to calculate the default RowsPerBuffer value
  • Obsoleted TypeSchemaCopyPolicy and moved its functionality to TypeDetail, and added BytesPerType
• Uses the new SqlCommand.EnableOptimizedParameterBinding feature to automatically avoid the Adb SqlClient Provider batch insert performance issue
• Adb SqlClient Provider and Adb ODBC Provider for SQL Server uses MaxValuesPerBatch=2098
• AdbInsertTarget maxRowsPerBatch is limited to 1000 rows, unless explicitly set with SetRowLimits(Int32, Int64)
• Removed obsolete AdbSqlClientBulkInsertTarget.SqlBulkCopyOptions
• IAdbInsertStatementService etc:
  • Added MaxRowsPerBatch and GetRowsPerBatch() to automatically limit rowsPerBatch to these max values
  • Removed obsoleted DefaultParametersPerStatement and MaxParametersPerStatement
• AdbSql92InsertStatementService, AdbOdbcProvider etc:
  • Removed all obsoleted *SingleRow* infrastructure. Removed GetMultiRowService().
  • Removed obsoleted AdbInsertTarget<TInputError> methods SetMaxRows and SetSingleRow()
• AdbProvider and derived providers:
  • Added With() overloads, removed With*() overloads
  • Added Get(...) overloads with insert statement parameters
• AdbSQLiteProvider.Get(): Reduced the default maxValuesPerBatch from 4000 to 999
• Adb Npgsql Provider supports System.Memory<System.Byte> to and from the database on .NET5+
• GetValueOutcome.Succeeded: Added ApplyToMatched parameter and property that shows if a Locator was used or not. This allows giving Locator settings higher precedence vs. global settings.
• Document support for dynamic row and column types (and their drawbacks) in Dataflow Columns
• MulticastTransform<TInputOutput>: Added factory methods with a column mapping parameter to allow avoiding copying column members with the SingleShallow copying policy (which would fail)

Fixes

• RowErrorCollection: Fixed a bug where errors were not copied when creating a new collection from old. This affected TypeColumnCopier<TFrom, TTo> workers.
• Fixed a TypeColumnCopier<TFrom, TTo> bug where *Join*Transform, MulticastTransform, and RepeatRowsSource workers could forward original SingleShallow/SingleShallowThenDeep column types instead of cloning them
• JoinMergeSortedTransformBase<TDerived, TLeftInput, TRightInput, TOutput>: Clone all copies (including the first one) since one input row can join with multiple rows from the other input

Package Dependency Changes

• FileHelpers 3.4.2 > 3.5.2
• Microsoft.Data.SqlClient 2.1.2 > 4.1.1
• MySql.Data 8.0.24 > 8.0.32.1
• MySqlConnector 1.3.3 > 1.3.14
• Newtonsoft.Json 12.0.3 > 13.0.3
• NLog 4.7.5 > 4.7.15
• NodaTime 2.4.13 > 3.1.9
• Npgsql 4.1.9 > 4.1.12
• System.Collections.Immutable 5.0.0 > 7.0.0
• System.Configuration.ConfigurationManager 5.0.0 > 7.0.0
• System.Data.Odbc 5.0.0 > 7.0.0
• System.Data.SQLite.Core 1.0.113.3 > 1.0.116
• System.Runtime.CompilerServices.Unsafe 5.0.0 > 6.0.0

Version 0.40.0

This release adds an additional MySQL/MariaDB provider that supports high performance bulk insert, adds new sample projects, and also provides a number of smaller additions, changes and fixes.

• Added new Samples projects:
  • AggregateCsvCreateInsertTable
  • CopyFileIfExists
  • MultipleWorkerSystemsDataflow
  • ReadSortWriteXlsx
• Added additional MySql/MariaDB provider Adb MySqlConnector Provider supporting high performance Bulk Insert
• Added transaction support to AdbMySqlConnectorBulkInsertTarget<TInput> and AdbSqlClientBulkInsertTarget<TInput>
• Added AdbSqlClientBulkInsertTarget.ColumnOrderHints and renamed SqlBulkCopyOptions to CopyOptions
• AdbInsertTarget<TInputError> related:
  • Reduced AdbInsertTarget database round-trips
  • Renamed SetMaxRows() to SetRowLimits(Int32, Int64)
  • Deprecated SetSingleRow()
  • Support setting default transaction size and disabling transactions
  • Fixed column mapping bug
  • ErrorOutput:
    • Fixed redirection of rows on insert exception
    • Linking disables creating transactions, and throws if there is a pre-existing transaction
• Added AdbProvider.WithSql92InsertStatement() to simplify setting default batch and transaction sizes for AdbInsertTarget<TInputError>
• IAdbInsertStatementService changes for AdbInsertTarget<TInputError>:
  • Renamed IAdbInsertStatementService.DefaultParametersPerStatement to DefaultValuesPerBatch
  • Renamed IAdbInsertStatementService.MaxParametersPerStatement to MaxValuesPerBatch
  • Added AConfigSetting InsertStatementValuesPerBatch and InsertStatementValuesPerTransaction
  • Added rowsPerBatch parameter to Create()
• ErrorOutputPort<TError>:
  • Added SendErrorBuffer() for sending a batch of error rows
  • Only log exception and message on the first row in a batch
  • Fixed bug when logging a null column
• Removed obsoleted InputPort.TakeRows()

Package dependencies:

• FileHelpers 3.4.1 > 3.4.2
• Microsoft.Data.SqlClient 2.0.1 > 2.1.2
• MySqlConnector 1.3.3 added
• MySql.Data 8.0.22 > 8.0.24
• NodaTime 2.4.8 > 2.4.13
• Npgsql 4.1.6 > 4.1.9
• System.Net.NameResolution removed
• System.Net.Primitives removed

Version 0.39.0

• Added a free Community edition
• Added .NET 5 support
• API changes:
  • Added ProgressStatusResult<TResult> and extended OutcomeStatusResult<TResult>
  • Changed OnOutputRowDemand() to return ProgressStatusResult<TResult>
• Added package dependency Microsoft.DotNet.PlatformAbstractions 3.1.6 and improved logging of .NET runtime version as well as operating system version
• Bumped package dependency versions:
  • Microsoft.Data.SqlClient 2.0.0 > 2.0.1
  • MySql.Data 8.0.21 > 8.0.22
  • NLog 4.7.3 > 4.7.5
  • Npgsql 4.1.4 > 4.1.6
  • System.CodeDom 4.7.0 > 5.0.0
  • System.Collections.Immutable 1.7.1 > 5.0.0
  • System.Configuration.ConfigurationManager 4.7.0 > 5.0.0
  • System.Data.Odbc 4.7.0 > 5.0.0
  • System.Data.SQLite.Core 1.0.113.1 > 1.0.113.6
  • System.Runtime.CompilerServices.Unsafe 4.7.1 > 5.0.0

Version 0.38.0

This release brings enhanced database support across all database providers, for both on-premises and cloud deployments:

• Improved GetTableColumnsAsync(String) and GetTableTypeAsync(String), which also helps AdbInsertTarget<TInputError> and AdbTableNonQueryWorker:
  • MariaDB™ and MySql™:
    • Fixed quoting bug.
  • MySql™:
    • Fixed table metadata bug for 4GB column types longblob and longtext.
  • PostgreSQL®:
    • Added support for temporary tables and PostgreSQL schema search_path.
    • Added support for BitArray to bit varying type mapping (in addition to the existing varbit mapping).
    • Fixed bug in GetTableColumnsAsync(String) overloads when no schema specified.
  • SQLite: Added support for views.
  • SQL Server®: Improved Precision, Scale, LengthInBytes, and LengthInCharacters column metadata from GetTableColumnsAsync(String) overloads.
• AdbExecuteScalarWorker<TResult> converts (with ChangeType(Object, Type)) the worker result if casting fails, thereby handling more data type mismatches.
• Verified actionETL on Azure for MariaDB, MySql, PostgreSQL, and Azure SQL Database.

Version 0.37.0

actionETL.nupkg

• Added commands to AdbTableNonQueryWorker for dropping views, and renamed table commands
• Added AdbTableType for detecting table-like type (table, view, or table-valued function). Renamed TableExistsAsync() to GetTableTypeAsync(String)
• Changed GetTableColumns() to asynchronous GetTableColumnsAsync(String) overloads.
• Added StringNumberRow dataflow row class to simplify extracting text lines with line numbers
• Fixed EnumerableSource<TOutput> dispose bug
• Deprecated InputPort<T>.TakeRows() since some use cases could deadlock dataflow. Use TakeBuffer() instead.

actionETL.templates.dotnet.nupkg

• Fixed the actionetl.console template getting the correct namespace

Version 0.36.1

• Added RepeatRowsSource.SendTemplateRows property for sending template rows without cloning them.
• Improved performance and fixed stack overflow for dataflows using >10,000 linked workers with the same parent.

Version 0.36.0

• Moved the library package and templates package distribution to nuget.org
• Added automatically making worker names unique when specified with trailing "/"
• Bumped package dependency versions:
  • MySql.Data 8.0.20 > 8.0.21
  • NLog 4.7.2 > 4.7.3
  • Npgsql 4.1.3.1 > 4.1.4
  • Sentry.PlatformAbstractions 1.1.0 > 1.1.1
  • System.Net.Primitives 4.3.0 > 4.3.1

Version 0.35.x

• Added .NET Standard and .NET Core support
• Added Linux support
• Added ‘dotnet new’ project templates
• Bumped package dependency versions:
  • EPPlus 4.5.3.2 > 4.5.3.3
  • Microsoft.Data.SqlClient 1.1.3 > 2.0.0
  • MySql.Data 8.0.19 > 8.0.20
  • NLog 4.6.8 > 4.7.0
  • NodaTime 2.4.7 > 2.4.8
  • Npgsql 4.1.2 > 4.1.3.1
  • System.Collections.Immutable 1.6.0 > 1.7.0
  • System.Data.SQLite.Core 1.0.112.2 > 1.0.113.1
  • System.Runtime.CompilerServices.Unsafe 4.6.0 > 4.7.1
  • System.Threading.Tasks.Extensions 4.5.3 > 4.5.4

Version 0.34.x

• Added PostgreSQL database support

Version 0.33.x

• Added SQLite database support

See Also

• actionETL Overview
• Common Tasks
• Getting Started
  • System Requirements
  • Dotnet Templates
  • Add actionETL Manually
  • Samples
  • Deploying Applications
  • Development Guidelines
• Worker System
• Workers
• Dataflow
• SQL Database Access
• Data Formats
• Transfer Protocols
• Troubleshooting
• API documentation