Class FileHelperStreamSource<TOutput>

A dataflow worker that reads delimited (e.g. CSV) or fixed format data rows from one or more streams, and sends them to a downstream worker.

Please see CSV Delimited and Fixed Format for how to specify the data format. Internally, this worker uses the http://www.filehelpers.net/ library to read the data, specifically the FileHelperAsyncEngine class.

Inheritance
Object
WorkerParent
WorkerBase
WorkerBase<FileHelperStreamSource<TOutput>>
FileHelperStreamSource<TOutput>
Implements
Inherited Members
Namespace: actionETL.FileHelper
Assembly: actionETL.dll
Syntax
[CLSCompliant(false)]
public class FileHelperStreamSource<TOutput> : WorkerBase<FileHelperStreamSource<TOutput>>, IDisposeOnFinished where TOutput : class
Type Parameters
Name Description
TOutput

A class that defines the format, columns, separators, column order etc. of the source data, as well as the type of the rows that will be passed to the downstream worker.

Constructors

FileHelperStreamSource(WorkerParent, String, IEnumerable<TextReader>)

Initializes a new instance of the FileHelperStreamSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more text streams, and outputs them to the downstream worker.

Declaration
public FileHelperStreamSource(WorkerParent workerParent, string workerName, IEnumerable<TextReader> textReaders)
Parameters
Type Name Description
WorkerParent workerParent

The parent worker or worker system that the new child worker will be added to. Cannot be null.
String workerName

Name of the worker.

Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

The name cannot otherwise contain "/", and cannot start with double underscore "__".

IEnumerable<TextReader> textReaders

The text readers.
Exceptions
Type Condition
ArgumentException

workerName:

  • Workers with the same parent must have unique names.
  • Worker and worker system names cannot contain '/' or start with double underscore '__'.
ArgumentNullException

workerParent - All workers must have a parent. The top level workers have the worker system as parent.
InvalidOperationException
  • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
  • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

FileHelperStreamSource(WorkerParent, String, Func<Boolean>, IEnumerable<TextReader>)

Initializes a new instance of the FileHelperStreamSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more text streams, and outputs them to the downstream worker.

Declaration
public FileHelperStreamSource(WorkerParent workerParent, string workerName, Func<bool> isStartableFunc, IEnumerable<TextReader> textReaders)
Parameters
Type Name Description
WorkerParent workerParent

The parent worker or worker system that the new child worker will be added to. Cannot be null.
String workerName

Name of the worker.

Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

The name cannot otherwise contain "/", and cannot start with double underscore "__".

Func<Boolean> isStartableFunc

Function to calculate the worker start constraint; it should return true for startable and false for not startable. Defaults to startable if null.
IEnumerable<TextReader> textReaders

The text readers.
Exceptions
Type Condition
ArgumentException

workerName:

  • Workers with the same parent must have unique names.
  • Worker and worker system names cannot contain '/' or start with double underscore '__'.
ArgumentNullException

workerParent - All workers must have a parent. The top level workers have the worker system as parent.
InvalidOperationException
  • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
  • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

FileHelperStreamSource(WorkerParent, String, Func<Boolean>, TextReader)

Initializes a new instance of the FileHelperStreamSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from a text stream, and outputs them to the downstream worker.

Declaration
public FileHelperStreamSource(WorkerParent workerParent, string workerName, Func<bool> isStartableFunc, TextReader textReader)
Parameters
Type Name Description
WorkerParent workerParent

The parent worker or worker system that the new child worker will be added to. Cannot be null.
String workerName

Name of the worker.

Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

The name cannot otherwise contain "/", and cannot start with double underscore "__".

Func<Boolean> isStartableFunc

Function to calculate the worker start constraint; it should return true for startable and false for not startable. Defaults to startable if null.
TextReader textReader

The text reader.
Exceptions
Type Condition
ArgumentException

workerName:

  • Workers with the same parent must have unique names.
  • Worker and worker system names cannot contain '/' or start with double underscore '__'.
ArgumentNullException

workerParent - All workers must have a parent. The top level workers have the worker system as parent.
InvalidOperationException
  • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
  • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

FileHelperStreamSource(WorkerParent, String, TextReader)

Initializes a new instance of the FileHelperStreamSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from a text stream, and outputs them to the downstream worker.

Declaration
public FileHelperStreamSource(WorkerParent workerParent, string workerName, TextReader textReader)
Parameters
Type Name Description
WorkerParent workerParent

The parent worker or worker system that the new child worker will be added to. Cannot be null.
String workerName

Name of the worker.

Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

The name cannot otherwise contain "/", and cannot start with double underscore "__".

TextReader textReader

The text reader.
Exceptions
Type Condition
ArgumentException

workerName:

  • Workers with the same parent must have unique names.
  • Worker and worker system names cannot contain '/' or start with double underscore '__'.
ArgumentNullException

workerParent - All workers must have a parent. The top level workers have the worker system as parent.
InvalidOperationException
  • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
  • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

Properties

Engine

Gets the underlying FileHelpers FileHelperAsyncEngine instance. Use this for customizing the data transfer, such as setting the text encoding and preprocess the data records. See CSV Delimited and Fixed Format for details.

Note: Retrieving and inspecting this property is thread-safe, but modifying the Engine state is not thread-safe, and should only be done from a single thread, before the worker has started.

Declaration
public FileHelperAsyncEngine<TOutput> Engine { get; }
Property Value
Type Description
FileHelpers.FileHelperAsyncEngine<TOutput>

The FileHelpers engine instance.
Remarks

Note that in rare cases, and only when using non-C# .NET languages, direct use of the Engine property can fail with a compile time error since the underlying FileHelpers library class (FileHelperAsyncEngine) is not CLS compliant (warning CS3003).

ErrorOutput

Gets the error output port for sending error rows to logging and an optional downstream worker.

Declaration
public ErrorOutputPort<StringRowErrors> ErrorOutput { get; }
Property Value
Type Description
ErrorOutputPort<StringRowErrors>

Output

Gets the output port for sending rows to the downstream worker.

Declaration
public OutputPort<TOutput> Output { get; }
Property Value
Type Description
OutputPort<TOutput>

TextReader

Sets the text reader. Overrides any TextReaders setting. Cannot be set after the worker has started running.

Note: This property is thread-safe.

Declaration
public TextReader TextReader { set; }
Property Value
Type Description
TextReader
Exceptions
Type Condition
InvalidOperationException

Cannot set the value after the worker has started running.

TextReaders

Sets the text readers. Overrides any TextReader setting. Cannot be set after the worker has started running.

Note: This property is thread-safe.

Declaration
public IEnumerable<TextReader> TextReaders { set; }
Property Value
Type Description
IEnumerable<TextReader>

The text readers.
Exceptions
Type Condition
InvalidOperationException

Cannot set the value after the worker has started running.

Methods

RunAsync()

This method can be overridden to add custom functionality to the derived worker that runs before and after the row processing. In this case, the base class base.RunAsync() must be called for the worker to function correctly.

Typically, this worker is used without overriding this method.

Declaration
protected override Task<OutcomeStatus> RunAsync()
Returns
Type Description
Task<OutcomeStatus>

A Task describing the success or failure of the worker. An asynchronous async implementation would e.g. return OutcomeStatus.Succeeded on success, while a synchronous implementation would return OutcomeStatus.SucceededTask.
Overrides

Implements

See Also