Search Results for

    Show / Hide Table of Contents

    Class FileHelperFileSource<TOutput>

    A dataflow worker that reads delimited (e.g. CSV) or fixed format data rows from one or more files, and sends them to a downstream worker.

    Please see CSV Delimited and Fixed Format for how to specify the data format. Internally, this worker uses the http://www.filehelpers.net/ library to read the data, specifically the FileHelperAsyncEngine class.

    Inheritance
    Object
    WorkerParent
    WorkerBase
    WorkerBase<FileHelperFileSource<TOutput>>
    FileHelperFileSource<TOutput>
    Implements
    IDisposeOnFinished
    Inherited Members
    WorkerBase<FileHelperFileSource<TOutput>>.AddCompletedCallback(Func<FileHelperFileSource<TOutput>, OutcomeStatus, Task<OutcomeStatus>>)
    WorkerBase<FileHelperFileSource<TOutput>>.AddRanCallback(Func<FileHelperFileSource<TOutput>, OutcomeStatus, WorkerParentChildrenState, Task<OutcomeStatus>>)
    WorkerBase<FileHelperFileSource<TOutput>>.AddStartingCallback(Func<FileHelperFileSource<TOutput>, Task<ProgressStatus>>)
    WorkerBase.AddCompletedCallback(Func<WorkerBase, OutcomeStatus, Task<OutcomeStatus>>)
    WorkerBase.AddRanCallback(Func<WorkerBase, OutcomeStatus, WorkerParentChildrenState, Task<OutcomeStatus>>)
    WorkerBase.AddStartingCallback(Func<WorkerBase, Task<ProgressStatus>>)
    WorkerBase.DefaultIsStartable()
    WorkerBase.ErroredPortErrorsWorkerProtected
    WorkerBase.ErrorOutputs
    WorkerBase.EscalateError
    WorkerBase.Inputs
    WorkerBase.IsStartable
    WorkerBase.Outputs
    WorkerBase.Parent
    WorkerBase.SucceededSequence<TLastWorker>(WorkerBase, WorkerBase, WorkerBase, WorkerBase, WorkerBase, TLastWorker)
    WorkerBase.SucceededSequence<TLastWorker>(WorkerBase, WorkerBase, WorkerBase, WorkerBase, TLastWorker)
    WorkerBase.SucceededSequence<TLastWorker>(WorkerBase, WorkerBase, WorkerBase, TLastWorker)
    WorkerBase.SucceededSequence<TLastWorker>(WorkerBase, WorkerBase, TLastWorker)
    WorkerBase.SucceededSequence<TLastWorker>(WorkerBase, TLastWorker)
    WorkerBase.SucceededSequence<TLastWorker>(TLastWorker)
    WorkerParent.AddChildCompletedCallback(Action<WorkerBase>)
    WorkerParent.AddStartingChildrenCallback(Func<WorkerParent, Task<ProgressStatus>>)
    WorkerParent.Children
    WorkerParent.DisposeOnFinished<TDisposable>(TDisposable)
    WorkerParent.GetDownstreamFactory<TInput>()
    WorkerParent.HasChildren
    WorkerParent.IsCanceled
    WorkerParent.IsCompleted
    WorkerParent.IsCreated
    WorkerParent.IsError
    WorkerParent.IsFailed
    WorkerParent.IsFatal
    WorkerParent.IsRunning
    WorkerParent.IsSucceeded
    WorkerParent.KeepChildrenLevels
    WorkerParent.Locator
    WorkerParent.LogFactory
    WorkerParent.Logger
    WorkerParent.MaxRunningChildren
    WorkerParent.Name
    WorkerParent.RemoveChildren()
    WorkerParent.RescheduleChildren()
    WorkerParent.RunChildrenAsync(Boolean)
    WorkerParent.RunChildrenAsync()
    WorkerParent.Status
    WorkerParent.Item[String]
    WorkerParent.ToString()
    WorkerParent.WorkerSystem
    WorkerParent.DebugCommands
    WorkerParent.AggregateErrorOutputRows
    WorkerParent.AggregateOutputRows
    WorkerParent.AggregateWorkersCompleted
    WorkerParent.InstantCompleted
    WorkerParent.InstantCreated
    WorkerParent.InstantStarted
    WorkerParent.RunningDuration
    Namespace: actionETL.FileHelper
    Assembly: actionETL.dll
    Syntax
    [CLSCompliant(false)]
    public class FileHelperFileSource<TOutput> : WorkerBase<FileHelperFileSource<TOutput>>, IDisposeOnFinished where TOutput : class
    Type Parameters
    Name Description
    TOutput

    A class that defines the format, columns, separators, column order etc. of the source data, as well as the type of the rows that will be passed to the downstream worker.

    Constructors

    FileHelperFileSource(WorkerParent, String, IEnumerable<String>)

    Initializes a new instance of the FileHelperFileSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more files, and outputs them to the downstream worker.

    Declaration
    public FileHelperFileSource(WorkerParent workerParent, string workerName, IEnumerable<string> fileNames)
    Parameters
    Type Name Description
    WorkerParent workerParent

    The parent worker or worker system that the new child worker will be added to. Cannot be null.

    String workerName

    Name of the worker.

    Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

    While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

    The name cannot otherwise contain "/", and cannot start with double underscore "__".

    IEnumerable<String> fileNames

    The file names.

    Exceptions
    Type Condition
    ArgumentException

    workerName:

    • Workers with the same parent must have unique names.
    • Worker and worker system names cannot contain '/' or start with double underscore '__'.
    ArgumentNullException

    workerParent - All workers must have a parent. The top level workers have the worker system as parent.

    InvalidOperationException
    • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
    • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

    FileHelperFileSource(WorkerParent, String, Func<Boolean>, IEnumerable<String>)

    Initializes a new instance of the FileHelperFileSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more files, and outputs them to the downstream worker.

    Declaration
    public FileHelperFileSource(WorkerParent workerParent, string workerName, Func<bool> isStartableFunc, IEnumerable<string> fileNames)
    Parameters
    Type Name Description
    WorkerParent workerParent

    The parent worker or worker system that the new child worker will be added to. Cannot be null.

    String workerName

    Name of the worker.

    Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

    While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

    The name cannot otherwise contain "/", and cannot start with double underscore "__".

    Func<Boolean> isStartableFunc

    Function to calculate the worker start constraint; it should return true for startable and false for not startable. Defaults to startable if null.

    IEnumerable<String> fileNames

    The file names.

    Exceptions
    Type Condition
    ArgumentException

    workerName:

    • Workers with the same parent must have unique names.
    • Worker and worker system names cannot contain '/' or start with double underscore '__'.
    ArgumentNullException

    workerParent - All workers must have a parent. The top level workers have the worker system as parent.

    InvalidOperationException
    • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
    • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

    FileHelperFileSource(WorkerParent, String, Func<Boolean>, String)

    Initializes a new instance of the FileHelperFileSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more files, and outputs them to the downstream worker.

    Declaration
    public FileHelperFileSource(WorkerParent workerParent, string workerName, Func<bool> isStartableFunc, string fileName)
    Parameters
    Type Name Description
    WorkerParent workerParent

    The parent worker or worker system that the new child worker will be added to. Cannot be null.

    String workerName

    Name of the worker.

    Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

    While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

    The name cannot otherwise contain "/", and cannot start with double underscore "__".

    Func<Boolean> isStartableFunc

    Function to calculate the worker start constraint; it should return true for startable and false for not startable. Defaults to startable if null.

    String fileName

    The file name.

    Exceptions
    Type Condition
    ArgumentException

    workerName:

    • Workers with the same parent must have unique names.
    • Worker and worker system names cannot contain '/' or start with double underscore '__'.
    ArgumentNullException

    workerParent - All workers must have a parent. The top level workers have the worker system as parent.

    InvalidOperationException
    • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
    • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

    FileHelperFileSource(WorkerParent, String, String)

    Initializes a new instance of the FileHelperFileSource<TOutput> dataflow worker, that reads delimited (e.g. CSV) or fixed format data rows from one or more files, and outputs them to the downstream worker.

    Declaration
    public FileHelperFileSource(WorkerParent workerParent, string workerName, string fileName)
    Parameters
    Type Name Description
    WorkerParent workerParent

    The parent worker or worker system that the new child worker will be added to. Cannot be null.

    String workerName

    Name of the worker.

    Set to a prefix plus a trailing "/" (e.g. "MyPrefix-/") to generate a unique name from the prefix plus an increasing number starting at 1.

    While less useful, set to null, whitespace or "/" to generate a unique name from the worker type plus an increasing number starting at 1.

    The name cannot otherwise contain "/", and cannot start with double underscore "__".

    String fileName

    The file name.

    Exceptions
    Type Condition
    ArgumentException

    workerName:

    • Workers with the same parent must have unique names.
    • Worker and worker system names cannot contain '/' or start with double underscore '__'.
    ArgumentNullException

    workerParent - All workers must have a parent. The top level workers have the worker system as parent.

    InvalidOperationException
    • Cannot add child worker to parent which has completed. Are you adding it to the correct parent?
    • Cannot add worker to parent, since its children have been started. Are you adding it to the correct parent?

    Properties

    Engine

    Gets the underlying FileHelpers FileHelperAsyncEngine instance. Use this for customizing the data transfer, such as setting the text encoding and preprocess the data records. See CSV Delimited and Fixed Format for details.

    Note: Retrieving and inspecting this property is thread-safe, but modifying the Engine state is not thread-safe, and should only be done from a single thread, before the worker has started.

    Declaration
    public FileHelperAsyncEngine<TOutput> Engine { get; }
    Property Value
    Type Description
    FileHelpers.FileHelperAsyncEngine<TOutput>

    The FileHelpers engine instance.

    Remarks

    Note that in rare cases, and only when using non-C# .NET languages, direct use of the Engine property can fail with a compile time error since the underlying FileHelpers library class (FileHelperAsyncEngine) is not CLS compliant (warning CS3003).

    ErrorOutput

    Gets the error output port for sending error rows to logging and an optional downstream worker.

    Declaration
    public ErrorOutputPort<StringRowErrors> ErrorOutput { get; }
    Property Value
    Type Description
    ErrorOutputPort<StringRowErrors>

    FileName

    Sets the name of the file. Overrides any FileNames setting. Cannot be set after the worker has started running.

    Note: This property is thread-safe.

    Declaration
    public string FileName { set; }
    Property Value
    Type Description
    String
    Exceptions
    Type Condition
    InvalidOperationException

    Cannot set the value after the worker has started running.

    FileNames

    Sets the file names. Overrides any FileName setting. Cannot be set after the worker has started running.

    Note: This property is thread-safe.

    Declaration
    public IEnumerable<string> FileNames { set; }
    Property Value
    Type Description
    IEnumerable<String>

    The file names.

    Exceptions
    Type Condition
    InvalidOperationException

    Cannot set the value after the worker has started running.

    Output

    Gets the output port for sending rows to the downstream worker.

    Declaration
    public OutputPort<TOutput> Output { get; }
    Property Value
    Type Description
    OutputPort<TOutput>

    Methods

    RunAsync()

    This method can be overridden to add custom functionality to the derived worker that runs before and after the row processing. In this case, the base class base.RunAsync() must be called for the worker to function correctly.

    Typically, this worker is used without overriding this method.

    Declaration
    protected override Task<OutcomeStatus> RunAsync()
    Returns
    Type Description
    Task<OutcomeStatus>

    A Task describing the success or failure of the worker. An asynchronous async implementation would e.g. return OutcomeStatus.Succeeded on success, while a synchronous implementation would return OutcomeStatus.SucceededTask.

    Overrides
    WorkerParent.RunAsync()

    Implements

    IDisposeOnFinished

    See Also

    FileHelperStreamSource<TOutput>
    FileHelperStringSource<TOutput>
    In This Article
    Back to top Copyright © 2020 Envobi Ltd