Transform Workers

A transform worker has both input and data output ports. It consumes and processes the input data rows, and outputs the same (modified or unmodified) rows it received, and/or generates new rows to output. The output ports can optionally have a different schema vs. the input ports.

Note
  • The library user instantiates transforms via factory methods, see Worker Instantiation for details
  • The type parameters describe what ports the transform has, e.g. *Error* means it has an error output port

Also see Source Workers and Target Workers, as well as Non-Dataflow Workers.

ActionTransform, ActionTwoInputTransform

Execute an Action or Func asynchronous callback once that consumes rows from the upstream worker and sends rows to the downstream worker. The workers have one or two input ports, respectively.

AggregateTransform

Aggregate and optionally group input rows, and output at most one row for all input rows (or one per grouping), calculating column aggregations (Sum, Average, Max etc.), row aggregations (First, Last, Single etc.), and custom aggregations.

CrossJoinTransform

Perform a CROSS JOIN on two inputs and send to the downstream worker.

DictionaryLookupTransform

Lookup a key in a dictionary (either provided or loaded from a DictionaryInput) for each row, with full or partial caching. Modify and redirect the row to the Output or ErrorOutput.

DictionaryLookupSplitTransform

Lookup a key in a dictionary (either provided or loaded from a DictionaryInput) for each row, with full or partial caching. Modify and redirect the row to the FoundOutput, NotFoundOutput, or ErrorOutput.

FullJoinMergeSortedTransform

Perform a FULL JOIN on two presorted inputs and send to the downstream worker.

InnerJoinMergeSortedTransform

Perform an INNER JOIN on two presorted inputs and send to the downstream worker.

LeftJoinMergeSortedTransform

Perform a LEFT JOIN on two presorted inputs and send to the downstream worker.

MergeSortedTransform

Merge multiple presorted inputs of the same type into a single sorted output. Any duplicates are preserved.

MulticastTransform

Clones input rows to one or more outputs, all of the same type.

RightJoinMergeSortedTransform

Perform a RIGHT JOIN on two presorted inputs and send to the downstream worker.

RowActionTransform

Execute an Action or Func callback for each input row before passing it to the downstream worker.

RowsActionTransform

Repeatedly execute an Action or Func callback when there is both rows to consume from the upstream worker and demand available from the downstream worker.

RowsTransformBase

Execute a virtual method when there are input rows and output demand available. Must be inherited.

RowTransformBase

Execute a virtual method for each input row before passing it to the downstream worker. Must be inherited.

SortTransform

Sort the incoming rows and pass them to the downstream worker, optionally removing duplicates.

SplitTransform

A worker with one input and one or more outputs, all of the same type, that sends each input row to a specific output, or discards the row, based on a supplied function.

TransformBase, TwoInputTransformBase

Workers with one or two inputs, and one output, for deriving dataflow transforms. Must be inherited.

UnionAllTransform

Combine multiple inputs of the same type into a single output. Any duplicates are preserved.

See Also