Transfer Protocols

There are many different transfer protocols (SFTP, SCP, FTP, rcp, FTPS, ...) in use for ETL applications, often with many connection and security options.

actionETL makes it easy to integrate with .NET libraries for these protocols, providing you access to their full capabilities; this article describes how to implement these integrations.

We'll use SFTP for our example, but the same approach can be used with other protocols; do search docs.microsoft.com and nuget.org for API and libraries supporting your particular protocol.

SFTP, SCP, SSH

SFTP and SCP are commonly used in ETL applications for secure file transfers. The free SSH.NET package and GitHub project is widely used, and we will use it to download files from an SFTP server:

using actionETL;
using actionETL.Logging;
using Renci.SshNet;
using System.IO;
using System.Threading.Tasks;

Put this code in a method:

var outcomeStatusTask = new WorkerSystem()
    .Root(ws =>
    {
        _ = new ActionWorker(ws, "Download files"
            , aw =>
            {
                using (var sftp = new SftpClient("demo.wftpserver.com", 2222, "demo", "demo"))
                {
                    sftp.Connect();
                    sftp.ChangeDirectory("download");

                    Download(aw, sftp, "Autumn.jpg", "autumn-local.jpg");
                    Download(aw, sftp, "Winter.jpg", "winter-local.jpg");

                    // Any other operations (upload, rename, stream, ...)
                }
            });
    })
    .StartAsync();
return outcomeStatusTask;

void Download(WorkerBase worker, SftpClient sftpClient
    , string remoteFileName, string localFileName)
{
    using (var file = File.OpenWrite(localFileName))
    {
        worker.Logger.Info(ALogCategory.ProgressWorker
            , $"Downloading '{remoteFileName}' to '{localFileName}'.");
        sftpClient.DownloadFile(remoteFileName, file);
    }
}

We used a Download() helper method to log and download each file. If you use this functionality in many different parts of your ETL you might create a custom worker that encapsulates creating the SftpClient instance.

You can increase throughput by running multiple workers in parallel, each operating on different sets of files. This is also a simple way of limiting number of parallel connections to your SFTP server, avoiding overloading it.

The SSH.NET library supports asynchronous operations, but uses the legacy style Asynchronous Programming Model (APM). This works, but is fairly verbose to use. If you need asynchronous operations, consider using the Renci.SshNet.Async wrapper which adds async-await functionality. In this case you would use the UsingActionWorker<TDisposable> instead of the above ActionWorker.

Note

SSH.NET provides a CHM documentation file with their release assets, but it's normally easier to use IntelliSense (or check the SftpClient source), and view the test cases, e.g. SftpFileTest.cs.

See Also