BulkWriter Library 3.0 release

We’re excited to announce the 3.0 release of the BulkWriter library. The library has proven very useful for loading data fast and efficiently, but it’s been in need of some updates given the changes in the .NET ecosystem in the last few years. ETL may not be the most glamorous of tasks, but it’s one we need often. We’ll walk you through the changes to BulkWriter featured in this latest release so you can use it to upgrade your own data-loading capabilities.

New feature: Processing pipelines

The single biggest change to BulkWriter is the addition of a new pipeline feature. BulkWriter previously focused only on the “L” in “ETL”.  With pipelines, the library now provides a compact way to define the “T”—your custom data transforms. The library will handle the internals of moving your data through the transforms in a fast and memory-efficient way so you can focus on the business logic.

Here’s some sample code to setup and run a transform pipeline:

using (var writer = new BulkWriter<PipelineTestsOtherTestClass>())
{
    var items = Enumerable.Range(1, 1000000).Select(i => new MyEntity { Id = i, Name = "Carol" });
    var pipeline = EtlPipeline
        .StartWith(items)
        .Aggregate(f => f.Max(c => c.Id))
        .Pivot(new BobFromIdPivot())
        .Project(i =>
        {
            var nameParts = i.Name.Split(' ');
            return new MyOtherEntity {Id = i.Id, FirstName = nameParts[0], LastName = nameParts[1] };
        })
        .TransformInPlace(i =>
        {
            i.Id -= 1;
            i.FirstName = "Alice";
            i.LastName = $"{i.Id}";
        })
        .LogWith(loggerFactory)
        .WriteTo(writer);

    await pipeline.ExecuteAsync();
}

Note the fluent interface for defining your pipeline. Your business logic will slot into the transform methods via Actions or Funcs, or via implementations of interfaces provided by the library. Head over to our Dev Corner for a demo that dives deeper into using the pipeline feature to create your custom ETL pipelines.

Better cross-platform support

.NET Core continues to improve on the cross-platform story for .NET apps. The platform has also really begun to stabilize now that there have been a few releases.

To that end, Bulk Writer now targets netstandard2.0.  We still have a build target for net461 to support the old Framework, but that will be removed in the next release.  We’ve also replaced the dependency on System.Data.SqlClient with Microsoft.Data.SqlClient. Microsoft has announced that as the Sql Server library of the future, and we can expect the cross-platform story to continue to evolve there.

Changes and improvements

We’ve also made a number of other changes and improvements since the last release.

  • Better, more comprehensive documentation (see http://headspringlabs.github.io/bulk-writer/)
  • Intellisense documentation is now in place for reference when you’re working in Visual Studio / Code
  • The ‘BulkWriter` class now provides full control over `BulkCopyOptions`
  • General cleanup of library interfaces: Previous releases included some artifacts that were not intended to be public
  • License change: The library license has changed from Apache 2.0 to MIT starting at this release.

Feedback welcome

We’d love to hear from you! How are you using the library? How can it be made better? PRs are always welcome: Submit a PR.
We hope to keep building up Bulk Writer with the help of our developer community!

Let's Talk