On Distributed Computing

How To Unit Test Framework Services In Orleans

Sun, 14 Jul 2019 00:00:00 +0000

Testing interactions between components in a distributed system context tends to involve full-scale integration testing rather than fast unit testing. This requires developers to wait for a deployment to execute, rather than having immediate feedback along the test-driven-development cycle. The use of the provider model in Orleans takes much of this pain away, as we can instead mock and fake the base services and providers, while relying on them to do the right thing at run time. We can still have deployment-time integration testing, but with Orleans, it is easy to have immediate testing feedback at development time.

This article describes how we can go about this, and it is based on the unit testing sample I’ve added to the Orleans repository.

TLDR

Clone the Orleans Unit Testing Sample
Explore the tests!

Overview

Though not obvious at first glance, Orleans allows mocking and faking of all of its own services and interfaces. This enables very high test coverage, which is can be hard to attain when developing a composed system.

Orleans favours testing using two approaches, Isolated Testing and Hosted Testing.

Isolated Testing

These are tests where both dependencies and Orleans services are mocked on a test-by-test basis. This style enables fine-grained isolated unit testing of the code under question. However, like any testing using mocks, it can also lead to more verbose test code due to redundant mocking constructs.

Hosted Testing

These are tests that run on the Orleans Test Cluster, where fake Orleans services are used. This style enables course-grained integration testing of multiple components, using fake data sources for higher coverage. It leads to both shorter and more reliable test code due to better reproduction of live running conditions. However, if using a shared test cluster, the developer must take care to prepare shared fake components and partition data as appropriate to avoid clashes during parallel testing. That said, the developer can spin up multiple test clusters as required and in parallel. The code samples in the appendix show how to wire this up in xUnit.

How It Works

The examples below demonstrate how to test various scenarios that depend on Orleans services. The actual grains under test are the same regardless of approach.

There is no setup required to code isolated unit tests. We call them isolated for a reason. Each one stands on its own.

However we do need some setup for the hosted integration tests - we need to spin up one or more test clusters where our tests can run. We also need to configure this cluster (or clusters!) with some fake services that we can verify afterwards. That said, this is easy to do - you can find that setup in the appendix at the end of this post.

Testing A Basic Grain

Here is a simple grain that allows setting and getting a value.

public class BasicGrain : Grain, IBasicGrain
{
    private int value;

    public Task<int> GetValueAsync() => Task.FromResult(value);

    public Task SetValueAsync(int value)
    {
        this.value = value;
        return Task.CompletedTask;
    }
}

Isolated Test

Testing this grain in isolation is no different from testing any other code.

[Fact]
public async Task Gets_And_Sets_Value()
{
    // create a new grain
    var grain = new BasicGrain();

    // assert the default value is zero
    Assert.Equal(0, await grain.GetValueAsync());

    // set a new value
    await grain.SetValueAsync(123);

    // assert the new value is as set
    Assert.Equal(123, await grain.GetValueAsync());
}

It may feel funny that while we never create grain ourselves in feature code, we’re doing it here. However this is still a valid - and fast - way of unit testing them.

Hosted Test

We can also test this grain on an in-memory test cluster. Again, see the appendix for instructions on how to set this up. This allows us to grab grains from said cluster at test time, in the same way we write feature code.

[Fact]
public async Task Gets_And_Sets_Value()
{
    // get a new basic grain from the cluster
    var grain = fixture.Cluster.GrainFactory.GetGrain<IBasicGrain>(Guid.NewGuid());

    // assert the default value is zero
    Assert.Equal(0, await grain.GetValueAsync());

    // set a new value
    await grain.SetValueAsync(123);

    // assert the new value is as set
    Assert.Equal(123, await grain.GetValueAsync());
}

In the example above, fixture.Cluster references the test cluster, which makes available a GrainFactory for us to use as we would in feature code.

This approach didn’t save any lines of code here (and we’ve had to pay to set this up), but now we’re testing a live grain, living in a live cluster and with the live behaviours we expect to see. We will get our investment money back soon enough.

Testing A Grain That Calls Another Grain

One of the neat things of actor frameworks like Orleans is the ability of actors (or grains in this case) to call each other across the cluster, regardless of where they are. This enables application modelling techniques very hard to do achieve in stateless designs. Here is a sample grain that keeps a running count and publishes that counter to some other grain when requested, wherever it is. And don’t worry if it looks simple - we’ll complicate this soon enough.

public class CallingGrain : Grain, ICallingGrain
{
    private int counter;

    public Task IncrementAsync()
    {
        counter += 1;
        return Task.CompletedTask;
    }

    public Task PublishAsync() => GrainFactory.GetGrain<ISummaryGrain>(Guid.Empty).SetAsync(GrainKey, counter);

    /// <summary>
    /// Opens up the grain factory for mocking.
    /// </summary>
    public virtual new IGrainFactory GrainFactory => base.GrainFactory;

    /// <summary>
    /// Opens up the grain key name for mocking.
    /// </summary>
    public virtual string GrainKey => this.GetPrimaryKeyString();
}

On Dependency Surfacing

Note how we are opening up the GrainFactory and GrainKey properties for mocking. Under normal circumstances, Orleans is the one providing these features, and hence they will not work when we instantiate the grain ourselves. We must therefore open them up for mocking in the isolated test code.

On the other hand, we don’t need this at all for hosted tests! If you’re going with the test cluster, by all means, remove that code. When grains are running in a live cluster, those features work just as you expect them to.

Isolated Test

Testing this grain in isolation now requires a bit more care…

[Fact]
public async Task Publishes_On_Demand()
{
    // mock a summary grain
    var summary = Mock.Of<ISummaryGrain>();

    // mock the grain factory
    var factory = Mock.Of<IGrainFactory>(_ => _.GetGrain<ISummaryGrain>(Guid.Empty, null) == summary);

    // mock the grain type so we can mock the base orleans methods
    var grain = new Mock<CallingGrain>() { CallBase = true };
    grain.Setup(_ => _.GrainFactory).Returns(factory);
    grain.Setup(_ => _.GrainKey).Returns("MyCounter");

    // increment the value in the grain
    await grain.Object.IncrementAsync();

    // publish the value to the summary
    await grain.Object.PublishAsync();

    // assert the summary was called as expected
    Mock.Get(summary).Verify(_ => _.SetAsync("MyCounter", 1));
}

As testing this grain now requires mocking of framework services, we now have to use var grain = new Mock<CallingGrain>() { CallBase = true }; in order to override them. Overriding also adds a few more lines of code. Still, the test remains isolated as we intend it to be.

Hosted Test

The hosted test on the other hand, looks as simple as feature code is with Orleans.

[Fact]
public async Task Publishes_On_Demand()
{
    // we should use a unique grain key as we are using a shared test cluster
    var key = Guid.NewGuid().ToString();

    // get a new grain from the test host
    var grain = fixture.Cluster.GrainFactory.GetGrain<ICallingGrain>(key);

    // increment the value in the grain
    await grain.IncrementAsync();

    // publish the value to the summary
    await grain.PublishAsync();

    // assert the summary was called as expected
    Assert.Equal(1, await fixture.Cluster.GrainFactory.GetGrain<ISummaryGrain>(Guid.Empty).TryGetAsync(key));
}

There are no framework services to fake in the unit test, as the test cluster is providing the correct functionality out-of-the-box, leading to cleaner test code.

Testing A Grain That Performs Work On A Timer

Another common scenario is for grains to perform some work on a timer tick. For the benefit of simplicity, here is a grain that just increments an internal counter on a schedule.

public class TimerGrain : Grain, ITimerGrain
{
    private int value;

    public Task<int> GetValueAsync() => Task.FromResult(value);

    public override Task OnActivateAsync()
    {
        RegisterTimer(_ =>
        {
            ++value;
            return Task.CompletedTask;
        }, null, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1));

        return base.OnActivateAsync();
    }

    /// <summary>
    /// This opens up the timer registration method for mocking.
    /// </summary>
    public virtual new IDisposable RegisterTimer(Func<object, Task> asyncCallback, object state, TimeSpan dueTime, TimeSpan period) =>
        base.RegisterTimer(asyncCallback, state, dueTime, period);
}

Note how we are opening up the RegisterTimer() framework method for mocking, just like we did before with other services. Again, we only need this for isolated tests and not for hosted tests.

Isolated Test

Testing this in isolation requires us to override the timer behaviour. This is because Orleans is no longer there to issue timer ticks in the first place.

[Fact]
public async Task Increments_Value_On_Timer()
{
    // mock the grain to override methods
    var grain = new Mock<TimerGrain>() { CallBase = true };

    // mock the timer registration method and capture the action
    Func<object, Task> action = null;
    object state = null;
    var dueTime = TimeSpan.FromSeconds(1);
    var period = TimeSpan.FromSeconds(1);
    grain.Setup(_ => _.RegisterTimer(It.IsAny<Func<object, Task>>(), It.IsAny<object>(), It.IsAny<TimeSpan>(), It.IsAny<TimeSpan>()))
        .Callback<Func<object, Task>, object, TimeSpan, TimeSpan>((a, b, c, d) => { action = a; state = b; dueTime = c; period = d; })
        .Returns(Mock.Of<IDisposable>());

    // simulate activation
    await grain.Object.OnActivateAsync();

    // assert the timer was registered
    Assert.NotNull(action);
    Assert.Null(state);
    Assert.Equal(TimeSpan.FromSeconds(1), dueTime);
    Assert.Equal(TimeSpan.FromSeconds(1), period);

    // assert the initial value is zero
    Assert.Equal(0, await grain.Object.GetValueAsync());

    // tick the timer
    await action(null);

    // assert the new value is one
    Assert.Equal(1, await grain.Object.GetValueAsync());
}

Here we are overriding the RegisterTimer() framework method in order to tick the timer when we want to, keeping the test very fast. Note as well how we are explicitly calling OnActivateAsync(). We must do this as Orleans is not there to call it for us as needed. This does start leading to more verbose code now, though nothing out of the ordinary for this type of functionality.

Hosted Test

Testing this using the test cluster is far more straightforward. However, this requires some common setup to work, which you can find in the appendix.

[Fact]
public async Task Increments_Value_On_Timer()
{
    // using a random key allows parallel testing on the shared cluster
    var key = Guid.NewGuid();

    // get a new instance of the grain
    var grain = fixture.Cluster.GrainFactory.GetGrain<ITimerGrain>(key);

    // assert the initial state is zero - this will activate the grain and register the timer
    Assert.Equal(0, await grain.GetValueAsync());

    // assert the timer was registered on some silo
    var timer = fixture.GetTimers(grain).SingleOrDefault();

    Assert.NotNull(timer);

    // tick the timer
    await timer.TickAsync();

    // assert the new value is one
    Assert.Equal(1, await grain.GetValueAsync());
}

As compared to the isolated test, this one looks far more readable. Get the grain proxy, call something to activate it, ensure the timer got registered, tick it, and check the results. Nothing to it.

Testing A Grain That Calls Another Grain On A Timer

Didn’t I say we’d start complicating soon enough? Well this is yet another common use of grains, calling some other grain on a schedule in order to do some work, often to propagate some transient state, or to act as a watchdog for some activity.

public class CallingTimerGrain : Grain, ICallingTimerGrain
{
    private int counter;

    /// <summary>
    /// Orleans calls this on grain activation.
    /// For isolated unit tests we must call this to simulate activation.
    /// However, the test host will call this on its own.
    /// </summary>
    public override Task OnActivateAsync()
    {
        // register a timer to call another grain every second
        RegisterTimer(_ => GrainFactory.GetGrain<ISummaryGrain>(Guid.Empty).SetAsync(GrainKey, counter),
            null, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1));

        return base.OnActivateAsync();
    }

    /// <summary>
    /// This opens up the grain key for mocking.
    /// </summary>
    public virtual string GrainKey => this.GetPrimaryKeyString();

    /// <summary>
    /// This opens up the grain factory property for mocking.
    /// </summary>
    public virtual new IGrainFactory GrainFactory =>
        base.GrainFactory;

    /// <summary>
    /// This opens up the timer registration method for mocking.
    /// </summary>
    public virtual new IDisposable RegisterTimer(Func<object, Task> asyncCallback, object state, TimeSpan dueTime, TimeSpan period) =>
        base.RegisterTimer(asyncCallback, state, dueTime, period);

    /// <summary>
    /// Increments the counter by one.
    /// </summary>
    public Task IncrementAsync()
    {
        counter += 1;
        return Task.CompletedTask;
    }
}

Again, the opening up of the GrainKey, GrainFactory and RegisterTimer services for mocking is only to enable isolated testing.

Isolated Test

Testing this scenario in isolation is now a matter of combining the mocking of the timer service with the mocking of the grain factory.

public class CallingTimerGrainTests
{
    [Fact]
    public async Task Publishes_Counter_To_Summary_On_Timer()
    {
        // mock the summary grain
        var summary = Mock.Of<ISummaryGrain>();

        // mock the grain factory and summary grain
        var factory = Mock.Of<IGrainFactory>(_ => _.GetGrain<ISummaryGrain>(Guid.Empty, null) == summary);

        // mock the grain under test and override affected orleans methods
        var grain = new Mock<CallingTimerGrain>() { CallBase = true };
        grain.Setup(_ => _.GrainFactory).Returns(factory);
        grain.Setup(_ => _.GrainKey).Returns("MyGrainKey");

        Func<object, Task> action = null;
        object state = null;
        var dueTime = TimeSpan.Zero;
        var period = TimeSpan.Zero;
        grain.Setup(_ => _.RegisterTimer(It.IsAny<Func<object, Task>>(), It.IsAny<object>(), It.IsAny<TimeSpan>(), It.IsAny<TimeSpan>()))
            .Callback<Func<object, Task>, object, TimeSpan, TimeSpan>((a, b, c, d) => { action = a; state = b; dueTime = c; period = d; })
            .Returns(Mock.Of<IDisposable>());

        // increment the value while simulating activation
        await grain.Object.OnActivateAsync();
        await grain.Object.IncrementAsync();

        // assert the timer was registered
        Assert.NotNull(action);
        Assert.Null(state);
        Assert.Equal(TimeSpan.FromSeconds(1), dueTime);
        Assert.Equal(TimeSpan.FromSeconds(1), period);

        // tick the timer
        await action(null);

        // assert the summary got the first result
        Mock.Get(factory.GetGrain<ISummaryGrain>(Guid.Empty)).Verify(_ => _.SetAsync("MyGrainKey", 1));

        // increment the value
        await grain.Object.IncrementAsync();

        // tick the timer again
        await action(null);

        // assert the summary got the next result
        Mock.Get(summary).Verify(_ => _.SetAsync("MyGrainKey", 2));
    }
}

Like mocking code always does, though, it does start getting quite verbose.

Hosted Test

Testing this grain on the hosted test cluster, however, still resembles feature code.

[Fact]
public async Task Publishes_Counter_To_Summary_On_Timer()
{
    // using a random key allows parallel testing on the shared cluster
    var key = Guid.NewGuid().ToString();

    // get a new grain instance to test
    var grain = fixture.Cluster.GrainFactory.GetGrain<ICallingTimerGrain>(key);

    // increment the counter - this will also activate the grain and register the timer
    await grain.IncrementAsync();

    // assert the timer was registered on some silo in the cluster
    var timer = fixture.GetTimers(grain).Single();

    // tick the timer once
    await timer.TickAsync();

    // assert the summary grain got the first result
    Assert.Equal(1, await fixture.Cluster.GrainFactory.GetGrain<ISummaryGrain>(Guid.Empty).TryGetAsync(key));

    // increment the counter again
    await grain.IncrementAsync();

    // tick the timer again
    await timer.TickAsync();

    // assert the summary grain got the second result
    Assert.Equal(2, await fixture.Cluster.GrainFactory.GetGrain<ISummaryGrain>(Guid.Empty).TryGetAsync(key));
}

We get the grain proxy, ask to increment the counter, tick the timer, ensure the target grain got the result and then do it again just to make sure. Nice and tidy, we’re starting to get our investment back.

Testing A Grain That Does Work On A Reminder

Reminders are just long-lived timers that survive the grain lifecycle. Testing grains that use reminders is therefore similiar to testing grain that use timers. Here is a sample grain that increments a counter upon a reminder tick.

public class ReminderGrain : Grain, IReminderGrain, IRemindable
{
    private int value;

    public override async Task OnActivateAsync()
    {
        await RegisterOrUpdateReminder(nameof(IncrementAsync), TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1));
        await base.OnActivateAsync();
    }

    public Task<int> GetValueAsync() => Task.FromResult(value);

    private Task IncrementAsync()
    {
        value += 1;
        return Task.CompletedTask;
    }

    public virtual new Task<IGrainReminder> RegisterOrUpdateReminder(string reminderName, TimeSpan dueTime, TimeSpan period) =>
        base.RegisterOrUpdateReminder(reminderName, dueTime, period);

    public async Task ReceiveReminder(string reminderName, TickStatus status)
    {
        switch (reminderName)
        {
            case nameof(IncrementAsync):
                await IncrementAsync();
                break;

            default:
                await UnregisterReminder(await GetReminder(reminderName));
                break;
        }
    }
}

Isolated Testing

Due to the way grains implement the IRemindable interface, ticking the reminder is as simple as calling the ReceiveReminder() method on its own with a given TickStatus value.

[Fact]
public async Task Increments_Value_On_Reminder()
{
    // mock the grain
    var grain = new Mock<ReminderGrain>() { CallBase = true };
    grain.Setup(_ => _.RegisterOrUpdateReminder("IncrementAsync", TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1)))
        .Returns(Task.FromResult(Mock.Of<IGrainReminder>(_ => _.ReminderName == "IncrementAsync")));

    // simulate activation
    await grain.Object.OnActivateAsync();

    // ensure the reminder was registered
    grain.Verify(_ => _.RegisterOrUpdateReminder("IncrementAsync", TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1)));

    // assert the initial value is zero
    Assert.Equal(0, await grain.Object.GetValueAsync());

    // tick the reminder
    await grain.Object.ReceiveReminder("IncrementAsync", new TickStatus());

    // assert the new value is one
    Assert.Equal(1, await grain.Object.GetValueAsync());
}

Overriding the RegisterOrUpdateReminder() method allows us to Verify() that the grain indeed registered the reminder as we expect.

Hosted Testing

We can also do this using the test cluster, again leading to cleaner code.

[Fact]
public async Task Increments_Value_On_Reminder()
{
    // get a new grain to test
    var grain = fixture.Cluster.GrainFactory.GetGrain<IReminderGrain>(Guid.NewGuid());

    // assert the initial value is zero - this will also activate the grain
    Assert.Equal(0, await grain.GetValueAsync());

    // assert the reminder was registered on one of the fake registries
    var reminder = fixture.GetReminder(grain, "IncrementAsync");
    Assert.NotNull(reminder);

    // tick the reminder
    await grain.AsReference<IRemindable>().ReceiveReminder("IncrementAsync", new TickStatus());

    // assert the new value is one
    Assert.Equal(1, await grain.GetValueAsync());
}

Testing A Grain That Persists State To Storage

As a final example, here is a grain that persists its own state to storage upon request.

public class PersistentGrain : Grain, IPersistentGrain
{
    private readonly IPersistentState<MyState> value;

    public PersistentGrain([PersistentState("State")] IPersistentState<MyState> value)
    {
        this.value = value;
    }

    public Task SetValueAsync(int value)
    {
        this.value.State.Value = value;
        return this.value.WriteStateAsync();
    }

    [ProtoContract]
    public class MyState
    {
        [ProtoMember(1)]
        public int Value { get; set; }
    }
}

The ProtoContract stuff relates to protocol buffers, not a target for testing right now.

Isolated Testing

For testing in isolation, we just mock the IPersistentState<PersistentGrain.MyState> and Verify its use afterwards.

[Fact]
public async Task Saves_State()
{
    // mock a persistent state item
    var state = Mock.Of<IPersistentState<PersistentGrain.MyState>>(_ => _.State == Mock.Of<PersistentGrain.MyState>());

    // create a new grain - we dont mock the grain here because we do not need to override any base methods
    var grain = new PersistentGrain(state);

    // set a new value
    await grain.SetValueAsync(123);

    // assert the state was saved
    Mock.Get(state).Verify(_ => _.WriteStateAsync());
    Assert.Equal(123, state.State.Value);
}

Hosted Testing

For hosting testing, we instead make use of the pre-registered fake storage provider to tell us what the state looks like.

[Fact]
public async Task Saves_State()
{
    // get a brand new grain to test
    var grain = fixture.Cluster.GrainFactory.GetGrain<IPersistentGrain>(Guid.NewGuid());

    // set its value to something we can check
    await grain.SetValueAsync(123);

    // assert that state was saved by one of the silos
    var state = fixture.GetGrainState(typeof(PersistentGrain), "State", grain);
    Assert.NotNull(state);

    // assert that state is of the corect type
    var obj = state.State as PersistentGrain.MyState;
    Assert.NotNull(obj);

    // assert that state has the correct value
    Assert.Equal(123, obj.Value);
}

Appendix: Setting Up Hosted Tests

While use of the test cluster makes for more straightforward testing, it does require some up-front setup. That said, this is a one-time cost. If you’re using xUnit you can probably lift the fixture code from the samples outright and go with that.

Let’s looks at the class we need to add to an xUnit test project

ClusterFixture

The ClusterFixture works as an xUnit managed fixture - a class that xUnit will instantiate and dispose of, while running a given collection of tests.

The fixture below enables spinning up multiple test clusters, each with its own set of silos.

As each cluster can have multiple silos, each with its own set of fake services, the fixture provides some helper methods to locate the right services in the right places during testing.

public class ClusterFixture : IDisposable
{
    /// <summary>
    /// Identifier for this test cluster instance to facilitate parallel testing with multiple clusters that need fake services.
    /// </summary>
    public string TestClusterId { get; } = Guid.NewGuid().ToString();

    /// <summary>
    /// Exposes the shared cluster for unit tests to use.
    /// </summary>
    public TestCluster Cluster { get; }

    /// <summary>
    /// Keeps all the fake grain storage instances in use by different clusters to facilitate parallel unit testing.
    /// </summary>
    public static ConcurrentDictionary<string, ConcurrentBag<FakeGrainStorage>> GrainStorageGroups { get; } = new ConcurrentDictionary<string, ConcurrentBag<FakeGrainStorage>>();

    /// <summary>
    /// Gets the fake grain storage item for the given grain by searching across all silos.
    /// </summary>
    public IGrainState GetGrainState(Type implementationType, string name, IGrain grain)
    {
        return GrainStorageGroups[TestClusterId]
            .SelectMany(_ => _.Storage)
            .Where(_ => _.Key.Item1 == $"{implementationType.FullName}{(name == null ? "" : $",{typeof(PersistentGrain).Namespace}.{name}")}")
            .Where(_ => _.Key.Item2.Equals((GrainReference)grain))
            .Select(_ => _.Value)
            .SingleOrDefault();
    }

    /// <summary>
    /// Keeps all the fake timer registries in use by different clusters to facilitate parallel unit testing.
    /// </summary>
    public static ConcurrentDictionary<string, ConcurrentBag<FakeTimerRegistry>> TimerRegistryGroups { get; } = new ConcurrentDictionary<string, ConcurrentBag<FakeTimerRegistry>>();

    /// <summary>
    /// Gets all the fake timers for the target grain across all silos.
    /// </summary>
    public IEnumerable<FakeTimerEntry> GetTimers(IGrain grain)
    {
        return TimerRegistryGroups[TestClusterId]
            .SelectMany(_ => _.GetAll())
            .Where(_ => _.Grain.GrainReference.Equals((GrainReference)grain));
    }

    /// <summary>
    /// Keeps all the fake reminder registries in use by different clusters to facilitate parallel unit testing.
    /// </summary>
    public static ConcurrentDictionary<string, ConcurrentBag<FakeReminderRegistry>> ReminderRegistryGroups { get; } = new ConcurrentDictionary<string, ConcurrentBag<FakeReminderRegistry>>();

    /// <summary>
    /// Gets the target fake reminder by searching across all silos.
    /// </summary>
    public FakeReminder GetReminder(IGrain grain, string name)
    {
        return ReminderRegistryGroups[TestClusterId]
            .Select(_ => _.GetReminder((GrainReference)grain, name).Result)
            .Where(_ => _ != null)
            .SingleOrDefault();
    }

    public ClusterFixture()
    {
        // prepare to receive the fake services from individual silos
        GrainStorageGroups[TestClusterId] = new ConcurrentBag<FakeGrainStorage>();
        TimerRegistryGroups[TestClusterId] = new ConcurrentBag<FakeTimerRegistry>();
        ReminderRegistryGroups[TestClusterId] = new ConcurrentBag<FakeReminderRegistry>();

        var builder = new TestClusterBuilder();

        // add the cluster id for this instance
        // this allows the silos to safely lookup shared data for this cluster deployment
        // without this we can only share data via static properties and that messes up parallel testing
        builder.ConfigureHostConfiguration(config =>
        {
            config.AddInMemoryCollection(new Dictionary<string, string>()
            {
                { nameof(TestClusterId), TestClusterId }
            });
        });

        // a configurator allows the silos to configure themselves
        // at this time, configurators cannot take injected parameters
        // therefore we must other means of sharing objects as you can see above
        builder.AddSiloBuilderConfigurator<SiloBuilderConfigurator>();

        Cluster = builder.Build();
        Cluster.Deploy();
    }

    private class SiloBuilderConfigurator : ISiloBuilderConfigurator
    {
        public void Configure(ISiloHostBuilder hostBuilder)
        {
            hostBuilder.ConfigureServices(services =>
            {
                // add the fake storage provider as default in a way that lets us extract it afterwards
                services.AddSingleton(_ => new FakeGrainStorage());
                services.AddSingleton<IGrainStorage>(_ => _.GetService<FakeGrainStorage>());

                // add the fake timer registry in a way that lets us extract it afterwards
                services.AddSingleton<FakeTimerRegistry>();
                services.AddSingleton<ITimerRegistry>(_ => _.GetService<FakeTimerRegistry>());

                // add the fake reminder registry in a way that lets us extract it afterwards
                services.AddSingleton<FakeReminderRegistry>();
                services.AddSingleton<IReminderRegistry>(_ => _.GetService<FakeReminderRegistry>());
            });

            hostBuilder.UseServiceProviderFactory(services =>
            {
                var provider = services.BuildServiceProvider();
                var config = provider.GetService<IConfiguration>();

                // grab the cluster id that owns this silo
                var clusterId = config[nameof(TestClusterId)];

                // extract the fake services from the silo so unit tests can access them
                GrainStorageGroups[clusterId].Add(provider.GetService<FakeGrainStorage>());
                TimerRegistryGroups[clusterId].Add(provider.GetService<FakeTimerRegistry>());
                ReminderRegistryGroups[clusterId].Add(provider.GetService<FakeReminderRegistry>());

                return provider;
            });
        }
    }

    public void Dispose() => Cluster.StopAllSilos();
}

To be able to tell xUnit where and when to use this fixture, we need to create at least one CollectionDefinition for it. This is an xUnit specific pattern.

[CollectionDefinition(nameof(ClusterCollection))]
public class ClusterCollection : ICollectionFixture<ClusterFixture>
{
}

Nothing stops us from creating more collection definitions, each with its own name, instead of the default in the example. This is what allows us to spin up multiple clusters at the same time.

This xUnit collection allows us to inject a cluster fixture instance into specifc xUnit unit tests.

[Collection(nameof(ClusterCollection))]
public class BasicGrainTests
{
    private readonly ClusterFixture fixture;

    public BasicGrainTests(ClusterFixture fixture)
    {
        this.fixture = fixture;
    }

    /* ... */
}

And voila - shared test cluster on-demand.

Fakes

The example test cluster fixture makes generous use of fake services to promote easier tests. Below are some example, and nothing stops us from adding more as we need to.

The FakeGrainStorage holds grain state in memory, in the same spirit of the built-in memory storage provider, but one that we can search across all silos for easy state verification.

public class FakeGrainStorage : IGrainStorage
{
    public ConcurrentDictionary<Tuple<string, GrainReference>, IGrainState> Storage { get; } = new ConcurrentDictionary<Tuple<string, GrainReference>, IGrainState>();

    public Task ClearStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
    {
        Storage.TryRemove(Tuple.Create(grainType, grainReference), out _);
        return Task.CompletedTask;
    }

    public Task ReadStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
    {
        Storage.TryGetValue(Tuple.Create(grainType, grainReference), out grainState);
        return Task.CompletedTask;
    }

    public Task WriteStateAsync(string grainType, GrainReference grainReference, IGrainState grainState)
    {
        Storage[Tuple.Create(grainType, grainReference)] = grainState;
        return Task.CompletedTask;
    }
}

The FakeReminder is used by the FakeReminderRegistry to hold reminder information.

public class FakeReminder : IGrainReminder
{
    public FakeReminder(string reminderName, TimeSpan dueTime, TimeSpan period)
    {
        ReminderName = reminderName;
        DueTime = dueTime;
        Period = period;
    }

    public string ReminderName { get; }
    public TimeSpan DueTime { get; }
    public TimeSpan Period { get; }
}

The FakeReminderRegistry overrides the built-in reminder services, storing reminder entries in-memory and in searchable form.

public class FakeReminderRegistry : GrainServiceClient<IReminderService>, IReminderRegistry
{
    private readonly ConcurrentDictionary<GrainReference, ConcurrentDictionary<string, FakeReminder>> reminders =
        new ConcurrentDictionary<GrainReference, ConcurrentDictionary<string, FakeReminder>>();

    public FakeReminderRegistry(IServiceProvider provider) : base(provider)
    {
    }

    private ConcurrentDictionary<string, FakeReminder> GetRemindersFor(GrainReference reference) =>
        reminders.GetOrAdd(reference, _ => new ConcurrentDictionary<string, FakeReminder>());

    #region Fake Service Calls

    public Task<IGrainReminder> GetReminder(string reminderName)
    {
        GetRemindersFor(CallingGrainReference).TryGetValue(reminderName, out var reminder);
        return Task.FromResult((IGrainReminder)reminder);
    }

    public Task<List<IGrainReminder>> GetReminders() =>
        Task.FromResult(GetRemindersFor(CallingGrainReference).Values.Cast<IGrainReminder>().ToList());

    public Task<IGrainReminder> RegisterOrUpdateReminder(string reminderName, TimeSpan dueTime, TimeSpan period)
    {
        var reminder = new FakeReminder(reminderName, dueTime, period);
        GetRemindersFor(CallingGrainReference)[reminderName] = reminder;
        return Task.FromResult((IGrainReminder)reminder);
    }

    public Task UnregisterReminder(IGrainReminder reminder)
    {
        GetRemindersFor(CallingGrainReference).TryRemove(reminder.ReminderName, out _);
        return Task.CompletedTask;
    }

    #endregion Unvalidated Service Calls

    #region Test Helpers

    public Task<FakeReminder> GetReminder(GrainReference grainRef, string reminderName)
    {
        GetRemindersFor(grainRef).TryGetValue(reminderName, out var reminder);
        return Task.FromResult(reminder);
    }

    #endregion Test Helpers
}

The FakeTimerEntry is used by the FakeTimerRegistry to hold timer information in a searchable form.

/// <summary>
/// Implements a fake timer entry to facilitate unit testing.
/// </summary>
public class FakeTimerEntry : IDisposable
{
    private readonly TaskScheduler scheduler;
    private readonly FakeTimerRegistry owner;
    public Grain Grain { get; }
    public Func<object, Task> AsyncCallback { get; }
    public object State { get; }
    public TimeSpan DueTime { get; }
    public TimeSpan DuePeriod { get; }

    public FakeTimerEntry(FakeTimerRegistry owner, TaskScheduler scheduler, Grain grain, Func<object, Task> asyncCallback, object state, TimeSpan dueTime, TimeSpan period)
    {
        this.scheduler = scheduler;
        this.owner = owner;

        Grain = grain;
        AsyncCallback = asyncCallback;
        State = state;
        DueTime = dueTime;
        DuePeriod = period;
    }

    /// <summary>
    /// Ticks the timer action within the activation context.
    /// </summary>
    public async Task TickAsync() => await await Task.Factory.StartNew(AsyncCallback, State, default, TaskCreationOptions.None, scheduler);

    public void Dispose()
    {
        try
        {
            owner.Remove(this);
        }
        catch (Exception)
        {
            // noop
        }
    }
}

The FakeTimerRegistry overrides the built-in timer registry and enables easier searching for timers from test code.

/// <summary>
/// Implements a fake timer registry to facilitate unit tests using the test cluster.
/// </summary>
public class FakeTimerRegistry : ITimerRegistry
{
    /// <summary>
    /// We dont have a ConcurrentHashSet yet so this does the job.
    /// </summary>
    private readonly ConcurrentDictionary<FakeTimerEntry, FakeTimerEntry> timers = new ConcurrentDictionary<FakeTimerEntry, FakeTimerEntry>();

    /// <summary>
    /// Registers a new fake timer entry and returns it.
    /// Note how we are capturing the activation task scheduler to ensure we can tick the fake timers within the activation context.
    /// </summary>
    public IDisposable RegisterTimer(Grain grain, Func<object, Task> asyncCallback, object state, TimeSpan dueTime, TimeSpan period)
    {
        var timer = new FakeTimerEntry(this, TaskScheduler.Current, grain, asyncCallback, state, dueTime, period);
        timers[timer] = timer;
        return timer;
    }

    /// <summary>
    /// Returns all fake timer entries.
    /// </summary>
    public IEnumerable<FakeTimerEntry> GetAll() => timers.Keys.ToList();

    /// <summary>
    /// Removes a timer.
    /// </summary>
    public void Remove(FakeTimerEntry entry) => timers.TryRemove(entry, out _);
}

Final Notes

To see everything running, go check out the Orleans Unit Testing Sample.

Any questions, let me know in the comments, or just pop over at the Orleans gitter channel.

Health Checking Microsoft Orleans

Mon, 10 Jun 2019 00:00:00 +0000

This article describes how to add ASP.NET Core Health Checks to Microsoft Orleans for distributed application health checking.

TLDR

Download the Orleans Health Checks Sample PR here
Start the Silo project
Open http://localhost:8880/health in the browser, or issue a GET with a tool such as Fiddler
Check for a response of Healthy, Degraded or Unhealthy
Look at the console log update every 30 seconds

Note: I’ll update the PR link to a sample link once the Orleans team has some time to review it.

Overview

The ASP.NET Core Health Checks pattern provides a standard way of reporting on the health of application components. Memory, storage, database and other sub-systems can be monitored for healthy status. This article describes how to set up the health check framework components and how to create custom health checks for Orleans-based sub-systems.

The Kestrel Web Server

To allow checking a silo’s health status in the first place, we need an endpoint to connect to. An easy way to achieve this is to start a Kestrel web server in the silo host process.

For simplicity and isolation, we can do this inside its own hosted service, which we then add to the main generic host.

The code below creates a web server that listens on the given port and serves health check requests at the given relative path.

public class HealthCheckHostedService : IHostedService
{
    private readonly IWebHost host;

    public HealthCheckHostedService(IClusterClient client, IMembershipOracle oracle, IOptions<HealthCheckHostedServiceOptions> myOptions)
    {
        host = new WebHostBuilder()
            .UseKestrel(options => options.ListenAnyIP(myOptions.Value.Port))
            .Configure(app =>
                {
                    app.UseHealthChecks(myOptions.Value.PathString);
                })
            /* ... */
            .Build();
    }

    public Task StartAsync(CancellationToken cancellationToken) => host.StartAsync(cancellationToken);
    public Task StopAsync(CancellationToken cancellationToken) => host.StopAsync(cancellationToken);
}

public class HealthCheckHostedServiceOptions
{
    public string PathString { get; set; } = "/health";
    public int Port { get; set; } = 8880;
}

We can then add this hosted service to the main generic host.

public static Task Main()
{
    return new HostBuilder()
        .ConfigureServices(services =>
        {
            services.AddHealthChecks();
            services.AddHostedService<HealthCheckHostedService>()
                .Configure<HealthCheckHostedServiceOptions>(options =>
                {
                    options.Port = healthCheckPort;
                    options.PathString = "/health";
                });
            /* ... */
        })
        /* ... */
        .RunConsoleAsync();
}

You can test the above by starting the silo and opening http://localhost:8880/health in a browser or a tool such as Fiddler.

This already fulfills a very basic check - that we can ping the kestrel server itself.

Under normal operations, the request will return Http Status Code 200 with one of the following strings as content:

Healthy
Degraded
Unhealthy

It can also return Http Status Code 500 (Internal Server Error) in case there is an error running the set of health checks.

Any unreasonable delay in the health check response requires treating as Degraded or Unhealthy by the monitoring tool in use. The default timeout is 30 seconds.

With this working, we can now add custom Orleans-based health checks.

The Health Checks

We create a custom health check class by inheriting from IHealthCheck and adding it to the service provider.

The code below adds four such classes to test different aspects of Orleans. We will go through each class in detail.

services.AddHealthChecks()
    .AddCheck<GrainHealthCheck>("GrainHealth")
    .AddCheck<SiloHealthCheck>("SiloHealth")
    .AddCheck<StorageHealthCheck>("StorageHealth")
    .AddCheck<ClusterHealthCheck>("ClusterHealth");

GrainHealthCheck

The GrainHealthCheck class verifies connectivity to a LocalHealthCheckGrain activation. As this grain is a [Stateless Worker], validation always occurs in the silo where the health check is issued.

public class GrainHealthCheck : IHealthCheck
{
    private readonly IClusterClient client;

    public GrainHealthCheck(IClusterClient client)
    {
        this.client = client;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await client.GetGrain<ILocalHealthCheckGrain>(Guid.Empty).PingAsync();
        }
        catch (Exception error)
        {
            return HealthCheckResult.Unhealthy("Failed to ping the local health check grain.", error);
        }
        return HealthCheckResult.Healthy();
    }
}

[StatelessWorker(1)]
public class LocalHealthCheckGrain : Grain, ILocalHealthCheckGrain
{
    public Task PingAsync() => Task.CompletedTask;
}

SiloHealthCheck

The SiloHealthCheck verifies if health-checkable Orleans services are healthy.

public class SiloHealthCheck : IHealthCheck
{
    private readonly IEnumerable<IHealthCheckParticipant> participants;

    private static long lastCheckTime = DateTime.UtcNow.ToBinary();

    public SiloHealthCheck(IEnumerable<IHealthCheckParticipant> participants)
    {
        this.participants = participants;
    }

    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var thisLastCheckTime = DateTime.FromBinary(Interlocked.Exchange(ref lastCheckTime, DateTime.UtcNow.ToBinary()));

        foreach (var participant in this.participants)
        {
            if (!participant.CheckHealth(thisLastCheckTime))
            {
                return Task.FromResult(HealthCheckResult.Degraded());
            }
        }

        return Task.FromResult(HealthCheckResult.Healthy());
    }
}

Such services implement the IHealthCheckParticipant interface.

For dependency service providers that do not handle discovering services by an arbitrary interface, we must collect these services ourselves.

At the time of writing this, only IMembershipOracle exists as a public implementation.

We can add these services to the service provider with this code:

public HealthCheckHostedService(IClusterClient client, IMembershipOracle oracle, IOptions<HealthCheckHostedServiceOptions> myOptions)
{
    host = new WebHostBuilder()
        /* ... */
        .ConfigureServices(services =>
        {
            services.AddSingleton(Enumerable.AsEnumerable(new IHealthCheckParticipant[] { oracle }));
        })
        .Build();
}

StorageHealthCheck

The StorageHealthCheck class verifies whether the StorageHealthCheckGrain can write, read, and clear state using the default storage provider.

This grain:

Is marked with PreferLocalPlacement;
Deactivates itself after each call;
Is called with a random key each time;

This ensures this test always happens in the silo under test.

public class StorageHealthCheck : IHealthCheck
{
    private readonly IClusterClient client;

    public StorageHealthCheck(IClusterClient client)
    {
        this.client = client;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        try
        {
            await client.GetGrain<IStorageHealthCheckGrain>(Guid.NewGuid()).CheckAsync();
        }
        catch (Exception error)
        {
            return HealthCheckResult.Unhealthy("Failed to ping the storage health check grain.", error);
        }
        return HealthCheckResult.Healthy();
    }
}

[PreferLocalPlacement]
public class StorageHealthCheckGrain : Grain, IStorageHealthCheckGrain
{
    private readonly IPersistentState<Guid> state;

    public StorageHealthCheckGrain([PersistentState("State")] IPersistentState<Guid> state)
    {
        this.state = state;
    }

    public async Task CheckAsync()
    {
        try
        {
            state.State = Guid.NewGuid();
            await state.WriteStateAsync();
            await state.ReadStateAsync();
            await state.ClearStateAsync();
        }
        finally
        {
            DeactivateOnIdle();
        }
    }
}

ClusterHealthCheck

The ClusterHealthCheck verifies whether any silos are unavailable by querying the ManagementGrain.

public class ClusterHealthCheck : IHealthCheck
{
    private readonly IClusterClient client;

    public ClusterHealthCheck(IClusterClient client)
    {
        this.client = client;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var manager = client.GetGrain<IManagementGrain>(0);
        try
        {
            var hosts = await manager.GetHosts();
            var count = hosts.Values.Where(x => x.IsUnavailable()).Count();
            return count > 0 ? HealthCheckResult.Degraded($"{count} silo(s) unavailable") : HealthCheckResult.Healthy();
        }
        catch (Exception error)
        {
            return HealthCheckResult.Unhealthy("Failed to get cluster status", error);
        }
    }
}

Health Check Publishers

The examples above are enough to support a pull health check model. This is where an external monitoring service or orchestrator polls all the silos for their health status on a preset schedule.

However, the framework also supports a push model. This allows each silo to publish their own health information to an external service, also on a preset schedule.

To use the push model, we create a class that inherits from IHealthCheckPublisher.

The sample LoggingHealthCheckPublisher class below publishes the summarized health report to the logging output.

public class LoggingHealthCheckPublisher : IHealthCheckPublisher
{
    private readonly ILogger<LoggingHealthCheckPublisher> logger;

    public LoggingHealthCheckPublisher(ILogger<LoggingHealthCheckPublisher> logger)
    {
        this.logger = logger;
    }

    public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
    {
        var id = Guid.NewGuid();
        var now = DateTime.UtcNow;

        logger.Log(report.Status == HealthStatus.Healthy ? LogLevel.Information : LogLevel.Warning,
            "Service is {@ReportStatus} at {@ReportTime} after {@ElapsedTime}ms with CorrelationId {@CorrelationId}",
            report.Status, now, report.TotalDuration.TotalMilliseconds, id);

        foreach (var entry in report.Entries)
        {
            logger.Log(entry.Value.Status == HealthStatus.Healthy ? LogLevel.Information : LogLevel.Warning,
                entry.Value.Exception,
                "{@HealthCheckName} is {@ReportStatus} after {@ElapsedTime}ms with CorrelationId {@CorrelationId}",
                entry.Key, entry.Value.Status, entry.Value.Duration.TotalMilliseconds, id);
        }

        return Task.CompletedTask;
    }
}

We then add this class to the service provider.

public HealthCheckHostedService(IClusterClient client, IMembershipOracle oracle, IOptions<HealthCheckHostedServiceOptions> myOptions)
{
    host = new WebHostBuilder()
        /* ... */
        .ConfigureServices(services =>
        {
            /* ... */
            services.AddSingleton<IHealthCheckPublisher, LoggingHealthCheckPublisher>()
                .Configure<HealthCheckPublisherOptions>(options =>
                {
                    options.Period = TimeSpan.FromSeconds(1);
                });
        })
        .Build();
}

We can configure reporting startup delay and frequency via the HealthCheckPublisherOptions.

However note that due to this issue, the value set for Period has no effect at the time of writing, and the default of 30 seconds will always apply. This other issue will fix this wobbly for .NET Core 3.

.Configure<HealthCheckPublisherOptions>(options =>
{
    options.Period = TimeSpan.FromSeconds(1);
});

Final Notes

The Orleans Health Check sample is now awaiting PR review from the core team. Once that’s done, you’ll find it in the official sample folder along with all the others. I’ll update this post at that time.

Latency-Adaptive Real-Time with Reactive Caching on Microsoft Orleans

Sun, 26 May 2019 00:00:00 +0000

The Reactive Caching pattern allows many service clients to stay in sync with the latest data snapshot from an origin, regardless of their relative network latency. This pattern exploits Reactive Polling, enables Real-Time CQRS Projection Streaming and opens the doors to opportunistic Reactive Replication.

This article talks about my team’s experience applying this approach with Microsoft Orleans to solve a multi-geography user latency challenge.

TLDR;

Overview

Reactive Caching is useful when the cost of streaming individual items to machines in disparate geographical locations causes these machines to go out of sync due to network latency. Reactive Caching allows machines in different locations to stay in sync with the same data snapshot, using a best-effort approach and as fast as their network allows.

When implemented in Orleans, everything happens within application memory, often uses the least number of networks hops necessary, and does not require external distributed caching services.

Background

In the industry I work in, relative network latency is a source of constant problems.

Here is an example. Two or more users are on a conference call, on far-between geographical locations, looking at the same data screens on their desktops. Due to their locations, each of their client’s network latency relative to a common data source can vary by a significant number.

graph LR; C[Service Cluster] --> |">100ms"| SC[Slow Client]; C --> |"~50ms"| AV[Average Client]; C --> |"<0ms"| FS[Fast Client];

Their screens are alive with real-time activity from various streaming data sources. Look at the header image. That’s a stock image but they’re kinda like that. This activity happens in bursts, with many events attempting to reach each machine in quick succession to feed their active real-time views.

Although these users are talking on the phone and looking at equivalent screens on their respective desktops, they will have a hard time seeing the same consistent data during periods of high streaming activity, when tens of thousands of individual events are attempting to reach each client machine per second. While the fastest client may indeed update in real-time, the slowest client can take several minutes to reach a consistent state.

It goes without saying, this can make our users very grumpy.

What’s Going On

While users near the source of data will often have latencies in the microsecond range, users in far-away geographical locations often have latencies over 100ms, even with expensive network links.

Classical streaming approaches to push individual data items to client applications will suffer from this when data volumes become significant. We haven’t yet commoditized quantum entanglement and there’s only so much you can send at the same time. Sending batches only helps up to a point. And geographically splitting your HPC cluster makes Alfred come with a can of worms on a silver platter, asking if we’d like some cookies and milk to help down them.

This is where the Reactive Caching pattern can give a hand.

Why Reactive Caching

Reactive Caching can replace streaming approaches when:

The client does not require the full event flow.
The client just wants to always get the latest snapshot of some data view as fast as their own network allows.
The client prefers skipping interim snapshots when the network cannot cope.

We happened to already use a CQRS-style projection solution to create data snapshots.

graph LR; ES1[Event Source 1] --> C1[Conflation 1] C1 --> P1[Always-On Projection 1] C1 --> P2[Always-On Projection 2] ES2[Event Source 2] --> C2[Conflation 2] C2 --> P2 C2 --> P3[Always-On Projection 3] P2 --> OP1(On-Demand Projection 1) P2 --> OP2(On-Demand Projection 2) P2 --> OP3(On-Demand Projection 3) P1 --> OP1 P1 --> OP2 P3 --> OP2 P3 --> OP3

Okay, so the above may look like a spider web, but don’t mind all the arrows.

In-Memory CQRS is super simple to implement in Orleans and it is what we use to compute and maintain real-time data aggregations plus point-in-time snapshots of such aggregations. This approach lets us turn millions of individual data items into many small aggregations of a few thousand items each tops.

That said, you don’t need CQRS to apply Reactive Caching. You need just need to have some snapshot data that you want to propagate.

Our underlying CQRS architecture meant that we already had access to latest data snapshots on-demand and in real-time. And we were already using these for plain request-response queries. All we needed was to solve the user latency problem.

What We Did

By applying the reactive caching approach, we moved from:

Streaming all data items to all clients, regardless of whom can handle them.

To:

Pushing the latest aggregated data snapshot each screen needs, at the point in time each client can handle it.

graph RL; C1[Fast Client] --> P[Projection] C2[Slow Client] --> P[Projection]

In this approach, both fast and slow clients will hang from whatever projections they are interested in. They do so by sending a message to the projection.

Upon this request, each projection decides whether to send the latest data immediately to an individual client or to delay response until some new data is available. This is very similar to long-polling in HTTP, but without the connection cost drawbacks. The MSR paper calls this reactive polling to highlight the difference.

In theory, while the slow client will skip snapshots as their own latency allows, the fast client may even flicker from too many snapshots coming through.

graph RL; C1[Fast Client] --> |Snapshot1, Snapshot2, Snapshot3| P[Projection] C2[Slow Client] --> |Snapshot1, Snapshot3| P[Projection]

In practice, the cluster will only expose at most one snapshot per projection per second, as faster updates made our users cry out for a freeze button in the user interface. That’s what happens when your real-time system is too real-time for its own good.

The neat thing about this pattern, is that there is nothing the slower or faster clients have to do, to regulate what exact snapshots they get. The immediate network latency itself acts a natural regulator of what snapshots will reach them.

How Reactive Caching Works

Here’s some C#-ish client-side pseudo-code, where the target service returns a null snapshot on graceful timeout.

while (keepGoing)
{
    string token = null;
    try
    {
        // request the latest snapshot - the first request provides a null token
        var snapshot = await service.GetNewSnapshotAsync(token);
        if (snapshot != null)
        {
            // keep the token for the next request
            token = snapshot.Token;

            // update own cache, fill a data grid, update a chart, etc
            await ApplySnapshotAsync(snapshot.Data);
        }
    }
    catch (Exception error)
    {
        logger.LogError(error, "Request failed!");
    }
}

The above is oversimplified. It goes without saying, don’t use while(true) style loops, they lead to the dark side of the Force, Luke.

The sample shows how to do this the right way, using non-reentrant timers.

Note that the above is a client concept. Reactive Caching is a pattern, not a technology. In our setup we have implemented this pattern both within Orleans and outside of it, having a mix of both .NET Windows-based and Angular-based browser clients. Orleans just happens to be an excelent fit for this.

In fact, we have expanded the base reactive caching pattern to something we’re calling Reactive Replication - the on-demand, temporaray replication of dynamic in-memory CQRS data projections across the Orleans cluster, using a well-defined hierarchy of short-lived beacon grains to minimize both data copying and the effect of in-cluster latency, while maximizing client responsiveness. This is to tie with Orleans geo-distribution. But that’s a story for another day.

Here is the server-side C#-ish pseudo-code for Orleans:

[Reentrant]
public class MyProjectionGrain: Grain, IMyProjectionGrain
{
    // this happens server-side
    public Task ApplyDataAsync(Data data)
    {
        // make use of the incoming data to the projection and create a new snapshot
        var newSnapshot = new Snapshot(
            data: /* do something smart here */,
            token: /* create a new token */
        );

        // now make the new snapshot available
        snapshot = newSnapshot;
        completion.TrySetResult(snapshot);
        completion = new TaskCompletionSource<Snapshot>();
    }

    // we use this to fulfil requests immediately
    private Snapshot snapshot = Snapshot.Empty;

    // we use this to delay fulfilment
    private TaskCompletionSource<Snapshot> completion = new TaskCompletionSnapshot<Snapshot>();

    // this is called by clients
    public Task<Snapshot> GetNewSnapshotAsync(token)
    {
        // see if the client already has the same version as the server
        if (token == snapshot.Token)
        {
            // the client already has the latest snapshot
            // leave the request pending until we have new data to provide or we reach a graceful timeout
            // on graceful timeout we just return null
            return completion.Task.WithDefaultOnTimeout(null, TimeSpan.FromSeconds(10));
        }
        else
        {
            // the client has a different version so fulfil this request now
            // note how there is no data processing going on here - it is already done
            return Task.FromResult(snapshot);
        }
    }
}

Again, this is pseudo-code, and the sample shows how to do this the right way.

Let’s see how this works step-by-step:

The client makes an initial request for a given data snapshot.
The server returns the latest data snapshot, plus a token representing that snapshot version. This token can be anything, as long as it is unique per version of the snapshot. Guid.NewGuid() works well, as do forever increasing integers.
The client handles the snapshot (e.g. updates its own cache, or paints a data grid, or updates a chart).
After this (or even during), the client asks for a snapshot again, now passing the token it received.
The server now looks at the token and decides whether the client already has the latest snapshot version or whether it needs another update.
- If the tokens do not match, the server responds immediately with the latest snapshot, along with the new token.
- If the tokens do match, the server will delay the response until a new snapshot is available or a graceful timeout is reached. As soon as a new snapshot version is created, the server fulfills that pending request.
Whenever the request resolves, the client makes use of that snapshot and then requests a new snapshot again, providing the new token. The loop repeats until the client stops it.

This client-side loop of requesting-waiting-applying is what causes natural back-pressure in the system and lets both server and client adapt to individual network latency.

And yes, delaying the response is similar to the ages-old long-polling in HTTP. However, unlike HTTP, Orleans does not incur the cost of establishing a connection per request, and instead simulates the long-poll via messaging between grains or cluster clients.

What Reactive Caching Solves

Individual data streaming, reactive observables, or any other kind of push-only approach without the ability to conflate data based on the back-pressure of an individual consumer, cannot cater for significant differences in network latency on the consumer side.

Pushing all individual data items to each consumer results in the slowest consumers struggling to keep up. If the amount of data you’re sending to a slow consumer is chronically more than it can handle, the consumer can enter a perpetual state of delay in handling that data.

Exploiting high network bandwith by pushing batches of items can help up to a point and is often enough on the server-side to scale-out streaming solutions. You may have no choice in this, if the consumer does require all individual items without exception.

However, when the consumer only cares about the latest dataset right now, then reactive caching allows the consumer to skip interim snapshots as latency forces it to, yet still adapting in real-time as the network quality changes.

In addition, some other problems just disappear with Reactive Caching. In a typical streaming scenario, you have to deal with subscriptions in some form. This forces you to think about:

To whom do I have to send the data to?
Are they alive?
Have heartbeats failed?
When must clients re-sub?
When must the server un-sub them?
Who’s getting too much data?
Who needs a reset to cope with too much data?

We tend to delegate these concerns to an external queue provider. That works fine, at the expense of a network double-hop to an external and often shared system, incurring the cost of its own internal latency, and of course, facing the main challenge with streaming detailed above - data volume on the client-side.

With Reactive Caching, we allow clients to connect directly to the server-side, without intermediates. We don’t manage pub-sub models because there is nothing to manage. That’s because each single request is the subscription, is the heartbeat and is the back-pressure. If a client does not receive a response to every single request within the agreed graceful timeout, then it knows immediately something is wrong.

Why Orleans

While the client-side implementation of this pattern is straightforward in any technology, the server-side implementation can be quite hard to acomplish without the help of an in-memory stateful system.

For the server-side to work, a number of things must be in the goldilocks zone:

MyProjectionGrain instances (or whatever equivalent) must be thread-safe (or you must take care of that yourself).
Instances must be long-lived - requests will keep hanging off them by design.
Instances must know how to recover themselves and establish initial state.
Instances must stay in-memory to minimize latency.
You must be able to scale while maintaining responsiveness. This means
- You must be able to create many instances across your cluster.
- For each request, you must discover what node holds what, so the request can hang off the correct instance.
- You must be able to create new instances on-demand as new requests for different projections come in.
- You must be able to dispose of these instances when no longer required.
- You must be able to replicate these instances as-needed (and not more than that) to cater for hot projections. Replicate too little and a node can get overwhelmed. Replicate too much and face memory issues.

The above is something stateless server-side technology by itself - such as ASP.NET - cannot provide. Distributed caching services can help to a point but become very expensive on the wallet early on.

Yet the above is something Orleans provides out-of-the-box, by virtue of being a virtual actor system. Implementing the server-side in Orleans means implementing one or more classes such as the pseudo-code above and letting the system do its thing. We also do not need expensive distributed caching providers - or caching at all for that matter. As an Actor System, Orleans is cache already. Cache we can program in plain C#.

To help you understand how simple this whole thing is, I’ve pushed a bare-bones sample to the Orleans repository. Just clone the whole thing, and follow the instructions in the readme file.

In our setup, we use both ASP.NET and Orleans in the same cluster, each one doing what they do best. Orleans handles the stateful in-memory projection stuff while ASP.NET handles the client-friendly protocols we provide to consumers.

I plan to update the sample, or create a new one, featuring an ASP.NET front-end and an example of the reactive replication pattern we’re ironing out. But that’s a story for another post.

Lessons Learned

Not all was sunshine and rainbows. We faced some challenges making this stuff work the first time. Here are some things to watch out for, if you’re going down this route.

#1: Beware Long-polling via HTTP

Long-polling on Orleans is very cheap. Long-polling on HTTP/1.1 is very expensive. HTTP/2 multiplexing requires goldilocks conditions to run, so it wasn’t even on our table. SignalR and WebSockets are there to hold your hand.

Why is this?

Orleans does not long-poll. Instead it simulates long-polling behaviour through message sending and receiving. An Orleans node or client does not need to establish a new connection on every single request. Most often, the connections are already established in the first place, and Orleans multiplexes messages back and forth through these connections. The cheapness of this underlying infrastructure is what allows long-polling to turn into reactive caching and so on.

HTTP/1.1 does long-poll. And it ain’t pretty. For every request, the browser must establish a connection and may decide to perform a DNS lookup beforehand. This makes the request lose valuable time even before the server knows about it.

In addition, browsers will impose a connection limit per domain. For Chrome this is six connections per domain. Each active long-poll counts towards this limit. If your application is issuing a lot of long-polls at the same time, then you will start seeing things not updating when they should. That’s not something you want your users to notice.

For .NET client applications, the default connection limit is two, but you can override it with the ServicePointManager. However, our load testing showed that abusing this will lead to random socket exhaustion. The HttpClientFactory mitigates this somewhat but one cannot rely on other teams to use specific tech to access standard REST APIs.

Why do it then?

Offering a long-poll-capable RESTful interface makes it easy for consumer-side developers to test reactive caching behaviour early on using the Swagger UI, as a human being just clicking things and seeing results. It works great for proof-of-concept-ing new functionality, before comitting to it. It’s also great for troubleshooting the server-side, while the debugger is attached.

What are the alternatives?

Once you’re serious, implement high-level SignalR or low-level WebSockets, or some other multiplexing-capable communication technology. Persistent connections are one of the basic building blocks of real-time applications.

#2: Reentrancy Will Catch You Off-Guard

The Orleans-side bits of this pattern require the use of reentrant grains. If the grains were not reentrant, a single reactive poll request would hang the grain until resolved.

However, reentrancy is a double-edge sword:

On one hand, a reentrant grain can handle multiple requests at the same time, while staying single-threaded. The grain just swaps requests whenever it encounters an await somewhere or it returns an unfulfilled task back to Orleans. This behaviour allows thousands of requests to “hang off” a single grain, waiting for a future response.

On the other hand, reentrant grains are prone to bugs from the unweary developer. I’ve lost count at the amount of bugs I created due to misattention. Once you know what to look out for in reentrant code, you’re good to go, but prepare to attach that debugger many times until you gain such foresight.

#3: Monitor Your Memory And Throughput (Even More!)

At their core, Orleans grains are concurrent programmable cache units. Running an Orleans cluster is a bit like running a distributed cache cluster - but one that you can program, and shard, and integrate with all sorts of other systems, without having your data leaving memory unless you say so. Unlike a distributed cache cluster, Orleans is cheap on the network and cheap on the wallet. Like a distributed cache cluster, memory is still your number one resource.

Our CQRS-style projection model enables the on-demand creation of many, small projections of data that update in real-time on the server-side. We implement these projections as grains in Orleans. Each projection instance can satisfy requests for a particular shape and shard of data. It holds only enough data to paint the screen area it represents, e.g. a grid or a chart. Many clients can hang off a single projection. A single client can also hang off multiple projections.

This can lead to a very high number of these projection grains spawning on the cluster. While Orleans can handle millions of grains across a cluster, and each projection is small, things do pile up, both in memory and throughput.

We have therefore found it important to develop a load test that focuses on hammering the model with both:

Requests that require new projections all the time - to monitor memory growth.
Requests that attempt to saturate a single projection - to monitor throughput.

It was the feedback gathered by this load test that lead us into the reactive replication pattern we are evaluating now.

Final Thoughts

Bringing Reactive Caching and CQRS together within Orleans is enabling us to create ground-breaking solutions in this industry for fractions of complexity and cost. This does not come without risk of failure and lots of head-scratching moments. Willingness to try new stuff is something one must convince paying customers of. It’s their money on the line. Disbelief is the main challenge to overcome when doing R&D.

When your application shows real-time data as you type filters to folk who are used to very slow, cumbersome systems, their initial reaction tends to be ah I see, you’re caching stuff, when does it refresh?.

Well, yes, we’re caching… An also no, we’re not caching. There is no cache refresh time or expiry date or consistency schedule or whatnot. All that stuff does not apply anymore. The data itself is cache. We are real-time caching, if that makes any sense.

Over-engineering Cyclic Array Rotation In C#

Sat, 23 Feb 2019 00:00:00 +0000

Rotating an array in an efficient way is a classical programming exercise.

It’s also one that’s ripe for over-engineering for funsies.

This article describes five approaches for rotating an array and ranks their performance against each other.

The Problem

We have some array of integers of length N, from zero to infinity and beyond.

We must rotate that array some K number of times.

If K is 3 then every element in the array will move three positions to the right, except the last three elements, which will move to the beginning.

For example, if we start with this array…

[0] [1] [2] [3] [4] [5]

…and rotate it by 3, we must end up with…

[3] [4] [5] [0] [1] [2]

Constraints

No in-place rotation - return a new array.
N can be of arbitrary size or even zero.
K can be of arbitrary size or zero but not negative - only rotate right.

A Naive Solution

There isn’t really one for this exercise. Even brute-force in-place rotation does not make sense as we need to create a new array anyway.

An Efficient Solution

An obvious solution to this exercise is to move all items to the new array by:

Calculating old index + distance
Getting its remainder over the array length to account for overflow.

public static int[] RotateByRemainderIndexing(int[] input, int distance)
{
    // validate
    if (input == null) throw new ArgumentNullException(nameof(input));
    if (distance < 0) throw new ArgumentOutOfRangeException(nameof(distance));
    if (input.Length == 0) return new int[0];

    // rotate
    var result = new int[input.Length];
    for (int i = 0; i < input.Length; ++i)
    {
        int j = (i + distance) % input.Length;
        result[j] = input[i];
    }
    return result;
}

This is already fit for purpose.

It’s a simple O(n) approach, easy to understand and scales in a linear fashion.

That said, we are still iterating the array and performing arithmetic on every single step. Yet when you think about it, all we are doing is swapping two array segments, nothing more.

Is there a way to avoid this extra leg work and go straight to copying memory?

Well, it turns out there are a number of ways to do just that.

A More Efficient Solution? Array.Copy

As a more efficient approach, we can divide an array into two segments at distance % input.Length and then copy over those segments to the new array.

public static int[] RotateByArrayCopy(int[] input, int distance)
{
    // validate
    if (input == null) throw new ArgumentNullException(nameof(input));
    if (distance < 0) throw new ArgumentOutOfRangeException(nameof(distance));
    if (input.Length == 0) return new int[0];

    // rotate
    var result = new int[input.Length];
    int diff = distance % input.Length;
    Array.Copy(input, 0, result, diff, input.Length - diff);
    Array.Copy(input, input.Length - diff, result, 0, diff);
    return result;
}

This solution does away with iterations in favour of Array.Copy to copy the underlying array data in one go.

Array.Copy is general-use and works for both value and reference types, performing boxing, unboxing and casting as required.

A More Efficient Solution? Buffer.BlockCopy

Another way of copying underlying memory is with Buffer.BlockCopy.

public static int[] RotateByBufferCopy(int[] input, int distance)
{
    // validate
    if (input == null) throw new ArgumentNullException(nameof(input));
    if (distance < 0) throw new ArgumentOutOfRangeException(nameof(distance));
    if (input.Length == 0) return new int[0];

    // rotate
    var size = sizeof(int);
    var result = new int[input.Length];
    int diff = distance % input.Length;
    Buffer.BlockCopy(input, 0, result, diff * size, (input.Length - diff) * size);
    Buffer.BlockCopy(input, (input.Length - diff) * size, result, 0, diff * size);
    return result;
}

Buffer.BlockCopy is a bit more low-level than Array.Copy and will copy the underlying bytes without regards to the type they represent.

Because of that, we must include the type size in the slice calculations.

There is also a Buffer.MemoryCopy method that takes memory pointers but as it is not CLS-compliant, I have not included it in this comparison.

A More Efficient Solution? Span.Slice.CopyTo

Yet another way to slice and copy an array is to use a Span<T>.

public static int[] RotateBySpanCopy(int[] input, int distance)
{
    // validate
    if (input == null) throw new ArgumentNullException(nameof(input));
    if (distance < 0) throw new ArgumentOutOfRangeException(nameof(distance));
    if (input.Length == 0) return new int[0];

    // rotate
    var result = new int[input.Length];
    var target = new Span<int>(result);
    var diff = distance % input.Length;
    var source1 = new Span<int>(input, 0, input.Length - diff);
    source1.CopyTo(target.Slice(diff, input.Length - diff));
    var source2 = new Span<int>(input, input.Length - diff, diff);
    source2.CopyTo(target.Slice(0, diff));

    return result;
}

Span<T> lets us work with contiguous segments of memory and enables neat stuff like in-place casting without boxing and slicing without allocation.

This algorithm uses that exact slicing ability to copy the underlying memory from the two original segments to the two target segments.

Span<T> is aware of the size it is supposed to represent (that’s what the <T> is for) and therefore we do not need to include the type size in the slice calculations.

A More Efficient Solution? Unsafe.CopyBlock

The final contender in this over-engineering race is the new(ish) Unsafe.CopyBlock from the System.Runtime.CompilerServices (aka “shush, I know what I’m doing”) assembly.

public unsafe static int[] RotateByUnsafeCopy(int[] input, int distance)
{
    // validate
    if (input == null) throw new ArgumentNullException(nameof(input));
    if (distance < 0) throw new ArgumentOutOfRangeException(nameof(distance));
    if (input.Length == 0) return new int[0];

    // prepare to rotate
    var result = new int[input.Length];
    int size = Unsafe.SizeOf<int>();
    int diff = distance % input.Length;

    // pin memory locations
    fixed (int* target1 = result.AsSpan(diff, input.Length - diff))
    fixed (int* slice1 = input.AsSpan(0, input.Length - diff))
    fixed (int* target2 = result.AsSpan(0, diff))
    fixed (int* slice2 = input.AsSpan(input.Length - diff, diff))
    {
        // copy the underlying memory in the array
        Unsafe.CopyBlock(target1, slice1, (uint)((input.Length - diff) * size));
        Unsafe.CopyBlock(target2, slice2, (uint)(diff * size));
    }

    return result;
}

Unsafe.CopyBlock is as brute-force as it gets.

It will copy memory from one place to the other without regard to the runtime shufling memory as it goes. That’s why we need to pin any memory we need to touch, lest the runtime swipes it off our algorithm’s feet.

To use it, we need to mark the method as unsafe and build the assembly with /unsafe turned on, just to prove how desperate we are.

Performance

Here is how these algorithms stack against each other…

Method	N	Mean	Error	StdDev	Ratio	RatioSD	Rank	Gen 0/1k Op	Gen 1/1k Op	Gen 2/1k Op	Allocated Memory/Op
RotateByRemainderIndexing	10	55.32 ns	1.2551 ns	1.6320 ns	1.00	0.00	**	0.0203	-	-	64 B
RotateByArrayCopy	10	54.73 ns	1.4390 ns	1.9698 ns	0.99	0.05	**	0.0203	-	-	64 B
RotateByBufferCopy	10	46.79 ns	0.9520 ns	0.8905 ns	0.84	0.03	***	0.0203	-	-	64 B
RotateBySpanCopy	10	44.12 ns	0.7237 ns	0.6416 ns	0.80	0.02	**	0.0203	-	-	64 B
RotateByUnsafeCopy	10	40.55 ns	0.5346 ns	0.5001 ns	0.73	0.02	*	0.0203	-	-	64 B

RotateByRemainderIndexing	100	490.21 ns	9.9104 ns	16.2831 ns	1.00	0.00	**	0.1345	-	-	424 B
RotateByArrayCopy	100	116.81 ns	2.5840 ns	3.0760 ns	0.24	0.01	***	0.1347	-	-	424 B
RotateByBufferCopy	100	97.67 ns	1.9820 ns	2.2824 ns	0.20	0.01	**	0.1347	-	-	424 B
RotateBySpanCopy	100	94.22 ns	1.9825 ns	2.2830 ns	0.19	0.01	*	0.1347	-	-	424 B
RotateByUnsafeCopy	100	92.16 ns	1.9450 ns	2.5965 ns	0.19	0.01	*	0.1347	-	-	424 B

RotateByRemainderIndexing	1000	4,674.12 ns	81.3798 ns	72.1410 ns	1.00	0.00	**	1.2741	-	-	4024 B
RotateByArrayCopy	1000	613.15 ns	10.8852 ns	9.6494 ns	0.13	0.00	*	1.2779	-	-	4024 B
RotateByBufferCopy	1000	604.01 ns	11.9966 ns	18.3201 ns	0.13	0.01	*	1.2779	-	-	4024 B
RotateBySpanCopy	1000	625.79 ns	12.2431 ns	10.8532 ns	0.13	0.00	*	1.2779	-	-	4024 B
RotateByUnsafeCopy	1000	599.51 ns	14.3827 ns	13.4535 ns	0.13	0.00	*	1.2779	-	-	4024 B

RotateByRemainderIndexing	10000	47,102.64 ns	1,061.3413 ns	1,993.4563 ns	1.00	0.00	**	12.6343	-	-	40024 B
RotateByArrayCopy	10000	6,383.94 ns	47.8780 ns	42.4426 ns	0.14	0.01	*	12.6572	-	-	40024 B
RotateByBufferCopy	10000	6,470.84 ns	127.7785 ns	119.5241 ns	0.14	0.01	*	12.6572	-	-	40024 B
RotateBySpanCopy	10000	6,677.21 ns	128.7819 ns	162.8678 ns	0.14	0.01	*	12.6572	-	-	40024 B
RotateByUnsafeCopy	10000	6,547.83 ns	126.1428 ns	172.6656 ns	0.14	0.01	*	12.6572	-	-	40024 B

RotateByRemainderIndexing	100000	461,920.31 ns	6,744.3065 ns	6,308.6284 ns	1.00	0.00	***	124.5117	124.5117	124.5117	400024 B
RotateByArrayCopy	100000	73,566.10 ns	1,464.5060 ns	1,686.5273 ns	0.16	0.00	**	124.8779	124.8779	124.8779	400024 B
RotateByBufferCopy	100000	73,716.92 ns	1,468.5753 ns	2,286.3948 ns	0.16	0.01	**	124.8779	124.8779	124.8779	400024 B
RotateBySpanCopy	100000	71,036.89 ns	802.9858 ns	626.9185 ns	0.15	0.00	*	124.8779	124.8779	124.8779	400024 B
RotateByUnsafeCopy	100000	73,232.43 ns	1,416.7108 ns	1,325.1922 ns	0.16	0.00	**	124.8779	124.8779	124.8779	400024 B

RotateByRemainderIndexing	1000000	5,715,133.83 ns	52,297.2313 ns	48,918.8626 ns	1.00	0.00	**	273.4375	273.4375	273.4375	4000024 B
RotateByArrayCopy	1000000	3,471,175.86 ns	40,279.0739 ns	35,706.3499 ns	0.61	0.01	*	152.3438	152.3438	152.3438	4000024 B
RotateByBufferCopy	1000000	3,484,901.70 ns	75,821.6356 ns	77,863.2374 ns	0.61	0.01	*	152.3438	152.3438	152.3438	4000024 B
RotateBySpanCopy	1000000	3,395,618.79 ns	67,417.6361 ns	184,554.6909 ns	0.57	0.07	*	152.3438	152.3438	152.3438	4000024 B
RotateByUnsafeCopy	1000000	3,492,599.63 ns	52,870.8936 ns	46,868.6701 ns	0.61	0.01	*	152.3438	152.3438	152.3438	4000024 B

RotateByRemainderIndexing	10000000	74,122,529.83 ns	1,659,361.8996 ns	2,037,845.4327 ns	1.00	0.00	**	111.1111	111.1111	111.1111	40000024 B
RotateByArrayCopy	10000000	36,266,718.27 ns	386,807.3179 ns	361,819.8052 ns	0.49	0.01	***	133.3333	133.3333	133.3333	40000024 B
RotateByBufferCopy	10000000	33,913,202.44 ns	349,838.7446 ns	327,239.3788 ns	0.46	0.01	*	125.0000	125.0000	125.0000	40000024 B
RotateBySpanCopy	10000000	35,197,233.69 ns	678,249.4610 ns	807,407.7360 ns	0.48	0.02	**	133.3333	133.3333	133.3333	40000024 B
RotateByUnsafeCopy	10000000	36,101,336.10 ns	719,881.3796 ns	739,265.1758 ns	0.49	0.02	***	133.3333	133.3333	133.3333	40000024 B

RotateByRemainderIndexing	100000000	650,980,738.30 ns	7,266,152.6154 ns	6,796,763.6647 ns	1.00	0.00	***	-	-	-	400000024 B
RotateByArrayCopy	100000000	334,371,656.80 ns	6,517,933.2712 ns	11,243,114.6469 ns	0.52	0.01	**	-	-	-	400000024 B
RotateByBufferCopy	100000000	325,036,779.43 ns	7,624,095.8569 ns	8,157,697.1495 ns	0.50	0.01	*	-	-	-	400000024 B
RotateBySpanCopy	100000000	318,656,163.20 ns	2,968,558.6617 ns	2,776,791.6140 ns	0.49	0.00	*	-	-	-	400000024 B
RotateByUnsafeCopy	100000000	319,322,045.17 ns	2,789,657.9564 ns	2,609,447.7833 ns	0.49	0.00	*	-	-	-	400000024 B

Here are some interesting bits from this benchmark:

Just about anything is better than remainder indexing.
The mean time of the memory copy methods is an order of magnitude lower than remainder indexing up 100k array size.
Said performance takes a hit at around 1M array size. The exact point may be specific to my own computer spec.
Copy mean time appears to stabilize at around half of remainder indexing. Again, the exact point may be specific to my own computer spec.
The garbage collector goes to sleep when we cross-over the 100M array size mark. That may be because we’re only allocating memory in the large object heap at this point, which goes straight into GC Generation 2.
None of the memory copy methods stand-out on the long run - they all behave the same.

In this case, it does not make sense to create a hybrid algorithm… But it does make sense to use one of the copy methods - which one wins, I’ll leave it up to you.

Takeaway

Sometimes one can benefit from over-engineering… Sometimes not… Sometimes both!

Neither of the fancy memory copy methods was significantly more performance than trusty old Array.Copy. Yet even Array.Copy by itself provided significant performance benefits over a naive remainder indexing approach. Why index what you can clone?

Resources

You can find all the code for this post in the Quicker repository, including all the benchmarks.

How To Disable Turbo Boost In Laptops

Fri, 22 Feb 2019 00:00:00 +0000

Intel Turbo-Boost and AMD Turbo CORE may be nice for gaming but they’re a nuisance for performance benchmarking.

While there is often a BIOS setting in desktop machines to turn this off, walled garden laptops like mine make it impossible to disable it via the user interface or even their own tuning software.

The good news is, you can use the power configuration manager utility in Windows to force it off outright.

Powercfg -setacvalueindex scheme_current sub_processor PERFBOOSTMODE 0
Powercfg -setactive scheme_current

This disables turbo-boost for the current power plan.

If you’re on Windows 10 and have the fancy power slider, you can now bring your power mode to best performance and your processor will keep working at the maximum standard clock rate, without boosting.

You can find more information about this on the Power and Performance Tuning page from the Performance Tuning Guidelines for Windows Server documentation.

Finding The Odd One Out In C#

Thu, 14 Feb 2019 00:00:00 +0000

Solutions to the finding the odd one out problem are of interest when one must find missing matches in some dataset, for example when performing data quality analysis for missing event pairs.

This article describes and benchmarks three approaches for a simple variation of this problem and then a hybrid approach for optimal performance.

The Problem

We have a variable-sized array of integer values, in the example below of nine items.

Values: 2 5 3 5 5 3 5 2 3
-------------------------
 Index: 0 1 2 3 4 5 6 7 8

We want to find the odd one out in the array.

These are the rules:

We can pair each item in the array against another item of equal value.
We can only pair each item once.

These are the constraints:

There can be any number of odd ones out in the array - items that don’t match with any other item.
There can also be no odd ones out, meaning all values have a corresponding pair.
The array can be of arbitrary length or even empty.
Each item can be of arbitrary value from minimum integer to maximum integer.

Given this setup, we want an algorithm that returns the value of the odd one out in the array, with the following behaviour:

If a single odd one out exists, return that value.
If multiple odd ones out exist, return any single one of those values.
If no odd one out exists, return null.

In the example above:

2 occurs twice, forming one pair, with no odd one out.
5 occurs four times, forming two pairs, with no odd one out.
3 occurs three times, forming one pair, with one odd one out.
There is one odd one out with value 3, so the algorithm returns 3.

A Simple Solution

One simple algorithm can be to iterate through all individual items and then count how many items of that value exist in the array.

public static int? FindOddOneOutByIterating(int[] values)
{
    // keep track of items already tested
    var tested = new HashSet<int>();

    // check each item in the array
    for (int i = 0; i < values.Length; ++i)
    {
        // check if we have tested this value already
        if (tested.Contains(values[i]))
        {
            continue;
        }

        // count the number of times this value shows in the array
        int count = 0;
        for (int j = 0; j < values.Length; ++j)
        {
            if (values[i] == values[j])
            {
                ++count;
            }
        }

        // check if it appears an odd number of times
        if (count % 2 != 0)
        {
            return values[i];
        }

        // mark this value as tested
        tested.Add(values[i]);
    }

    // we did not find an unpaired item
    return null;
}

The above algorithm is self-explanatory. It brute-forces the problem by testing every single value in the array for an odd number of occurences until it finds one. To avoid testing the same value twice, it makes use of a HashSet<T> to keep track of values already tested. Time complexity varies from a best-case scenario O(n) when the odd one out is on the first position, to a worst-case scenario O((n*n)/2) when the odd one out is on the last position.

This works, but can we do better?

A Better Solution

We can attempt to improve upon the brute-force aspect of the previous solution by sorting the data first, and then counting the occurrences of each item in sequential fashion.

public static int? FindOddOneOutByOrdering(int[] values)
{
    // we need at least one item for this algorithm to make sense
    int count = 0;
    int? last_value = null;

    // copy and sort the array
    var sorted = new int[values.Length];
    Array.Copy(values, sorted, values.Length);
    Array.Sort(sorted);

    // look for odd occurrences of a value until we find one
    for (int i = 0; i < sorted.Length; ++i)
    {
        if (sorted[i] == last_value)
        {
            ++count;
        }
        else
        {
            if (count % 2 != 0)
            {
                return sorted[i];
            }
            else
            {
                count = 1;
                last_value = sorted[i];
            }
        }
    }

    // extra check for the last value
    return (count % 2 != 0 ? last_value : null);
}

The above algorithm makes use of the Array.Sort() method to sort a copy of the input array first. Then it tests each value in the sorted sequence until it finds a value with an odd number of occurrences.

Array.Sort() selects an appropriate algorithm based on length between Insertion Sort, Heapsort and Quicksort, with an overall worst-case time complexity of O(n*log(n)). We then pay a further O(n) in a worst-case to find an odd one left out.

This looks a bit better than the worst-case scenario of the first approach, but can we do even better?

An Efficient Solution

We can achieve a more efficient algorithm by keeping track of value occurrences while making a single pass over the array. In fact, we do not even need to know how many occurrences of each value there are, unlike what the brute-force algorithm would make us believe. We only need to know whether there is an current odd one left out for each value that we track.

public static int? FindOddOneOutByHashing(int[] values)
{
    // prepare a lookup for on-the-go checks
    var candidates = new HashSet<int>();

    // check each item in the enumeration
    for (int i = 0; i < values.Length; ++i)
    {
        // check if we have found this value once before
        if (candidates.Contains(values[i]))
        {
            // if yes then it is no longer a candidate for odd occurrences
            candidates.Remove(values[i]);
        }
        else
        {
            // if not then it is either the first time it appears in the enumeration
            // or all previous appearances have found a pair
            // in that case we add it to the lookup as a new candidate for odd occurences
            candidates.Add(values[i]);
        }
    }

    // check if any candidate survived
    if (candidates.Count > 0)
    {
        return candidates.First();
    }
    else
    {
        return null;
    }
}

The above algorithm makes use of a HashSet<T> collection as a cheap state-machine for keeping track of whether a given value has been matched or not. Contains() and Remove() are O(1) operations, while Add() approaches a O(1) operation, given a decent GetHashCode() implementation to minimize hash bucket sizes.

The HashSet<T>’s approaching O(1) time complexity combined with the single-pass iteration over the array makes this algorithm approach O(n), at the expense of allocating the HashSet<T> at startup.

At the end there is an extra O(1) operation to fetch any identified odd one out.

Performance

Performance comparison for the three approaches isn’t as straightforward as one would expect.

The simple algorithm varies in performance depending on what position the odd item out is in the array. Its best-case scenario may be a nice O(n) but its worst-case scenario is a not so nice O((n*n)/2).
The ordering algorithm brings complexity down to a more manageable O(n*log(n) + n) worst-case scenario, with some up-front cost to sort the array.
The hashing algorithm brings complexity down to almost O(n), albeit at a price of creating and maintaining a HashSet<T> over the single pass. But, then again, so does the brute force algorithm pay for it.

Let’s see how performance compares on a best-case scenario and a worst-case scenario.

You can also run these benchmarks yourself from the Quicker repository.

Best Case Scenario

In a best-case scenario, the odd element is at the start of the input array, which is ideal for the brute scenario.

Each test array contains a pattern similiar to:

[0] [1] [2] [3] [4] [0] [1] [2] [3] [4] [0]

Here are some numbers, lower is better.

Method	N	Mean	Ratio	Rank
FindOddOneOutByIterating	1	19.73 ns	1.00	*
FindOddOneOutByOrdering	1	35.33 ns	1.79	**
FindOddOneOutByHashing	1	102.13 ns	5.15	***

FindOddOneOutByIterating	11	26.97 ns	1.00	*
FindOddOneOutByOrdering	11	118.27 ns	4.32	**
FindOddOneOutByHashing	11	449.65 ns	16.47	***

FindOddOneOutByIterating	101	100.84 ns	1.00	*
FindOddOneOutByOrdering	101	1,066.39 ns	10.63	**
FindOddOneOutByHashing	101	3,402.77 ns	33.71	***

FindOddOneOutByIterating	1001	656.41 ns	1.00	*
FindOddOneOutByOrdering	1001	22,699.50 ns	34.62	**
FindOddOneOutByHashing	1001	31,599.56 ns	48.05	***

FindOddOneOutByIterating	10001	6,183.70 ns	1.00	*
FindOddOneOutByOrdering	10001	546,227.42 ns	88.33	***
FindOddOneOutByHashing	10001	349,001.85 ns	56.44	**

FindOddOneOutByIterating	100001	61,403.81 ns	1.00	*
FindOddOneOutByOrdering	100001	7,183,987.40 ns	117.00	***
FindOddOneOutByHashing	100001	3,155,303.61 ns	51.40	**

For a best-case scenario:

The brute-force algorithm shows a clear advantage in all cases where the first item is the odd one out. As it iterates only once through the array in that case, it outpaces the setup time required for sorting and the maintenance time required for hash pairing.
The ordering algorithm does start out better than hashing, but then it loses pace as the array size becomes significant.
The hashing algorithm shows the greatest initial cost, but its performance benchmark remains stable as we add more items.

Worst Case Scenario

However, the tables turn in a dramatic fashion for a worst-case scenario, where the odd one out is at the end of the array.

Each test array contains a pattern similiar to:

[0] [1] [2] [3] [4] [0] [1] [2] [3] [4] [5]

Here are some numbers, lower is better.

Method	N	Mean	Ratio	Rank
FindOddOneOutByIterating	1	19.86 ns	1.00	*
FindOddOneOutByOrdering	1	34.18 ns	1.72	**
FindOddOneOutByHashing	1	91.43 ns	4.62	***

FindOddOneOutByIterating	11	328.65 ns	1.00	**
FindOddOneOutByOrdering	11	136.84 ns	0.42	*
FindOddOneOutByHashing	11	429.78 ns	1.31	***

FindOddOneOutByIterating	101	7,284.18 ns	1.00	***
FindOddOneOutByOrdering	101	1,323.07 ns	0.18	*
FindOddOneOutByHashing	101	3,368.68 ns	0.46	**

FindOddOneOutByIterating	1001	348,152.80 ns	1.00	***
FindOddOneOutByOrdering	1001	22,426.86 ns	0.06	*
FindOddOneOutByHashing	1001	31,421.15 ns	0.09	**

FindOddOneOutByIterating	10001	31,160,440.83 ns	1.00	***
FindOddOneOutByOrdering	10001	476,150.85 ns	0.02	**
FindOddOneOutByHashing	10001	347,095.77 ns	0.01	*

FindOddOneOutByIterating	100001	3,072,503,593.03 ns	1.000	***
FindOddOneOutByOrdering	100001	7,446,879.34 ns	0.002	**
FindOddOneOutByHashing	100001	3,091,703.46 ns	0.001	*

The charts now make clear the difference in scalability between the three algorithms.

While brute-forcing works for just above 10 items, it becomes unusable after that.
The ordering algorithm holds its ground well, event if struggling a bit as the array grows. It also shows lower initial cost for small arrays than the hashing alternative.
However, when array length becomes significant, the hashing algorithm leaves all contenders behind with no apologies. It runs in the same time regardless of where the odd one out is and it scales in a linear fashion. For a 100K-long array, the hashing algorithm takes less than 0.03% of the time of brute-forcing and less than half the time of ordering.

The Winner

… is not what you expect.

The most interesting bit in these tests is not the scalability of the hashing algorithm. It’s the performance cross-overs between 10 and 100 items and then between 1K and 10K items. Not one of these algorithms is better than all the others ones all the time - each one has their particular performance domain.

This realization opens the door for a hybrid algorithm, a pseudo-algorithm that elects the better suited approach depending on the incoming number of items. Similar to the sort technique selection in Array.Sort(), this can provide the best performance for any count of input elements.

Here is one way to implement this:

public static int? FindOddOneOutHybrid(int[] values, int orderingThreshold = 10, int hashingThreshold = 10000)
{
    if (values.Length > hashingThreshold) return FindOddOneOutByHashing(values);
    if (values.Length > orderingThreshold) return FindOddOneOutByOrdering(values);
    return FindOddOneOutByIterating(values);
}

The default thresholds are taken from earlier testing and we can always optimize them with finer-grained benchmarking.

Here are the results of a new benchmark run with default thresholds. There is a slight variance in values as I have run this on a laptop, but the overall performance ranking holds.

Method	N	Mean	Ratio	Rank
FindOddOneOutByIterating	1	20.65 ns	1.00	**
FindOddOneOutByOrdering	1	35.98 ns	1.81	***
FindOddOneOutByHashing	1	90.05 ns	4.38	**
FindOddOneOutHybrid	1	19.92 ns	0.97	*

FindOddOneOutByIterating	11	345.30 ns	1.00	**
FindOddOneOutByOrdering	11	137.49 ns	0.39	*
FindOddOneOutByHashing	11	461.86 ns	1.37	***
FindOddOneOutHybrid	11	136.58 ns	0.39	*

FindOddOneOutByIterating	101	7,509.89 ns	1.00	**
FindOddOneOutByOrdering	101	1,350.84 ns	0.18	*
FindOddOneOutByHashing	101	3,491.06 ns	0.46	***
FindOddOneOutHybrid	101	1,378.60 ns	0.18	**

FindOddOneOutByIterating	1001	351,577.94 ns	1.00	***
FindOddOneOutByOrdering	1001	23,636.44 ns	0.07	*
FindOddOneOutByHashing	1001	32,036.17 ns	0.09	**
FindOddOneOutHybrid	1001	23,039.26 ns	0.07	*

FindOddOneOutByIterating	10001	31,723,898.86 ns	1.00	***
FindOddOneOutByOrdering	10001	481,376.16 ns	0.02	**
FindOddOneOutByHashing	10001	348,620.23 ns	0.01	*
FindOddOneOutHybrid	10001	353,378.17 ns	0.01	*

FindOddOneOutByIterating	100001	3,106,550,714.05 ns	1.000	**
FindOddOneOutByOrdering	100001	7,459,762.64 ns	0.002	***
FindOddOneOutByHashing	100001	3,290,452.46 ns	0.001	**
FindOddOneOutHybrid	100001	3,138,249.50 ns	0.001	*

Hint: You can click on the series names in the chart to hide and show individual lines.

The charts speak for themselves. The true winner of this contest is not an individual algorithm - but the use of the most appropriate algorithm for the given situation.

Takeaway

This article described a simple variation of the finding the odd one out problem and compared three possible solutions for it.

As the benchmarks show, it can pay off by orders of magnitude to invest in some setup time at the beginning of an algorithm.

It also pays off to create a hybrid algorithm - one that elects the most appropriate approach for the situation at hand.

Resources

You can find all the code for this post in the Quicker repository, including all the benchmarks.

Finding Binary Gaps In C#

Wed, 13 Feb 2019 00:00:00 +0000

Variations of the binary gap problem are a classic at coding challenge sites nowadays.

This easy-to-solve, not-so-obvious-to-master problem can highlight how a programmer is aware of binary manipulation and how it affects performance.

This article describes two solutions for a variation of this problem and how their performance stacks against each other.

The Problem

Consider some integer, for example 14512.

Its binary representation is 11100010110000, with the most significant bit first.

A binary gap is a sequence of consecutive bits of the same value (ones or zeros) that are surrounded by bits of opposite values. For example, a zero binary gap is a sequence of unset bits (zeroes) that are surrounded by set bits (ones).

The number 14512, or 11100010110000 in binary, has two zero binary gaps, one of length 3 and one of length 1.

11100010110000
   ^^^ ^

The right-most sequence of 4 unset bits does not count, as it is not surrounded by set bits on both sides.

A maximum binary gap length is thus the length of the longest binary gap within the number’s binary representation.

This concept may look simple at first, but variations of it become interesting in the domain of compression algorithms that look for long chunks of similar data.

It is also a fine example of how not all O(N)-ish algorithms are born the same.

The precise way one writes code to achieve something can have a dramatic impact on how fast that code achieves that end.

Let’s see an example of that by attempting to find the length of the longest binary gap in any given number.

A Simple Approach

One simple approach can be to convert the integer into its binary representation as a string, and then iterate the characters one-by-one.

Here is a C# algorithm that performs just that.

public static int BinaryGapMaxLengthByIterating(int value, BinaryGapBit bit = BinaryGapBit.Unset)
{
    // trackers
    int max_gap = 0;
    int gap = 0;
    char mask = bit == BinaryGapBit.Unset ? '1' : '0';

    // convert the value to a binary string
    string binary = Convert.ToString(value, 2);

    // iterate until the first set bit
    int i = 0;
    while (i < binary.Length && binary[i] != mask)
    {
        ++i;
    }

    // we are either at the first set bit or have run out of bits
    for (; i < binary.Length; ++i)
    {
        // check if the current bit is set
        if (binary[i] == mask)
        {
            // check if the current gap is greater than the max candidate
            if (gap > max_gap)
            {
                // promote the current gap count to max candidate
                max_gap = gap;
            }

            // close the running gap count
            gap = 0;
        }
        else
        {
            // if zero then increase the running gap count
            ++gap;
        }
    }

    // at this point we have the greatest gap length or zero
    return max_gap;
}

public enum BinaryGapBit
{
    /// <summary>
    /// Look for gaps where the bits are unset.
    /// </summary>
    Unset = 0,

    /// <summary>
    /// Look for gaps where the bits are set.
    /// </summary>
    Set = 1
}

The algorithm above allows finding gaps of either zeroes or ones via the BinaryGapBit parameter, without affecting the iteration itself.

This is how it works for a zero-based gap:

Convert the integer number to its binary representation as a string.
Iterate through characters until the first '1'.
For each character in the string…
- Is it '1'? If so…
  - If the current gap length is longer than the maximum gap length, then promote it to the maximum gap length.
  - Reset the current gap length.
- Is it '0'? If so, increment the current gap length.
Return the maximum gap length.

It is a simple to understand algorithm, with apparent time-complexity O(N), if you consider an arbitrary number of bits to cover.

However, that apparent O(N) is hit by a curve ball from Convert.ToString(value, 2);, of which you can see the code here.

Looking at that source code, it becomes difficult to discern the actual time-complexity of this solution without empirical testing over a larger dataset than a single integer.

This is in addition to the string allocation by string binary = ... on the algorithm itself.

If one is writing performance-sensitive code, this can be one allocation too many.

An Efficient Approach

One efficient approach comes from realizing that one does not need to convert anything to binary at all.

Integer numbers are already binary values to start with, and many languages allow for some form of direct binary manipulation.

The algorithm below exploits this by creating a binary mask to check wheather a given bit is set or not, and then shifting it to simulate an iteration through all the bits.

public static int BinaryGapMaxLengthByShifting(this int value, BinaryGapBit bit = BinaryGapBit.Unset)
{
    // trackers
    int max_gap = 0;
    int gap = 0;
    int mask = 1;

    // if searching for gaps of ones just flip the bits in the search space
    if (bit == BinaryGapBit.Set)
    {
        value = ~value;
    }

    // shift until the first set bit
    while ((mask & value) == 0 && mask != 0)
    {
        mask <<= 1;
    }

    // shift one more time to skip it
    // this avoid a duplicate comparison on the next step
    mask <<= 1;

    // we are either at the first set bit or have run out of bits
    for (; mask != 0; mask <<= 1)
    {
        // check if the current bit is set
        if ((mask & value) == mask)
        {
            // check if the current gap is greater than the max candidate
            if (gap > max_gap)
            {
                // promote the current gap count to max candidate
                max_gap = gap;
            }

            // close the running gap count
            gap = 0;
        }
        else
        {
            // if zero then increase the running gap count
            ++gap;
        }
    }

    // at this point we have the greatest gap length or zero
    return max_gap;
}

While the algorithm above is a bit harder to read that the previous, its behaviour still follows the same train of thought.

Step 1 - Find First Set Bit

Create a mask of value 1 - this translates to binary 00000000000001 (plus other 18 other zeroes to left for a 32-bit integer).
While the mask is not zero…
- Apply a binary & (AND) operation between the mask and the value to see if the bit at the mask position is set.
- If yes, exit the loop.
- If not, shift the mask to the left and continue the loop.

The first iteration does not find a set bit in the last position because the & operation between both returns zero.

Value: 11100010110000
 Mask: 00000000000001
 --------------------
  AND: 00000000000000
                    ^

We therefore shift the mask one bit to the left with mask <<= 1 and try again…

Value: 11100010110000
 Mask: 00000000000010
 --------------------
  AND: 00000000000000
                   ^

Still no luck, so we shift and test a few more times. This time, the & operation returns a value that is equal to mask itself. This is our tell that we have found the first set bit.

Value: 11100010110000
 Mask: 00000000010000
 --------------------
  AND: 00000000010000
                ^--

We thus stop fast-scanning for a starting point and start counting gaps. We also shift the mask one bit further as there is no need to test the current bit again.

Value: 11100010110000
 Mask: 00000000100000
               ^

Step 2 - Count Gaps

This is also similiar to the simple approach but now we use binary shifting.

While the mask is not zero…
Test the mask against the value with the & (AND) operator.
- If the bit is set…
  - If the current gap length is longer than the maximum gap length, promote it to the maximum gap length.
  - Reset the current gap length.
- If the bit is not set, increment the current gap length.
Return the maximum gap length.

This is what it looks like…

The first test finds a set bit at the mask position. We therefore close the current gap count, which is still zero at this point.

Value: 11100010110000
 Mask: 00000000100000
 --------------------
  AND: 00000000100000
               ^

The next test returns zero. This means we can starting counting the current gap length.

Value: 11100010110000
 Mask: 00000001000000
 --------------------
  AND: 00000000000000
              ^

The next test returns the mask value. This means we found a set bit, so we take the current gap count of one, and promote it the maximum.

Value: 11100010110000
 Mask: 00000010000000
 --------------------
  AND: 00000010000000
             ^

We rinse and repeat a few more times, which will put the maximum gap length at 3 - the length of the gap between the fourth and sixth bit in the representation below.

Value: 11100010110000
 Mask: 10000000000000
 --------------------
  AND: 10000000000000
       ^-----

As we shift the mask to the left one last time, its set bit disappears, and the mask value itself becomes zero. This is our tell to stop the algortihm and return the current maximum length of 3.

Value: 11100010110000
 Mask: 00000000000000

Performance

What look like very similar algorithms can show dramatic performance differences.

This is a benchmark of the sample bit-shifting approach against the string-iteration approach as a baseline:

Method	Mean	Error	StdDev	Ratio
BinaryGapMaxLengthByIterating	658.9 ns	1.1043 ns	1.0330 ns	1.00
BinaryGapMaxLengthByShifting	144.2 ns	0.8616 ns	0.8060 ns	0.22

You can run the benchmark on your own machine from the Quicker repository.

The bit-shifting algorithm, by saving on the allocation, runs in less than a quarter of the time, or over four times faster… even for a single integer.

Takeaway

It pays off to obsess about performance, even in simple cases like this.

Through the compounding effect, a better algorithm can make a significant difference to our systems, our users and our business, if we just keep looking for things to improve.

And that is what problems such as this attempt to highlight.

Increasing Message Throughput In Akka.NET Actor Systems

Thu, 21 Jun 2018 00:00:00 +0000

When you are building a real-time Actor System in Akka.NET, message throughput numbers will be one of your top engineering priorities. One wouldn’t be using a performance scalpel such as Akka.NET if speed wasn’t a main business concern. This article details four approaches you can use right now to improve the message processing and delivery throughput of your actor system.

Routing

One of the main points of using Akka.NET is its ability to do some good old massively parallel processing for you at a unit of work level, far lower than at the process level of many other architectures.

Doing stuff in a serial manner? Break it down to the core and route it out.

Routers allow you to distribute units of work across routees, in addition to a myriad of other use cases. Routers can send messages in-process, remotely or across a cluster. Some routers can even deploy new children in new nodes you add to a live cluster.

Router Families

There are two main families of routers in Akka.Net:

Pool Routers
Group Routers

This is how they compare:

A Pool Router…	A Group Router…
Creates and uses its own routees as children	Routes to existing routees in the system or cluster
Supervises its own routees	Cannot supervise its routees
Can manage routee recovery	Has no control on routee recovery
Can only handle a single underlying props type at a time	Can route to any actor regardless of underlying props type
Can scale out its routee pool automatically	Has no control over scaling

Router Strategies

A routing strategy decides which routee a router forwards a message to. Akka.NET ships with several in-built routers, covering different routing strategies. Most strategies are available for both Pool or Group router families, with some exceptions.

Round Robin

This strategy forwards messages to each target actor in a sequential fashion.

This is a good default strategy for most cases. Use this to distribute units of work that take equal time to complete.

Broadcast

This strategy forwards the same message to all target actors.

Use this when you need to notify all actors of some global event.

Random

This strategy forwards messages in random fashion.

You can use this when:

Distributing work with unknown completion time.
Using a sequence of Round Robin routers is pushing many messages to a single routee.

However it is best to consider other options before resorting to random.

Consistent Hashing

This strategy forwards messages based on some key to the same target routee.

Use this to ensure ordered processing of messages for a given key.

Notes:

The key is derived from some data in the message. You decide what the key is.
The hashing and distribution algorithm is based on the key and the total number of routees. Therefore, if the pool/group size changes, the key to routee mapping will also change.
A single target actor may receive more than one message type, should the distribution algorithm hash the key to the same value.
Despite all the above, message delivery order is maintained.

Tail Chopping

This strategy keeps forwarding a given message to a random routee, until some routee sends a reply back. The reply is then handed to the original message sender. The strategy allows some time for a routee to reply before attempting the next one.

Use this when you want to brute-force a fast reply to the sender and have the resources to spare.

Notes:

Any replies after the first one are ignored.
To reduce too many wasted resources, have the routees implement some timeout of their own.

Scatter Gather First Completed

This strategy forwards a message to all routees - like broadcast - and forwards the first reply back to the original sender.

Use this when Tail Chopping isn’t brute-force enough for you.

Notes:

Any replies after the first one are ignored.

Smallest Mailbox

This strategy forwards each message to the routee with the smallest mailbox.

Use this when attempting to balance equal load across different capacity hardware. This strategy only works will pool routers, though it works across cluster nodes.

Prioritization is as follows:

Routee with empty mailbox and not processing a message right now.
Routee with empty mailbox regardless of processing a message right now.
Routee with lowest inbox message count.
Routee in a remote actor system.

Automatic Resizing

Pool Routers can scale out and back based on load. This enables the actor system to adapt its responsiveness to tolerate under heavy load without letting a flood of requests overwhelm it outright.

Example:

var resizingRouter =
    system.ActorOf(
        Props.Create()
            .WithRouter(
                new RoundRobinPool(10)
                .WithResizer(
                    new DefaultResizer(
                        10, /* lower bound */
                        100 /* upper bound */
                    )
                )
            )
    );

Stashing

Stashing allows an actor to put away the present message for processing at a later time.

Use it to prioritize certain messages over others, on a particular criteria, or to delay processing on external systems.

Stashing can help with:

Improve system responsiveness by prioritizing high-importance or fast messages over low-importance or slow ones.
Improve system resilience by holding on to messages that require work on an external system, while the external system is down or recovering from a crash. Extra useful when combined with a Backoff Supervision Strategy.
Optimize capacity use of external systems by limiting throughput of certain messages during certain times of day.

Stashing works in two steps:

Stash the current message: This puts a message on a temporary store - a list of messages, ordered in the same way they came in.
Unstash one or all messages: This puts all the stashed messages in front of the queue, in the same order they came in.

Notes:

Unstashed messages keep their original delivery order, regardless of unstashing one by one or all at the same time.
Unstashed messages are inserted at the head of the inbox and will therefore take precedence over any normal messages already queued.

Example:

using System;
using System.Threading.Tasks;
using Akka;
using Akka.Actor;

public class SomeStashingActor : ReceiveActor, IWithUnboundedStash
{
    public IStash Stash { get; set; }

    public SomeStashingActor()
    {
        // request a wake up every second
        var today = DateTime.Today;
        Context.System.Scheduler.ScheduleTellRepeatedly(
            TimeSpan.FromSeconds(1),
            TimeSpan.FromSeconds(1),
            Self,
            ProcessHeavyMessagesNow.Instance,
            Self);

        // process light messages at any time
        Receive(m =>
        {
            // do a little work here
            Task.Delay(100).Wait();
        });

        // stash or process heavy messages
        Receive(m =>
        {
            // check if we can we process a heavy message right now
            if (InPeakTime())
            {
                // stash the heavy message for future processing
                Stash.Stash();
            }
            else
            {
                // do lots of work here
                Task.Delay(1000).Wait();
            }
        });

        // unstash all heavy messages for processing
        Receive(m =>
        {
            // check if we can unstash now
            if (!InPeakTime())
            {
                // good to go now
                Stash.UnstashAll();
            }
        });
    }

    private bool InPeakTime()
    {
        var now = DateTime.Now.TimeOfDay;
        return now >= _peakTimeStart && now < _peakTimeEnd;
    }

    private readonly TimeSpan _peakTimeStart = new TimeSpan(6, 0, 0);
    private readonly TimeSpan _peakTimeEnd = new TimeSpan(21, 0, 0);

    private class ProcessHeavyMessagesNow
    {
        private ProcessHeavyMessagesNow() { }

        public static readonly ProcessHeavyMessagesNow Instance = new ProcessHeavyMessagesNow();
    }
}

public class MessageThatRequiresLittleWork { }

public class MessageThatRequiresLotsOfWork { }

Blocking

Blocking an actor on long running operations can and will degrade throughput.

Long running operations can happen when dealing with I/O, external system calls and even calling and waiting for replies of other actors.

Blocking an actor thread on operations such as these will limit message throughput by:

Making the queued messages wait to start processing.
Making the blocked thread unavailable to process other work.

There are a couple of Akka-friendly ways of handling this…

ReceiveAsync

Use ReceiveAsync coupled with an await/async handler to free up the thread while processing.

Pros:

Guarantees message completion order.
Frees up the thread during the async/await context switch.
Enables use of a router to limit parallel calls to an external system.

Cons:

Requires the use of a router to scale out at all.
Queued messages will still wait for prior ones to complete before even starting.
Pays an overhead cost for the async/await context switch.

Example:

using System.Threading.Tasks;
using Akka.Actor;

public class SomeAsyncActor : ReceiveActor
{
    public SomeAsyncActor()
    {
        ReceiveAsync(async m =>
        {
            // perform some work on the external system
            await Task.Delay(1000);

            // ack the sender
            Sender.Tell(WorkComplete.Instance);
        });
    }
}

public class MessageThatRequiresExternalSystem
{
}

public class WorkComplete
{
    private WorkComplete() { }
    public static readonly WorkComplete Instance = new WorkComplete();
}

PipeTo

Use PipeTo() to pipe the result of some background work to either the current actor or straight to a target actor.

Pros:

Frees up the thread without paying for async/await overhead.
Scales out without a pool.
Queued messages do not wait for earlier completion to start processing.
Can send the result straight to another actor in the background.

Cons:

Can scale out too much and overwhelm an external system.
Does not guarantees message completion order - that depends on the external system doing the work.

Notes:

PipeTo() works with any TPL Task;
Therefore, one can PipeTo() the result of Ask();

Example:

[code language=”csharp”] using System; using System.Threading.Tasks; using Akka.Actor;

public class SomePipingActor : ReceiveActor { public SomePipingActor() { Receive(m => { // capture the original sender for piping var sender = Sender;

        // perform some work on the external system
        // and send the result to self
        // this does not block the thread
        Task.Delay(1000)
            .ContinueWith(t => new ExternalSystemResult())
            .PipeTo(
                // target - can be self or some other actor
                Self,

                // sender for the message
                // using the original sender here to make things easier
                sender,

                // optional - factory for a success envelope
                // we can use it if we want to add additional information
                wr => new NotifyExternalWorkCompleted(wr),

                // optional - factory for an error envelope
                cause => new NotifyExternalWorkFailed(cause)
            );
    });

    // handle notification that the external system work completed
    Receive(m =>
    {
        Sender.Tell(WorkComplete.Instance);
    });

    // handle notification that the external system work failed
    Receive(m =>
    {
        Sender.Tell(WorkFailed.Instance);
    });
}

private class NotifyExternalWorkCompleted
{
    public NotifyExternalWorkCompleted(ExternalSystemResult result)
    {
        Result = result;
    }

    public ExternalSystemResult Result { get; }
}

private class NotifyExternalWorkFailed
{
    public NotifyExternalWorkFailed(Exception cause)
    {
        Cause = cause;
    }

    public Exception Cause { get; }
} }

public class MessageThatRequiresExternalSystem { }

public class WorkComplete { private WorkComplete() { } public static readonly WorkComplete Instance = new WorkComplete(); }

public class WorkFailed { private WorkFailed() { } public static readonly WorkFailed Instance = new WorkFailed(); }

public class ExternalSystemResult { public Guid SomeResult { get; } = Guid.NewGuid(); } [/code]

Serialization

Pushing messages through the network makes you pay three-fold:

The sender must pay CPU time to serialize a message.
The network must pay transfer time to transfer all the bytes in the message to the receiver.
The receiver must pay CPU time to deserialize the message.

All the above add up. You can therefore improve message throughput by using message serializers that reduce the metrics above.

The most common serializers in Akka.NET are:

Newtonsoft.Json: The default Akka.NET serializer for user messages as of writing.
Hyperion: Scheduled to become the default serializer for user messages from Akka.NET v1.5+.
Google.Protobuf: Akka.NET uses this for optimized internal messages but not for user messages.

This is how they compare:

Properties	JSON	Hyperion	Protobuf
Human-Readable	Yes	No	No
Version-Tolerant	Limited	No	Yes
Platform-independent	Yes	No	Yes
Payload Size	Baseline	Smaller	Smallest
Supports POCOs	Yes	Yes	No
Supports Complex Object Graphs	Limited	Yes	Yes
Requires Specialized Development	No	No	Yes

There are others as well, which are similar to one of the archetypes above, though varying in unique development features and performance.

Less Bytes More Game

When it comes to maximizing message throughput, human-readability is an afterthought - with the notable exception of your message destination being a scale-out JSON document store to begin with.

After your storage systems and external systems, your slowest resource is often the network. The less bytes on the pipe, the faster a message can arrive at its destination. If the destination is a persistent storage system, the less bytes there are there to persist, the faster they can get persisted, of course depending on system specifics.

This, among other reasons, lies behind the plan to switch to Hyperion for user messages

So this becomes a no-brainer:

Use Hyperion by default for messages where performance is not critical or for fast solution prototyping.
Use Google Protocol Buffers for message transfer or persistence where performance or version tolerance is of critical importance.

As Google Protocol buffers uses its own syntax and compiler, it will require additional build setup and development time than using POCOs with Hyperion. However the investment you make in time, you will get back in increased performance numbers.

Switching To Hyperion In Visual Studio / .NET Framework

Run this on the NuGet Package Manager Console:

Install-Package Akka.Serialization.Hyperion -pre

Hyperion is formally in pre-release status at the time of writing, hence the -pre flag.

Now apply this to your application’s hocon files across all the nodes in the cluster:

akka {
  actor {
    serializers {
      hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
    }
    serialization-bindings {
      "System.Object" = hyperion
    }
  }
}

Switching to Hyperion In Visual Studio Code / .NET Core

Run his on the terminal:

dotnet add package Akka.Serialization.Hyperion

Hyperion is in pre-release at the time of writing. If it is still in pre-release as you are reading this, then the command above will fail. That’s fine. Take note of the nearest package version that the error states and run the next command, replacing the version below with the version you see in the terminal:

dotnet add package Akka.Serialization.Hyperion -v 1.1.3.31-*

Now apply this to your application’s hocon files across all the nodes in the cluster:

akka {
  actor {
    serializers {
      hyperion = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
    }
    serialization-bindings {
      "System.Object" = hyperion
    }
  }
}

Using Google Protocol Buffers

This requires a tutorial of its own, and is out of the scope of this article.

You can start with the tutorial for C# to understand how Protocol Buffers work.

Thanks

As you can see, Akka.NET has several tools to aid you in maximizing message throughput in your actor system. The sky is the limit. Or your network. Whichever bottlenecks first.

Version Tolerant Binary Serialization With Microsoft Bond In C#

Tue, 27 Mar 2018 00:00:00 +0000

You’ve been there too. A month ago you wrapped up your first release. You were happy and went for a beer with your workmates. The next month things didn’t go so well. As you start integration testing again, you realize something has gone wrong. You dig down, and discover your app had serialized some useful data to disk not so long ago. Yet, as the new version does its thing, it crashes with a vengeance, when it tries to get that data back from the same place. As you sigh, and realize there won’t be any beer that night, you notice that those serialized bits of data have changed schema between deployments. You think - I do want the app to re-use that data without breaking… So how on Earth do I get around this?

The answer to this is remarkably simple - if your serialization code needs to tolerate versions… Then make it version tolerant!

Now there are many ways of making serialization version tolerant, and they all depend on how exactly you are serializing your data.

A direct approach can be to just JSON.NET-the-lot and go with that. And that’s fine, it works and pays the bills. Happy camping, Homer.

But what if you care about the milliseconds? What if you do want the brute speed and small size of binary serialization… and the benefits of text based version tolerance?

Well that’s a what version tolerant binary serializer is there for. (Again, the answer is so simple, it should be a criminal offense).

Below I’ll show you how to setup and use the Microsoft Bond serializer, so you get the best of both worlds.

My name is Bond… Microsoft Bond. Binary Martini… Shaken, not shifted.

Bond is Microsoft’s equivalent of Google’s Protocol Buffers and they both share the same traits:

Some annoying platform independent script compilation (though Visual Studio makes it a lot better) - this is in exchange for…
Fast binary (de)serialization by default. Bond uses a layered approach so you can mix and match features. The more you use, the more you pay, but it goes the other way too.
Compact binary format.
Platform and language agnostic - well kinda.

You can find the full sales pitch here.

Anyway, onwards to the how-to:

Install Bond.CSharp

The first step is to go grab the Bond.CSharp NuGet package.

This package will bring the Bond dependency lot with it and make life easier for you. If for some reason you need to micromanage your dependencies, you can also go with the several other Bond.* packages.

Restart Visual Studio

The Bond Package will activate a new Custom Build option in Visual Studio. You need to restart the IDE for this to take effect.

Create a Bond File

Now create a dot bond file to hold your serializable types. And don’t shake your head now, stick with me here.

Bond shares the same necessary evil of Protocol Buffers. As it is a platform-independant serializer, you need to specify its types in a platform-independant way. Bond needs to know what types actually are, regardless of the platform in use. One platform may use Big Endian encoding while another uses Little Endian. One platform may encode strings with a length prefix while another platform uses an end marker. Who knows? Yet you still want both platform to speak to each other, even in high speed binary. Both Bond and Protobuf’s answer to this is to use their own uniform types and translate as needed for each different platform.

Set the file’s Build Action to BondCodegen</h2>

This will let Bond do some magic in the background and generate C# classes for you that mirror the types you wrote. You can also compile these bond files with the bond compiler, but I won’t dive into that here.

Write Some Code To Save

This will serialize an instance of that SomeBondClass to disk - but you can also send the bytes via the pipe, of course.

Note the lack of the using clauses. That surprised me at first, streams being all IDisposable and all. Not so much with Bond, it cleans after itself. If only my dog did that as well.

Check That File Up

This binary file stands at a measly 15 bytes. Serialize this tiny same class as JSON and you end up with 50 bytes. You are using only 30% of bytes as you would with JSON. Won’t even bother with XML at this point.

Write Some Code To Load

This will read the serialized data back into a class for you. Here’s the result:

As expected.

Mess It Up

Okay, this is all nice but we haven’t made of use of any version tolerance whatsoever. So let’s make it so and break things up.

Introduce a change into your Bond class:

I just added a new property here for show, but do whatever you want.

Read It Back Again

This:

Outputs this:

Which is the behavior you’d get with say, JSON.NET - though for a much smaller file size, which is ideal for piping over or persisting to disk.

That’s It

If you want to dig in further, go check out the project at GibHub or the full manual at GitHub. Because GitHub.

I’m growing fond of Bond over time as another useful tool for implementing real-time systems. Anything that helps grab that extra millisecond helps. I may even start drinking Martini again.

Till next time.

Bucketizing All The Things In C#

Mon, 26 Mar 2018 00:00:00 +0000

The Partition Problem is a very old conundrum in computer science. The usual question goes: given some buckets of some capacity, and some items of some size, how can I distribute the items by the buckets in an optimal manner?

The answer to this is more complex than it seems at first glance, as it depends on what you mean by some buckets and by optimal.

Do you know in advance how many buckets you have? Or their size?
Do you have to fill more buckets with fewer items or less buckets with more items?
Do you allow bucket overflow or enforce capacity?
Do you want the perfect distribution regardless of performance or is fast-and-furious good enough for you?

There are lots of ways of answering these questions and even more ways of implementing solutions for them, from simple greedy algorithms to more complex heuristics based ones.

Today I’ll get you started the basics for a simple and generic distribution algorithm based on a descending greedy approach, with a couple of variations:

Fixed bucket count: Distribute all items up to available buckets. This is a common scenario and a good place to start.
Fixed bucket size, uncapped bucket count: Prioritize filling up buckets up to their size, before adding new ones.
Auto-capped bucket size, uncapped bucket count: Cap bucket size by the largest item, then behave as the previous method. This is useful when you want to adjust some work to the size of the longest task, so all tasks complete together as possible.

So let’s set this up.

Bucket

First, we create a helper class to act as a bucket:

public class Bucket
{
    public int TotalCost { get; set; } = 0;
    public IList Items { get; } = new List();

    private Func cost;

    public Bucket(Func cost)
    {
        this.cost = cost;
    }

    public void Add(T item)
    {
        Items.Add(item);
        TotalCost += cost(item);
    }
}

This class abstracts the bucket concept out of the algorithm and helps keep things simple. Note how the bucket accepts a cost function. This makes it simple to keep the current bucket cost updated without having to recalculate things all the time.

Bucketizer

Now we add a Bucketizer class to hold the algorithms. You can also call it Potatoes, but I’ll leave that up to you.

public class Bucketizer
{
}

Bucketize By Bucket Count

So let’s say you know how many buckets you have available upfront. This variation will distribute any number of items over a fixed number of buckets. If you have fewer items than buckets, then the method returns only those buckets that got filled.

public Bucket[] BucketizeByBucketCount(IEnumerable items, Func cost, int maxBucketCount)
{
    // validate
    if (items == null) throw new ArgumentNullException(nameof(items));
    if (cost == null) throw new ArgumentNullException(nameof(cost));
    if (maxBucketCount  cost(x)).ToArray();

    // check if there is anything to bucket
    if (ordered.Length < 1)
    {
        // return the empty bucket array
        return buckets.ToArray();
    }

    // how many items we have vs how many buckets
    var needs = (ordered.Length < maxBucketCount ? ordered.Length : maxBucketCount);

    // distribute items by bucket count
    foreach (var item in ordered)
    {
        // can we still add buckets
        if (buckets.Count  x.TotalCost).First().Add(item);
        }
    }

    // done
    return buckets.ToArray();
}

This simple algorithm takes all of the items, orders them by descending order, and then puts each item into the bucket with the most space available. However, it will create a new bucket first, up to the maximum number, before reusing existing buckets. The magic here is the descending order - this will give you very decent results most of the time while keeping the code easy to understand.

Note that the code above is not super fast - it has (buckets * items) max iterations plus the initial cost of ordering - but you can optimize it with a dictionary (keyed on current bucket weight) once you adjust it to your needs.

I’ll leave that exercise up to you.

Bucketize By Bucket Size

So you don’t care about the number of buckets. Instead you want to fill up as few buckets as possible up a maximum bucket size. In that case, you can go with this:

public Bucket[] BucketizeByBucketSize(IEnumerable items, Func cost, int maxBucketSize)
{
    // validate
    if (items == null) throw new ArgumentNullException(nameof(items));
    if (cost == null) throw new ArgumentNullException(nameof(cost));
    if (maxBucketSize  cost(x)).ToArray();

    // check if there is anything to bucket
    if (ordered.Length  cost(i)).Max();

    // bucketize by max bucket size using top item cost
    return BucketizeByBucketSize(items, cost, top);
}

All this code does is to discover the item with the highest cost, and the delegate the rest of the logic to the previous method. Nothing to it.

There are many other ways of bucketizing items and in far more optimal ways that the examples above - but I hope this gives you a head start.

Till next time.