Working with a cluster (.NET)

Configuring a cluster

To use a cluster, we need to configure it first in the ActorSystem.

That is what a basic configuration looks like:

using Proto;
using Proto.Cluster;
using Proto.Remote;

var actorSystemConfig = ActorSystemConfig.Setup();

var remoteConfig = GrpcNetRemoteConfig.BindTo(/* ... */);

var clusterConfig = ClusterConfig.Setup(
    clusterName: "MyCluster",
    clusterProvider: new TestProvider(new TestProviderOptions(), new InMemAgent()),
    identityLookup: new PartitionIdentityLookup()
);

var actorSystem = new ActorSystem(actorSystemConfig)
    .WithRemote(remoteConfig)
    .WithCluster(clusterConfig);

To learn more about Actor System configuration, read the Actors section of Proto.Actor’s documentation.

Remote must be configured when using a cluster. To learn more about its configuration, read the Remote section of Proto.Actor’s documentation.

Setup parameters:

1. clusterName - a name of the cluster; it should be the same in all cluster members you want to work together.

2. clusterProvider - provides information about currently available members in a cluster; read more about it in the Cluster Providers section.

3. identityLookup - allows a cluster to locate grains (virtual actors); read more about it in the Identity Lookup section.

Registering grains (virtual actors)

Grains are not explicitly spawned. Instead, they are spawned (activated) by a cluster (in one of the cluster members), when a first message is sent to them. In order for a cluster member to know how to spawn a grain of a given kind, we need to register it in a ClusterConfig:

var clusterConfig = ClusterConfig
    .Setup(/* ... */);
    .WithClusterKind(
        kind: "user",
        prop: Props.FromProducer(() => new UserGrainActor())
    );

WithClusterKind parameters:

1. kind - a type of grain.

2. prop - define how an actor and its context is created. You can read more about Props here.

You can create any kind of actor using Props. However, implementing grains by hand comes with a lot of gotchas, e.g. you have to make sure that grain always responds to certain kinds of messages (also, with the correct response message type). For this reason, the recommended way of creating grains is by using the Proto.Cluster.CodeGen package, which solves most of these problems. Read more about it here.

Getting a Cluster object

To work with a cluster, we need a Cluster object.

You can get it from an ActorSystem:

using Proto;
using Proto.Cluster;

Cluster cluster = actorSystem.Cluster();

Or from IContext:

using Proto;
using Proto.Cluster;

Cluster cluster = context.Cluster();

Starting and shutting down a cluster member

Cluster members need to be explicitly started and shut down.

To start a new cluster member:

await _actorSystem
    .Cluster()
    .StartMemberAsync();

If you only want to send messages to a cluster, you can start it as a client:

await _actorSystem
    .Cluster()
    .StartClientAsync();

To shut down a cluster:

await _actorSystem
    .Cluster()
    .ShutdownAsync();

Sending messages to grains (virtual actors)

Grains require request-response-based messaging to ensure that the message was delivered and/or that the grain was properly activated. This allows Proto.Actor to re-try getting the PID by calling the actor until it succeeds.

You do this using:

using Proto.Cluster;

BlockUserResponse response = await cluster.RequestAsync<BlockUserResponse>(
    identity: "150",
    kind: "user",
    message: new BlockUser(),
    ct: cancellationToken
);

The result will be one of the following:

1. null - when a request timeouts.
2. A grain’s response message.

As mentioned before, the recommended way of creating grains is by using the Proto.Cluster.CodeGen package. A generated grain will include Cluster extension methods for sending requests to it:

BlockUserResponse response = await cluster
    .Cluster()
    .GetUserGrain("150")
    .BlockUser(ct: cancellationToken);

Read more about generating grains here.

Handling timeouts

Timeouts are handled with cancellation tokens. Proto.Actor has a CancellationTokens utility that efficiently creates such tokens:

using Proto;
using Proto.Cluster;

BlockUserResponse response = await cluster.RequestAsync<BlockUserResponse>(
    identity: "150",
    kind: "user",
    message: new BlockUser(),
    ct: CancellationTokens.WithTimeout(TimeSpan.FromSeconds(2))
);

Using a cluster in an ASP.NET Core app

It’s recommended to configure and register an ActorSystem in a service collection as a singleton:

services.AddSingleton(provider =>
{
    var actorSystemConfig = ActorSystemConfig.Setup();

    var remoteConfig = GrpcNetRemoteConfig.BindTo(/* ... */);
    
    var clusterConfig = ClusterConfig.Setup(/* ... */);

    return new ActorSystem(actorSystemConfig)
        .WithServiceProvider(provider)
        .WithRemote(remoteConfig)
        .WithCluster(clusterConfig);
});

For convinience, you can also register a Cluster object:

services.AddSingleton(provider => provider
    .GetRequiredService<ActorSystem>()
    .Cluster()
);

The recommended way of starting and shutting down a cluster member in an ASP.NET Core app is by using a Hosted Service.

To create a hosted service:

public class ActorSystemClusterHostedService : IHostedService
{
    private readonly ActorSystem _actorSystem;

    public ActorSystemClusterHostedService(ActorSystem actorSystem)
    {
        _actorSystem = actorSystem;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        await _actorSystem
            .Cluster()
            .StartMemberAsync(); // or StartClientAsync()
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        await _actorSystem
            .Cluster()
            .ShutdownAsync();
    }
}

To register it:

services.AddHostedService<ActorSystemClusterHostedService>();