Download Riak 2.0 Beta 1

Multi

Overview

Riak allows you to run multiple backends within a single Riak instance. This is very useful for two very different use cases.

  1. You may want to use different backends for different buckets or
  2. You may need to use the same storage engine in different ways for different buckets.

The Multi backend allows you to configure more than one backend at the same time on a single cluster.

Installing Multi-Backend Support

Riak ships with Multi backend support included within the distribution so there is no separate installation required.

Configuring Multiple Backends

Modify the default behavior by adding these settings in your app.config. The multi_backend configuration must be within the riak_kv section of the app.config.

%% Riak KV config
{riak_kv, [
    %% ...
    %% Use the Multi Backend
    {storage_backend, riak_kv_multi_backend},
    %% ...
]}

Then later anywhere in the riak_kv section (but you'll likely want this in the section with other backend-related information) add a section to configure the multiple backends.

If you are defining multiple file based backends of the same type, each of these must have a separate data_root defined.

Organizing Configuration

While these configuration directives can be placed anywhere within the riak_kv section of app.config, we recommend that you place them in the section with other backend-related settings to keep the settings organized.

%% Use bitcask by default
{riak_kv, [
    %% ...
    {multi_backend_default, <<"bitcask_mult">>},
    {multi_backend, [
        %% Here's where you set the individual multiplexed backends
        {<<"bitcask_mult">>,  riak_kv_bitcask_backend, [
                         %% bitcask configuration
                         {data_root, "/var/lib/riak/bitcask_mult/"},
                         {config1, ConfigValue1},
                         {config2, ConfigValue2}
        ]},
        {<<"bitcask_expiry_mult">>,  riak_kv_bitcask_backend, [
                         %% bitcask configuration
                         {data_root, "/var/lib/riak/bitcask_expiry_mult/"},
                         {expiry_secs, 86400},
                         {config1, ConfigValue1},
                         {config2, ConfigValue2}
        ]},
        {<<"eleveldb_mult">>, riak_kv_eleveldb_backend, [
                         %% eleveldb configuration
                         {config1, ConfigValue1},
                         {config2, ConfigValue2}
        ]},
        {<<"second_eleveldb_mult">>,  riak_kv_eleveldb_backend, [
                         %% eleveldb with a different configuration
                         {config1, ConfigValue1},
                         {config2, ConfigValue2}
        ]},
        {<<"memory_mult">>,   riak_kv_memory_backend, [
                         %% memory configuration
                         {config1, ConfigValue1},
                         {config2, ConfigValue2}
        ]}
    ]},
    %% ...
]},
Multi-Backend Memory Use
Each backend has settings for how much memory the backend can use. It might be for caching, like in LevelDB, or for the entire set of data, like in the Memory Backend. Each of these backends suggests allocating up to 50% of available memory for this. When using Multi Backend, it is important that the sum of all backend memory use is at 50% or less. Three backends each set to use 50% of available memory would cause problems.
Multi-Backend settings

Certain settings, such as Bitcask's merge_window, are set per-node, rather than per-backend, and as such must be set in the top level backend sections of your app.config.

Once configured start the Riak cluster. Riak will use the multi_backend_default for all new bucket storage unless you configure a bucket to use one of the alternate storage engines. This can be done using either the Erlang console or the HTTP interface, both methods simply change the bucket properties. Here are two examples:

Note that if you change the default bucket storage engine via the app.config settings, then you will need to restart the node, or nodes, for that change to take effect.

Changing Backends

The backend bucket property designates which backend should be used for storing objects in the specified bucket. This setting should be configured before loading the bucket with data. Changing a bucket's backend after data has been loaded will cause the data to appear lost, as the bucket will be associated with a separate backend that does not contain previously inserted keys.

There is no built-in functionality in Riak to automatically change the underlying datastore of a bucket. In order to migrate keys from one backend to another, the keys must be manually migrated using GET and PUT operations.