MapR 5.1 is at End of Life (EOL) and no longer supported. Please see the latest documentation. This documentation is not being updated.

Home
5.1 DB & Streams
This section covers the MapR-DB, MapR-Streams, and MapR-Gateways features. Each feature may contain both administrator and developer topics.
MapR Streams
MapR Streams brings integrated publish/subscribe messaging to the MapR Converged Data Platform.
Replicating MapR Streams
You can replicate streams to other MapR clusters worldwide or to other streams within a MapR cluster.
Configuring MapR Clusters for Replication Between Streams
Configuring clusters for participation in the replication of MapR Streams streams involves configuring two or more gateways on destination clusters and, if the clusters are secure, setting up secure communications between the clusters.

MapR 5.1 Documentation

5.1 DB & Streams
This section covers the MapR-DB, MapR-Streams, and MapR-Gateways features. Each feature may contain both administrator and developer topics.
- MapR-DB
- MapR Streams
  MapR Streams brings integrated publish/subscribe messaging to the MapR Converged Data Platform.
  - MapR Streams Concepts
  - Life of a Message
    To show how the MapR Streams concepts fit together, here is an example of the flow of one message from a producer to a consumer.
  - Getting Started with MapR Streams
    If you have a basic understanding of MapR Streams's components and the typical flow of messages from producers to consumers, you can give MapR Streams a try.
  - MapR Streams in Detail
    If you understand the basics about the components of MapR Streams, the flow of messages from producers to consumers, and how producers and consumers are coded, it's time to cover them in more detail.
  - Monitoring the Flow of Messages
    The speed at which messages flow from producers to partitions and from partitions to consumers depends on the performance of your producers, cluster nodes hosting partitions, and your consumers.
  - Replicating MapR Streams
    You can replicate streams to other MapR clusters worldwide or to other streams within a MapR cluster.
    - Modes of Replication of Streams
      You can replicate streams in one of two replication modes. You specify the mode per source-replica pair.
    - Security for Replication of Streams
      You can configure security measures in these locations:
    - Configuring MapR Clusters for Replication Between Streams
      Configuring clusters for participation in the replication of MapR Streams streams involves configuring two or more gateways on destination clusters and, if the clusters are secure, setting up secure communications between the clusters.
    - Setting Up Master-Slave Replication of MapR Streams Streams
      In this topology, you replicate in one direction. Replica streams can be in a remote MapR cluster or in the MapR cluster where their source streams are located.
    - Setting Up Multi-Master Replication of MapR Streams Streams
      In this replication topology, there are two master-slave relationships, with each stream playing both the role of a master and a slave. Client applications update both streams and each stream replicates updates to the other.
  - Querying Messages in MapR Streams
    You can develop Java applications or modify existing Java applications to query messages that are in MapR streams by using the methods in the Streams class.
  - Migrating Apache Kafka 0.9.0 Applications to MapR Streams
    There are only two steps that you need to follow to migrate applications written with the Apache Kafka 0.9.0 Java API to MapR Streams.
  - Mirroring Topics with Apache Kafka's MirrorMaker
    Use Apache Kafka's MirrorMaker utility either to mirror topics that are in Apache Kafka clusters to streams that are in MapR clusters or to Mirror topics that are in MapR clusters to Apache Kafka clusters.
  - Utilities for MapR Streams
    MapR Streams provides the following utilities.
- MapR Gateways
  A MapR gateway mediates one-way communication between a source MapR cluster and a destination cluster. MapR-DB binary tables and MapR Streams streams can be replicated.

Configuring MapR Clusters for Replication Between Streams

Configuring clusters for participation in the replication of MapR Streams streams involves configuring two or more gateways on destination clusters and, if the clusters are secure, setting up secure communications between the clusters.

Prerequisites

Plan which replication topology you want to use: basic master-slave, multi-master, or a combination of these.
Ensure that you have administrative authority on the clusters that you plan to use.
In the mapr-clusters.conf file on every node in your source cluster, add an entry that lists the CLDB nodes that are in the destination cluster. This step is required for using maprcli commands and the utilities, which bypass gateways and communicate directly with the destination cluster's CLDB nodes. See mapr-clusters.conf for the format to use for the entries.
In the mapr-clusters.conf file on every node in your destination cluster, add an entry that lists the CLDB nodes that are in the source cluster. This step is required for running maprcli commands, as when you add upstream sources to replicas or when the destination cluster also serves as a source cluster in bi-directional replication or multi-master replication. See the topic mapr-clusters.conf for the format to use for the entry.

Procedure

To configure clusters for replication between streams:

Configure gateways on the destination clusters.
See Configuring a MapR Gateway Master-Slave Topology.
Optional: If your clusters are secure, configure source clusters so that you can locally run maprcli commands that are executed on the destination cluster.
See Running Commands on Remote Secure Clusters.
If your clusters are secure, add one cross-cluster ticket to each source cluster for each cluster that it replicates to.
See Adding Cross-Cluster Tickets to Secure Clusters.
Ensure that the user ID of the person who starts the replication process has the readAce permission on the volume where the source streams are located and the writeAce permission on the volumes where the replica streams are located. For information about how to set permissions on volumes, see Setting/Modifying Whole Volume ACEs.