MapR 5.1 is at End of Life (EOL) and no longer supported. Please see the latest documentation. This documentation is not being updated.

Home
5.1 DB & Streams
This section covers the MapR-DB, MapR-Streams, and MapR-Gateways features. Each feature may contain both administrator and developer topics.
MapR Streams
MapR Streams brings integrated publish/subscribe messaging to the MapR Converged Data Platform.
MapR Streams in Detail
If you understand the basics about the components of MapR Streams, the flow of messages from producers to consumers, and how producers and consumers are coded, it's time to cover them in more detail.
Producers
Producers are data-generating applications, such as sensors in automobiles or activity loggers in servers.
The Two Modes of Publishing
The producer client library batches messages into publish requests that it sends to the MapR Streams server. By default, the client sends multiple publish requests without waiting for acknowledgements from the MapR Streams server. This behavior is determined by the per producer configuration parameter streams.parallel.flushers.per.partition, which defaults to true.

MapR 5.1 Documentation

5.1 DB & Streams
This section covers the MapR-DB, MapR-Streams, and MapR-Gateways features. Each feature may contain both administrator and developer topics.
- MapR-DB
- MapR Streams
  MapR Streams brings integrated publish/subscribe messaging to the MapR Converged Data Platform.
  - MapR Streams Concepts
  - Life of a Message
    To show how the MapR Streams concepts fit together, here is an example of the flow of one message from a producer to a consumer.
  - Getting Started with MapR Streams
    If you have a basic understanding of MapR Streams's components and the typical flow of messages from producers to consumers, you can give MapR Streams a try.
  - MapR Streams in Detail
    If you understand the basics about the components of MapR Streams, the flow of messages from producers to consumers, and how producers and consumers are coded, it's time to cover them in more detail.
    - Creating Topics
      There are two methods by which topics can be created in a stream.
    - Default Number of Partitions in New Topics
      When you create or edit a stream, the -defaultpartitions parameter determines how many partitions are created in each new topic in the stream. If you do not explicitly set this value, each new topic is created with one partition.
    - Security
      The topics that are in a stream are all protected by a single set of security permissions.
    - Time-to-Live for Messages
      Messages can persist in the partitions of a stream for as long as you need them to, and can be read or reread by consumers during that time. However, if you know that after a certain period of time the messages in a stream will no longer be needed, you can specify a length of time for messages to persist until they expire.
    - Encrypting Streams
      Apply an additional layer of security to streams by encrypting them.
    - Deciding Which Streams to Create
      Your decision about what streams to create should take into account whatever topics you want to replicate. Replication is between streams, not individual topics.
    - Producers
      Producers are data-generating applications, such as sensors in automobiles or activity loggers in servers.
      - Configuration Parameters for Producers
        You can control some of the aspects of how producers publish messages by setting various configuration parameters every time you start a producer.
      - When Messages Are Published
        To publish a message, a producer sends a record to the producer client library, which batches the records before sending them to the server. The producer client library will send the records to the server when any of the four following conditions are met.
      - The Two Modes of Publishing
        The producer client library batches messages into publish requests that it sends to the MapR Streams server. By default, the client sends multiple publish requests without waiting for acknowledgements from the MapR Streams server. This behavior is determined by the per producer configuration parameter streams.parallel.flushers.per.partition, which defaults to true.
      - How Partitions Are Chosen for Messages
        Because the number of partitions in a topic can change over time, producers regularly refresh the information that they have about the topics that they know of. This interval is controlled by the metadata.max.age.ms configuration parameter.
      - Supported Apache Kafka 0.90 Java APIs for Producers
        MapR Streams supports these classes and interfaces for producers.
    - Consumers
      Consumers are applications that you create, such as analytics applications, reporting tools, or enterprise dashboards.
    - Compiling and Launching Producers and Consumers
      You can compile producers and consumers by using the required JAR file from the MapR Maven repository or from the MapR installation. Launch producers and consumers with the mapr command.
  - Monitoring the Flow of Messages
    The speed at which messages flow from producers to partitions and from partitions to consumers depends on the performance of your producers, cluster nodes hosting partitions, and your consumers.
  - Replicating MapR Streams
    You can replicate streams to other MapR clusters worldwide or to other streams within a MapR cluster.
  - Querying Messages in MapR Streams
    You can develop Java applications or modify existing Java applications to query messages that are in MapR streams by using the methods in the Streams class.
  - Migrating Apache Kafka 0.9.0 Applications to MapR Streams
    There are only two steps that you need to follow to migrate applications written with the Apache Kafka 0.9.0 Java API to MapR Streams.
  - Mirroring Topics with Apache Kafka's MirrorMaker
    Use Apache Kafka's MirrorMaker utility either to mirror topics that are in Apache Kafka clusters to streams that are in MapR clusters or to Mirror topics that are in MapR clusters to Apache Kafka clusters.
  - Utilities for MapR Streams
    MapR Streams provides the following utilities.
- MapR Gateways
  A MapR gateway mediates one-way communication between a source MapR cluster and a destination cluster. MapR-DB binary tables and MapR Streams streams can be replicated.

The Two Modes of Publishing

The producer client library batches messages into publish requests that it sends to the MapR Streams server. By default, the client sends multiple publish requests without waiting for acknowledgements from the MapR Streams server. This behavior is determined by the per producer configuration parameter streams.parallel.flushers.per.partition, which defaults to true.

With this behavior turned on, it is possible for messages to arrive to partitions out of order due to the presence of multiple network interface controllers, network errors, or retries.

For example, suppose a producer is sending messages that are specifically for partition 1. The producer client library buffers the messages and sends a batch to partition 1. Meanwhile, the producer keeps sending messages for partition 1 and the client continues to buffer them. The next time the producer client library has enough messages for partition 1, the client sends another batch, whether or not the MapR Streams server has acknowledged the previous batch.

If you always want messages to arrive to partitions in the order in which they were sent, set the configuration parameter streams.parallel.flushers.per.partition to false. The producer client library will wait for acknowledgements from the MapR Streams server before sending subsequent publish requests.