Stream Design

The design of streams should take into account replication and security permissions (using access-control expressions) requirements. Because data replication and security permission settings take place at the stream-level, these requirements can impact the design of streams.

For example, suppose that you plan to create the stream pollution_monitors to collect various measurements about pollution levels in European cities. However, during a planning session, the representative from Amsterdam says that her country wants to analyze of the data for its cities and would like your company to replicate the data to its own MapR cluster, where its own consumers can read the replicated messages.

In this scenario, you might do the following:
  • Create a stream dedicated to the Netherland's pollution data or even for every country you are monitoring. For example, create streams named pollution_monitors_netherlands, pollution_monitors_sweden, pollution_monitors_france, and so on.
  • Within each stream, create topics for each city in that county. For example, a create topic named amsterdam which contains data from Amsterdam's pollution sensors.
  • Since, in this scenario, the Amsterdam representive also requested stream replication to their own MapR cluster, you would set up stream replication from your MapR cluster to Amsterdam's MapR cluster. See Managing Stream Replication for information about setting up and managing replication.

Alternatively, perhaps the Netherlands didn't request replication to their own MapR cluster, however, you want to restrict access to the pollution data where consumers could read only pollution data for their respective country.

In this scenario, you might do the following:
  • Create streams for each country.
  • Create topics for each city in that country.
  • Set each stream's consumeperm permission for consumers associated with that country. See Stream Security for more information about security permissions used with MapR-ES streams. For general information about access-control expressions, see ACE Syntax..