Topic Creation

Topics are created in streams. They can be created either automatically through your producer application or manually through the MCS or maprcli.

Automatic Creation

If the topic does not already exist, a topic is created automatically when a producer first publishes a message to it. This is the default behavior.

For example, you created the stream anonymous_usage that you intend to use to collect data about the usage of a software application that is soon to be released. However, you did not create any stream topics because the topics were not known at the time. After the software is released to the public, at some point, a producer application starts publishing messages to a topic that is created based on the range within which the producer's IP address falls. At another point in time, another producer application starts publishing messages to a topic based on a different range of IP addresses. Eventually, the stream contains a number of topics for different IP address ranges.

Note: Automatic creation of topics can be turned off by setting the autocreate parameter to false either when creating the stream or by editing the stream. See the maprcli stream create or stream edit command for more information. If you turn off automatic creation, you must manually create the stream topic, otherwise, the publishing of a message fails.

Manual Creation

To create topics manually, use either the MCS or maprcli. See Administering Streams for information about managing MapR-ES streams, topics and replication. See the stream topic create command for specific information about creating stream topics with maprcli.

For example, you created a stream called systemMetrics that you intend to use to collect operational metrics from systems in your enterprise. You did create several topics based on system, location, company department, project, or some other criterion. In this case, you could create topics in advance because they were pre-planned.

Note: When you manually create a topic, you can have the option of customizing the number of topic partitions used, otherwise, the default number of partitions is inherited from the stream.

Topic Partitions

The default number of partitions for MapR-DB streams and topics can impact performance. Depending on the volume of messages being published to a topic, the default number of partitions might be increased for efficient consumption.

When there is a high volume of messages being published to a topic:
  • Multiple consumers, in consumer groups, reading from multiple partitions are handled more efficiently.
  • Individual consumers each reading from a single partition are handled less efficiently.
When creating or editing a stream, a default number of partitions can be specified for that stream's topics. Topics inherit the stream's partition default, however, topics can also override the stream's partition default by setting the number of partitions to be used.

For More Information

  • See the maprcli stream create for information about creating streams with the -defaultpartitions parameter..
  • See the maprcli stream edit for information about editing streams with the -defaultpartitions parameter.
  • See the maprcli stream topic create for information about creating topics with the -partitions parameter.
  • See the maprcli stream topic edit for information about modifying topics with the -partitions parameter.
  • See the MapR-ES Java API Library for the methods used to create and edit streams and to create and edit topics.

Restrictions

  • After a topic is created in a stream, it is not possible to move that topic to a different stream.

    For example, suppose you create the topic structural_integrity_sensors_us_western_region, one of a number of topics that collect data from sensors that keep watch over various measurements for bridges, buildings, and other structures. However, you've mistakenly created the topic in the stream ventilation_systems instead of the stream structural_integrity_sensors.

    Remember: There is no command that will move a topic from its current stream to a different stream.
    To rectify the mistake:
    • You must delete the topic and recreate it in the other stream.
    • Any producers that published messages to the topic and any consumers that read messages from the topic must be modified to point to the new location of the topic. This is because producers and consumers refer to topics with a combination of stream name and topic name.
  • Only the following characters are allowed for stream topic names:
    • Alphanumeric characters
    • Period, underscore, and dash
  • When producing or consuming stream topic messages, you must specify the stream's path and name along with the topic name, like the following:
    /<stream name>:<topic name>

    If a topic is specified but the stream's path and name is not, depending the application's programming language, you might get an error or nothing. If nothing happens and you are using Java, the assumption is that you are publishing to Apache Kafka.