Understanding Topic Creation

Topics are created in streams. They can be created either automatically through your producer application or manually through the MCS or maprcli.

Automatic Creation

If the topic does not already exist, a topic is created automatically when a producer first publishes a message to it. This is the default behavior.

For example, you created the stream anonymous_usage that you intend to use to collect data about the usage of a software application that is soon to be released. However, you did not create any stream topics because the topics were not known at the time. After the software is released to the public, at some point, a producer application starts publishing messages to a topic that is created based on the range within which the producer's IP address falls. At another point in time, another producer application starts publishing messages to a topic based on a different range of IP addresses. Eventually, the stream contains a number of topics for different IP address ranges.

Note: Automatic creation of topics can be turned off by setting the autocreate parameter to false either when creating the stream or by editing the stream. See the maprcli stream create or stream edit command for more information. If you turn off automatic creation, you must manually create the stream topic, otherwise, the publishing of a message fails.

Manual Creation

To create topics manually, use either the MCS or maprcli. See Administering Streams for information about managing MapR-ES streams, topics and replication. See the stream topic create command for specific information about creating stream topics with maprcli.

For example, you created a stream called systemMetrics that you intend to use to collect operational metrics from systems in your enterprise. You did create several topics based on system, location, company department, project, or some other criterion. In this case, you could create topics in advance because they were pre-planned.

Note: When you manually create a topic, you can have the option of customizing the number of topic partitions used, otherwise, the default number of partitions is inherited from the stream.

Restrictions

  • After a topic is created in a stream, it is not possible to move that topic to a different stream.

    For example, suppose you create the topic structural_integrity_sensors_us_western_region, one of a number of topics that collect data from sensors that keep watch over various measurements for bridges, buildings, and other structures. However, you've mistakenly created the topic in the stream ventilation_systems instead of the stream structural_integrity_sensors.

    Remember: There is no command that will move a topic from its current stream to a different stream.
    To rectify the mistake:
    • You must delete the topic and recreate it in the other stream.
    • Any producers that published messages to the topic and any consumers that read messages from the topic must be modified to point to the new location of the topic. This is because producers and consumers refer to topics with a combination of stream name and topic name.
  • Only the following characters are allowed for stream topic names:
    • Alphanumeric characters
    • Period, underscore, and dash
  • When producing or consuming stream topic messages, you must specify the stream's path and name along with the topic name, like this:
    /<stream name>:<topic name>

    If a topic is specified but the stream's path and name is not, depending the application's programming language, you might get an error or nothing. If nothing happens and you are using Java, the assumption is that you are publishing to Apache Kafka.