Creating Topics

There are two methods by which topics can be created in a stream.

You must first create a stream before creating a topic. To do so, you can use either the maprcli command stream create or the MapR Streams Java API Admin.createStream(String streamPathAndName, StreamDescriptor desc).
After a topic is created in a stream, it is not possible to move that topic to a different stream.

For example, suppose you create the topic structural_integrity_sensors_us_western_region, one of a number of topics that collect data from sensors that keep watch over various measurements for bridges, buildings, and other structures. However, you've mistakenly created the topic in the stream ventilation_systems instead of the stream structural_integrity_sensors. There is no command that would move the topic from its current stream to the other stream. You must delete the topic and recreate it in the other stream.

Moreover, any producers that have published messages to the topic and any consumers that have read messages to the topic will need to be modified to point to the new location of the topic. Producers and consumers refer to topics with a combination of stream name and topic name.

Restrictions for the names of topics

A name can include alphanumeric characters and the following characters: . , _ , and - .

A producer can specify the name of a stream together with the name of a topic to write to, like this:

However, if the stream is not specified, the value of this configuration parameter is assumed to be the stream in which the topic is located.

If the producer specifies the name of a topic without also providing the path and name of the stream, and there is no value for this configuration parameter, MapR Streams assumes that the topic specified is in Apache Kafka and does nothing.

You can create topics by using either of these methods:

  • Automatic creation

    By default, a topic is created automatically when a producer creates the first message for it and the topic does not already exist.

    For example, you might have the stream anonymous_usage that is intended to collect data about the use of a software application that is about to be released. The administrator did not create any topics when creating the stream, but producers will create topics automatically by publishing to topics for ranges of IP addresses. After the software is released to the public, at some point a producer application starts publishing messages to a topic that is created based on the range within which the producer's IP address falls. At another point in time, a producer starts publishing messages to a topic for a different range of IP addresses. Eventually, the stream contains a number of topics for different ranges, and multiple producers are publishing to each topic.

    You can turn off the automatic creation of topics for a stream. If you do this, the publishing of a message to a non-existent topic fails.

    You can use the command maprcli stream edit to change this setting after you create a stream.

  • Manual creation
    The other method of creating topics is to use the stream topic create command.

    For example, if you are creating a stream to collect operational metrics from systems in your enterprise, you might have already planned on a set number of topics based on system, location, company department, project, or some other criterion. You could create these topics after creating the stream.

    When you create a topic this way, you can either accept the default number of partitions for all new topics in the current stream, or you can override that default and specify a number of partitions for the new topic.