Configuration Parameters for Producers

You can control some of the aspects of how producers publish messages by setting various configuration parameters every time you start a producer.

Here is a list of the parameters, along with brief descriptions. Four of them are originally from Apache Kafka and are supported by MapR Streams. The remaining three are specific to MapR Streams. They are all described further as the details of consumers are explained.

Configuration parameters supported from Apache Kafka

buffer.memory: The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are generated faster than they can be delivered to the server the producer will block.; Default: 33554432
client.id: Producers can tag records with a client ID that identifies the producer. Consumers can then be aware of which producer sent a message or set of messages. Apache Drill or other analytic tools querying messages can include this ID in the filters for their queries.; Default: No client ID.
key.serializer: The name of the appropriate serialization class in the org.apache.kafka.common.serialization package or a class that implements the Serializer interface for serializing keys.
metadata.max.age.ms: The producer generally refreshes the topic metadata from the server when there is a failure. It will also poll for this data regularly.; Default: 600 * 1000 msec
value.serializer: The class that implements the Serializer interface for serializing values.

Configuration parameters specific to MapR Streams

streams.buffer.max.time.ms

Messages are buffered in the producer for at most the specified time. A thread will flush all the messages that have been buffered for more than the time specified.

Default: 3 * 1000 msec

streams.parallel.flushers.per.partition

If enabled, producer may have multiple parallel send requests to the server for each topic partition. If this setting is set to true, it is possible for messages to be sent out of order.

Default: true

streams.partitioner.class

The class that implements the StreamsPartitioner interface. This interface lets you write custom algorithms for determining which topic and partition to use for messages that match specific criteria. Use this configuration parameter only for producers that are written in Java.

streams.producer.default.stream

Specifies the stream that the producer will use by default if the producer does not provide the name of a stream when specifying a topic to write to.

The syntax is

/mapr/<cluster name>/<volume name>/<stream name>

fs.mapr.hardmount

Specifies whether to use a hard mount or a soft mount for connections to the MapR Streams server.

The default is true.

If a value for this parameter is set in the core-site.xml file, the value in that file is ignored.

fs.mapr.rpc.timeout

Specifies the length of time in seconds to wait for a response from the MapR Streams server if the configuration parameter fs.mapr.hardmount is set to false.

The default value is 300. The minimum value is 30.

If a soft mount is used, the time expires while a producer waits for a response from the MapR Streams server, and the producer used the KafkaProducer.send(ProducerRecord<K,V> record, Callback callback) method, the callback is invoked with the error EAGAIN, which means "Resource temporarily unavailable."

If a value for this parameter is set in the core-site.xml file, the value in that file is ignored.