Logical Schema of Messages

Each message has the same logical schema: _id, topic, partition, offset, timestamp, producer, key, and value.

Because the logical schema of each message is the same, analytics applications can run queries on these fields. See MapR-ES Java API Library for information about querying messages and mapr streamanalyzer for a sample application used to query and count messages in topics.

{
          "_id":<STRING>,
          "topic":<STRING>,
          "partition":<SHORT>,
          "offset":<LONG>,
          "timestamp":<LONG>,
          "producer":<VARCHAR>,
          "key":<BINARY>,
          "value":<VARBINARY>
          }
Field Description
_id A STRING value that represents the ID of the topic in which the message is located.
topic A STRING value that represents the name of the topic in which the message is located.
partition A SHORT value that represents the index of the partition in the topic.
offset A LONG value that represents the position of the message within a partition.
timestamp A LONG value that represents the date and time at which the message was sent to the stream.
producer A VARCHAR value that represents the value of the client.id configuration parameter for the producer that published the message. MapR-ES does not require a value for this configuration parameter, so the value for this field could be empty.
key A BINARY value that represents the key of the message. MapR-ES does not require each message to have a key, so this value could be empty. The configuration parameter key.serializer for the producer that published the message specifies the means by which the key was serialized.

Your application can deserialize the key by using the appropriate deserialization class in the org.apache.kafka.common.serialization package or a class that implements the Deserializer interface.

value A VARBINARY value that represents the value of the message. The configuration parameter value.serializer for the producer that published the message specifies the means by which the value was serialized.

Your application can deserialize the value by using the appropriate deserialization class in the org.apache.kafka.common.serialization package or a class that implements the Deserializer interface.