cldbguts

As of version 4.0.2, you can run the cldbguts utility to monitor the activity of the CLDB. This utility prints information about of the CLDB service that is running on the node from which you run the utility. Monitoring the progress of the CLDB may be useful when troubleshooting cluster issues.

The cldbguts utility prints information about active container reports, full container reports, registration requests, MapR-FS heartbeats, NFS server heartbeats, and containers. You can run cldbguts from any CLDB node; however, running this command from the CLDB master node provides the most relevant information.

NOTE: After you run cldbguts, it continues to print the output until you kill the process.

Syntax:

/opt/mapr/bin/cldbguts

Output Fields:

Field Description

acr

Represents active container requests(ACR).

This column includes the following information:

  • nr: Number of ACRs completed in the past 1 second. The first entry displays total number of ACRs completed since the start of the CLDB service on the node.
  • pt: Processing time (in milliseconds) for the ACRs completed in the past 1 second. The first entry displays the total time (in milliseconds) spent processing the ACRs since the start of the CLDB service on the node.
  • to: Number of ACRs that took longer than expected in the past 1 second. The first entry displays the total number of ACRs that took longer than expected since start of the CLDB service on the node.
  • d: Number of duplicate ACRs received in the past 1 second.The first entry displays the total number of duplicate ACRs since start of the CLDB service on the node.
  • dp: Number of duplicate ACRs that required additional work in the past 1 second. The first entry displays total number of duplicate ACRs that required additional work since start of the CLDB service on the node.

fcr

Represents full container report (FCR).

This column includes the following information:

  • nr: Number of FCRs completed in the past 1 second. The first entry displays total number of FCRs completed since the start of the CLDB service on the node.

  • pt: Processing time (in milliseconds) for the FCRs completed in the past 1 second. The first entry displays total time (in milliseconds) spent processing the FCRs since the start of the CLDB service on the node.

  • to: Number of FCRs that took longer than expected in the past 1 second. The first entry displays the total number of FCRs that took longer than expected since the start of the CLDB service on the node.

regn

Represents registration requests.

This column includes the following information:

  • nr: Number of registration requests completed in the past 1 second. The first entry displays total number of registration requests completed since the start of the CLDB service on the node.

  • pt: Processing time (in milliseconds) for the registration requests completed in the past 1 second. The first entry displays the total time (in milliseconds) spent processing the registration requests since the start of the CLDB service on the node.

  • to: Number of registration requests that took longer than expected in the past 1 second.The first entry displays total number of registration requests that took longer than expected since the start of the CLDB service on the node.

mfs hb

Information about MapR-FS heartbeats.

This column includes the following information:

  • nr: Number of MapR-FS heartbeats completed in the past 1 second. The first entry displays total number of MapR-FS heartbeats completed since the start.

  • pt: Processing time (in microseconds) for the MapR-FS heartbeats completed in the past 1 second.The first entry displays total time (in microseconds) spent processing MapR-FS heartbeats since the start of the CLDB service on the node.

  • to: Number of MapR-FS heartbeats that took longer than expected in the past 1 second.The first entry displays total number of MapR-FS heartbeats that took longer than expected since the start of the CLDB service on the node.

nfs hb

Information about NFS server heartbeats.

This column includes the following information:

  • nr: Number of NFS server heartbeats completed in the past 1 second. The first entry displays total number of NFS server heartbeats completed since the start.

  • pt: Processing time (in microseconds) for the NFS server heartbeats completed in the past 1 second.The first entry displays the total time (in microseconds) spent processing MapR-FS heartbeats since the start of the CLDB service on the node.

assigns

This column includes the following information:

  • nr: Number of container assign requests in the past 1 second.The first entry displays the total number of container assign requests since the start of the CLDB service on the node.

  • nc: Number of containers created as part of the above container assign requests in the past 1 second. The first entry displays the total number of containers created since the start of the CLDB service on the node.

  • nrt: Number of container assign requests for tablets in the past 1 second.The first entry displays total number of container assign requests for tablets since the start of the CLDB service on the node.

  • nct: Number of containers created as part of the above container assign requests for tablets in past 1 second. The first entry displays the total number of container created in tablets since the start of the CLDB service on the node.

roles

Represents the roles of the various replica containers.

This column includes the following information:

  • bm: Number of replica containers that are in the process of becoming master.

  • ms: Number of replica containers that the CLDB thinks have valid masters.

  • wr: Number of replica containers that are waiting for CLDB to assign a role to them.

  • rs: Number of replica containers that are resyncing.

  • vr: Number of non-master replica containers that have finished resynchronization.

  • uu: Number of replica containers that are unused. For example, the number of replica containers that are on nodes or storage pools which have been offline or unavailable for more than an hour.

Note: It may takes some time for the CLDB to be aware of role changes.

progress

This column includes the following information:

  • m%: Percentage of containers that have valid masters.

  • uc: Number of unique containers.

  • v%: Percentage of replica containers that are valid (i.e., have completed resynchronization).

  • tr: Total number of replica containers.

con-chain

This column includes the following information:

  • ms: Number of unique containers that have a master.

  • 1r: Number of unique containers that have 2 valid copies of the data.

  • 2r: Number of unique containers that have 3 valid copies of the data.

Example Output:

# /opt/mapr/bin/cldbguts
2014-11-20 02:17:38,088
           acr                 fcr          regn             mfs hb              nfs hb             assigns                 roles                     progress             con-chain  
   nr    pt   to  d  dp    nr   pt  to   nr   pt  to     nr      pt    to     nr      pt       nr  nc  nrt  nct    bm  ms  wr  rs   vr  uu      m%    uc     v%     tr    ms  1r  2r  
  4956  2143   0  0   0    26  167   0   30  113   0   16209  3962833   0    3173  86214899     7  40   7    25     0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    3     0    0  0   0     0   0    0    0   0    0     4      1390    0      3    128038      0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    1     0    0  0   0     0   0    0    0   0    0     5      1108    0      1     27623      0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    0     0    0  0   0     0   0    0    0   0    0     4      920     0      0       0        0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    3     0    0  0   0     0   0    0    0   0    0     4      866     0      0       0        0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    1     0    0  0   0     0   0    0    0   0    0     4      884     0      2     62007      0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    0     0    0  0   0     0   0    0    0   0    0     4      890     0      1     25073      0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    3     1    0  0   0     0   0    0    0   0    0     4      891     0      1     24613      0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    1     0    0  0   0     0   0    0    0   0    0     4      884     0      0       0        0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37  
    0     0    0  0   0     0   0    0    0   0    0     4      868     0      0       0        0   0   0    0      0  65   0   0  102   0   100.00%  65  100.00%  167    65  65  37