Turning Compression On or Off on Directories

You can turn compression on or off for a given directory in two ways:

  • Set the value of the Compression attribute in the .dfs_attributes file at the top level of the directory.
    • Set Compression=lzf|lz4|zlib to turn compression on for a directory.
    • Set Compression=false to turn compression off for a directory.
  • Use the command hadoop mfs -setcompression on|off/lzf/lz4/zlib <dir>.

If you choose -setcompression on without specifying an algorithm, lz4 is used by default. This algorithm has improved compression speeds for MapR's block size of 64 KB.

Example

Suppose the volume test is NFS-mounted at /mapr/my.cluster.com/projects/test. You can turn off compression by editing the file /mapr/my.cluster.com/projects/test/.dfs_attributes and setting Compression=false. To accomplish the same thing from the hadoop shell, use the following command:

hadoop mfs -setcompression off /projects/test

You can view the compression settings for directories using the hadoop mfs -ls command. For example,

# hadoop mfs -ls /
        Found 23 items
        vrwxr-xr-x Z   - root root         13 2012-04-29 10:24  268435456 /.rw
        p mapr.cluster.root writeable 2049.35.16584 -> 2049.16.2  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        vrwxr-xr-x U   - root root          7 2012-04-28 22:16   67108864 /hbase
        p mapr.hbase default 2049.32.16578 -> 2050.16.2  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        drwxr-xr-x Z   - root root          0 2012-04-29 09:14  268435456 /tmp
        p 2049.41.16596  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
        vrwxr-xr-x Z   - root root          1 2012-04-27 22:59  268435456 /user
        p users default 2049.36.16586 -> 2055.16.2  scale-50.scale.lab:5660 scale-52.scale.lab:5660 scale-51.scale.lab:5660
        drwxr-xr-x Z   - root root          1 2012-04-27 22:37  268435456 /var
        p 2049.33.16580  scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660

The symbols for the various compression settings are explained here:

Symbol

Compression Setting

Z

lz4

z

zlib

L

lzf

U

Uncompressed, or previously compressed by another algorithm