hadoop mfs
The hadoop mfs
command performs operations on
directories in the cluster.
The main purposes of hadoop
mfs
are to display directory information and contents, to
create symbolic links, to set, get, and remove Access Control Expressions (ACE) on
files and directories, and to set compression and chunk size on a
directory.
Syntax
hadoop mfs
[ -delace [-R] <path> ]
[ -getace [-R] <path> ]
[ -help <command> ]
[ -ln <target> <symlink> ]
[ -ls <path> ]
[ -lsd <path> ]
[ -lsr <path> ]
[ -Lsr <path> ]
[ -lsrv <path> ]
[ -lss <path> ]
[ -rmr <path> ]
[ -setace [-R]
[-readfile <ace>] [-writefile <ace>] [-executefile <ace>]
[-addchild <ace>] [-deletechild <ace>] [-lookupdir <ace>] [-readdir <ace>]
[-aces "[rf:<ace>],[wf:<ace>],[ef:<ace>],[ac:<ace>],[dc:<ace>],[rd:<ace>],[ld:<ace>]"]
[-preservemodebits <true|false>] [-setinherit <true|false>] <path> ]
[ -setaudit on|off <dir|file|table> ]
[ -setcompression on|off|lzf|lz4|zlib <dir> ]
[ -setchunksize <size> <dir> ]
[ -setnetworkencryption on|off <target> ]
[ -help <command> ]
Parameters
The normal command syntax is to specify a single option from the following table, along with its corresponding arguments. If compression and chunk size are not set explicitly for a given directory, the values are inherited from the parent directory.
Parameter |
Description |
||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
-delace [-R] <path>
|
Deletes all ACEs associated with the specified file or directory and sets
ACEs for the specified file or directory to the default value, which is the
empty string. Here:
NOTE: You cannot delete specific access types with this parameter.
Instead, if necessary, reset the value for the specific access type to
the empty string using -setace parameter. If the empty
string is used to deny a specific type of access, that type of access
will be denied to all users. To deny specific types of access to specific
users only, use the negation operator ( ! ). The mode bits
corresponding to the ACEs being deleted will not change. |
||||||||||||||
-getace [-R] <path>
|
Returns the permissions -- POSIX mode bits and ACEs -- for
the given file or (recursively) for the directory. Recursion is enabled only
if -R is specified; if -R is not specified, returns the permissions for the
given directory only. Here:
NOTE: If one or more ACEs are available for the file or directory, a
plus sign ( + ), which indicates that both ACEs and POSIX
mode bits are set for the given file or directory, will also be returned.
If the ACE on the file or directory is an empty string, the plus sign
will not be returned. |
||||||||||||||
-help <command>
|
Displays help for the hadoop mfs command. |
||||||||||||||
-ln <target> <symlink>
|
Creates a symbolic link <symlink> that points to
the target path <target> , similar to the standard
Linux ln -s command. |
||||||||||||||
-ls <path>
|
Lists files in the directory specified by <path> .
The hadoop mfs -ls command corresponds to the standard
hadoop fs -ls command, but provides the following
additional information:
|
||||||||||||||
-lsd <path>
|
Lists files in the directory specified by <path> ,
and also provides information about the specified directory itself:
|
||||||||||||||
-lsr <path>
|
Lists files in the directory and subdirectories specified by
<path> , recursively, including dereferencing
symbolic links. The hadoop mfs -lsr command corresponds to
the standard hadoop fs -lsr command, but provides the
following additional information:
|
||||||||||||||
-Lsr <path>
|
Equivalent to lsr, but additionally dereferences symbolic links | ||||||||||||||
-lsrv <path>
|
Lists all paths recursively without crossing volume links. | ||||||||||||||
-lss <path>
|
Lists files in the directory specified by <path> ,
with an additional column that displays the number of disk blocks per file.
Disk blocks are 8192 bytes. |
||||||||||||||
-rmr <path> |
Deletes files and directories in the specified path recursively. This option is particularly useful when one or more directories in the specified path contains many (millions of) files. | ||||||||||||||
-setace [-R] [-readfile <ace>] [-writefile <ace>]
[-executefile <ace>] [-addchild <ace>] [-deletechild
<ace>] [-lookupdir <ace>] [-readdir <ace>] [-aces
"[rf:<ace>],[wf:<ace>],[ef:<ace>],[ac:<ace>],[dc:<ace>],[rd:<ace>],[ld:<ace>]"]
[-preservemodebits <true|false>] [-setinherit <true|false>]
<path>
|
Sets or modifies the read, write, and/or execute permissions
for files or directories. This argument will
NOTE: ACEs must be specified immediately after the
Here: -setace parameter. All other arguments, if specified,
must be given after the ACEs.
|
||||||||||||||
-setaudit on|off <dir|file|table>
|
Enables auditing of the specified directory, file, or MapR-DB
table. Enabling auditing of a directory does not enable auditing of files and subdirectories that exist in the directory. You must enable auditing on those existing files and subdirectories. However, any new files and subdirectories that you create will automatically be enabled for auditing. See Checking Whether Auditing is Enabled for a Directory, File, or MapR-DB Table. For operations on the object to be logged, auditing also needs to be enabled on the cluster and the volume in which the object is located. See Enabling Auditing for details. If auditing is enabled for a directory, new files and directories created within that directory are also enabled for auditing. |
||||||||||||||
-setchunksize <size> <dir>
|
Sets the chunk size in bytes for the directory specified in
<dir> . The <size> parameter
must be a multiple of 65536. |
||||||||||||||
-setcompression on|off|lzf|lz4|zlib <dir>
|
Turns compression on or off on the directory specified in
<dir> , and sets the compression type:
|
||||||||||||||
-setnetworkencryption on|off <target>
|
Sets network encryption on or
off for the filesystem object defined in
<target> . The cluster encrypts network target to
or from a file, directory, or MapR table with network security enabled. |
||||||||||||||
-help <command>
|
Displays help for the hadoop mfs command. |
Output
When used with -ls
, -lsd
,
-lsr
, or -lss
, hadoop mfs
displays information about files and directories. For each file or
directory hadoop mfs
displays a line of basic
information followed by lines listing the chunks that make up the
file, in the following format:
{mode} {compression} {encryption} {audit} {replication} {owner} {group} {size} {date}
{chunk size} {name} {chunk} {fid} {host} [{host}...] {chunk} {fid} {host} [{host}...]
...
Volume links are displayed as follows:
{mode} {compression} {encryption} {audit} {replication} {owner} {group} {size} {date}
{chunk size} {name} {chunk} {target volume name} {writability} {fid} -> {fid}
[{host}...]
The following table describes the values:
mode |
A text string indicating the read, write, and execute permissions for the owner, group, and other permissions. See also Managing Permissions. |
compression |
|
encryption | U: unencrypted; E: encrypted |
audit | U: disabled; A: enabled |
replication |
The replication factor of the file (directories display a dash instead) |
owner |
The owner of the file or directory |
group |
The group of the file of directory |
size |
The size of the file or directory |
date |
The date the file or directory was last modified |
chunk size |
The chunk size of the file or directory |
name |
The name of the file or directory |
chunk |
The chunk number. The first chunk is a primary chunk labeled
" |
fid |
The chunk's file ID, which consists of three parts:
NOTE: For volume links, the first
fid is the chunk that
stores the volume link itself; the fid after the arrow
(-> ) is the first chunk in the target volume. |
host |
The host on which the chunk resides. When several hosts are listed, the first host is the first copy of the chunk and subsequent hosts are replicas. |
target volume name |
The name of the volume pointed to by a volume link. |
writability |
Displays whether the volume is writable. |
Examples
View File Information
The hadoop mfs
command is used to view file contents. You can use this
command to check if compression is turned off in a directory or mounted volume. For
example,
# hadoop mfs -ls /
Found 23 items
vrwxr-xr-x Z U U 3 root root 13 2015-11-29 10:24 268435456 /.rw
p mapr.cluster.root writeable 2049.35.16584 -> 2049.16.2 scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
vrwxr-xr-x U U U 3 root root 7 2015-11-28 22:16 67108864 /hbase
p mapr.hbase default 2049.32.16578 -> 2050.16.2 scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
drwxr-xr-x Z U U 3 root root 0 2015-11-29 09:14 268435456 /tmp
p 2049.41.16596 scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
vrwxr-xr-x Z U A 3 root root 1 2015-11-27 22:59 268435456 /user
p users default 2049.36.16586 -> 2055.16.2 scale-50.scale.lab:5660 scale-52.scale.lab:5660 scale-51.scale.lab:5660
drwxr-xr-x Z U U 3 root root 1 2015-11-27 22:37 268435456 /var
p 2049.33.16580 scale-50.scale.lab:5660 scale-51.scale.lab:5660 scale-52.scale.lab:5660
In the above example, the letter Z
indicates LZ4
compression on the directory; the letter U
indicates
that the directory is uncompressed. In the following example, the
listed item is both uncompressed (first U
) and
unencrypted (second U
).
[root@node1-302 ~]# hadoop mfs -ls /hbase
Found 10 items
drwxr-xr-x U U U 3 mapr mapr 3 2015-11-28 12:05 67108864 /hbase/-ROOT-
p 2050.34.3674200 node2-302:5660 node1-302:5660 node3-302:5660
...
Set ACEs
Example 1: The following command shows how to set separate read, write, and execute permissions (using ACE) on a file:
hadoop mfs -setace -readfile p -writefile 'g:group1&!u:user1' -executefile p /file
- Read access is set for owner, owning group, and others.
- Write access is set for none.
- Execute access is set for owner, owning group, others.
hadoop mfs -setace -aces "rf:u:root,wf:group1&!user1,ef:p,rd:u:m7user1" -setinherit true /dir
- Read access is set to owner/user.
- Write access is set to none.
- Execute access set for others.
hadoop mfs -setace -R -aces "rf:p,wf:g:group1&!u:user1,ef:p" -preservemodebits true /dir
When the command shown above runs, the POSIX mode bits are not modified to match the ACE setting.
writefile
, which was set in the first example above, without
removing all other access types associated with the file. The empty string used in the
following example will deny write access to all users, roles, and groups.
hadoop mfs -setace -writefile "" -preservemodebits false /file
When the command shown above runs, the POSIX mode bit for writing to the file is set to 0.
Get ACEs
hadoop mfs -getace /m7user1/file1.txt
Output
Path: /m7user1/file1.txt
readfile: !u:m7user1
writefile: !u:m7user1
executefile: !u:m7user1
mode: ---------
Delete ACEs
hadoop mfs -delace /file
hadoop mfs -delace /dir
hadoop mfs -delace -R /dir