HDFS文件系统JAVA api访问接口（基于hadoop大数据平台）

id="article_content" class="article_content clearfix"> id="content_views" class="htmledit_views">

        在搭建完hadoop大数据系统（我是使用CDH5.16.1进行安装）后，如何访问hdfs文件系统上的数据呢？那当然是通过构建maven项目 使用java api接口进行文件了。为此，特别进行了hdfs文件系统java api访问的整理。下面就附录上我的CDH5.16.1平台上安装的各组件版本说明。说明：如果需要CDH5.16.1安装教程的，可以给我留言哦~

  1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1">
class="hljs-ln-code"> class="hljs-ln-line">jdk 版本：1.8.0_211
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2">
    • class="hljs-ln-code"> class="hljs-ln-line">hadoop 版本：2.6.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">hive 版本：1.1.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">impala 版本：2.12.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">hbase 版本：1.2.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">spark 版本：2.4.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">oozie 版本：4.1.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line">hue 版本：3.9.0
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="9"> class="hljs-ln-code"> class="hljs-ln-line">zk 版本：3.4.5
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="10"> class="hljs-ln-code"> class="hljs-ln-line">kafka 版本：2.1.0
    •  class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

         下面将通过java api进行hdfs的访问，我是通过maven project方式创建的项目，因为这样便于管理。

    项目代码原文件链接下载地址：HDFS文件系统JAVA api访问接口（基于hadoop大数据平台）

    附上核心代码：

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">package com.hadoop.hdfs;
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line"> 
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">import java.io.File;
    4.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">import java.io.FileInputStream;
    5.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">import java.io.IOException;
    6.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">import java.net.URI;
    7.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">import java.net.URISyntaxException;
    8.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line"> 
    9.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="9"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.commons.lang.StringUtils;
    10.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="10"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.conf.Configuration;
    11.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="11"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.BlockLocation;
    12.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="12"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.FSDataInputStream;
    13.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="13"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.FSDataOutputStream;
    14.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="14"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.FileStatus;
    15.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="15"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.FileSystem;
    16.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="16"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.fs.Path;
    17.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="17"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.hadoop.io.IOUtils;
    18.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="18"> class="hljs-ln-code"> class="hljs-ln-line"> 
    19.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="19"> class="hljs-ln-code"> class="hljs-ln-line">public class HdfsApi {
    20.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="20"> class="hljs-ln-code"> class="hljs-ln-line">	private static String HDFS_URL = "hdfs://node-01:8020";
    21.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="21"> class="hljs-ln-code"> class="hljs-ln-line">	private static FileSystem fs = null;
    22.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="22"> class="hljs-ln-code"> class="hljs-ln-line"> 
    23.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="23"> class="hljs-ln-code"> class="hljs-ln-line">	// 获取FS
    24.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="24"> class="hljs-ln-code"> class="hljs-ln-line">	public static FileSystem getFs() throws IOException {
    25.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="25"> class="hljs-ln-code"> class="hljs-ln-line"> 
    26.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="26"> class="hljs-ln-code"> class="hljs-ln-line">		if (fs != null) {
    27.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="27"> class="hljs-ln-code"> class="hljs-ln-line">			return fs;
    28.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="28"> class="hljs-ln-code"> class="hljs-ln-line">		}
    29.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="29"> class="hljs-ln-code"> class="hljs-ln-line"> 
    30.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="30"> class="hljs-ln-code"> class="hljs-ln-line">		// 获取配置文件
    31.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="31"> class="hljs-ln-code"> class="hljs-ln-line">		Configuration conf = new Configuration();
    32.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="32"> class="hljs-ln-code"> class="hljs-ln-line">		if (StringUtils.isBlank(HDFS_URL)) {
    33.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="33"> class="hljs-ln-code"> class="hljs-ln-line">			// 返回默认文件系统 如果在 Hadoop集群下运行，使用此种方法可直接获取默认文件系统
    34.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="34"> class="hljs-ln-code"> class="hljs-ln-line">			try {
    35.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="35"> class="hljs-ln-code"> class="hljs-ln-line">				fs = FileSystem.get(conf);
    36.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="36"> class="hljs-ln-code"> class="hljs-ln-line">			} catch (IOException e) {
    37.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="37"> class="hljs-ln-code"> class="hljs-ln-line">				e.printStackTrace();
    38.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="38"> class="hljs-ln-code"> class="hljs-ln-line">			}
    39.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="39"> class="hljs-ln-code"> class="hljs-ln-line">		} else {
    40.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="40"> class="hljs-ln-code"> class="hljs-ln-line">			// 返回指定的文件系统,如果在本地测试，需要使用此种方法获取文件系统
    41.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="41"> class="hljs-ln-code"> class="hljs-ln-line">			try {
    42.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="42"> class="hljs-ln-code"> class="hljs-ln-line">				URI uri = new URI(HDFS_URL.trim());
    43.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="43"> class="hljs-ln-code"> class="hljs-ln-line">				fs = FileSystem.get(uri, conf);
    44.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="44"> class="hljs-ln-code"> class="hljs-ln-line">			} catch (URISyntaxException | IOException e) {
    45.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="45"> class="hljs-ln-code"> class="hljs-ln-line">				e.printStackTrace();
    46.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="46"> class="hljs-ln-code"> class="hljs-ln-line">			}
    47.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="47"> class="hljs-ln-code"> class="hljs-ln-line">		}
    48.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="48"> class="hljs-ln-code"> class="hljs-ln-line"> 
    49.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="49"> class="hljs-ln-code"> class="hljs-ln-line">		return fs;
    50.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="50"> class="hljs-ln-code"> class="hljs-ln-line">	}
    51.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="51"> class="hljs-ln-code"> class="hljs-ln-line"> 
    52.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="52"> class="hljs-ln-code"> class="hljs-ln-line">	// 关闭hdfs文件系统
    53.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="53"> class="hljs-ln-code"> class="hljs-ln-line">	public static void closeFs() throws IOException {
    54.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="54"> class="hljs-ln-code"> class="hljs-ln-line">		if (fs != null) {
    55.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="55"> class="hljs-ln-code"> class="hljs-ln-line">			fs.close();
    56.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="56"> class="hljs-ln-code"> class="hljs-ln-line">		}
    57.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="57"> class="hljs-ln-code"> class="hljs-ln-line">	}
    58.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="58"> class="hljs-ln-code"> class="hljs-ln-line"> 
    59.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="59"> class="hljs-ln-code"> class="hljs-ln-line">	// 判断hdfs文件系统，文件是否存在
    60.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="60"> class="hljs-ln-code"> class="hljs-ln-line">	public static boolean fileExists(String dirName) throws IOException {
    61.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="61"> class="hljs-ln-code"> class="hljs-ln-line">		// 获取 FileSystem
    62.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="62"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    63.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="63"> class="hljs-ln-code"> class="hljs-ln-line"> 
    64.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="64"> class="hljs-ln-code"> class="hljs-ln-line">		return fs.exists(new Path(dirName));
    65.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="65"> class="hljs-ln-code"> class="hljs-ln-line">	}
    66.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="66"> class="hljs-ln-code"> class="hljs-ln-line"> 
    67.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="67"> class="hljs-ln-code"> class="hljs-ln-line">	// 读取hdfs文件信息，并打印在控制台
    68.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="68"> class="hljs-ln-code"> class="hljs-ln-line">	public static void readFile(String src) throws IOException {
    69.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="69"> class="hljs-ln-code"> class="hljs-ln-line">		// 获取 FileSystem
    70.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="70"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    71.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="71"> class="hljs-ln-code"> class="hljs-ln-line">		// 读的路径
    72.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="72"> class="hljs-ln-code"> class="hljs-ln-line">		Path readPath = new Path(src);
    73.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="73"> class="hljs-ln-code"> class="hljs-ln-line">		FSDataInputStream inStream = null;
    74.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="74"> class="hljs-ln-code"> class="hljs-ln-line">		try {
    75.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="75"> class="hljs-ln-code"> class="hljs-ln-line">			// 打开输入流
    76.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="76"> class="hljs-ln-code"> class="hljs-ln-line">			inStream = fs.open(readPath);
    77.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="77"> class="hljs-ln-code"> class="hljs-ln-line">			IOUtils.copyBytes(inStream, System.out, 4096, false);
    78.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="78"> class="hljs-ln-code"> class="hljs-ln-line">		} catch (Exception e) {
    79.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="79"> class="hljs-ln-code"> class="hljs-ln-line">			e.printStackTrace();
    80.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="80"> class="hljs-ln-code"> class="hljs-ln-line">		} finally {
    81.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="81"> class="hljs-ln-code"> class="hljs-ln-line">			IOUtils.closeStream(inStream);
    82.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="82"> class="hljs-ln-code"> class="hljs-ln-line">		}
    83.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="83"> class="hljs-ln-code"> class="hljs-ln-line">	}
    84.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="84"> class="hljs-ln-code"> class="hljs-ln-line"> 
    85.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="85"> class="hljs-ln-code"> class="hljs-ln-line">	// 上传文件至hdfs文件系统中
    86.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="86"> class="hljs-ln-code"> class="hljs-ln-line">	public static void copyFileToHDFS(String src, String dst)
    87.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="87"> class="hljs-ln-code"> class="hljs-ln-line">			throws IOException {
    88.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="88"> class="hljs-ln-code"> class="hljs-ln-line">		// 获取filesystem
    89.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="89"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    90.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="90"> class="hljs-ln-code"> class="hljs-ln-line">		// 本地路径
    91.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="91"> class="hljs-ln-code"> class="hljs-ln-line">		File inFile = new File(src);
    92.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="92"> class="hljs-ln-code"> class="hljs-ln-line">		// 目标路径
    93.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="93"> class="hljs-ln-code"> class="hljs-ln-line">		Path outFile = new Path(dst);
    94.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="94"> class="hljs-ln-code"> class="hljs-ln-line">		FileInputStream inStream = null;
    95.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="95"> class="hljs-ln-code"> class="hljs-ln-line">		FSDataOutputStream outStream = null;
    96.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="96"> class="hljs-ln-code"> class="hljs-ln-line">		try {
    97.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="97"> class="hljs-ln-code"> class="hljs-ln-line">			// 打开输入流
    98.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="98"> class="hljs-ln-code"> class="hljs-ln-line">			inStream = new FileInputStream(inFile);
    99.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="99"> class="hljs-ln-code"> class="hljs-ln-line">			// 打开输出流
    100.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="100"> class="hljs-ln-code"> class="hljs-ln-line">			outStream = fs.create(outFile);
    101.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="101"> class="hljs-ln-code"> class="hljs-ln-line"> 
    102.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="102"> class="hljs-ln-code"> class="hljs-ln-line">			IOUtils.copyBytes(inStream, outStream, 4096, false);
    103.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="103"> class="hljs-ln-code"> class="hljs-ln-line">		} catch (Exception e) {
    104.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="104"> class="hljs-ln-code"> class="hljs-ln-line">			// TODO: handle exception
    105.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="105"> class="hljs-ln-code"> class="hljs-ln-line">		} finally {
    106.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="106"> class="hljs-ln-code"> class="hljs-ln-line">			IOUtils.closeStream(inStream);
    107.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="107"> class="hljs-ln-code"> class="hljs-ln-line">			IOUtils.closeStream(outStream);
    108.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="108"> class="hljs-ln-code"> class="hljs-ln-line">		}
    109.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="109"> class="hljs-ln-code"> class="hljs-ln-line">	}
    110.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="110"> class="hljs-ln-code"> class="hljs-ln-line"> 
    111.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="111"> class="hljs-ln-code"> class="hljs-ln-line">	// 从hdfs文件系统中下载文件
    112.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="112"> class="hljs-ln-code"> class="hljs-ln-line">	public static void downLoadFromHDFS(String src, String dst)
    113.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="113"> class="hljs-ln-code"> class="hljs-ln-line">			throws IOException {
    114.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="114"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    115.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="115"> class="hljs-ln-code"> class="hljs-ln-line">		// 设置下载地址和目标地址
    116.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="116"> class="hljs-ln-code"> class="hljs-ln-line">		fs.copyToLocalFile(false, new Path(src), new Path(dst), true);
    117.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="117"> class="hljs-ln-code"> class="hljs-ln-line">	}
    118.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="118"> class="hljs-ln-code"> class="hljs-ln-line"> 
    119.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="119"> class="hljs-ln-code"> class="hljs-ln-line">	// 对hdfs文件系统中的文件,进行重命名和移动
    120.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="120"> class="hljs-ln-code"> class="hljs-ln-line">	public static void renameMV(String src, String dst) throws IOException {
    121.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="121"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    122.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="122"> class="hljs-ln-code"> class="hljs-ln-line">		fs.rename(new Path(src), new Path(dst));
    123.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="123"> class="hljs-ln-code"> class="hljs-ln-line">	}
    124.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="124"> class="hljs-ln-code"> class="hljs-ln-line"> 
    125.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="125"> class="hljs-ln-code"> class="hljs-ln-line">	// 对hdfs文件系统中的文件,进行删除
    126.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="126"> class="hljs-ln-code"> class="hljs-ln-line">	public static void delete(String fileName) throws IOException {
    127.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="127"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    128.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="128"> class="hljs-ln-code"> class="hljs-ln-line">		fs.deleteOnExit(new Path(fileName));
    129.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="129"> class="hljs-ln-code"> class="hljs-ln-line">	}
    130.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="130"> class="hljs-ln-code"> class="hljs-ln-line"> 
    131.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="131"> class="hljs-ln-code"> class="hljs-ln-line">	// 对hdfs文件系统中的文件,获取文件列表
    132.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="132"> class="hljs-ln-code"> class="hljs-ln-line">	public static FileStatus[] listFile(String dirName) throws IOException {
    133.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="133"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    134.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="134"> class="hljs-ln-code"> class="hljs-ln-line">		FileStatus[] fileStatuses = fs.listStatus(new Path(dirName));
    135.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="135"> class="hljs-ln-code"> class="hljs-ln-line">		// for (FileStatus fileName : fileStatuses) {
    136.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="136"> class="hljs-ln-code"> class="hljs-ln-line">		// System.out.println(fileName.getPath().getName());
    137.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="137"> class="hljs-ln-code"> class="hljs-ln-line">		// }
    138.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="138"> class="hljs-ln-code"> class="hljs-ln-line">		return fileStatuses;
    139.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="139"> class="hljs-ln-code"> class="hljs-ln-line">	}
    140.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="140"> class="hljs-ln-code"> class="hljs-ln-line"> 
    141.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="141"> class="hljs-ln-code"> class="hljs-ln-line">	/**
    142.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="142"> class="hljs-ln-code"> class="hljs-ln-line">	 * 查找某个文件在 HDFS集群的位置
    143.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="143"> class="hljs-ln-code"> class="hljs-ln-line">	 * 
    144.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="144"> class="hljs-ln-code"> class="hljs-ln-line">	 * @param filePath
    145.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="145"> class="hljs-ln-code"> class="hljs-ln-line">	 * @return BlockLocation[]
    146.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="146"> class="hljs-ln-code"> class="hljs-ln-line">	 */
    147.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="147"> class="hljs-ln-code"> class="hljs-ln-line">	public static BlockLocation[] getFileBlockLocations(String filePath) {
    148.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="148"> class="hljs-ln-code"> class="hljs-ln-line">		// 文件路径
    149.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="149"> class="hljs-ln-code"> class="hljs-ln-line">		Path path = new Path(filePath);
    150.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="150"> class="hljs-ln-code"> class="hljs-ln-line"> 
    151.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="151"> class="hljs-ln-code"> class="hljs-ln-line">		// 文件块位置列表
    152.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="152"> class="hljs-ln-code"> class="hljs-ln-line">		BlockLocation[] blkLocations = new BlockLocation[0];
    153.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="153"> class="hljs-ln-code"> class="hljs-ln-line">		try {
    154.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="154"> class="hljs-ln-code"> class="hljs-ln-line">			// 返回FileSystem对象
    155.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="155"> class="hljs-ln-code"> class="hljs-ln-line">			FileSystem fs = getFs();
    156.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="156"> class="hljs-ln-code"> class="hljs-ln-line">			// 获取文件目录
    157.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="157"> class="hljs-ln-code"> class="hljs-ln-line">			FileStatus filestatus = fs.getFileStatus(path);
    158.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="158"> class="hljs-ln-code"> class="hljs-ln-line">			// 获取文件块位置列表
    159.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="159"> class="hljs-ln-code"> class="hljs-ln-line">			blkLocations = fs.getFileBlockLocations(filestatus, 0,
    160.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="160"> class="hljs-ln-code"> class="hljs-ln-line">					filestatus.getLen());
    161.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="161"> class="hljs-ln-code"> class="hljs-ln-line">		} catch (IOException e) {
    162.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="162"> class="hljs-ln-code"> class="hljs-ln-line">			e.printStackTrace();
    163.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="163"> class="hljs-ln-code"> class="hljs-ln-line">		}
    164.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="164"> class="hljs-ln-code"> class="hljs-ln-line">		return blkLocations;
    165.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="165"> class="hljs-ln-code"> class="hljs-ln-line">	}
    166.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="166"> class="hljs-ln-code"> class="hljs-ln-line"> 
    167.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="167"> class="hljs-ln-code"> class="hljs-ln-line">	// 对hdfs文件系统中的文件,创建目录
    168.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="168"> class="hljs-ln-code"> class="hljs-ln-line">	public static void mkdir(String dirName) throws IOException {
    169.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="169"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    170.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="170"> class="hljs-ln-code"> class="hljs-ln-line"> 
    171.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="171"> class="hljs-ln-code"> class="hljs-ln-line">		if (fs.exists(new Path(dirName))) {
    172.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="172"> class="hljs-ln-code"> class="hljs-ln-line">			System.out.println("Directory already exists!");
    173.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="173"> class="hljs-ln-code"> class="hljs-ln-line">			fs.close();
    174.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="174"> class="hljs-ln-code"> class="hljs-ln-line">			return;
    175.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="175"> class="hljs-ln-code"> class="hljs-ln-line">		}
    176.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="176"> class="hljs-ln-code"> class="hljs-ln-line">		fs.mkdirs(new Path(dirName));
    177.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="177"> class="hljs-ln-code"> class="hljs-ln-line">	}
    178.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="178"> class="hljs-ln-code"> class="hljs-ln-line"> 
    179.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="179"> class="hljs-ln-code"> class="hljs-ln-line">	// 对hdfs文件系统中的文件,删除目录
    180.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="180"> class="hljs-ln-code"> class="hljs-ln-line">	public static void deletedir(String dirName, boolean recursive)
    181.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="181"> class="hljs-ln-code"> class="hljs-ln-line">			throws IOException {
    182.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="182"> class="hljs-ln-code"> class="hljs-ln-line">		FileSystem fs = getFs();
    183.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="183"> class="hljs-ln-code"> class="hljs-ln-line">		// true表示递归删除目录下所有文件
    184.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="184"> class="hljs-ln-code"> class="hljs-ln-line">		// false就只能删除空目录
    185.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="185"> class="hljs-ln-code"> class="hljs-ln-line">		fs.delete(new Path(dirName), recursive);
    186.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="186"> class="hljs-ln-code"> class="hljs-ln-line">	}
    187.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="187"> class="hljs-ln-code"> class="hljs-ln-line"> 
    188.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="188"> class="hljs-ln-code"> class="hljs-ln-line">	public static void main(String[] args) throws IOException {
    189.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="189"> class="hljs-ln-code"> class="hljs-ln-line">		HdfsApi.getFs();
    190.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="190"> class="hljs-ln-code"> class="hljs-ln-line"> 
    191.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="191"> class="hljs-ln-code"> class="hljs-ln-line">		// boolean fileExist = fileExists("/user/root/spark.txt");
    192.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="192"> class="hljs-ln-code"> class="hljs-ln-line">		// System.out.println(fileExist);
    193.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="193"> class="hljs-ln-code"> class="hljs-ln-line">		//
    194.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="194"> class="hljs-ln-code"> class="hljs-ln-line">		// readFile("/user/root/spark.txt");
    195.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="195"> class="hljs-ln-code"> class="hljs-ln-line"> 
    196.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="196"> class="hljs-ln-code"> class="hljs-ln-line">		// copyFileToHDFS();
    197.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="197"> class="hljs-ln-code"> class="hljs-ln-line"> 
    198.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="198"> class="hljs-ln-code"> class="hljs-ln-line">		// copyFileToHDFS("d://test.jpg", "/user/root/up_test.jpg");
    199.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="199"> class="hljs-ln-code"> class="hljs-ln-line"> 
    200.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="200"> class="hljs-ln-code"> class="hljs-ln-line">		// downLoadFromHDFS("/user/root/up_test.jpg", "d://down_test.jpg");
    201.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="201"> class="hljs-ln-code"> class="hljs-ln-line"> 
    202.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="202"> class="hljs-ln-code"> class="hljs-ln-line">		// renameMV("/user/root/up_test.jpg", "/user/root/up_test_rename.jpg");
    203.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="203"> class="hljs-ln-code"> class="hljs-ln-line"> 
    204.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="204"> class="hljs-ln-code"> class="hljs-ln-line">		// delete("/user/root/up_test_rename.jpg");
    205.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="205"> class="hljs-ln-code"> class="hljs-ln-line"> 
    206.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="206"> class="hljs-ln-code"> class="hljs-ln-line">		listFile("/user/root/");
    207.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="207"> class="hljs-ln-code"> class="hljs-ln-line"> 
    208.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="208"> class="hljs-ln-code"> class="hljs-ln-line">		// BlockLocation[] blockBlockLocations =
    209.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="209"> class="hljs-ln-code"> class="hljs-ln-line">		// getFileBlockLocations("/user/root/spark.txt");
    210.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="210"> class="hljs-ln-code"> class="hljs-ln-line">		// for (BlockLocation block: blockBlockLocations){
    211.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="211"> class="hljs-ln-code"> class="hljs-ln-line">		// System.out.println(block.toString());
    212.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="212"> class="hljs-ln-code"> class="hljs-ln-line">		// }
    213.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="213"> class="hljs-ln-code"> class="hljs-ln-line"> 
    214.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="214"> class="hljs-ln-code"> class="hljs-ln-line">		// mkdir("/user/root/mkdir_Test");
    215.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="215"> class="hljs-ln-code"> class="hljs-ln-line"> 
    216.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="216"> class="hljs-ln-code"> class="hljs-ln-line">		// deletedir("/user/root/mkdir_Test", true);
    217.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="217"> class="hljs-ln-code"> class="hljs-ln-line"> 
    218.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="218"> class="hljs-ln-code"> class="hljs-ln-line">		HdfsApi.closeFs();
    219.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="219"> class="hljs-ln-code"> class="hljs-ln-line">	}
    220.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="220"> class="hljs-ln-code"> class="hljs-ln-line"> 
    221.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="221"> class="hljs-ln-code"> class="hljs-ln-line">}
     class="hide-preCode-box"> class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    文章最后，给大家推荐一些受欢迎的技术博客链接：

    1. JAVA相关的深度技术博客链接
    2. Flinak 相关技术博客链接
    3. Spark 核心技术链接
    4. 设计模式 —— 深度技术博客链接
    5. 机器学习 —— 深度技术博客链接
    6. Hadoop相关技术博客链接
    7. 超全干货--Flink思维导图，花了3周左右编写、校对
    8. 深入JAVA 的JVM核心原理解决线上各种故障【附案例】
    9. 请谈谈你对volatile的理解？--最近小李子与面试官的一场“硬核较量”
    10. 聊聊RPC通信，经常被问到的一道面试题。源码+笔记，包懂
    11. 深入聊聊Java 垃圾回收机制【附原理图及调优方法】

    欢迎扫描下方的二维码或 搜索 公众号“大数据高级架构师”，我们会有更多、且及时的资料推送给您，欢迎多多交流！

                                               

           

    >>
    注：本文转载自blog.csdn.net的不埋雷的探长的文章"https://blog.csdn.net/weixin_32265569/article/details/91473453"。版权归原作者所有，此博客不拥有其著作权，亦不承担相应法律责任。如有侵权，请联系我们删除。
    复制链接
    复制链接
    相关推荐
    发表评论

    评论记录：

    未查询到任何数据！