大数据（一）MaxCompute

id="article_content" class="article_content clearfix"> id="content_views" class="htmledit_views">

一、引言

        作者后面会使用MaxCompute，所以在进行学习研究，总会有一些疑问产生，这里讲讲作者的疑问和思路

二、介绍

MaxCompute（原名 ODPS - Open Data Processing Service）是阿里云提供的大数据处理平台，专门用于批量数据存储和大规模并行计算。它广泛应用于数据分析和处理任务，为企业级数据处理提供高效的解决方案。

下面是 MaxCompute 的一些主要功能和应用场景：

  1. 大规模数据存储：MaxCompute 提供高效的数据存储解决方案，可以存储 PB 级别的数据，支持结构化和半结构化数据。它的存储系统具有高可用性和高可靠性。

  2. 大数据计算引擎：MaxCompute 提供了强大的计算能力，能够处理大规模数据集。它支持 SQL 、MapReduce 、Graph 和 UDF（用户自定义函数）等多种计算模型，为复杂数据分析提供支持。

  3. 高效的资源管理：MaxCompute 通过资源组、任务队列等资源管理机制，能够实现资源的高效调度和利用，确保计算任务的高效进行。

  4. 安全和权限管理：MaxCompute 提供了完善的安全机制，包括数据访问权限控制、数据加密等，确保数据的安全性。

  5. 数据处理和分析：适用于多种数据处理和分析任务，包括数据清洗、预处理、数据挖掘、统计分析等。它可以与阿里云其他大数据产品（如 DataWorks 、AnalyticDB）集成，形成完整的大数据解决方案。

  6. 扩展性和高可用性：MaxCompute 具有很强的扩展性，可以根据需要动态扩展计算资源。此外，它具有高可用性设计，确保服务的稳定运行。

  7. 多语言支持：MaxCompute 支持多种编程语言，如 SQL 、Python 、Java 等，方便用户编写数据处理和分析任务

三、问答

看了一段时间官方文档之后，常见的SQL问题以及优化示例_云原生大数据计算服务 MaxCompute(MaxCompute)-阿里云帮助中心作者有很多疑问

 1、长尾问题是什么？

一般来说数据分布中存在少量出现频率非常高的“热点”数据，而同时也有大量出现频率低但多样性高的“长尾”数据

但是MaxCompute将Fuxi Instance 耗费时长高于平均值 2倍的实例判定为长尾

也就是将任务分配和执行的引擎实例时间执行过长的认为长尾

比如这个sql他就举例说会引起长尾问题

SELECT shop_id,sum(is_open) AS 营业天数 FROM table_xxx_diWHERE dt BETWEEN 'bizdate365′AND′bizdate3​65′AND′{bizdate}'GROUP BY shop_id; class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

按照社区请教了一些朋友，这种可能是数据倾斜造成，比如一些 shop_id 的数据量远大于其他 shop_id，GROUP BY shop_id的处理时间会明显超过平均值，从而导致长尾现象

解决方案一般是

  1. 数据预处理和分区调整：

    尝试在数据写入阶段使用均匀分布的分区键，如果可能，重新设计数据模型以减轻倾斜。

  2. 计算逻辑优化：

    • 将复杂计算在 ETL 阶段提前处理或者分解为多个合理的小任务。
    • 使用窗口函数或其他优化 SQL 技术来减少不必要的数据重计算。

这些说起来都比较笼统，作者对于这个长尾还是有一些模糊

比如数据是

class="table-box">
transaction_iduser_iditem_idamountcategory_id
1100150110010
2100250215010
3100350320020
...............
99999100150410010

sql查询是

  1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1">
class="hljs-ln-code"> class="hljs-ln-line">SELECT user_id, COUNT(*) AS transaction_count
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">FROM transactions
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">GROUP BY user_id
  • class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">ORDER BY transaction_count DESC;
    • class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    这样会因为user_id 1001 存在大量记录（数据倾斜），导致数据在计算节点上分布不均。部分计算节点会因为数据量过大而成为瓶颈，其他节点却几乎无事可做。这会拖慢整个查询的执行时间，造成长尾问题

    对应的解决方案是

    1. 盐值处理（作者觉得类似hash进行key的分散）

    将热点用户的记录通过添加附加前缀或后缀进行分散，来分配给不同的计算节点。

    修改 SQL

    首先在数据预处理时对user_id进行变换：

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">INSERT INTO transactions_with_salt
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">SELECT CONCAT(user_id, '_', CAST(RAND() * 10 AS INT)) AS user_salt, amount
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">FROM transactions;
     class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    然后在查询时进行逆变换处理：

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">SELECT SUBSTR(user_salt, 0, LENGTH(user_salt) - 2) AS user_id,
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">       SUM(transaction_count) AS total_transactions
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">FROM (
    4.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">    SELECT user_salt, COUNT(*) AS transaction_count
    5.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">    FROM transactions_with_salt
    6.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">    GROUP BY user_salt
    7.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">) AS salted_data
    8.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line">GROUP BY SUBSTR(user_salt, 0, LENGTH(user_salt) - 2)
    9.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="9"> class="hljs-ln-code"> class="hljs-ln-line">ORDER BY total_transactions DESC;
     class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    这样相当于是多遍计算，因为要进行随机化再分组，然后把结果给去除之前处理进行重新分组，但是把计算压力给到了多个节点

    2. 分区与并行化

    对user_id分区或者引入合适的分区键，比如category_id，以在存储阶段给予更好的数据分配

    2、为什么MapReduce需要本地Combiner？

    看到示例的MapReduce，作息的理解是进行分布式的数据录入处理，然后进行集中处理，感觉很像是java的Fork/Join 框架进行分治

    作者理解是吧线程映射成了机器，进行分布式处理然后合并

    但是他为什么有个本地Combiner呢？

    作者一开始是不理解的，后面再思考发现最明显的是作用范围不一样，

    Combiner作用于单个 Mapper 节点的本地数据，仅对本节点的数据进行局部汇总，他主要是减少数据传输，本地先算一下会快很多

    Reducer对所有 Mapper 节点输出的数据进行处理

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">package com.aliyun.odps.mapred.open.example;
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">import java.io.IOException;
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">import java.util.Iterator;
    4.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.data.Record;
    5.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.data.TableInfo;
    6.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.JobClient;
    7.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.MapperBase;
    8.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.ReducerBase;
    9.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="9"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.conf.JobConf;
    10.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="10"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.utils.InputUtils;
    11.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="11"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.utils.OutputUtils;
    12.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="12"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.mapred.utils.SchemaUtils;
    13.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="13"> class="hljs-ln-code"> class="hljs-ln-line">public class WordCount {
    14.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="14"> class="hljs-ln-code"> class="hljs-ln-line">    public static class TokenizerMapper extends MapperBase {
    15.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="15"> class="hljs-ln-code"> class="hljs-ln-line">        private Record word;
    16.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="16"> class="hljs-ln-code"> class="hljs-ln-line">        private Record one;
    17.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="17"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    18.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="18"> class="hljs-ln-code"> class="hljs-ln-line">            public void setup(TaskContext context) throws IOException {
    19.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="19"> class="hljs-ln-code"> class="hljs-ln-line">            word = context.createMapOutputKeyRecord();
    20.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="20"> class="hljs-ln-code"> class="hljs-ln-line">            one = context.createMapOutputValueRecord();
    21.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="21"> class="hljs-ln-code"> class="hljs-ln-line">            one.set(new Object[] { 1L });
    22.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="22"> class="hljs-ln-code"> class="hljs-ln-line">            System.out.println("TaskID:" + context.getTaskID().toString());
    23.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="23"> class="hljs-ln-code"> class="hljs-ln-line">        }
    24.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="24"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    25.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="25"> class="hljs-ln-code"> class="hljs-ln-line">            public void map(long recordNum, Record record, TaskContext context)
    26.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="26"> class="hljs-ln-code"> class="hljs-ln-line">            throws IOException {
    27.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="27"> class="hljs-ln-code"> class="hljs-ln-line">            for (int i = 0; i < record.getColumnCount(); i++) {
    28.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="28"> class="hljs-ln-code"> class="hljs-ln-line">                word.set(new Object[] { record.get(i).toString() });
    29.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="29"> class="hljs-ln-code"> class="hljs-ln-line">                context.write(word, one);
    30.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="30"> class="hljs-ln-code"> class="hljs-ln-line">            }
    31.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="31"> class="hljs-ln-code"> class="hljs-ln-line">        }
    32.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="32"> class="hljs-ln-code"> class="hljs-ln-line">    }
    33.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="33"> class="hljs-ln-code"> class="hljs-ln-line">    /**
    34.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="34"> class="hljs-ln-code"> class="hljs-ln-line">       * A combiner class that combines map output by sum them.
    35.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="35"> class="hljs-ln-code"> class="hljs-ln-line">       **/
    36.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="36"> class="hljs-ln-code"> class="hljs-ln-line">    public static class SumCombiner extends ReducerBase {
    37.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="37"> class="hljs-ln-code"> class="hljs-ln-line">        private Record count;
    38.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="38"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    39.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="39"> class="hljs-ln-code"> class="hljs-ln-line">            public void setup(TaskContext context) throws IOException {
    40.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="40"> class="hljs-ln-code"> class="hljs-ln-line">            count = context.createMapOutputValueRecord();
    41.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="41"> class="hljs-ln-code"> class="hljs-ln-line">        }
    42.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="42"> class="hljs-ln-code"> class="hljs-ln-line">        /**Combiner实现的接口和Reducer一样，是可以立即在Mapper本地执行的一个Reduce，作用是减少Mapper的输出量。*/
    43.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="43"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    44.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="44"> class="hljs-ln-code"> class="hljs-ln-line">            public void reduce(Record key, Iterator values, TaskContext context)
    45.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="45"> class="hljs-ln-code"> class="hljs-ln-line">            throws IOException {
    46.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="46"> class="hljs-ln-code"> class="hljs-ln-line">            long c = 0;
    47.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="47"> class="hljs-ln-code"> class="hljs-ln-line">            while (values.hasNext()) {
    48.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="48"> class="hljs-ln-code"> class="hljs-ln-line">                Record val = values.next();
    49.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="49"> class="hljs-ln-code"> class="hljs-ln-line">                c += (Long) val.get(0);
    50.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="50"> class="hljs-ln-code"> class="hljs-ln-line">            }
    51.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="51"> class="hljs-ln-code"> class="hljs-ln-line">            count.set(0, c);
    52.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="52"> class="hljs-ln-code"> class="hljs-ln-line">            context.write(key, count);
    53.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="53"> class="hljs-ln-code"> class="hljs-ln-line">        }
    54.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="54"> class="hljs-ln-code"> class="hljs-ln-line">    }
    55.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="55"> class="hljs-ln-code"> class="hljs-ln-line">    /**
    56.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="56"> class="hljs-ln-code"> class="hljs-ln-line">       * A reducer class that just emits the sum of the input values.
    57.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="57"> class="hljs-ln-code"> class="hljs-ln-line">       **/
    58.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="58"> class="hljs-ln-code"> class="hljs-ln-line">    public static class SumReducer extends ReducerBase {
    59.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="59"> class="hljs-ln-code"> class="hljs-ln-line">        private Record result = null;
    60.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="60"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    61.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="61"> class="hljs-ln-code"> class="hljs-ln-line">            public void setup(TaskContext context) throws IOException {
    62.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="62"> class="hljs-ln-code"> class="hljs-ln-line">            result = context.createOutputRecord();
    63.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="63"> class="hljs-ln-code"> class="hljs-ln-line">        }
    64.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="64"> class="hljs-ln-code"> class="hljs-ln-line">        @Override
    65.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="65"> class="hljs-ln-code"> class="hljs-ln-line">            public void reduce(Record key, Iterator values, TaskContext context)
    66.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="66"> class="hljs-ln-code"> class="hljs-ln-line">            throws IOException {
    67.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="67"> class="hljs-ln-code"> class="hljs-ln-line">            long count = 0;
    68.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="68"> class="hljs-ln-code"> class="hljs-ln-line">            while (values.hasNext()) {
    69.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="69"> class="hljs-ln-code"> class="hljs-ln-line">                Record val = values.next();
    70.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="70"> class="hljs-ln-code"> class="hljs-ln-line">                count += (Long) val.get(0);
    71.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="71"> class="hljs-ln-code"> class="hljs-ln-line">            }
    72.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="72"> class="hljs-ln-code"> class="hljs-ln-line">            result.set(0, key.get(0));
    73.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="73"> class="hljs-ln-code"> class="hljs-ln-line">            result.set(1, count);
    74.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="74"> class="hljs-ln-code"> class="hljs-ln-line">            context.write(result);
    75.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="75"> class="hljs-ln-code"> class="hljs-ln-line">        }
    76.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="76"> class="hljs-ln-code"> class="hljs-ln-line">    }
    77.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="77"> class="hljs-ln-code"> class="hljs-ln-line">    public static void main(String[] args) throws Exception {
    78.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="78"> class="hljs-ln-code"> class="hljs-ln-line">        if (args.length != 2) {
    79.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="79"> class="hljs-ln-code"> class="hljs-ln-line">            System.err.println("Usage: WordCount  ");
    80.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="80"> class="hljs-ln-code"> class="hljs-ln-line">            System.exit(2);
    81.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="81"> class="hljs-ln-code"> class="hljs-ln-line">        }
    82.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="82"> class="hljs-ln-code"> class="hljs-ln-line">        JobConf job = new JobConf();
    83.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="83"> class="hljs-ln-code"> class="hljs-ln-line">        job.setMapperClass(TokenizerMapper.class);
    84.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="84"> class="hljs-ln-code"> class="hljs-ln-line">        job.setCombinerClass(SumCombiner.class);
    85.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="85"> class="hljs-ln-code"> class="hljs-ln-line">        job.setReducerClass(SumReducer.class);
    86.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="86"> class="hljs-ln-code"> class="hljs-ln-line">        /**设置Mapper中间结果的key和value的Schema, Mapper的中间结果输出也是Record的形式。*/
    87.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="87"> class="hljs-ln-code"> class="hljs-ln-line">        job.setMapOutputKeySchema(SchemaUtils.fromString("word:string"));
    88.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="88"> class="hljs-ln-code"> class="hljs-ln-line">        job.setMapOutputValueSchema(SchemaUtils.fromString("count:bigint"));
    89.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="89"> class="hljs-ln-code"> class="hljs-ln-line">        /**设置输入和输出的表信息。*/
    90.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="90"> class="hljs-ln-code"> class="hljs-ln-line">        InputUtils.addTable(TableInfo.builder().tableName(args[0]).build(), job);
    91.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="91"> class="hljs-ln-code"> class="hljs-ln-line">        OutputUtils.addTable(TableInfo.builder().tableName(args[1]).build(), job);
    92.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="92"> class="hljs-ln-code"> class="hljs-ln-line">        JobClient.runJob(job);
    93.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="93"> class="hljs-ln-code"> class="hljs-ln-line">    }
    94.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="94"> class="hljs-ln-code"> class="hljs-ln-line">}
     class="hide-preCode-box"> class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    3、Graph图识别使用场景？

    看到他有个图识别的功能，作者挺感兴趣的，主要是想看看他在什么场景下可以使用

    不过看起来也不多，程序示例只有三个，比较经典的是PageRank识别

    看着里面的代码还有点疑惑，比如hasEdges这个没有具体的实现，想了一下应该是Vertex这个父类里面应该做了一些通用的实现

    PageRank(i)为什么等于 0.15/TotalNumVertices+0.85*sum？

    这也是一个不容易理解的点，他会考虑其连接的其他页面对其排名的“投票”权重，同时也引入了一个“随机跳转”的概率

    作者想的是为什么会有自由跳转，每个页面被多少页面引用不是应该是确定的嘛？

    作者理解他主要是防呆的

    1. 防止死循环和陷阱页面：

      在某些情况下，一个算法可能会遇到“死循环”或者陷入一个“陷阱”。例如，一个圈子内的页面互相链接，但与外界没有链接。在没有自由跳转的情况下，PageRank 值可能会无限在这个局部循环中传播，而无法扩展到其他页面。

    2. 处理无出链接页面（dead-ends）：

    3. 算法的数学稳定性：

    4. 现实浏览行为的模拟：

    sendMessageToNeighbors会将节点的 PageRank 值发送到所有直接连接的邻居节点，作者又有问题了：如果相邻节点已经计算过，发送过去有什么用呢？

    作者理解这就是区分了三个if的迭代过程，算好的值还会接受相邻节点再进行结合计算

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">import java.io.IOException;
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line"> 
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">import org.apache.log4j.Logger;
    4.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line"> 
    5.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.WritableRecord;
    6.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.ComputeContext;
    7.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.GraphJob;
    8.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.GraphLoader;
    9.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="9"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.MutationContext;
    10.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="10"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.Vertex;
    11.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="11"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.graph.WorkerContext;
    12.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="12"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.DoubleWritable;
    13.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="13"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.LongWritable;
    14.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="14"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.NullWritable;
    15.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="15"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.data.TableInfo;
    16.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="16"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.Text;
    17.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="17"> class="hljs-ln-code"> class="hljs-ln-line">import com.aliyun.odps.io.Writable;
    18.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="18"> class="hljs-ln-code"> class="hljs-ln-line"> 
    19.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="19"> class="hljs-ln-code"> class="hljs-ln-line">public class PageRank {
    20.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="20"> class="hljs-ln-code"> class="hljs-ln-line"> 
    21.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="21"> class="hljs-ln-code"> class="hljs-ln-line">  private final static Logger LOG = Logger.getLogger(PageRank.class);
    22.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="22"> class="hljs-ln-code"> class="hljs-ln-line"> 
    23.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="23"> class="hljs-ln-code"> class="hljs-ln-line">  public static class PageRankVertex extends
    24.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="24"> class="hljs-ln-code"> class="hljs-ln-line">      Vertex {
    25.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="25"> class="hljs-ln-code"> class="hljs-ln-line"> 
    26.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="26"> class="hljs-ln-code"> class="hljs-ln-line">    @Override
    27.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="27"> class="hljs-ln-code"> class="hljs-ln-line">    public void compute(
    28.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="28"> class="hljs-ln-code"> class="hljs-ln-line">        ComputeContext context,
    29.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="29"> class="hljs-ln-code"> class="hljs-ln-line">        Iterable messages) throws IOException {
    30.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="30"> class="hljs-ln-code"> class="hljs-ln-line">      if (context.getSuperstep() == 0) {
    31.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="31"> class="hljs-ln-code"> class="hljs-ln-line">        setValue(new DoubleWritable(1.0 / context.getTotalNumVertices()));
    32.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="32"> class="hljs-ln-code"> class="hljs-ln-line">      } else if (context.getSuperstep() >= 1) {
    33.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="33"> class="hljs-ln-code"> class="hljs-ln-line">        double sum = 0;
    34.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="34"> class="hljs-ln-code"> class="hljs-ln-line">        for (DoubleWritable msg : messages) {
    35.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="35"> class="hljs-ln-code"> class="hljs-ln-line">          sum += msg.get();
    36.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="36"> class="hljs-ln-code"> class="hljs-ln-line">        }
    37.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="37"> class="hljs-ln-code"> class="hljs-ln-line">        DoubleWritable vertexValue = new DoubleWritable(
    38.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="38"> class="hljs-ln-code"> class="hljs-ln-line">            (0.15f / context.getTotalNumVertices()) + 0.85f * sum);
    39.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="39"> class="hljs-ln-code"> class="hljs-ln-line">        setValue(vertexValue);
    40.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="40"> class="hljs-ln-code"> class="hljs-ln-line">      }
    41.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="41"> class="hljs-ln-code"> class="hljs-ln-line">      if (hasEdges()) {
    42.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="42"> class="hljs-ln-code"> class="hljs-ln-line">        context.sendMessageToNeighbors(this, new DoubleWritable(getValue()
    43.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="43"> class="hljs-ln-code"> class="hljs-ln-line">            .get() / getEdges().size()));
    44.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="44"> class="hljs-ln-code"> class="hljs-ln-line">      }
    45.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="45"> class="hljs-ln-code"> class="hljs-ln-line">    }
    46.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="46"> class="hljs-ln-code"> class="hljs-ln-line"> 
    47.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="47"> class="hljs-ln-code"> class="hljs-ln-line">    @Override
    48.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="48"> class="hljs-ln-code"> class="hljs-ln-line">    public void cleanup(
    49.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="49"> class="hljs-ln-code"> class="hljs-ln-line">        WorkerContext context)
    50.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="50"> class="hljs-ln-code"> class="hljs-ln-line">        throws IOException {
    51.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="51"> class="hljs-ln-code"> class="hljs-ln-line">      context.write(getId(), getValue());
    52.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="52"> class="hljs-ln-code"> class="hljs-ln-line">    }
    53.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="53"> class="hljs-ln-code"> class="hljs-ln-line">  }
    54.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="54"> class="hljs-ln-code"> class="hljs-ln-line"> 
    55.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="55"> class="hljs-ln-code"> class="hljs-ln-line">  public static class PageRankVertexReader extends
    56.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="56"> class="hljs-ln-code"> class="hljs-ln-line">      GraphLoader {
    57.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="57"> class="hljs-ln-code"> class="hljs-ln-line"> 
    58.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="58"> class="hljs-ln-code"> class="hljs-ln-line">    @Override
    59.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="59"> class="hljs-ln-code"> class="hljs-ln-line">    public void load(
    60.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="60"> class="hljs-ln-code"> class="hljs-ln-line">        LongWritable recordNum,
    61.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="61"> class="hljs-ln-code"> class="hljs-ln-line">        WritableRecord record,
    62.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="62"> class="hljs-ln-code"> class="hljs-ln-line">        MutationContext context)
    63.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="63"> class="hljs-ln-code"> class="hljs-ln-line">        throws IOException {
    64.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="64"> class="hljs-ln-code"> class="hljs-ln-line">      PageRankVertex vertex = new PageRankVertex();
    65.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="65"> class="hljs-ln-code"> class="hljs-ln-line">      vertex.setValue(new DoubleWritable(0));
    66.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="66"> class="hljs-ln-code"> class="hljs-ln-line">      vertex.setId((Text) record.get(0));
    67.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="67"> class="hljs-ln-code"> class="hljs-ln-line">      System.out.println(record.get(0));
    68.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="68"> class="hljs-ln-code"> class="hljs-ln-line"> 
    69.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="69"> class="hljs-ln-code"> class="hljs-ln-line">      for (int i = 1; i < record.size(); i++) {
    70.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="70"> class="hljs-ln-code"> class="hljs-ln-line">        Writable edge = record.get(i);
    71.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="71"> class="hljs-ln-code"> class="hljs-ln-line">        System.out.println(edge.toString());
    72.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="72"> class="hljs-ln-code"> class="hljs-ln-line">        if (!(edge.equals(NullWritable.get()))) {
    73.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="73"> class="hljs-ln-code"> class="hljs-ln-line">          vertex.addEdge(new Text(edge.toString()), NullWritable.get());
    74.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="74"> class="hljs-ln-code"> class="hljs-ln-line">        }
    75.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="75"> class="hljs-ln-code"> class="hljs-ln-line">      }
    76.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="76"> class="hljs-ln-code"> class="hljs-ln-line">      LOG.info("vertex edgs size: "
    77.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="77"> class="hljs-ln-code"> class="hljs-ln-line">          + (vertex.hasEdges() ? vertex.getEdges().size() : 0));
    78.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="78"> class="hljs-ln-code"> class="hljs-ln-line">      context.addVertexRequest(vertex);
    79.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="79"> class="hljs-ln-code"> class="hljs-ln-line">    }
    80.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="80"> class="hljs-ln-code"> class="hljs-ln-line"> 
    81.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="81"> class="hljs-ln-code"> class="hljs-ln-line">  }
    82.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="82"> class="hljs-ln-code"> class="hljs-ln-line"> 
    83.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="83"> class="hljs-ln-code"> class="hljs-ln-line">  private static void printUsage() {
    84.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="84"> class="hljs-ln-code"> class="hljs-ln-line">    System.out.println("Usage:   [Max iterations (default 30)]");
    85.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="85"> class="hljs-ln-code"> class="hljs-ln-line">    System.exit(-1);
    86.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="86"> class="hljs-ln-code"> class="hljs-ln-line">  }
    87.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="87"> class="hljs-ln-code"> class="hljs-ln-line"> 
    88.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="88"> class="hljs-ln-code"> class="hljs-ln-line">  public static void main(String[] args) throws IOException {
    89.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="89"> class="hljs-ln-code"> class="hljs-ln-line">    if (args.length < 2)
    90.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="90"> class="hljs-ln-code"> class="hljs-ln-line">      printUsage();
    91.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="91"> class="hljs-ln-code"> class="hljs-ln-line"> 
    92.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="92"> class="hljs-ln-code"> class="hljs-ln-line">    GraphJob job = new GraphJob();
    93.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="93"> class="hljs-ln-code"> class="hljs-ln-line"> 
    94.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="94"> class="hljs-ln-code"> class="hljs-ln-line">    job.setGraphLoaderClass(PageRankVertexReader.class);
    95.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="95"> class="hljs-ln-code"> class="hljs-ln-line">    job.setVertexClass(PageRankVertex.class);
    96.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="96"> class="hljs-ln-code"> class="hljs-ln-line">    job.addInput(TableInfo.builder().tableName(args[0]).build());
    97.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="97"> class="hljs-ln-code"> class="hljs-ln-line">    job.addOutput(TableInfo.builder().tableName(args[1]).build());
    98.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="98"> class="hljs-ln-code"> class="hljs-ln-line"> 
    99.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="99"> class="hljs-ln-code"> class="hljs-ln-line">    // default max iteration is 30
    100.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="100"> class="hljs-ln-code"> class="hljs-ln-line">    job.setMaxIteration(30);
    101.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="101"> class="hljs-ln-code"> class="hljs-ln-line">    if (args.length >= 3)
    102.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="102"> class="hljs-ln-code"> class="hljs-ln-line">      job.setMaxIteration(Integer.parseInt(args[2]));
    103.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="103"> class="hljs-ln-code"> class="hljs-ln-line"> 
    104.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="104"> class="hljs-ln-code"> class="hljs-ln-line">    long startTime = System.currentTimeMillis();
    105.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="105"> class="hljs-ln-code"> class="hljs-ln-line">    job.run();
    106.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="106"> class="hljs-ln-code"> class="hljs-ln-line">    System.out.println("Job Finished in "
    107.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="107"> class="hljs-ln-code"> class="hljs-ln-line">        + (System.currentTimeMillis() - startTime) / 1000.0 + " seconds");
    108.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="108"> class="hljs-ln-code"> class="hljs-ln-line">  }
    109.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="109"> class="hljs-ln-code"> class="hljs-ln-line">}
     class="hide-preCode-box"> class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    4、他的存储是什么结构？

    根据资料，他使用的分布式文件系统专为大规模数据存储和处理设计，具备高可用性、高可靠性和高性能的特点。

    在存储模型上，类似于对象存储，数据以对象的形式存储，便于快速访问和处理

    为优化查询性能和存储效率，MaxCompute 在存储层面支持列式存储格式

    看起来很类似 Hadoop 的 HDFS

    1. NameNode：

      负责管理元数据和文件系统的命名空间。 NameNode 知道每个文件和目录中每个数据块的位置。它是系统的中心，并不存储实际数据或文件

    2. DataNode：

      负责存储实际数据以及数据块。每个 DataNode 定期向 NameNode 报告其存储的块信息。在集群规模扩大时，可以通过增加 DataNode 来增加存储能力

    3. Secondary NameNode：

      协助 NameNode 的检查点和元数据快照工作，减少 NameNode 的启动时间

    当然具体是不是，还需要看源代码才知道，他没有开源，作者后续再进行源码研究

    5、他的索引结构是什么样子?

    他是列式存储（列存）来提高查询性能，对数据进行分区（例如按日期、地理位置等字段），减小查询范围，加快查询速度。这种机制在做范围查询或按条件过滤时比较高效。

    存储文件可能会被分成多个区块，每个区块内的数据在物理存储上是连续的，通常会根据统计信息（如最小值和最大值）对这些区块建立索引，从而快速跳过不相关的数据块。

    对于一些离散型类别数据，常使用数据字典技术来进行压缩并加速查询。

    6、histogram有什么用？

    他介绍过一个sql优化，意思是拿直方图来作为分析

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">ANALYZE TABLE <tablename> compute statistics 
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">FOR columns WITH histogram 256 buckets;
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line"> 
     class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    256 BUCKETS 表示直方图中将数据分为 256 个桶，这个就是横坐标

    比如数据值的范围是 0到 100，我们用 256 个桶来描述这列数据的直方图。那么可能的分布形式会类似于：

    类似于

    1.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="1"> class="hljs-ln-code"> class="hljs-ln-line">  │
    2.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="2"> class="hljs-ln-code"> class="hljs-ln-line">    │        │
    3.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="3"> class="hljs-ln-code"> class="hljs-ln-line">    │   │    │
    4.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="4"> class="hljs-ln-code"> class="hljs-ln-line">    │   │  │ │
    5.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="5"> class="hljs-ln-code"> class="hljs-ln-line">    │   │  │ │     │
    6.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="6"> class="hljs-ln-code"> class="hljs-ln-line">    │   │  │ │  │  │
    7.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="7"> class="hljs-ln-code"> class="hljs-ln-line">    └───┴──┴───┴───┴──
    8.  class="hljs-ln-numbers"> class="hljs-ln-line hljs-ln-n" data-line-number="8"> class="hljs-ln-code"> class="hljs-ln-line">     0   20  40  60  80  100
     class="hljs-button signin active" data-title="登录复制" data-report-click="{"spm":"1001.2101.3001.4334"}" onclick="hljs.signin(event)">

    但是作者觉得还是不太理解这在什么时候要用到，等后面进项目看看

    四、总结

            他的内容很多，作者暂时看了四五分之一吧，作者会持续研究分享

            目前没有进项目，也没有看到实际源码，很多问题和分析都是作者YY的，过几天进项目了再看看源码，很多疑问就会有确定的答案

            作者认为带着问题去学习研究才是最快的，大胆假设疑问，小心分析求证，原理都是相通的，区别不会很大

    data-report-view="{"mod":"1585297308_001","spm":"1001.2101.3001.6548","dest":"https://blog.csdn.net/m0_69270256/article/details/144140073","extend1":"pc","ab":"new"}">>
    注：本文转载自blog.csdn.net的胖当当技术的文章"https://blog.csdn.net/m0_69270256/article/details/144140073"。版权归原作者所有，此博客不拥有其著作权，亦不承担相应法律责任。如有侵权，请联系我们删除。
    复制链接
    复制链接
    相关推荐
    发表评论

    评论记录：

    未查询到任何数据！