Youluo's Blog

Appearance

部署目标 ​

  • 安装JDK8

  • 安装zookeeper集群

  • 安装CK集群，使用Zookeeper作协调节点，使用 4分片1副本

    注：CK 不依赖JDK。

环境描述 ​

软件列表 ​

序号软件名称版本安装包名称Md5下载地址备注
1JDKjdk-8u281-linux-x64jdk-8u281-linux-x64.tar.gzc96c8c4da5f7c4befd8c26750f152dbeJAVA运行环境
2Zookeeperv3.9.0zookeeper-3.4.14.tar.gzb0ecf30e2cd83eac8de3454e06bcae28http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz分布式协调组件
3clickhouse-common-static23.3.8.21clickhouse-common-static-23.3.8.21.x86_64.rpm417a0dc3223dc99ac0c4953db61d2383https://packages.clickhouse.com/rpm/lts/clickhouse-common-static-23.3.8.21.x86_64.rpmCK通用基础包
4clickhouse-common-static-dbg23.3.8.21clickhouse-common-static-dbg-23.3.8.21.x86_64.rpm2d52ee1052bd327a5b1b3530b95a6a1chttps://packages.clickhouse.com/rpm/lts/clickhouse-common-static-dbg-23.3.8.21.x86_64.rpmCK通用基础包
5clickhouse-server23.3.8.21clickhouse-server-23.3.8.21.x86_64.rpm69c38c84ef2e49ff0b77e68d3dc973dehttps://packages.clickhouse.com/rpm/lts/clickhouse-server-23.3.8.21.x86_64.rpmCK server
6clickhouse-client23.3.8.21clickhouse-client-23.3.8.21.x86_64.rpmfb984adc2a06c1cbde420875b66e76aehttps://packages.clickhouse.com/rpm/lts/clickhouse-client-23.3.8.21.x86_64.rpmCK client

硬件列表 ​

生产环境中ZK和CK应该分开。

IPhostname操作系统版本备注
192.168.0.44ck01CentOS 7.6.1810CK
192.168.0.45ck02CentOS 7.6.1810CK
192.168.0.46ck03CentOS 7.6.1810CK，和 ck01互为副本
192.168.0.47ck04CentOS 7.6.1810CK，和 ck02互为副本
192.168.0.48ck05CentOS 7.6.1810CK
192.168.0.49ck06CentOS 7.6.1810CK
192.168.0.50ck07CentOS 7.6.1810CK，和 ck05互为副本
192.168.0.51ck08CentOS 7.6.1810CK，和 ck06互为副本
192.168.0.52keeper01CentOS 7.6.1810JDK /ZK
192.168.0.53keeper02CentOS 7.6.1810JDK /ZK
192.168.0.54keeper03CentOS 7.6.1810JDK /ZK

假设所有机器的数据盘都挂在 data01目录下。

安装步骤 ​

均使用root账号操作。

准备工作 ​

所有主机均执行

shell
# 修改系统nofile和nproc限制
echo "修改系统nofile和nproc限制开始"
cp /etc/security/limits.conf /etc/security/limits.conf.bak
echo '*   soft    nofile   165535' >> /etc/security/limits.conf
echo '*   hard    nofile   165535' >> /etc/security/limits.conf
echo '*   soft    nproc    65535' >> /etc/security/limits.conf
echo '*   hard    nproc    65535' >> /etc/security/limits.conf
echo '*   soft    memlock  unlimited' >> /etc/security/limits.conf
echo '*   hard    memlock  unlimited' >> /etc/security/limits.conf
echo 'clickhouse   soft    memlock  unlimited' >> /etc/security/limits.conf
echo 'clickhouse   hard    memlock  unlimited' >> /etc/security/limits.conf
echo "修改系统nofile和nproc限制结束"

# 查看是否生效，默认为1024，看是否修改成了添加的内容的数字(如果没有生效，需要重启机器，可以延迟到 `准备工作` 全部结束再重启主机)
ulimit -n

# 修改vm
echo "修改vm开始"
sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" > /etc/sysctl.conf
echo "vm.swappiness=1" >> /etc/sysctl.conf
sysctl -p
echo "修改vm结束"

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 设置selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

# 禁用透明大页
echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

安装JDK ​

keeper01/keeper02/keeper033台机器均执行

shell
SOFTWARE_INSTALL_PATH=/opt/software/jdk8
mkdir -p ${SOFTWARE_INSTALL_PATH}

tar zxvf jdk-8u371-linux-x64.tar.gz -C ${SOFTWARE_INSTALL_PATH}

echo '' >> /etc/profile
echo '# JDK环境' >> /etc/profile
echo "export JAVA_HOME=${SOFTWARE_INSTALL_PATH}/jdk1.8.0_371" >> /etc/profile
echo 'export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >> /etc/profile
echo 'export PATH=$JAVA_HOME/bin:$PATH' >> /etc/profile

# 验证
source /etc/profile
java -version

安装zookeeper ​

keeper01/keeper02/keeper033台机器均执行

配置主机别名 ​

shell
echo '192.168.0.52 keeper01' >> /etc/hosts
echo '192.168.0.53 keeper02' >> /etc/hosts
echo '192.168.0.54 keeper03' >> /etc/hosts

# 验证
tail /etc/hosts
ping -c 2 keeper01
ping -c 2 keeper02
ping -c 2 keeper03

执行安装 ​

shell
SOFTWARE_INSTALL_PATH=/opt/software/zk
ZK_NODE_1=keeper01
ZK_NODE_2=keeper02
ZK_NODE_3=keeper03
mkdir -p ${SOFTWARE_INSTALL_PATH}
tar -zxvf zookeeper-3.4.14.tar.gz -C ${SOFTWARE_INSTALL_PATH}

#创建zk存储数据目录
mkdir -p ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data
#创建zk日志文件目录
mkdir -p ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data/logs
#修改zk配置文件
cd ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/conf
#文件改名
cp zoo_sample.cfg zoo.cfg

#更新datadir
sed -i "s#dataDir=/tmp/zookeeper#dataDir=${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data#g" zoo.cfg
#增加logdir
sed -i "13i dataLogDir=${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data/logs" zoo.cfg
#打开注释
#ZK提供了自动清理事务日志和快照文件的功能，这个参数指定了清理频率，单位是小时
sed -i "s|#autopurge.purgeInterval=1|autopurge.purgeInterval=1|g" zoo.cfg
#增加集群配置
##server.服务器ID=服务器IP地址：服务器之间通信端口：服务器之间投票选举端口
echo "server.1=${ZK_NODE_1}:2888:3888" >> zoo.cfg
echo "server.2=${ZK_NODE_2}:2888:3888" >> zoo.cfg
echo "server.3=${ZK_NODE_3}:2888:3888" >> zoo.cfg

添加myid配置【差异执行】 ​

在 keeper01 执行

shell
# ZK_NODE_1
echo 1 > ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data/myid

在 keeper02 执行

shell
# ZK_NODE_2
echo 2 > ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data/myid

在 keeper03 执行

shell
# ZK_NODE_3
echo 3 > ${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14/data/myid

配置环境变量 ​

3台zookeeper节点均执行

shell
cho '' >> /etc/profile
echo '# ZooKeeper环境' >> /etc/profile
echo "export ZOOKEEPER_HOME=${SOFTWARE_INSTALL_PATH}/zookeeper-3.4.14" >> /etc/profile
echo 'export PATH=$PATH:$ZOOKEEPER_HOME/bin' >> /etc/profile
source /etc/profile

修改zookeeper端允许占用内存 ​

vi /clickhouse/zookeeper/bin/zkEnv.sh 搜索 ZK_SERVER_HEAP ，默认设置大小约 1G，实际生产环境需设大，zookeeper数据处理使用，不超过32G，一般可设置为物理内存一半，因此不建议将clickhouse和zk放在一台机器上

bash
ZK_SERVER_HEAP="${ZK_SERVER_HEAP:-1000}"  # 生产需改大

启动 ​

3台zookeeper节点均执行

shell
source /etc/profile
zkServer.sh start

验证 ​

3台zookeeper节点均执行

shell
source /etc/profile
zkServer.sh status

预期结果：有一台主机输出leader，另外两台输出follower。

安装CK ​

在 ck01/ck02/ck03/ck04 ，4台主机数据盘均为 data01

配置主机别名 ​

shell
echo '192.168.0.44 ck01' >> /etc/hosts
echo '192.168.0.45 ck02' >> /etc/hosts
echo '192.168.0.46 ck03' >> /etc/hosts
echo '192.168.0.47 ck04' >> /etc/hosts
echo '192.168.0.48 ck05' >> /etc/hosts
echo '192.168.0.49 ck06' >> /etc/hosts
echo '192.168.0.50 ck07' >> /etc/hosts
echo '192.168.0.51 ck08' >> /etc/hosts
echo '192.168.0.52 keeper01' >> /etc/hosts
echo '192.168.0.53 keeper02' >> /etc/hosts
echo '192.168.0.54 keeper03' >> /etc/hosts

# 验证
tail /etc/hosts
ping -c 2 ck01
ping -c 2 ck02
ping -c 2 ck03
ping -c 2 ck04
ping -c 2 ck05
ping -c 2 ck06
ping -c 2 ck07
ping -c 2 ck08
ping -c 2 keeper01
ping -c 2 keeper02
ping -c 2 keeper03

执行安装 ​

shell
rpm -ivh ./*.rpm
  • 输入密码： Your_Password

更改数据目录 ​

通过软链接进行关联

shell
mkdir -p /data01/clickhouse-server/data/
mv /var/lib/clickhouse /data01/clickhouse-server/data/
ln -s /data01/clickhouse-server/data/clickhouse /var/lib/clickhouse
chown -R clickhouse:clickhouse /data01/clickhouse-server/data/clickhouse

mkdir -p /data01/clickhouse-server/log/
mv /var/log/clickhouse-server/ /data01/clickhouse-server/log/
ln -s /data01/clickhouse-server/log/clickhouse-server /var/log/clickhouse-server
chown -R clickhouse:clickhouse  /data01/clickhouse-server/log/

修改配置 ​

包括普通、分片、副本等配置，配置文件位于 /etc/clickhouse-server/config.xml。

  • 将 log的 <level>trace</level>改为 <level>warning</level>

  • 放开注释的 <listen_host>0.0.0.0</listen_host>，表示对所有主机开放CK服务

  • 放开 <timezone>UTC</timezone>，并修改为：<timezone>Asia/Shanghai</timezone>

  • 在 <remote_servers>注释掉 <test_shard_localhost>等一众shard，并添加如下：

    xml
            <cluster1>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>ck01</host>
                    <port>9000</port>
                     <user>default</user>
                    <password>Your_Password</password>
                </replica>
                <replica>
                    <host>ck03</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>ck02</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
                <replica>
                    <host>ck04</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>ck05</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
                <replica>
                    <host>ck07</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>ck06</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
                <replica>
                    <host>ck08</host>
                    <port>9000</port>
                    <user>default</user>
                    <password>Your_Password</password>
                </replica>
            </shard>
        </cluster1>

  • 在 <zookeeper> 节点下添加：

    xml
              <zookeeper>
                <node>
                    <host>keeper01</host>
                    <port>2181</port>
                </node>
                <node>
                    <host>keeper02</host>
                    <port>2181</port>
                </node>
                <node>
                    <host>keeper03</host>
                    <port>2181</port>
                </node>
            </zookeeper>

  • 【差异执行】添加宏定义

    • （ck01）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>01</shard>
        <replica>ck01</replica>
    </macros>

    • （ck02）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>02</shard>
        <replica>ck02</replica>
    </macros>

    • （ck03）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>01</shard>
        <replica>ck03</replica>
    </macros>

    • （ck04）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>02</shard>
        <replica>ck04</replica>
    </macros>

    • （ck05）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>03</shard>
        <replica>ck05</replica>
    </macros>

    • （ck06）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>04</shard>
        <replica>ck06</replica>
    </macros>

    • （ck07）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>03</shard>
        <replica>ck07</replica>
    </macros>

    • （ck08）搜索关键字 macros，改为如下：

      xml
          <macros>
        <shard>04</shard>
        <replica>ck08</replica>
    </macros>

  • config.xml其他参数优化：

    https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings#background_schedule_pool_size

    • 后台用于merge的线程池大小

      xml
      <background_pool_size>128</background_pool_size>

    • 用于不断执行复制表、Kafka 流和 DNS 缓存更新的一些轻量级定期操作的最大线程数

      xml
      <background_schedule_pool_size>128</background_schedule_pool_size>

    • 将用于执行分布式发送的最大线程数。

      xml
      <background_distributed_schedule_pool_size>128</background_distributed_schedule_pool_size>

    • 开启Prometheus监控

      vim /etc/clickhouse-server/config.xml，搜索 prometheus关键字，去掉 <prometheus>块头部注释 <!--及尾注释 -->。无需重启，实时生效。

      验证：预期有日志输出

      shell
      curl http://127.0.0.1:9363/metrics
      shell
      cat << 'EOF' >>  /etc/prometheus/prometheus.yml
  - job_name: 'ck-monitor'
    static_configs:
      - targets: ['192.168.0.44:9363','192.168.0.45:9363','192.168.0.46:9363','192.168.0.47:9363','192.168.0.48:9363','192.168.0.49:9363','192.168.0.50:9363','192.168.0.51:9363']
EOF

tail /etc/prometheus/prometheus.yml

  • users.xml其他参数优化

    https://clickhouse.com/docs/en/operations/settings/query-complexity

    • 完整参考如下：

      xml
      <clickhouse>
    <profiles>
        <default>
          <receive_timeout>1800</receive_timeout>
          <send_timeout>1800</send_timeout>
          <max_memory_usage>100G</max_memory_usage>
          <max_bytes_before_external_group_by>50G</max_bytes_before_external_group_by>
        </default>
        <readonly>
            <readonly>1</readonly>
        </readonly>
    </profiles>
    <users>
        <default>
            <password></password>
            <networks>
                <ip>::/0</ip>
            </networks>
            <profile>default</profile>
            <quota>default</quota>
          	<access_management>1</access_management>
            <named_collection_control>1</named_collection_control>
            <show_named_collections>1</show_named_collections>
            <show_named_collections_secrets>1</show_named_collections_secrets>
        </default>
    </users>
    <quotas>
        <default>
            <interval>
                <duration>3600</duration>
                <queries>0</queries>
                <errors>0</errors>
                <result_rows>0</result_rows>
                <read_rows>0</read_rows>
                <execution_time>0</execution_time>
            </interval>
        </default>
    </quotas>
</clickhouse>

启动 ​

shell
systemctl start clickhouse-server

验证 ​

  • 查看启动状态

    shell
    systemctl status clickhouse-server

  • 使用 clickHouse-client 验证

    shell
    clickhouse-client -m
# 输入密码：Your_Password

# 退出使用 Ctrl + D

    • 验证分片

      sql
      SELECT cluster, host_name FROM system.clusters;

      预期有 cluster1

    • 验证宏定义

      sql
      SELECT * FROM system.macros;

      预期 ck01、ck03 上 shard为 01, ck02、ck04 上shard为 02，ck05、ck07 上shard为 03，ck06、ck08 上shard为 04，replica为各主机名称。

    • 验证副本

      sql
      ---- 测试database自动创建
-- 在ck01上执行建database语句
CREATE DATABASE IF NOT EXISTS cluster_db ON CLUSTER cluster1;
-- 查看其余所有ck主机，预期会在ck02/ck03/ck04/ck05/ck06/ck07/ck08上自动建database，名为cluster_db
show databases;

---- 测试table自动创建
-- 在ck01上执行建table语句
CREATE TABLE cluster_db.test_1_local ON CLUSTER cluster1(
    ID String,
    URL String,
    EventTime Date
)ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test_1_local', '{replica}')
ORDER BY ID;
-- 查看其余所有ck主机，预期会在ck02/ck03/ck04/ck05/ck06/ck07/ck08上自动建table，名为test_1_local
use cluster_db;
show tables;

---- 测试相同分片自动INSERT
-- 在ck01上执行建INSERT语句
INSERT INTO TABLE cluster_db.test_1_local VALUES('A001', 'www.baidu.com','2019-05-10 00:00:00');
-- 查看其余所有ck主机，预期会在ck03/ck04上自动建table，名为test_1_local
select * from cluster_db.test_1_local;

安装CK其他生态 ​

安装Prometheus监控 ​

TODO

安装Lighthouse ​

TODO

基于 知识共享 CC BY-NC-SA 许可发布

版权所有 © 2019-2024 叶莜落