欢迎您访问365答案网,请分享给你的朋友!
生活常识 学习资料

搭建基于clickhouse集群的zookeeper+kafka的数据分析架构

时间:2023-05-05

1 安装java环境

yum search java

yum install java-1.8.0

2 安装zookeeper

mkdir /app/

cd /app/

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

上面的下载不了用下面的

tar -zxf apache-zookeeper-3.7.0-bin.tar.gz 解压

3 更改配置文件

cd /conf

cp zoo.sample.cfg zoo.cfg 复制配置文件

vim zoo.cfg 修改

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/app/apache-zookeeper-3.7.0-bin/datadataLog=/app/apache-zookeeper-3.7.0-bin/log# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable auto purge feature#autopurge.purgeInterval=1## Metrics Providers## https://prometheus.io Metrics Exporter#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider#metricsProvider.httpPort=7000#metricsProvider.exportJvmInfo=trueserver.1=ch201:2888:3888server.2=ch202:2888:3888server.3=ch203:2888:3888

4 修改hosts

192.168.211.100 ch201

192.168.211.188 ch202

192.168.211.186 ch203

5 修改myid,每台机器不同

mkdir -p /app/apache-zookeeper-3.7.0-bin/data

mkdir -p /app/apache-zookeeper-3.7.0-bin/log

cd /app/apache-zookeeper-3.7.0-bin/data

vim myid 只写一个数字字符

以上三台机器修改为1,2,3

6 全部启动zookeeper

到bin下面 ./zkServer.sh start

然后./zkServer.sh status 查看状态

安装clickhouse,参考另一篇教程

vim /etc/clickhouse-server/config.xml

::/etc/clickhouse-server/config.d/metrika.xml

vim /etc/clickhouse-server/config.d/metrika.xml

<?xml version="1.0"?> true ch201 9000 true ch202 9000 true ch203 9000 ch201 2181 ch202 2181 ch203 2181 01 01 ch201 ::/0 10000000000 0.01 lz4

修改配置文件

以上三台机器同样配置,macro改掉

同样执行sudo clickhouse start.启动集群

看到服务启动后登录clickhouse查看

clickhouse-client直接登录

select * from system.clusters查看集群状态

安装kafka

cd /app

wget Apache Downloads

wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.13-2.8.1.tgz 版本不同,皆可用

tar -zxf kafka_2.13-2.8.1.tgz解压

cd /config/

vim server.properties

broker.id=1 这个值为myid的值,每天机器唯一

listeners=PLAINTEXT://192.168.211.100:9092 本机ip,kafka的地址

zookeeper.connect=192.168.211.100:2181,192.168.211.186:2181,192.168.211.188:2181 集群三台机器的ip

cd /bin

./kafka-server-start.sh /app/kafka_2.13-2.8.1/config/server.properties

如果报错日志问题,将日志目录删除重新启动即可

nohup ./kafka-server-start.sh /apps/kafka_2.13-2.8.1/config/server.properties >>/tmp/kafka-logs/kafka.log 2>&1 & 后台启动,可查看/tmp下面的日志

Copyright © 2016-2020 www.365daan.com All Rights Reserved. 365答案网 版权所有 备案号:

部分内容来自互联网,版权归原作者所有,如有冒犯请联系我们,我们将在三个工作时内妥善处理。