引言
Apache Pulsar越来越多的公司使用，与Apache Kafka、Apache RocketMQ并列成为消息领域三家马车，有必要对其研究一番。下面以笔者曾在生产环境使用的配置梳理，内容提要：
Pulsar的安装与部署
Pulsar集群的验证
生产环境机器配置
生产环境内存分配
生产环境broker配置项调整
生产环境bookie配置项调整
一、Pulsar安装与部署
1.下载安装包Pulsar安装包包含了zookeeper、broker、bookie三个组件。
下载Pulsar二进制包
https://pulsar.apache.org/download/
解压压缩包
tar -zvxf apache-pulsar-2.9.1-bin.tar.gz
2.部署zookeeper 2.1 修改zookeeper配置创建目录
mkdir -p data/zookeeperecho 1 > data/zookeeper/myid
修改zk配置，文件位于conf/zookeeper.conf
# 数据目录dataDir=data/zookeeper# 日志目录dataLogDir=data/zookeeper/logs# zk集群配置，server.1~nserver.1=127.0.0.1:2888:3888
2.2 后台启动zookeeperbin/pulsar-daemon start zookeeperdoing start zookeeper ...starting zookeeper, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-zookeeper-M-C02GL1NTQ05P.logNote: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.
通过pulsar-daemon管理pulsar组件
bin/pulsar-daemon helpError: no enough arguments provided.Usage: pulsar-daemon (start|stop|restart)
备注：可以通过pulsar-daemon命令对broker、bookie、zookeeper等组件启动、关闭或者重启。
2.3 检查zookeeper是否启动成功zookeeper启动日志和查看zookeeper进程
ps axu | grep zookeeper
3.元数据初始化 3.1 初始化命令说明bin/pulsar initialize-cluster-metadata --cluster pulsar-cluster-1 --zookeeper 127.0.0.1:2181 --configuration-store 127.0.0.1:2181 --web-service-url http://127.0.0.1:8080 --web-service-url-tls https://127.0.0.1:8443 --broker-service-url pulsar://127.0.0.1:6650 --broker-service-url-tls pulsar+ssl://127.0.0.1:6651
参数说明
备注：生产环境可以使用域名。
3.2 查看初始化结果bin/pulsar zookeeper-shell[zk: localhost:2181(CONNECTED) 1] ls /[admin, bookies, ledgers, pulsar, stream, zookeeper]
4.部署BookKeeper集群 4.1 配置修改bindAddress=127.0.0.1advertisedAddress=127.0.0.1zkServers=127.0.0.1:2181
参数说明
bin/pulsar-daemon start bookiedoing start bookie ...starting bookie, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-bookie-M-C02GL1NTQ05P.logNote: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.
4.3 测试bookie集群bin/bookkeeper shell simpletest --ensemble 1 --writeQuorum 1 --ackQuorum 1 -- numEntries 1000...2022-02-19T23:43:03,391+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 722 entries written2022-02-19T23:43:03,983+0800 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand - 1000 entries written to ledger 02022-02-19T23:43:04,041+0800 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for 127.0.0.1:3181...
备注：通过simpletest命令向bookie集群写入测试数据，完成测试后会自动删除。
5.部署Broker集群 5.1 修改配置zookeeperServers=127.0.0.1:2181configurationStoreServers=127.0.0.1:2181bindAddress=127.0.0.1# 默认InetAddress.getLocalHost().getHostName()advertisedAddress=127.0.0.1clusterName=pulsar-cluster-1
5.2 启动brokerbin/pulsar-daemon start brokerdoing start broker ...starting broker, logging to /Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-broker-M-C02GL1NTQ05P.logNote: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.
5.3 验证集群查看集群节点
bin/pulsar-admin brokers list cluster-1"172.17.13.184:8080"
发送测试消息
bin/pulsar-client produce persistent://public/default/test -n 1 -m "Hello Pulsar"...2022-02-20T13:31:18,469+0800 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced...
消费测试消息
bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive"...----- got message -----key:[null], properties:[], content:Hello Pulsar...
小结：至此测试集群搭建完成，下文将介绍生产环境配置的调整项。
二、生产环境配置
1.机器配置下面为生产环境搭建Pulsar集群，由3个zookeeper节点、3个broker节点和5个bookie节点构成。
备注：每个组件集群部署时可以同城跨可用区部署，提高高可用。broker不存储消息100G即可，bookie存储消息通常需要较大磁盘，比如3T，具体根据消息量计算。
2.内存优化2.1 broker内存调整
以内存64G大小，在文件conf/pulsar_env.sh修改如下内容：
PULSAR_MEM=${PULSAR_MEM:-"-Xms10g -Xmx10g -XX:MaxDirectMemorySize=20g"}
2.2 bookie内存调整
以内存64G大小，在文件conf/bkenv.sh修改如下内容：
BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms10g -Xmx10g -XX:MaxDirectMemorySize=20g"}}
3.Broker调整项默认为-1表示没有限制，可以通过set-message-ttl设置过期时间，防⽌磁盘爆满backlogQuotaDefaultRetentionPolicy=producer_request_hold保持默认，未被消费确认的消息超过存储⼤⼩的策略managedLedgerDefaultEnsembleSize=3创建Ledger时指定Ensemble的⼤⼩managedLedgerDefaultWriteQuorum=3创建Ledger时指定Quorum的⼤⼩managedLedgerDefaultAckQuorum=2创建Ledger时指定ack Quorum的⼤⼩dispatcherMaxReadBatchSize=500⼀次从bookkeeper读取的数量，默认为100条loadBalancerAutoBundleSplitEnabled=false关闭auto bundle split功能，提⾼客户端稳定性loadBalancerAutoUnloadSplitBundlesEnabled=false关闭auto bundle split功能，提⾼客户端稳定性loadBalancerSheddingEnabled=false禁⽌Pulsar⾃动均衡loadBalancerEnabled=false禁⽌Pulsar⾃动均衡
备注：参数根据实际情况调整，在线上开启负载均衡时，发现有重复消息，此处先将其关闭。
4.Bookie调整项备注：journalDirectory和ledgerDirectories在条件允许的情况可以配置到不同的磁盘。