在Linux环境下安装Kafka

在Linux环境下安装Kafka⽬录

1.1 Java环境为前提

1.2 Zookeeper的安装配置

1.3 Kafka 的安装与配置

1.4 启动Kafka

1.5 重新开⼀个窗⼝，查看Zookeeper的节点

1.6 此时Kafka是前台模式启动，要停⽌，使⽤Ctrl+C⼆、⽣产与消费

2.1 kafka-topics.sh ⽤于管理主题

2.2 kafka-console-consumer.sh⽤于消费消息

2.3 kafka-console-producer.sh⽤于⽣产消息

2.4 具体操作

⼀、环境准备

jdk下载地址链接：

zookeeper下载地址链接：

kafka下载地址链接：

1.1 Java环境为前提

1.1.1 上传jdk-8u261-linux-x64.rpm到服务器并安装

# 安装命令rpm -ivh jdk-8u261-linux-x64.rpm

1.1.2 配置环境变量

# 编辑配置⽂件,jdk的bin⽬录到/etc/profile⽂件，对所有⽤户的shell有效

vim /etc/profile

# ⽣效

source /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_261-amd64export PATH=$PATH:$JAVA_HOME/bin

# 验证java -version

⾄此，jdk安装成功。

1.2 Zookeeper的安装配置

1.2.1 上传zookeeper-3.4.14.tar.gz到服务器，解压到/opt

# 解压zk到指定⽬录

tar -zxf zookeeper-3.4.14.tar.gz -C /opt

1.2.2 修改Zookeeper保存数据的⽬录，dataDir

# 进⼊conf配置⽬录

cd /opt/zookeeper-3.4.14/conf

# 复制zoo_sample.cfg命名为zoo.cfg

cp zoo_sample.cfg zoo.cfg

# 编辑zoo.cfg⽂件

vim zoo.cfgdataDir=/var/riemann/zookeeper/data

1.2.3 编辑/etc/profile，使配置⽣效

设置环境变量ZOO_LOG_DIR，指定Zookeeper保存⽇志的位置；

ZOOKEEPER_PREFIX指向Zookeeper的解压⽬录；

将Zookeeper的bin⽬录添加到PATH中：

export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14

export PATH=$PATH:$ZOOKEEPER_PREFIX/bin

export ZOO_LOG_DIR=/var/riemann/zookeeper/log

配置完以后再⽣效配置：

source /etc/profile

1.2.4 启动Zookeeper，确认Zookeeper的状态

zkServer.sh start

⾄此，zookeeper安装成功。

1.3 Kafka 的安装与配置

1.3.1 上传kafka_2.12-1.0.2.tgz到服务器并解压tar -zxf kafka_2.12-1.0.2.tgz -C /opt

1.3.2 配置环境变量并⽣效

vim /etc/profile

export KAFKA=/opt/kafka_2.12-1.0.2

export PATH=$PATH:$KAFKA/bin

source /etc/profile

1.3.3 配置/opt/kafka_2.12-1.0.2/config中的server.properties⽂件vi /opt/kafka_2.12-1.0.2/config/server.properties

Kafka连接Zookeeper的地址，此处使⽤本地启动的Zookeeper实例

连接地址是localhost:2181

后⾯的 myKafka 是Kafka在Zookeeper中的根节点路径

配置kafka存储持久化数据⽬录

log.dirs=/var/riemann/kafka/kafka-logs

创建上述持久化数据⽬录

mkdir -p /var/riemann/kafka/kafka-logs

1.4 启动Kafka

进⼊Kafka安装的根⽬录，执⾏如下命令：kafka-server-start.sh ../config/server.properties

启动成功，可以看到控制台输出的最后⼀⾏的started状态：此时kafka安装成功。

1.5 重新开⼀个窗⼝，查看Zookeeper的节点

1.6 此时Kafka是前台模式启动，要停⽌，使⽤Ctrl+C

如果要后台启动，使⽤命令：

kafka-server-start.sh -daemon config/server.properties

查看Kafka的后台进程：

ps -ef | grep kafka

停⽌后台运⾏的Kafka：

kafka-server-stop.sh

⼆、⽣产与消费

查看zookeeper状态，zookeeper启动成功，再启动kafka。

2.1 kafka-topics.sh ⽤于管理主题

查看命令的帮助信息

[root@master1 bin]# kafka-topics.sh

Create, delete, describe, or change a topic.

Option Description

------ -----------

--alter Alter the number of partitions,

replica assignment, and/or

configuration for the topic.

--config A topic configuration override for the

topic being created or altered.The

following is a list of valid

configurations:

cleanup.policy

compression.type

delete.retention.ms

file.delete.delay.ms

flush.messages

flush.ms

follower.replication.throttled.

replicas

index.interval.bytes

leader.replication.throttled.replicas

max.message.bytes

message.format.version

message.timestamp.difference.max.ms

message.timestamp.type

min.cleanable.dirty.ratio

min.compaction.lag.ms

min.insync.replicas

preallocate

retention.bytes

retention.ms

segment.bytes

segment.index.bytes

segment.jitter.ms

segment.ms

unclean.leader.election.enable

See the Kafka documentation for full

details on the topic configs.

--create Create a new topic.

--delete Delete a topic

--delete-config A topic configuration override to be

removed for an existing topic (see

the list of configurations under the

--config option).

--describe List details for the given topics.

--disable-rack-aware Disable rack aware replica assignment

--force Suppress console prompts

--help Print usage information.

--if-exists if set when altering or deleting

topics, the action will only execute

if the topic exists

--if-not-exists if set when creating topics, the

action will only execute if the

topic does not already exist

--list List all available topics.

--partitions The number of partitions for the topic

being created or altered (WARNING:

If partitions are increased for a

topic that has a key, the partition

logic or ordering of the messages

will be affected

--replica-assignment

broker_id_for_part1_replica1 : assignments for the topic being

broker_id_for_part1_replica2 , created or altered.

broker_id_for_part2_replica1 :

broker_id_for_part2_replica2 , ...>

--replication-factor

replication factor> partition in the topic being created.

--topic The topic to be create, alter or

describe. Can also accept a regular