Nifi configuration mysql_binlog reads

这个就是mysqlcdc Suitable for real-time reading mysqlThe data is then transferred to different databases 如es hbase kudu

准备工作

一、mysql_binlog介绍

binlog三种模式

①.statement

每一条会修改数据的sql语句会记录到binlog中.优点是并不需要记录每一条sql语句和每一行的数据变化,减少了binlog日志量,节约IO,提高性能.缺点是在某些情况下会导致master-slave中的数据不一致(如sleep()函数, last_insert_id(),以及user-defined functions(udf)等会出现问题.

In fact, this sentence is copied by me,I didn't know what to say for a long time,简单的翻译下,

一般开启binlogMaster-slave replication will be performed,The flaw in this is

主 insert into table  a values(1,rand())  插入了1,0.2

从  insert into table  a values(1,rand()) 插入了1,0.3 

Because this is just a recordsql.

② ROW模式（RBR）

不记录每条sql语句的上下文信息,仅需记录哪条数据被修改了,修改成什么样了.而且不会出现某些特定情况下的存储过程、或function、或trigger的调用和触发无法被正确复制的问题.缺点是会产生大量的日志,尤其是alter table的时候会让日志暴涨.

③ MIXED模式（MBR）

以上两种模式的混合使用,一般的复制使用STATEMENT模式保存binlog,对于STATEMENT模式无法复制的操作使用ROW模式保存binlog,MySQL会根据执行的SQL语句选择日志保存方式.

我们需要

show variables like '%binlog%'  --查看binlog的一些配置信息

show variables like '%log_bin%' --查看到binlogThe log is status

修改配置文件

vim /etc/cnf

[mysqld]

character-set-server=utf8
##server_id 主从不一致
server_id = 1
##上文介绍的
binlog_format=row
##Just record thatdb的信息. 我是为了测试 不写默认所有
binlog_do_db=cc

[client]

default-character-set=utf8

[mysql]

default-character-set=utf8

 查看binlog

cd /var/lib/mysql

 here beforebinlog很多的 我删了一部分,因为出现了bug,后面细说

进入mysql客户端可以查看binlog的一些基本信息

show binary logs;--获取binlog文件列表

show master status;--查看当前正在写入的binlog文件

show binlog events in 'mysqlbin.000102' from 1 limit 0,10;

binlog查看

通过mysqlbinlog 工具

基础版本  /usr/bin/mysqlbinlog /var/lib/mysql/mysqlbin.102

高级版本 /usr/bin/mysqlbinlog --no-defaults --base64-output=decode-rows -v /var/lib/mysql/mysqlbin.000102  

This is also Baidu.Do your own research

binlog删除

（1）手动删除 直接 rm 另外修改mysqlbin.index的内容.

（2）通过mysqlprovided tools to remove

Check it out before deletingpurge的用法：help purge;

删除举例：

RESET MASTER;//删除所有binlog日志,The new log number starts from the beginning

PURGE MASTER LOGS TO 'mysql-bin.010';//删除mysql-bin.010All previous logs

PURGE MASTER LOGS BEFORE '2003-04-02 22:46:26';// 删除2003-04-02 22:46:26之前产生的所有日志.

Why delete it here？Because I have a lot of them beforebinlog,It seems to be set at firststatement 后面改为row了,在使用nifi读取的时候,Discovery read the first onebinlog01can start to read7w条数据,然后就一直报错,没法,Just delete all the previous onesbinglog,Keep the latestbinlog.The previous error log

COMMIT event received while not processing a transaction (i.e. no corresponding BEGIN event). This could indicate that your binlog position is invalid. 

仅供参考,I tested it myselfmysq So it doesn't matter if you delete it,公司的mysql注意了.

下面开始nifi布置.

CaptureChangeMySQL | Apache NIFI中文文档

46.实时同步Mysql到Hive-3_哔哩哔哩_bilibili

I have been having issues with the configuration before...后面bStop watching the video of the dark horse.You can also watch it directly,I'm just recording myself here.

核心CaptureChangeMySQL

 这里说几点,

scheduling 的runschedulem默认是0s Set a fews Otherwise it runs very fast cpufull of.

properties里有个 Distributed Map Cache Client This thing is actually similari一个缓存,Note that an error will occur if this data is not configured.

如何配置

注意这里是个service ,Usually there will be oneserver

在项目空白处右键->configure

然后选择service-> 点击+号

搜索distributed 出现了server和service

然后先点击server. Then use the default 

再新建service 采用默认配置 只用修改host 

 Then remember to start firstserver 再enable service.

其实现在cdcThe basic configuration has been completed.

懒得写了.Go watch the video,The video is more detailed than mine.