• Pull方式

Flume Agent 编写

# Name the components on this agentsimple-agent.sources = netcat-sourcesimple-agent.sinks = spark-sinksimple-agent.channels = memory-channel# Describe/configure the sourcesimple-agent.sources.netcat-source.type = netcatsimple-agent.sources.netcat-source.bind = localhostsimple-agent.sources.netcat-source.port = 44444# Describe the sinksimple-agent.sinks.spark-sink.type = org.apache.spark.streaming.flume.sink.SparkSinksimple-agent.sinks.spark-sink.hostname =localhostsimple-agent.sinks.spark-sink.port =41414simple-agent.sinks.spark-sink.channel = memoryChannel# Use a channel which buffers events in memorysimple-agent.channels.memory-channel.type = memory# Bind the source and sink to the channelsimple-agent.sources.netcat-source.channels = memory-channelsimple-agent.sinks.spark-sink.channel = memory-channel复制代码

启动Flume

flume-ng agent \--name simple-agent \--conf conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/flume_pull_streaming.conf \-Dflume.root.logger=INFO,console &复制代码

• Push方式

Flume Agent的编写

# Name the components on this agentsimple-agent.sources = netcat-sourcesimple-agent.sinks = avro-sinksimple-agent.channels = memory-channel# Describe/configure the sourcesimple-agent.sources.netcat-source.type = netcatsimple-agent.sources.netcat-source.bind = localhostsimple-agent.sources.netcat-source.port = 44444# Describe the sinksimple-agent.sinks.avro-sink.type = avrosimple-agent.sinks.avro-sink.hostname = localhostsimple-agent.sinks.avro-sink.port = 41414# Use a channel which buffers events in memorysimple-agent.channels.memory-channel.type = memory# Bind the source and sink to the channelsimple-agent.sources.netcat-source.channels = memory-channelsimple-agent.sinks.avro-sink.channel = memory-channel复制代码

启动flume

flume-ng agent \--name simple-agent \--conf conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/conf/flume_push_streaming.conf \-Dflume.root.logger=INFO,console &复制代码

==注意在本地和服务器上切换的时候需要修改flume的sink的hostname==

本地测试总结

1. 启动SparkStreaming作业
2. 启动flume agent
3. 通过telnet输入数据，观察IDEA控制台的输出

提交到服务器

spark-submit \--class com.gwf.spark.FlumePushWordCount \--master local[2] \--packages org.apache.spark:spark-streaming-flume_2.11:2.2.0 \/Users/gaowenfeng/Documents/IDE/newsell/spark-train/target/spark-train-1.0-SNAPSHOT.jar localhost 41414复制代码

spark-submit \--class com.gwf.spark.FlumePushWordCount \--master local[2] \/Users/gaowenfeng/Documents/IDE/newsell/spark-train/target/spark-train-1.0-SNAPSHOT.jar localhost 41414复制代码