Disruptor详解
发布日期:2021-06-30 12:27:56 浏览次数:2 分类:技术文章

本文共 19773 字,大约阅读时间需要 65 分钟。

LMAX是一种新型零售金融交易平台,它能够以很低的延迟产生大量交易。这个系统是建立在JVM平台上,其核心是一个业务逻辑处理器,它能够在一个线程里每秒处理6百万订单。业务逻辑处理器完全是运行在内存中,使用事件源驱动方式。业务逻辑处理器的核心是Disruptor。

Disruptor它是一个开源的并发框架,并获得2011 Duke’s 程序框架创新奖,能够在无锁的情况下实现网络的Queue并发操作。
Disruptor是一个高性能的异步处理框架,或者可以认为是最快的消息框架(轻量的JMS),也可以认为是一个观察者模式的实现,或者事件监听模式的实现。
在Disruptor中,我们想实现hello world 需要如下几步骤:
第一:建立一个Event类
第二:建立一个工厂Event类,用于创建Event类实例对象
第三:需要有一个监听事件类,用于处理数据(Event类)
第四:我们需要进行测试代码编写。实例化Disruptor实例,配置一系列参数。然后我们对Disruptor实例绑定监听事件类,接受并处理数据。
第五:在Disruptor中,真正存储数据的核心叫做RingBuffer,我们通过Disruptor实例拿到它,然后把数据生产出来,把数据加入到RingBuffer的实例对象中即可。
Event类:数据封装类

public class LongEvent {
private Long value; public Long getValue() {
return value; } public void setValue(Long value) {
this.value = value; } }

工厂Event类:实现EventFactory<>接口的实现类

public class LongEventFactory implements EventFactory
{
@Override public LongEvent newInstance() {
return new LongEvent(); }}

EventHandler类:数据处理类实现EventHandler<>接口

/** * 消费者,事件监听 * @author Administrator * */public class LongEventHandler implements EventHandler
{
@Override public void onEvent(LongEvent longEvent, long l, boolean b) throws Exception {
//消费,数据处理 System.out.println(longEvent.getValue()); }}

数据生产类:

public class LongEventProducer {
private final RingBuffer
ringBuffer; public LongEventProducer(RingBuffer
ringBuffer) {
this.ringBuffer=ringBuffer; } public void onData(ByteBuffer bb) {
//可以把ringBuffer看做一个事件队列,那么next就是得到下面一个事件槽 long sequence=ringBuffer.next(); try {
//用上面的索引取出一个空的事件用于填充 LongEvent l=ringBuffer.get(sequence); l.setValue(bb.getLong(0)); }catch (Exception e) {
}finally {
ringBuffer.publish(sequence); } }}

测试类:

public class LongEventTest {
public static void main(String[] args) {
ExecutorService executor=Executors.newCachedThreadPool(); LongEventFactory eventFactory=new LongEventFactory(); //必须2的N次方 int ringBufferSize = 1024*1024; /** //BlockingWaitStrategy 是最低效的策略,但其对CPU的消耗最小并且在各种不同部署环境中能提供更加一致的性能表现 WaitStrategy BLOCKING_WAIT = new BlockingWaitStrategy(); //SleepingWaitStrategy 的性能表现跟BlockingWaitStrategy差不多,对CPU的消耗也类似,但其对生产者线程的影响最小,适合用于异步日志类似的场景 WaitStrategy SLEEPING_WAIT = new SleepingWaitStrategy(); //YieldingWaitStrategy 的性能是最好的,适合用于低延迟的系统。在要求极高性能且事件处理线数小于CPU逻辑核心数的场景中,推荐使用此策略;例如,CPU开启超线程的特性 WaitStrategy YIELDING_WAIT = new YieldingWaitStrategy(); */ Disruptor
dis=new Disruptor<>(eventFactory, ringBufferSize, executor, ProducerType.SINGLE, new YieldingWaitStrategy()); dis.handleEventsWith(new LongEventHandler()); dis.start(); RingBuffer
ringBuffer=dis.getRingBuffer(); LongEventProducer producer=new LongEventProducer(ringBuffer); //LongEventProducerWithTranslator producer = new LongEventProducerWithTranslator(ringBuffer); ByteBuffer bb=ByteBuffer.allocate(8); for (int i = 0; i < 100; i++) {
bb.putLong(0,i); producer.onData(bb); } dis.shutdown(); executor.shutdown(); } }EventProducerWithTranslator实现方式:public class LongEventProducerWithTranslator {
//一个translator可以看做一个事件初始化器,publicEvent方法会调用它 //填充Event private static final EventTranslatorOneArg
TRANSLATOR= new EventTranslatorOneArg
() {
@Override public void translateTo(LongEvent event, long sequence, ByteBuffer buffer) {
event.setValue(buffer.getLong(0)); } }; private final RingBuffer
ringBuffer; public LongEventProducerWithTranslator(RingBuffer
ringBuffer) { this.ringBuffer=ringBuffer; } public void onData(ByteBuffer buffer) { ringBuffer.publishEvent(TRANSLATOR,buffer); }}

Disruptor术语说明

RingBuffer: 被看作Disruptor最主要的组件,然而从3.0开始RingBuffer仅仅负责存储和更新在Disruptor中流通的数据。对一些特殊的使用场景能够被用户(使用其他数据结构)完全替代。

Sequence: Disruptor使用Sequence来表示一个特殊组件处理的序号。和Disruptor一样,每个消费者(EventProcessor)都维持着一个Sequence。大部分的并发代码依赖这些Sequence值的运转,因此Sequence支持多种当前为AtomicLong类的特性。
Sequencer: 这是Disruptor真正的核心。实现了这个接口的两种生产者(单生产者和多生产者)均实现了所有的并发算法,为了在生产者和消费者之间进行准确快速的数据传递。
SequenceBarrier: 由Sequencer生成,并且包含了已经发布的Sequence的引用,这些的Sequence源于Sequencer和一些独立的消费者的Sequence。它包含了决定是否有供消费者来消费的Event的逻辑。
WaitStrategy:决定一个消费者将如何等待生产者将Event置入Disruptor。
Event:从生产者到消费者过程中所处理的数据单元。Disruptor中没有代码表示Event,因为它完全是由用户定义的。
EventProcessor:主要事件循环,处理Disruptor中的Event,并且拥有消费者的Sequence。它有一个实现类是BatchEventProcessor,包含了event loop有效的实现,并且将回调到一个EventHandler接口的实现对象。
EventHandler:由用户实现并且代表了Disruptor中的一个消费者的接口。
Producer:由用户实现,它调用RingBuffer来插入事件(Event),在Disruptor中没有相应的实现代码,由用户实现。
WorkProcessor:确保每个sequence只被一个processor消费,在同一个WorkPool中的处理多个WorkProcessor不会消费同样的sequence。
WorkerPool:一个WorkProcessor池,其中WorkProcessor将消费Sequence,所以任务可以在实现WorkHandler接口的worker吃间移交
LifecycleAware:当BatchEventProcessor启动和停止时,于实现这个接口用于接收通知。

EventProcessor使用:

handler消费类:

public class TradeHandler implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade event) throws Exception {
//生成订单id event.setId(UUID.randomUUID().toString()); System.out.println(event); } @Override public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
this.onEvent(event); }}

Trade数据封装类:

public class Trade {
private String id;//id private String name;//名称 private double price;//金额 private AtomicInteger count=new AtomicInteger(0); public String getId() {
return id; } public void setId(String id) {
this.id = id; } public String getName() {
return name; } public void setName(String name) {
this.name = name; } public double getPrice() {
return price; } public void setPrice(double price) {
this.price = price; } public AtomicInteger getCount() {
return count; } public void setCount(AtomicInteger count) {
this.count = count; } }

EventProcessorMain测试类:

public static void main(String[] args) throws InterruptedException, ExecutionException {
int BUFFER_SIZE=1024; int THREAD_NUMBERS=4; /* * createSingleProducer创建一个单生产者的RingBuffer, * 第一个参数叫EventFactory,从名字上理解就是"事件工厂",其实它的职责就是产生数据填充RingBuffer的区块。 * 第二个参数是RingBuffer的大小,它必须是2的指数倍 目的是为了将求模运算转为&运算提高效率 * 第三个参数是RingBuffer的生产都在没有可用区块的时候(可能是消费者(或者说是事件处理器) 太慢了)的等待策略 */ final RingBuffer
ringBuffer=RingBuffer.createSingleProducer(new EventFactory
() {
@Override public Trade newInstance() {
return new Trade(); } }, BUFFER_SIZE,new YieldingWaitStrategy()); //创建一个线程池 ExecutorService executors=Executors.newFixedThreadPool(THREAD_NUMBERS); //创建SequenceBarrier SequenceBarrier sequenceBarrier=ringBuffer.newBarrier(); //创建消息处理器 BatchEventProcessor
transProcessor=new BatchEventProcessor
(ringBuffer, sequenceBarrier, new TradeHandler()); //这一步的目的是把消费者的位置信息引用注入到生产者 如果只有一个消费者的情况可以省略 ringBuffer.addGatingSequences(transProcessor.getSequence()); //把消息处理器提交到线程池 executors.submit(transProcessor); //如果存在多个消费者,那么重复执行上面三行代码,把TradeHandler换成其他消费者类 Future
future=executors.submit(new Callable
() {
@Override public Trade call() throws Exception {
long seq; for(int i=0;i<10;i++) {
seq=ringBuffer.next();//占一个坑-----ringBuffer一个可用区块 ringBuffer.get(seq).setPrice(Math.random()*9999);//给这个区块放入数据 ringBuffer.publish(seq);//发布这个区块的数据使handler(consumer)可见 } return null; } }); future.get();//等待生成者结束 Thread.sleep(1000);//等待一秒,等消费者处理完成 transProcessor.halt();//通知事件(或者说消息)处理器,可以结束了(并不是马上结束) executors.shutdown();//终止线程 }

WorkProcessor使用:

WorkProcessorMain测试类:

import java.util.concurrent.Callable;import java.util.concurrent.ExecutionException;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.concurrent.Future;import com.lmax.disruptor.EventFactory;import com.lmax.disruptor.IgnoreExceptionHandler;import com.lmax.disruptor.RingBuffer;import com.lmax.disruptor.SequenceBarrier;import com.lmax.disruptor.WorkHandler;import com.lmax.disruptor.WorkerPool;public class WorkProcessorMain {
public static void main(String[] args) throws InterruptedException, ExecutionException {
int BUFFER_SIZE = 1024; int THREAD_NUMBERS = 4; EventFactory
eventFactory = new EventFactory
() {
@Override public Trade newInstance() {
return new Trade(); } }; final RingBuffer
ringBuffer = RingBuffer.createSingleProducer(eventFactory, BUFFER_SIZE); SequenceBarrier sequenceBarrier = ringBuffer.newBarrier(); ExecutorService executors = Executors.newFixedThreadPool(THREAD_NUMBERS); WorkHandler
handler = new TradeHandler(); WorkerPool
workerPool = new WorkerPool<>(ringBuffer, sequenceBarrier, new IgnoreExceptionHandler(), handler); workerPool.start(executors); // 如果存在多个消费者,那么重复执行上面三行代码,把TradeHandler换成其他消费者类 Future
future = executors.submit(new Callable
() { @Override public Trade call() throws Exception { long seq; for (int i = 0; i < 10; i++) { seq = ringBuffer.next();// 占一个坑-----ringBuffer一个可用区块 ringBuffer.get(seq).setPrice(Math.random() * 9999);// 给这个区块放入数据 ringBuffer.publish(seq);// 发布这个区块的数据使handler(consumer)可见 } return null; } }); future.get();// 等待生成者结束 Thread.sleep(1000);// 等待一秒,等消费者处理完成 workerPool.halt();// 通知事件(或者说消息)处理器,可以结束了(并不是马上结束) executors.shutdown();// 终止线程 }}

并行计算 - 多边形高端操作

菱形操作

Disruptor可实现串并行同时编码。

在复杂场景下使用RingBuffer(希望P1生产的数据给C1、C2并行执行,最后C1、C2执行结束后C3执行)

  • C1和C2并行执行。

六边形操作

C1h和C2并行执行,C4和C5并行执行,并行执行完后执行C3

示例:
C1:

import com.lmax.disruptor.EventHandler;import com.lmax.disruptor.WorkHandler;import com.moudle.disruptorDemo.generate1.Trade;public class Handler1 implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade trade) throws Exception {
System.out.println("handler1 set name:"); trade.setName("h1"); Thread.sleep(1000); } @Override public void onEvent(Trade arg0, long arg1, boolean arg2) throws Exception {
this.onEvent(arg0); }}

C2

import com.lmax.disruptor.EventHandler;import com.lmax.disruptor.WorkHandler;import com.moudle.disruptorDemo.generate1.Trade;public class Handler2 implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade trade) throws Exception {
System.out.println("handler2 set price:"); trade.setPrice(17); Thread.sleep(1000); } @Override public void onEvent(Trade arg0, long arg1, boolean arg2) throws Exception {
this.onEvent(arg0); }}

C3

import com.lmax.disruptor.EventHandler;import com.lmax.disruptor.WorkHandler;import com.moudle.disruptorDemo.generate1.Trade;public class Handler3 implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade event) throws Exception {
System.out.println("handler3: name: " + event.getName() + " , price: " + event.getPrice() + "; instance: " + event.toString()); } @Override public void onEvent(Trade arg0, long arg1, boolean arg2) throws Exception {
this.onEvent(arg0); }}

C4

import com.lmax.disruptor.EventHandler;import com.lmax.disruptor.WorkHandler;import com.moudle.disruptorDemo.generate1.Trade;public class Handler4 implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade trade) throws Exception {
System.out.println("handler4 set addName:"); trade.setName(trade.getName()+"h4"); } @Override public void onEvent(Trade arg0, long arg1, boolean arg2) throws Exception {
this.onEvent(arg0); }}

C5

import com.lmax.disruptor.EventHandler;import com.lmax.disruptor.WorkHandler;import com.moudle.disruptorDemo.generate1.Trade;public class Handler5 implements EventHandler
,WorkHandler
{
@Override public void onEvent(Trade trade) throws Exception {
System.out.println("handler5 set add price:"); trade.setPrice(trade.getPrice()+3); } @Override public void onEvent(Trade arg0, long arg1, boolean arg2) throws Exception {
this.onEvent(arg0); }}

P1(生产者)

import java.util.Random;import java.util.concurrent.CountDownLatch;import com.lmax.disruptor.EventTranslator;import com.lmax.disruptor.dsl.Disruptor;import com.moudle.disruptorDemo.generate1.Trade;public class TradePublisher implements Runnable{
Disruptor
disruptor; private CountDownLatch latch; private static int count =1;//模拟百万次交易的发生 public TradePublisher(Disruptor
disruptor,CountDownLatch latch){
this.disruptor=disruptor; this.latch=latch; } @Override public void run() {
TradeEventTranslator translator=new TradeEventTranslator(); for(int i=0;i
{
private Random random=new Random(); @Override public void translateTo(Trade trade, long arg1) {
this.generateTrade(trade); } private Trade generateTrade(Trade trade){
trade.setPrice(random.nextDouble()*9999); return trade; }}

Main:

package com.moudle.disruptorDemo.generate2;import java.util.concurrent.CountDownLatch;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import com.lmax.disruptor.BusySpinWaitStrategy;import com.lmax.disruptor.EventFactory;import com.lmax.disruptor.dsl.Disruptor;import com.lmax.disruptor.dsl.EventHandlerGroup;import com.lmax.disruptor.dsl.ProducerType;import com.moudle.disruptorDemo.generate1.Trade;public class Main {
public static void main(String[] args) throws InterruptedException {
long beginTime=System.currentTimeMillis(); int bufferSize=1024; ExecutorService executor=Executors.newFixedThreadPool(8); Disruptor
disruptor=new Disruptor<>(new EventFactory
() {
@Override public Trade newInstance() {
return new Trade(); } }, bufferSize, executor, ProducerType.SINGLE, new BusySpinWaitStrategy()); //菱形操作 //使用disruptor创建消费者组C1,C2 EventHandlerGroup
handlerGroup=disruptor.handleEventsWith(new Handler1(),new Handler2()); //声明在C1,C2完事之后执行JMS消息发送操作 也就是流程走到C3 handlerGroup.then(new Handler3()); //輸出結果:// handler1 set name:// handler2 set price:// handler3: name: h1 , price: 17.0; instance: com.moudle.disruptorDemo.generate1.Trade@220a5c4d /*//六边形操作 Handler1 h1 = new Handler1(); Handler2 h2 = new Handler2(); Handler3 h3 = new Handler3(); Handler4 h4 = new Handler4(); Handler5 h5 = new Handler5(); disruptor.handleEventsWith(h1,h2); disruptor.after(h1).handleEventsWith(h4); disruptor.after(h2).handleEventsWith(h5); disruptor.after(h4,h5).handleEventsWith(h3); //输出结果:// handler1 set name:// handler2 set price:// handler4 set addName:// handler5 set add price:// handler3: name: h1h4 , price: 20.0; instance: com.moudle.disruptorDemo.generate1.Trade@5e6d6957 */ /* //顺序执行 disruptor.handleEventsWith(new Handler1()). handleEventsWith(new Handler2()). handleEventsWith(new Handler3()); //输出结果:// handler1 set name:// handler2 set price:// handler3: name: h1 , price: 17.0; instance: com.moudle.disruptorDemo.generate1.Trade@331d6441 */ disruptor.start();//启动 CountDownLatch latch=new CountDownLatch(1); //生产者准备 executor.submit(new TradePublisher(disruptor, latch)); latch.await();//等待生产完成 disruptor.shutdown(); executor.shutdown(); }}

多生产者多消费者的使用:

Order订单类:

package com.moudle.disruptorDemo.multi;public class Order {
private String id;//id private String name;// private double price;// public String getId() {
return id; } public void setId(String id) {
this.id = id; } public String getName() {
return name; } public void setName(String name) {
this.name = name; } public double getPrice() {
return price; } public void setPrice(double price) {
this.price = price; } }

Producer生产者:

package com.moudle.disruptorDemo.multi;import com.lmax.disruptor.RingBuffer;public class Producer {
private final RingBuffer
ringBuffer; public Producer(RingBuffer
ringBuffer){
this.ringBuffer=ringBuffer; } /** * onData用来发布事件,每调用一次就发布一次事件 * 它的参数会用过事件传递给消费者 */ public void onData(String data){
//可以把ringBuffer看做一个事件队列,那么next就是得到下面一个事件槽 long sequence=ringBuffer.next(); try {
//用上面的索引取出一个空的事件用于填充(获取该序号对应的事件对象) Order order=ringBuffer.get(sequence); //获取要通过事件传递的业务数据 order.setId(data); } catch (Exception e) {
}finally{
//发布事件 //注意,最后的 ringBuffer.publish 方法必须包含在 finally 中以确保必须得到调用;如果某个请求的 sequence 未被提交,将会堵塞后续的发布操作或者其它的 producer。 ringBuffer.publish(sequence); } } }

Consumer消费者:

package com.moudle.disruptorDemo.multi;import java.util.concurrent.atomic.AtomicInteger;import com.lmax.disruptor.WorkHandler;public class Consumer implements WorkHandler
{
private String consumerId; private static AtomicInteger count=new AtomicInteger(0); public Consumer(String consumerId){
this.consumerId=consumerId; } @Override public void onEvent(Order order) throws Exception {
System.out.println("当前消费者:"+this.consumerId+",消费消息:"+order); count.incrementAndGet(); } public int getCount(){
return count.get(); } }

Main测试类:

package com.moudle.disruptorDemo.multi;import java.util.UUID;import java.util.concurrent.CountDownLatch;import java.util.concurrent.Executor;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import com.lmax.disruptor.EventFactory;import com.lmax.disruptor.ExceptionHandler;import com.lmax.disruptor.RingBuffer;import com.lmax.disruptor.SequenceBarrier;import com.lmax.disruptor.WorkerPool;import com.lmax.disruptor.YieldingWaitStrategy;import com.lmax.disruptor.dsl.ExceptionHandlerWrapper;import com.lmax.disruptor.dsl.ProducerType;public class Main {
public static void main(String[] args) throws Exception{
// RingBuffer
ringBuffer=RingBuffer.create(// ProducerType.MULTI, new EventFactory
() {
// @Override// public Order newInstance() {
// return new Order();// }// }, 1024*1024, new YieldingWaitStrategy()); //创建ringBuffer RingBuffer
ringBuffer=RingBuffer.createMultiProducer(new EventFactory
() {
@Override public Order newInstance() {
return new Order(); } }, 1024*1024, new YieldingWaitStrategy()); //创建SequenceBarrier SequenceBarrier barriers=ringBuffer.newBarrier(); //创建3个消费者实例 Consumer[] consumers=new Consumer[3]; for (int i = 0; i < consumers.length; i++) {
consumers[i]=new Consumer("c"+i); } WorkerPool
workerPool=new WorkerPool<>( ringBuffer, barriers, new IntEventExceptionHandler(), consumers); //这一步的目的是把消费者的位置信息引用注入到生产者 如果只有一个消费者的情况可以省略。 //workerPool.getWorkerSequences()获取Sequence集合 ringBuffer.addGatingSequences(workerPool.getWorkerSequences()); //创建线程池 ExecutorService executorService=Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()); workerPool.start(executorService); final CountDownLatch latch=new CountDownLatch(1); for (int i = 0; i < 100; i++) {
final Producer producer=new Producer(ringBuffer); new Thread(new Runnable() {
@Override public void run() {
try {
//等待生产者100个线程启动 latch.await(); for (int j = 0; j < 100; j++) {
//生产数据 producer.onData(UUID.randomUUID().toString()); } } catch (InterruptedException e) {
e.printStackTrace(); } } }).start(); } //等待两秒,等生产者的100个线程启动 Thread.sleep(2000); System.out.println("---------------开始生产-----------------"); latch.countDown(); Thread.sleep(5000); System.out.println("总数:"+consumers[0].getCount()); executorService.shutdown(); } static class IntEventExceptionHandler implements ExceptionHandler
{ @Override public void handleEventException(Throwable arg0, long arg1, Order arg2) { } @Override public void handleOnShutdownException(Throwable arg0) { } @Override public void handleOnStartException(Throwable arg0) { } }}

参考

  • https://programtip.com/zh/art-111663

转载地址:https://javaedge.blog.csdn.net/article/details/109007314 如侵犯您的版权,请留言回复原文章的地址,我们会给您删除此文章,给您带来不便请您谅解!

上一篇:Homebrew替换国内镜像,高速下载!
下一篇:单一职责原则(Single Responsibility Principle,SRP)

发表评论

最新留言

能坚持,总会有不一样的收获!
[***.219.124.196]2024年05月05日 18时10分55秒

关于作者

    喝酒易醉,品茶养心,人生如梦,品茶悟道,何以解忧?唯有杜康!
-- 愿君每日到此一游!

推荐文章