Overview about situation

Imagine we are building a microservices system with BPMN workflow engine (Camunda Zeebe) at the heart for service orchestration. The system is completely event-driven, so we are going to put kafka into the picture acting as a middleware for all messages going back and forth. At the end, this causes below problems with workflow engine orchestration

• WE does not support kafka connection out of the box. So, we going to use WE embeded mode in serverlet container to leverage kafka connection implementation.
• However, it's not recommended by the provider by a list of disadvantages list here. Then, we tried to implement a separated component by follow the reference architecture from official kafka connect.
• This is nice architecture. According to this, each service task need another intermediate catched event to continue the process. It turns out we had a double long process compared to what actual business process should be. And, it's technical aspect that business people don't want to know.
• The minior consequence from this architecture is the microservice is receiving unexpected data model due to containing the headers as key-value from WE

So, the problem we are going to solve is shorten the bussiness process and keep the technical communication asynchronously (even-driven).

Enhancement on kafka connect

The simplest way to shorten the bussiness process is activating and completing the jobs in a synchronous manner, from the kafka connect. Additionally, kafka connect has to wait for the response from the microservices and to complete the exact activated job. We came up with following architecture

As the image depicts, kafka connect need to store the running activated jobs. By doing this, kafka connect could communicate with WE in synchronous manner. The activated jobs are managed by kafka connect internally. The new issue came from this architecture is making the running jobs high available and resilient enough for rebooting or unexpected crash.