On Linux, it’s quite easy to get a single Kafka node cluster up & running with Docker. Confluence gives a great primer in their documentation:. On macOS, things get a little bit more complicated because containers are not directly supported in the OS.
To use docker on macOS, one must use the docker-machine (older method) or the new Docker for Mac. This post presents at method on both. The result will be a single node cluster. You will be able to:. produce/consume from another container. produce/consume from the host. Note: with a few changes to the kafka docker image and the docker-compose.yml file, it's possible to have a multiple nodes cluster.
The problem The problem on Mac is that Docker runs with a VM. It’s true in the case of the Docker Machine but it’s also true in the case of Docker for Mac. The implementation is different but the result is essentially the same: we need an address:port that is resolvable from the host and from other containers. On linux, there is no problem, you slap -network host arguments when you run the containers. This way, anybody can use localhost:port to communicate. On Mac, -network host is useless. With the Docker Machine, you get an IP, which is the IP of the virtual machine where docker runs.
The Web application is built with Docker (docker-compose), and it is a way to access the API of the local environment. I will explain with the assumption that you use Docker for Mac.
You can use this IP from your host, but it will be not work from another container. With Docker for Mac, the VM implementation is different and from the host perspective, things are mapped on localhost. But from within a container, that’s not true, localhost resolves on the container and not on the host. The solution I found uses the famous.
The solution To resolves the issue, I did what kubernetes does to allow multiple containers in one pod, to talk to each other using localhost. We will use the pause container to do port forwarding from the host to the containers. The pause container will expose port on the host, map them on it’s localhost interface. The other containers will use the pause container’s network interface as their network interface. Kafka works with Zookeeper and they both need at least one port exposed, 9092 and 2181 respectively.