↓Skip to main content
IT Blog IT Blog
  1. All Posts/

fast-data-dev

Tools Shell

fast-data-dev / kafka-lenses-dev (Lenses Box)

lensesio/fast-data-dev


lensesio/box (lensesio/box)


Join the Slack Lenses.io Community!
Apache Kafka docker image for developers; with
Lenses
(lensesio/box)
or Lenses.io’s open source UI tools
(lensesio/fast-data-dev). Have
a full fledged Kafka installation up and running in seconds and top it off with
a modern streaming platform (only for kafka-lenses-dev), intuitive UIs and extra
goodies. Also includes Kafka Connect, Schema Registry, Lenses.io’s Stream Reactor
25+ Connectors and more.

View latest demo on-line or get a free license for Lenses Box

Introduction

When you need:

  1. A Kafka distribution with Apache Kafka, Kafka Connect, Zookeeper, Confluent Schema Registry and REST Proxy
  2. Lenses.io Lenses or kafka-topics-ui, schema-registry-ui, kafka-connect-ui
  3. Lenses.io Stream Reactor, 25+ Kafka Connectors to simplify ETL processes
  4. Integration testing and examples embedded into the docker

just run:

docker run --rm --net=host lensesio/fast-data-dev

That’s it. Visit http://localhost:3030 to get into the fast-data-dev environment

All the service ports are exposed, and can be used from localhost / or within
your IntelliJ. The kafka broker is exposed by default at port 9092, zookeeper
at port 2181, schema registry at 8081, connect at 8083. As an example, to
access the JMX data of the broker run:

jconsole localhost:9581

If you want to have the services remotely accessible, then you may need to pass
in your machine’s IP address or hostname that other machines can use to access
it:

docker run --rm --net=host -e ADV_HOST=<IP> lensesio/fast-data-dev

Hit control+c to stop and remove everything

Mac and Windows users (docker-machine)

Create a VM with 4+GB RAM using Docker Machine:

docker-machine create --driver virtualbox --virtualbox-memory 4096 lensesio

Run docker-machine ls to verify that the Docker Machine is running correctly. The command’s output should be similar to:

$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS
lensesio     *        virtualbox   Running   tcp://192.168.99.100:2376           v17.03.1-ce

Configure your terminal to be able to use the new Docker Machine named lensesio:

eval $(docker-machine env lensesio)

And run the Kafka Development Environment. Define ports, advertise the hostname and use extra parameters:

docker run --rm -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 
       -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=192.168.99.100 
       lensesio/fast-data-dev:latest

That’s it. Visit http://192.168.99.100:3030 to get into the fast-data-dev environment

Run on the Cloud

You may want to quickly run a Kafka instance in GCE or AWS and access it from your local
computer. Fast-data-dev has you covered.
Start a VM in the respective cloud. You can use the OS of your choice, provided it has
a docker package. CoreOS is a nice choice as you get docker out of the box.
Next you have to open the firewall, both for your machines but also for the VM itself.
This is important!
Once the firewall is open try:

docker run -d --net=host -e ADV_HOST=[VM_EXTERNAL_IP] 
           -e RUNNING_SAMPLEDATA=1 lensesio/fast-data-dev

Alternatively just export the ports you need. E.g:

docker run -d -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 
           -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=[VM_EXTERNAL_IP] 
           -e RUNNING_SAMPLEDATA=1 lensesio/fast-data-dev

Enjoy Kafka, Schema Registry, Connect, Lensesio UIs and Stream Reactor.

Customize execution

Fast-data-dev and kafka-lenses-dev support custom configuration and extra features
via environment variables.

fast-data-dev / kafka-lenses-dev advanced configuration

Optional Parameters
Description

CONNECT_HEAP=3G
Configure the maximum (-Xmx) heap size allocated to Kafka Connect. Useful when you want to start many connectors.

<SERVICE>_PORT=<PORT>
Custom port <PORT> for service, `` will disable it. <SERVICE> one of ZK, BROKER, BROKER_SSL, REGISTRY, REST, CONNECT.

<SERVICE>_JMX_PORT=<PORT>
Custom JMX port <PORT> for service, `` will disable it. <SERVICE> one of ZK, BROKER, BROKER_SSL, REGISTRY, REST, CONNECT.

USER=username
Run in combination with PASSWORD to specify the username to use on basic auth.

PASSWORD=password
Protect the fast-data-dev UI when running publicly. If USER is not set, the default username is kafka.

SAMPLEDATA=0
Do not create topics with sample avro and json records; (e.g do not create topics sea_vessel_position_reports, reddit_posts).

RUNNING_SAMPLEDATA=1
In the sample topics send a continuous (yet very low) flow of messages, so you can develop against live data.

RUNTESTS=0
Disable the (coyote) integration tests from running when container starts.

FORWARDLOGS=0
Disable running the file source connector that brings broker logs into a Kafka topic.

RUN_AS_ROOT=1
Run kafka as root user – useful to i.e. test HDFS connector.

DISABLE_JMX=1
Disable JMX – enabled by default on ports 9581 – 9585. You may also disable it individually for services.

ENABLE_SSL=1
Generate a CA, key-certificate pairs and enable a SSL port on the broker.

SSL_EXTRA_HOSTS=IP1,host2
If SSL is enabled, extra hostnames and IP addresses to include to the broker certificate.

CONNECTORS=<CONNECTOR>[,<CON2>]
Explicitly set which connectors* will be enabled. E.g hbase, elastic (Stream Reactor version)

DISABLE=<CONNECTOR>[,<CON2>]
Disable one or more connectors*. E.g hbase, elastic (Stream Reactor version), elasticsearch (Confluent version)

BROWSECONFIGS=1
Expose service configuration in the UI. Useful to see how Kafka is setup.

DEBUG=1
Print stdout and stderr of all processes to container’s stdout. Useful for debugging early container exits.

SUPERVISORWEB=1
Enable supervisor web interface on port 9001 (adjust via SUPERVISORWEB_PORT) in order to control services, run tail -f, etc.

*Available connectors are: azure-documentdb, blockchain, bloomberg, cassandra,
coap, druid, elastic, elastic5, ftp, hazelcast, hbase, influxdb, jms, kudu,
mongodb, mqtt, pulsar, redis, rethink, voltdb, couchbase, dbvisitreplicate,
debezium-mongodb, debezium-mysql, debezium-postgres, elasticsearch, hdfs,
jdbc, s3, twitter.
To programmatically get a list, run:

docker run --rm -it lensesio/fast-data-dev 
       find /opt/landoop/connectors -type d -maxdepth 2 -name "kafka-connect-*"

Optional Parameters (unsupported)
Description

WEB_ONLY=1
Run in combination with --net=host and docker will connect to the kafka services running on the local host. Please use our UI docker images instead.

TOPIC_DELETE=0
Configure whether you can delete topics. By default topics can be deleted. Please use KAFKA_DELETE_TOPIC_ENABLE=false instead.

Configure Kafka Components

You may configure any Kafka component (broker, schema registry, connect, rest proxy) by converting the configuration option to uppercase, replace dots with underscores and prepend with
<SERVICE>_.
As example:

  • To set the log.retention.bytes for the broker, you would set the environment
    variable KAFKA_LOG_RETENTION_BYTES=1073741824.
  • To set the kafkastore.topic for the schema registry, you would set
    SCHEMA_REGISTRY_KAFKASTORE_TOPIC=_schemas.
  • To set the plugin.path for the connect worker, you would set
    CONNECT_PLUGIN_PATH=/var/run/connect/connectors/stream-reactor,/var/run/connect/connectors/third-party,/connectors.
  • To set the schema.registry.url for the rest proxy, you would set
    KAFKA_REST_SCHEMA_REGISTRY_URL=http://localhost:8081.

We also support the variables that set JVM options, such as KAFKA_OPTS, SCHEMA_REGISTRY_JMX_OPTS, etc.
Lensesio’s Kafka Distribution (LKD) supports a few extra flags as well. Since in
the Apache Kafka build, both the broker and the connect worker expect JVM
options at the default KAFKA_OPTS, LKD supports using BROKER_OPTS, etc for
the broker and CONNECT_OPTS, etc for the connect worker. Of course
KAFKA_OPTS are still supported and apply to both applications (and…



↑