Import Streaming Data from Any REST API Into Apache Kafka Using JDBC

Connect to and query any REST API directly from Apache Kafka without writing a single line of code using Progress DataDirect Autonomous REST Connector.

Strike while the iron is hot is how the adage goes. Literally, a blacksmith can only manipulate metal when it is just out of the forge. After that, if the metal becomes too cool it can’t be used until the temperature is once again correct.

Your data is no different. Whether it is real-time stock market data for a financial services company or real-time sensor data for an IoT operation, your business thrives or dies on its ability to keep up with the dynamic swings of its source data.

Apache’s Kafka application has quickly emerged as the preeminent open source platform for fast, scalable processing of streaming data. Businesses of all kinds rely on this speed and dependability to create, monitor and report on their streaming data clusters. As with any data application, connectivity and access become a vital part of its success.

In this tutorial, we will show you how to import stock market data from any REST API to Kafka with a little help from Progress DataDirect Autonomous REST Connector. Autonomous REST Connector allows you to seamlessly and codelessly connect REST API data sources to your BI application. While Kafka has its own JDBC connector, it cannot connect to REST APIs. Progress DataDirect Autonomous REST Connector can connect to and query any REST API using SQL.


Our tutorial’s stock market example uses PRGS (that’s us!), but feel free to substitute with your own favorite ticker symbol. Once you’re up and running, Kafka will run your JDBC source connectors and continue polling the REST API via the DataDirect Autonomous REST Connector every five seconds to fetch new data. How’s that for real-time information?

Everything we build at Progress DataDirect is to help you improve your business. This tutorial is meant to show you one of many possibilities.