Deploying Confluent Connectors on AWS ECS

Overview

Apache Kafka serves as a critical component in constructing scalable, distributed data pipelines. Confluent extends Kafka's capabilities through connectors that facilitate seamless data movement between Kafka topics and external systems. This guide demonstrates deploying Confluent Self-Managed Connectors on AWS ECS using CloudFormation.

Prerequisites

Before beginning deployment:

• AWS account with appropriate permissions
• Docker installed locally
• AWS CLI installed
• Confluent Platform installed locally or on a separate cluster

Step 1: Prepare Docker Images

Create Docker images for Confluent Self-Managed Connectors following Confluent's documentation. A sample Dockerfile for a JMS connector:

FROM confluentinc/cp-kafka-connect-base:6.2.1
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-jms:11.0.11
USER root
RUN yum install -y jq findutils
RUN mkdir /usr/share/connector
COPY connect-distributed.properties /usr/share/connector/
RUN chown -R appuser:appuser /usr/share/connector/
USER appuser
COPY connect-log4j.properties /etc/kafka/
COPY ./lib/12.1.0.2/*.jar /usr/share/confluent-hub-components/confluentinc-kafka-connect-jms/lib

Step 2: Build and Push to Amazon ECR

Build and push Docker images to Amazon ECR:

#!/bin/sh -e
ECR_REPO=$(aws sts get-caller-identity --query Account --output text).dkr.ecr.ap-southeast-2.amazonaws.com/%REPOSITORY_NAME%

echo "Docker login"
aws ecr get-login-password --region ap-southeast-2 | docker login --username AWS --password-stdin $ECR_REPO

echo "Building docker image"
docker build -t %CONNECTOR_NAME% .
docker tag %CONNECTOR_NAME% $ECR_REPO/$CONNECTOR_NAME:$BUILD_VERSION

echo "Pushing docker image"
docker push $ECR_REPO/$CONNECTOR_NAME:$BUILD_VERSION

Step 3: Deploy to AWS ECS

Deploy using CloudFormation:

#!/bin/sh -e
IMAGE_VERSION=$1
STACK_NAME='%STACK_NAME%'

if [[ -z "$IMAGE_VERSION" ]]; then
    echo "Please provide IMAGE_VERSION"
    exit 1;
fi

cmd="docker run --rm \
    -v `pwd`:/cwd \
    -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
    -e AWS_REGION \
    realestate/stackup:1.4.6 $STACK_NAME up \
    -t fargate.yml \
    -o ImageVersion=$IMAGE_VERSION"

echo "updating $STACK_NAME"
eval "$cmd"

Step 4: Create JMS Connector Configuration

Reference Confluent's official documentation for JMS Source and Sink Connector configuration specifics.

Step 5: Confluent Cluster Setup

Configure appropriate access control lists for topics and consumer groups using Confluent CLI or Terraform.

Step 6: Deploy JMS Connector

Deploy via REST API:

if [ ! -f ${CONNECTOR_FILE_PATH}/connector.json ]; then
    echo "ERROR: Failed to find Connector definition"
    exit 1
fi

echo "... deploying Connector [ ${CONNECTOR_NAME} ]"
curl -i -s -X POST -H "%API_DETAILS%: ${WES_API_HEADER}" \
    -H "X-ENTITY-NAME: ${PROJECT_NAME}" \
    -H "Content-Type: application/json" \
    ${CONNECT_URL}/connectors --data "$RESULT"

Conclusion

Successfully deploying Confluent Connectors on AWS ECS requires careful attention to Docker configuration, ECR integration, and CloudFormation orchestration. LimePoint, recognized as a Confluent Premier Partner, offers expertise in cluster setup, connector configuration, and cloud-based solutions using Confluent Kafka technology.