We’re looking for a hands-on Data Engineer with strong experience building reliable, real-time data pipelines using Amazon Kinesis and Databricks.
You’ll work with modern AWS data technologies to develop and optimise streaming pipelines, moving high-volume data from Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.
What you’ll do:
- Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
- Develop streaming solutions in Databricks using Spark Structured Streaming.
- Ingest and manage streaming data within Delta tables.
- Optimise pipelines for performance, latency, reliability and cost.
- Build monitoring and alerting using AWS CloudWatch.
- Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
- Develop clean, testable Python and SQL code.
- Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.
What you’ll bring:
- Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
- Strong Python and SQL skills.
- Experience with Apache Spark Structured Streaming.
- Hands-on experience with Databricks, ideally including Delta Lake and Unity Catalog.
- Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
- Understanding of scalable and reliable data-pipeline design.
- Strong troubleshooting and problem-solving skills.
- Experience with Kafka is beneficial
Qualifications
Additional Requirements:
Security Clearance Requirement: Given the secure nature of this role, we are seeking candidates who are Australian Citizens with an active security clearance.
This requirement ensures that all team members are cleared to handle sensitive information and contribute to maintaining the highest standards of security in our operations.
Benefits of working at Accenture:
- 18 weeks paid parental leave
- Long & short-term career break opportunities
- Structured career development program
- Local and international career opportunities.
- Certified as a Family Inclusive Workplace™
- Flexible Work Arrangements - centred around Accenture’s Truly Human ethos and our commitment to supporting the health and wellbeing of our people.
- We are proud to be in the top 3 of last year’s Diversity & Inclusion Index!
#LI_GM