Senior Data Engineer
Location: Atlanta, Georgia, Redmond, Washington, Reston, Virginia
Job Type: Full time
Microsoft runs on trust. Earning and keeping that trust has never been more important.
In Customer Security and Trust (CST), program managers, engineers, analysts, investigators, data scientists, attorneys, and business professionals are responsible for some of the most exciting projects at Microsoft focused on protecting our customers.
- We actively partner with law enforcement and security organizations around the globe to disrupt bad actors who are using malware, phishing scams, IP abuse, tech support fraud, and other means to target our customers.
- We ensure that governments who are requesting data from Microsoft in support of their criminal investigations through legal orders follow due process, striking a balance between fighting cybercrime and protecting customer data.
- We provide solutions to demonstrate that Microsoft is in compliance with cybersecurity practices in support of government and industry customers.
Our CST Engineering team is growing. We are looking for a Senior Data Engineer to join our team and help deliver on our vision of creating a safer digital experience for every person and organization on the planet, which in turn, promotes trust in Microsoft.
As part of the CST Engineering team, you will be hands-on building products from inception to launch and everything in-between.
The successful candidate will have a strong sense of teamwork, personal integrity, accountability, and the ability to understand business functions and requirements, translating to innovative working applications while navigating competing priority tradeoffs
- Leads collaborations with appropriate stakeholders across product groups and uses understanding of tradeoffs and project costs to recommend appropriate data requirements.
- Applies in-depth knowledge of the business to design a data model that is appropriate for the project and translates business requirements into design specification documents to model the flow and storage of data across multiple data pipelines.
- Identifies multiple, complex data sources and builds advanced code to extract raw data from identified upstream sources using query languages, tools, or machine learning algorithms, while assuring accuracy, validity, and reliability of the data across the pipeline.
- Uses and contributes to the development of advanced and reusable modification techniques and operations.
- Collaborates with others to identify opportunities to optimize data tools used to transform, manage, and access data across teams. Ensures the effectiveness and placement of performance monitoring protocols across multiple data pipelines.
- Leads root cause analysis in response to detected problems/anomalies to identify the reason for alerts and implement advanced solutions that prevent recurring points of failure.
- Anticipate the need for data governance and works with designated committees to ensure that data modeling and data handling procedures are compliant with applicable laws and policies across responsibilities in the data pipeline.
- Uses technical expertise of existing tools’ strengths and limitations to improve data tools used to transform, manage, and access data across the organization
- Leads the negotiation of complex agreements with partners and system owners to align on project delivery, data ownership between both parties, and the shape and cadence of data extraction for a complex product
- Embody our culture and a demonstrated commitment to the company’s core values, including a commitment to high ethical standards, unquestioned integrity, and a commitment to advancing the company’s diversity & inclusion commitments.
- A bachelor’s degree with 8+ years of solution development using Agile development methodologies OR equivalent experience.
- Demonstrated experience working across the data stack on a variety of complex projects, while providing technical and thought leadership.
- 7+ years of experience with SQL
- 5+ years of experience with Python
- 2+ years of experience with Graph databases
- 2+ years of experience with Big Data or Hyper Scale processing offering in Azure
- 2+ years of experience in big data platforms and tools (Hadoop, Hive, Spark, Kafka).
- 5+ years of experience in the BI/Analytics space with deep experience using a combination of tools/tech including but not limited to NoSQL, SQL Server, Kusto, Azure Synapse Analytics, Data Explorer, Cosmos, App Insights, and Snowflake (or similar).
- Experience developing best practices on designing components of the data model for a functional area of a project.
- US Citizenship.
Nice to Have / Preferred Qualifications:
- Uses technical expertise of existing tools’ strengths and limitations to improve data tools used to transform, manage, and access data across the organization.
- Self-starter with excellent attention to detail and desire to learn and master new technologies.
- Ability to organize, schedule work and communicate effectively in a fast-paced environment.
- Ability to maintain confidentiality.
- Collaborate effectively with product management, program management, engineers and stakeholders.
- Excellent analytical and critical thinking skills.
Interprets laws and works with engineers to understand and develop data modeling and data management, access, and visualization procedures that ensure compliance with applicable laws and policies across the whole data pipeline.
- Experience with microservices architectures, container technologies like docker and Kubernetes.
- Expertise in the areas of CI/CD, Continuous testing, and site reliability.
- Working experience in AI/ML Ops.
- Ability to influence without authority and thrive in an ambiguous environment.
If hired for this position, the team you would be joining is part of our Cloud organization and/or works with government contracts and as such has a unique background check requirement, detailed below. Please note you will be provided with steps for completing the check if you accept a role on the team.
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
- Citizenship Verification: This position requires verification of US Citizenship to meet federal government security requirements.
Citizenship Verification: This position requires verification of US Citizenship to meet federal government security requirements.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.
Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.