Sr. Data Engineer
Location: Redmond, Washington
Job Type: Full time
Are you passionate about building and delivering big data analytics and machine learning solutions using cutting edge technologies? Are you looking for an opportunity to be a part of an elite team of data engineers to transform data into critical business insights for one of the most impactful organizations in Azure?
The Insights and Analytics team within the Azure Data organization is looking for a Senior Data Engineer. Azure Data is one of the most innovative organizations that builds Microsoft’s industry-leading data platform products including SQL Server and Azure services such as SQL Database, Synapse and Cosmos DB. You’ll be joining a team of elite data professionals focused on building and delivering big data analytics solutions to support all aspects of our data-driven business decision processes. The Insights and Analytics team builds, curates, and consumes data for reporting, deep data analysis as well as advanced data science. We develop insights that are used in scenarios spanning increasing customer satisfaction, defining, and tracking key business metrics and informing the future roadmap for the Data business.
As a Sr. Data Engineer on the Insights and Analytics team, you are an authority, familiar with all data analytics and warehousing technical components, infrastructure, and their integration. You are an expert in both high-level design and experience with data modeling and operationalizing ML models as well as deeply technical in building durable data pipelines with the ability to scale elegantly with data volume growth. You will be responsible for designing, developing, and maintaining data pipelines and back-end services for real-time decisioning, reporting, optimization, data collection, and related functions. The code you write will enable our users to get data in a timely manner. You'll work on a variety of tools and systems, including cloud-based applications and data-platform components (e.g., data pipelines, data flows, data factories, notebooks, visualization, reporting, etc.). In this role, you will be working in Agile development method for faster, quick-wins with the quality of design, development and operating the Analytics & BI components you develop.
We are a highly collaborative team that strives to build lasting relationships with stakeholders across the company from product management to engineering, finance, marketing and more. Our team includes a diverse set of roles and as such, we value the diversity of skills we each bring to the team and apply towards our common goals. We work together to build an inclusive environment that values a variety of perspectives and backgrounds.
In this role, you’ll partner with program managers, engineers and leaders across the Azure Data and broader Azure teams to enable timely and actionable insights aligned with the business’s key strategic programs as well as perform deep analyses to understand customer usage patterns and behaviors in conjunction with our broader analytics and data science team. More specifically, you will:
- Design, architect and maintain key datasets in lake houses that provide structured and timely access to actionable business information with the needs of the end customer always in view.
- Design, develop, and maintain data pipelines to curate, clean and process data from various telemetry streams, data lakes and supporting data sources and craft a unified dimensional or data model for analytics and reporting.
- Develop a deep understanding of vast data sources (existing on the cloud) and know exactly how, when, and which data to use to tackle business problems.
- Leverage data products and services including Azure Data Factory, Synapse, SQL Database, Data Flow and Pipelines, Cosmos Scope, Power BI, Azure Purview, Spark Notebook, Python and other big data related tools to develop a reference architecture of an Analytics, BI + DSML solution.
- Establishes and builds processes and structures based on business and technical requirements to channel data from multiple inputs, route appropriately and store using any combination of distributed (cloud) structures, local databases, and other applicable storage forms as required.
- Develops technical tools and programming that leverage artificial intelligence, machine learning and big-data techniques to cleanse, organize and transform data and to maintain, defend and update data structures and integrity on an automated basis.
- Creates and establishes design standards and assurance processes for software, systems and applications development to ensure compatibility and operability of data connections, flows and storage requirements.
- Reviews internal and external business and product requirements for data operations and activity and suggests changes and upgrades to systems and storage to accommodate ongoing needs.
To be successful in this role, you must have strong skills in written and oral communications, a can-do, growth mindset attitude and the willingness to tackle hard problems in innovative ways. You also thrive in a team environment that values cross team collaboration and building on the success of others.
- 8+ years of hands-on data engineering experience building, developing and supporting large-scale big data processing pipelines and analytics solutions leveraging data products/services including Azure Data Factory, Synapse, SQL Database, Spark, Data Flow and Pipelines.
- 8+ years of hands-on experience in designing and implementing large scale enterprise data warehouses.
- 8+ years of hands-on experience using programing languages like C#, Python or Scala.
- 8+ years of hands-on experience using scripting languages like Cosmos Scope or SQL.
- 5+ years of experience in Synapse Spark Notebooks or Jupyter Notebooks.
- 5+ years of experience in using Power BI features including datasets and dataflow.
- Bachelor’s degree in computer science or engineering, database systems, mathematics, physics, or statistics.
- Master’s degree in computer science or engineering, database systems, mathematics, physics, or statistics.
- Strong written and verbal communication and presentation skills.
- 10+ years of hands-on data engineering experience building, developing and supporting large-scale big data processing pipelines and analytics solutions leveraging data products/services including Azure Data Factory, Synapse, SQL Database, Spark, Data Flow and Pipelines.
- 10+ years of hands-on experience in Python, Cosmos Scope and SQL.
- 8+ years of hands-on experience Spark Notebooks or Jupyter Notebooks.
- 8+ years of hands-on experience in Power BI including developing datasets, datamarts and tabular models.
- 6+ years of experience designing and building data lake or lake houses solutions.
- 6+ years of experience developing and deploying products or systems at multiple points in the product cycle from ideation to shipping.
- 6+ years of practical experience in data modeling including developing tabular and multi-dimensional data models.
- 6+ years of experience operationalizing and supporting ML pipelines.
- Working experience in Azure Databrick, Snowflake and Azure Purview is a plus.
- Exceptional problem solving, technical and data analysis skills.
- Be self-driven, and show ability to deliver on ambiguous projects with incomplete or dirty data.
- Ability to work in a team environment that promotes collaboration.
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.
Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.