Azure Data Engineer

Key Responsibilities<\/span><\/span><\/b>
<\/span><\/span><\/p>

Fabric Implementation<\/span><\/span><\/b>
<\/span><\/span><\/p>

• Work on the fabrics platform to design and implement robust data solutions, including One Lake architecture for efficient data storage and processing.<\/span>
  <\/span><\/span> <\/div>

  <\/span>
  <\/span><\/span> <\/div><\/li> <\/ul>

  Build & Optimize Data Pipelines<\/span><\/span><\/b>
  <\/span><\/span><\/p>

  • Design, develop, and maintain scalable ingestion and transformation pipelines using Microsoft Fabric (Data Factory in Fabric \/ Pipelines), ADF\/Synapse Pipelines, OneLake storage patterns, PySpark, Python, and SQL across structured and unstructured data.<\/span>
    <\/span><\/span> <\/div>

    <\/span>
    <\/span><\/span> <\/div><\/li> <\/ul>

    API‑Driven & Scheduled Workflows<\/span><\/span><\/b>
    <\/span><\/span><\/p>

    • Develop pipelines that ingest data from external APIs on a scheduled basis and initiate end\-to\-end downstream processing, supporting one or multiple daily runs through to curated and consumption\-ready layers.<\/span>
      <\/span><\/span> <\/div>

      <\/span>
      <\/span><\/span> <\/div><\/li> <\/ul>

      Data Ingestion & Integration<\/span><\/span><\/b>
      <\/span><\/span><\/p>

      • Integrate data from cloud and on\-prem sources including databases, third\-party systems, files, and REST\/SOAP APIs (auth, throttling, pagination, retries, and error handling).<\/span>
        <\/span><\/span> <\/div>

        <\/span>
        <\/span><\/span> <\/div><\/li> <\/ul>

        Transformation & Data Modelling<\/span><\/span><\/b>
        <\/span><\/span><\/p>

        • Build curated layers and consumption\-ready models; implement incremental and batch processing logic; apply data modelling and transformation best practices aligned to reporting\/analytics needs.<\/span>
          <\/span><\/span> <\/div>

          <\/span>
          <\/span><\/span> <\/div><\/li> <\/ul>

          SQL Development & Tuning<\/span><\/span><\/b>
          <\/span><\/span><\/p>

          • Develop and optimize complex queries, stored procedures, views, and datasets for efficient analytics and reporting; partner with analytics teams to meet performance SLAs.<\/span>
            <\/span><\/span> <\/div>

            <\/span>
            <\/span><\/span> <\/div><\/li> <\/ul>

            Performance Tuning & Cost Optimization<\/span><\/span><\/b>
            <\/span><\/span><\/p>

            • Tune Spark jobs, ADF data flows and SQL workloads (partitioning, caching, parallelism, cluster sizing\/configs) to improve reliability and reduce runtime\/cost.<\/span>
              <\/span><\/span> <\/div>

              <\/span>
              <\/span><\/span> <\/div><\/li> <\/ul>

              Business Logic Implementation<\/span><\/span><\/b>
              <\/span><\/span><\/p>

              • Translate requirements into scalable rules (validation, eligibility, availability calculations), manage exceptions, audit logging, and ensure data consistency across systems.<\/span>
                <\/span><\/span><\/li> <\/ul>

                Data Quality & Validation<\/span><\/span><\/b>
                <\/span><\/span><\/p>

                • Implement automated data quality checks, validation frameworks, reconciliation, and monitoring to ensure trusted datasets.<\/span>
                  <\/span><\/span> <\/div>

                  <\/span>
                  <\/span><\/span> <\/div><\/li> <\/ul>

                  Security & Compliance<\/span><\/span><\/b>
                  <\/span><\/span><\/p>

                  • Implement secure access via Azure AD, Managed Identities, RBAC, least privilege, and secure connectivity to data lake, Fabric\/Synapse, and APIs.<\/span>
                    <\/span><\/span><\/li> <\/ul>

                    Automation & CI\/CD<\/span><\/span><\/b>
                    <\/span><\/span><\/p>

                    • Build deployment automation using Azure DevOps\/Git, promoting code across environments with consistent release practices; support testing and release activities.<\/span>
                      <\/span><\/span><\/li> <\/ul>

                      Monitoring & Troubleshooting<\/span><\/span><\/b>
                      <\/span><\/span><\/p>

                      • Monitor pipelines and jobs using Spark UI and Azure Log Analytics; triage failures, perform root\-cause analysis, and improve resiliency\/runbooks.<\/span>
                        <\/span><\/span> <\/div>

                        <\/span>
                        <\/span><\/span> <\/div><\/li> <\/ul>

                        Collaboration<\/span><\/span><\/b>
                        <\/span><\/span><\/p>

                        • Work closely with architects, platform\/DevOps engineers, analysts, and data scientists; participate in design sessions and code reviews; operate within Agile\/Scrum delivery.<\/span>
                          <\/span><\/span><\/li> <\/ul>
                          • 
                            <\/span><\/span><\/li> <\/ul>

                            Tools & Technologies<\/span><\/span><\/b>
                            <\/span><\/span><\/p>

                            Fabric<\/span><\/span><\/b>
                            <\/span><\/span><\/p>

                            • Microsoft Fabric Workspaces<\/span>
                              <\/span><\/span><\/li>