Senior Data Engineer (Developer)

Verified Job On Employer Career Site
Job Summary: Corgan is a design-focused company that values collaboration and creativity. They are seeking a Senior Data Engineer to design, build, and maintain scalable data infrastructure and AI-powered data pipelines that align with corporate strategic objectives. The role involves working closely with IT staff, data scientists, and business analysts to ensure effective data solutions and governance.
Responsibilities: • Design and build enterprise-scale data architectures supporting structured, semi-structured, and unstructured data across multi-cloud environments (Azure, AWS) • Implement scalable data lakes and data warehouses optimized for both batch and real-time analytics workloads • Develop and maintain data mesh architectures that enable self-service analytics while ensuring data governance and security • Architect cloud-native solutions leveraging serverless computing, containerization, and microservices patterns • Build robust, fault-tolerant data pipelines using modern ELT methodologies and orchestration tools • Implement real-time data streaming solutions using Apache Kafka, Apache Pulsar, and cloud-native streaming services • Design and maintain automated data quality frameworks with comprehensive monitoring, alerting, and auto-remediation capabilities • Develop CI/CD pipelines for data engineering workflows, including automated testing, deployment, and rollback procedures • Integrate machine learning workflows into data pipelines, supporting feature engineering, model training, and inference at scale • Implement MLOps practices including model versioning, A/B testing frameworks, and automated retraining pipelines • Build data infrastructure to support generative AI applications, including vector databases and retrieval-augmented generation (RAG) systems • Collaborate with developers, engineers, and data scientists to produce machine learning models and ensure scalable inference capabilities • Implement comprehensive data governance frameworks including data lineage tracking, metadata management, and data cataloging • Ensure compliance with data privacy regulations (GDPR, CCPA) and implement data masking, encryption, and access controls • Establish data quality standards and automated validation rules across all data assets • Design and maintain audit trails for data processing activities and model predictions • Optimize data processing performance through query tuning, indexing strategies, and cost-effective resource allocation • Implement comprehensive observability solutions for data pipelines, including metrics, logging, and distributed tracing • Conduct root cause analysis for data quality issues and system performance bottlenecks • Establish SLAs for data freshness, accuracy, and system availability • Collaborate with cross-functional teams, including developers, analysts, and business stakeholders, to understand requirements and deliver solutions • Provide technical leadership and mentoring to junior developers and analysts • Develop and maintain technical documentation, data architecture diagrams, and best practices guidelines • Lead technical design reviews and contribute to technology strategy decisions
Qualifications:
Required: • 7+ years of hands-on data engineering experience with a proven track record of building production-scale data systems • 5+ years of hands-on experience with Microsoft Dynamics 365 (CRM) • 5+ years of experience with cloud platforms (Azure, AWS), including data services and infrastructure management • 5+ years of advanced SQL experience including query optimization, performance tuning, and complex analytical queries • 3+ years of experience with big data frameworks (Apache Spark, Hadoop ecosystem, Databricks) • 3+ years of experience with real-time data processing and streaming technologies • Strong programming skills in Python, C#, and/or Scala with a focus on data processing and automation • Expert-level proficiency in SQL, NoSQL, and NewSQL databases (PostgreSQL, MongoDB, Cassandra, Snowflake) • Advanced experience with ETL/ELT tools and orchestration platforms (Apache Airflow, Azure Data Factory, Fabric Dataflow Gen2 and Data Pipelines, AWS Glue, dbt) • Deep understanding of data modeling techniques for both transactional and analytical workloads • Experience with data warehousing concepts including dimensional modeling, star/snowflake schemas, and slowly changing dimensions • Hands-on experience with cloud-native data services (Azure Synapse, AWS Redshift/Athena) • Proficiency with Infrastructure as Code (Bicep, Terraform, CloudFormation) and containerization (Docker, Kubernetes) • Experience with serverless computing architectures and event-driven data processing • Understanding of cloud security, networking, and cost optimization strategies • Expert-level Apache Spark development using PySpark, Scala, or Java • Experience with real-time streaming platforms (Azure Event Hubs, Apache Kafka, Apache Pulsar, AWS Kinesis) • Knowledge of distributed systems concepts and fault tolerance patterns • Experience with data lakehouse architectures and formats (Delta Lake, Apache Iceberg, Apache Hudi) • Experience integrating machine learning workflows into data pipelines • Understanding of MLOps practices and tools (MLflow, Kubeflow, SageMaker) • Knowledge of feature stores and model serving architectures • Familiarity with vector databases and embedding techniques for AI applications • Advanced Git workflows, code review processes, and collaborative development practices • Experience with CI/CD pipelines for data engineering (Azure DevOps, GitHub Actions, Jenkins) • Proficiency with monitoring and observability tools (DataDog, Splunk, Prometheus, Grafana) • Understanding of agile development methodologies and project management tools
Preferred: • A Master's degree in Computer Science, Data Science, Engineering, or related technical field • Experience with graph databases and knowledge graph technologies • Background in financial services, architecture, engineering, or construction industry data systems • Knowledge of data privacy and compliance frameworks • Microsoft Certified: Azure Data Engineer Associate (DP-203) • Microsoft Certified: Fabric Data Engineer Associate (DP-700) • Microsoft Certified: Fabric Analytics Engineer Associate (DP-600) • AWS Certified Data Engineer – Associate • Databricks Certified: Data Engineer Associate • Databricks Certified: Data Engineer Professional • SnowPro Advanced: Data Engineer (DEA-C02) • SnowPro Advanced: Architect (ARA-C01)
Company: Corgan is an architecture and design firm with expertise in branding, aviating planning and interior design. Founded in 1938, the company is headquartered in Dallas, Texas, USA, with a team of 1001-5000 employees. The company is currently Late Stage. Corgan has a track record of offering H1B sponsorships.