W2 - 15+ Lead Data Engineer

Position Lead Data Engineer Main skills SQL, Databricks, ADF, Datastage (or other ETL tool), SSAS cubes, Cognos/other, Tableau/other, Thoughtspot/other Employment typ W2 Project duratio 9 months with possible extension Location San Francisco, CA Work mode 1-2 days per week from office Travel No Recruitment process General -> Technical Interview -> Manager Interview -> Client Interview Required start date ASAP Level Senior/Lead level   Work authorization statu H1B and TN visa candidates can be considered. Any applicant needs to be on your W2. Description Job Description We are seeking a Lead Data Engineer to design, implement, and maintain our data infrastructure and business intelligence solutions. This role requires a technical expert who can architect scalable data solutions while collaborating across teams to drive business insights.   Required Experience · 8-10 years of software development and deployment experience · At least 7 years of hands-on experience with: · SQL · Databricks · Azure Data Factory (ADF) · Datastage (or other ETL tools) · SSAS cubes · BI tools (Cognos, Tableau, ThoughtSpot, etc.) · Strong Database knowledge (SQL & No-SQL preferred) · Advanced SQL and Python skills · Experience working with APIs for data collection/ingestion   Key Responsibilities Technical Development · Write SQL for processing raw data, Kafka ingestions, ADF pipelines, data validation, and QA · Lead, design and implement innovative analytical solutions using Tableau, SQL, and Big Data technologies · Experience with storing, joining, filtering, and analyzing data using SQL, Spark, Hive, etc.   Data Architecture & Design · Design comprehensive data architecture and data modeling strategies for BI systems · Create logical and physical data models using best practices to ensure high data quality and reduced redundancy · Collaborate with stakeholders to translate business requirements into robust, scalable data models · Perform detailed analysis of business problems and technical environments to design quality solutions   Integration & Implementation · Integrate data from various sources (databases, warehouses, external systems) · Develop strategies for ETL processes ensuring data accuracy, consistency, and availability · Evaluate and select appropriate BI tools based on organizational needs · Run POCs and implement BI software, including visualization tools and analytics solutions   Optimization & Governance · Monitor and optimize BI system performance for efficient data retrieval, processing, and reporting · Identify and resolve performance bottlenecks through query tuning and data structure optimization · Establish data governance policies ensuring data quality, consistency, and security · Define access controls, privacy measures, and retention policies to meet regulatory requirements   Collaboration & Leadership · Build partnerships with architects, technical leads, and stakeholders across functional groups · Actively participate in code reviews and testing to ensure adherence to best practices · Work with product and engineering teams to evaluate requirements and new features · Build and foster a high-performance engineering culture · Mentor team members and provide tools and motivation for success   Additional Skills & Requirements · Experience building data ingestion pipelines and data warehouse architecture · Ability to write design documentation, source-to-target mapping documentation, and manage Confluence pages · Experience converting business functionalities into technical Jira stories · Strong experience with data modeling, design patterns, and building scalable BI solutions · Knowledge of cloud platforms (Azure, AWS or equivalent) · Experience analyzing large datasets to identify trends, patterns, and outliers