Need - Pyspark Developer - Indianapolis, IN(Onsite) - MK

The following requirement is open with our client. Title : Pyspark Developer Location : Indianapolis, IN (Onsite) Duration : 6+ Months Visa Status : H1B, Job Description

• Create and set best practices for data ingestion, integration, and access patterns to support both real-time and batch-based consumer data needs
• Assist with design and lead development on scalable, high-performance data architecture solutions that supports both the consumer side of the business as well as analytic use cases.
• Create comprehensive documentation for design, and processes to support ongoing maintenance and knowledge sharing for both GMP and non-GMP solutions.
• Drive continuous data transformation to minimize technical debt
• Responsible for creation of test protocols / test scripts and other validation deliverables.
• Provide technical support to local end users on Data pipelines and Advanced Analytics Solutions developed.
• Strong experience with programming languages, such as Python, SQL & Spark.
• Experience with building batch and streaming pipelines using complex SQL, PySpark, Pandas, and similar frameworks.
• Develop, refine, and optimize Advanced Analytics Solutions using machine learning models to extract insights from complex data sources.
• Transform data using SQL, NoSQL, and Python.
• Visualizing data using a diverse tool set including but not limited to Python and R.
• Experience with cloud services in AWS and/or Microsoft Azure.
• Experience with message brokers and event-driven architectures (e.g, MQTT, Kafka, RabbitMQ).
• Experience in handling data streams, APIs, events, container orchestration products such as OpenShift, EKS, ECS.
• Experience testing, troubleshooting & establishing API connectivity utilizing software documentation and tools such as Postman.
• Strong experience transforming data using common ETL/ELT patterns
• Experience with orchestrating complex workflows and data pipelines using like Airflow or similar tools
• Knowledge and/or experience in predictive modeling and machine learning is a plus.

Must Have:

• Pyspark
• AWS
• SQL
• ETL
• Kafka

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity. Report this job

• Dice Id: 10117479
• Position Id: 8686409