Data Engineer Job at EXL, Philadelphia, PA

ZVlCNWZkajUzQ1dyN0FRZTFkbjFqMWho
  • EXL
  • Philadelphia, PA

Job Description

Job Title: Data Engineer

Location: Philadelphia, PA (Hybrid)

Experience: 5+ years

Role Summary:

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities:

Data Pipeline Management & Support

  • Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
  • Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

  • Design, develop, and optimize PySpark jobs for efficient large-scale data processing.
  • Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

  • Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.
  • Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.
  • Work with MinIO/S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

  • Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.
  • Provide daily coordination and technical oversight to offshore engineering teams.
  • Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

  • Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.
  • Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications:

Primary Skills (Must-Have)

  • PySpark: Advanced proficiency in transformations, performance tuning, and Spark internals.
  • SQL: Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).
  • Python: Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

  • Kubernetes: Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log/monitoring tools.
  • Databricks: Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.
  • MinIO / S3: Familiarity with bucket configuration, policies, and efficient ingestion patterns.
  • DevOps: Experience with Git, CI/CD, and cloud environments (Azure preferred).

Job Tags

Similar Jobs

Wimmer Solutions

Project Scheduling Manager Job at Wimmer Solutions

 ...PROJECT SCHEDULING MANAGER REMOTE 24100 Our client is looking for a Project Scheduling Manager who will be responsible for developing, implementing, and maintaining the companys enterprise-wide scheduling process. This individual will serve as the subject matter... 

TORP Group

Building Engineer Office Building, Class A Property! Job at TORP Group

 ...with specialty projects Special Skills: Ability to work well with others Strong people skills Must Have: ~3+ years of Building Maintenance, ideally commercial, but flexible to residential or even janitorial background ~ Ability to work on automated... 

The TJX Companies, Inc.

District Loss Prevention Manager Job at The TJX Companies, Inc.

 ...DLPM takes the complex current LP scenario head-on by building proven partnerships with store and senior regional leadership and law enforcement agencies and is responsible for establishing LP policies and procedures to ensure profitability with customer experience in... 

Insight Global

Business Systems Analyst Job at Insight Global

 ...Vancouver, WA Permanent/Direct Placement Position: Business Systems Analyst (financial/accounting) On-site 3 Days per week, Remote 2 Days per week. Required Skills & Experience ~4+ years of experience as a Business System Analyst with integrations for financial... 

LVI Associates

National Business / Channel Development Manager - Data Centers (Remote) Job at LVI Associates

 ...a industry leader on a national scale?*This role is a fully remote position, candidates can be based in any location with travel expected...  ...negotiation skills ~ Proficiency with CRM platforms such as Salesforce ~ Ability to influence stakeholders and deliver results in a...