Senior Software Engineer, Gluten Engine

We are seeking highly skilled Senior Engineers to join our team working on the Apache Gluten project, a cutting-edge initiative aimed at enhancing the performance of JVM-based SQL engines like Apache Spark by offloading execution to native engines. This role requires deep expertise in distributed systems, native code development, and familiarity with big data frameworks.

Responsibilities:Architecture & Development: Design, develop, and optimize the core components of the Apache Gluten project, focusing on the transformation of Spark SQL's physical plans to Substrait plans, and their execution using native libraries like Velox and ClickHouse.
Performance Optimization: Lead efforts to offload performance-critical data processing tasks from JVM-based SQL engines to native libraries, improving overall query execution efficiency.
Integration & Support: Work on integrating Gluten with various backends and frameworks, such as Apache Arrow for data format standardization, and extend support for additional backends and file formats.
Collaboration: Collaborate with other contributors, including those from leading tech companies like Intel and Kyligence, to enhance the project's functionality and ensure robust performance across various environments.
Documentation & Community Engagement: Contribute to the project's documentation, assist in community engagement, and mentor junior engineers.