Software Architect – Distributed Data Processing

Role OverviewAs a Software Architect, you will own the design and implementation of the high-level software layers that connect open-source big data engines such as Apache Spark to DualBird's custom hardware accelerators. This includes the Spark plugin, query planning and optimization, execution orchestration, and the interfaces between the engine, the cloud environment, and the accelerated runtime.This is a hands-on architecture role. You will set technical direction and make system-wide design decisions, but you will also write a significant share of the code yourself while driving complex features end-to-end. You will work closely with the compiler, runtime, hardware, and infrastructure teams to ensure the full stack delivers predictable, out-of-the-box performance on large-scale customer workloads running on AWS.Responsibilities· Design and evolve the architecture of DualBird's Spark integration and query processing layers.· Lead the design and implementation of new features across planning, optimization, and execution, from concept to production.· Extend support for Spark and SQL semantics, expressions, and operators.· Own correctness, compatibility, and performance across Spark versions and AWS data platforms (EMR, EKS, Glue, S3).· Review designs and code, mentor engineers, and shape engineering practices across the team.· Work with customers' real workloads to identify gaps, prioritize, and deliver improvements.