The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn.
This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen.
As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you.
We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it.
Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform.
Make it fast and make it scale. Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads.
Run everywhere the customer's data lives. Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds.
Power the AI data agents. Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one.
Own reliability and observability. Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on.
Lead technically. Set architecture and standards, drive cross-team designs, and mentor engineers. Own features and initiatives end-to-end, as a founder would , from the requirement to delivery. We don't believe in hierarchy or years of experience for their own sake; we believe in ownership and impact.
What You'll Need
- 5+ years of backend engineering experience building distributed systems or data-intensive services, with a track record of solving deep-tech problems
- Deep expertise in distributed systems fundamentals: scheduling, task coordination, concurrency, resource management, and fault tolerance will help you succeed.
- Strong proficiency in a systems language such as Go, Scala, or Java (our runtime is primarily Go and Python; functional-programming experience is a plus)
- Hands-on experience running data at scale , with Spark and/or cloud data warehouses (Databricks, Snowflake, BigQuery) and query execution
- A strong foundation in data structures, algorithms, and software design, with excellent analytical and debugging skills
- Experience owning the reliability, performance, and observability of a production service
- A demonstrated ability to lead technically , driving architecture, influencing across teams, and raising the quality bar for others
- Comfort working in short, agile, iterative development cycles
- Entrepreneurial spirit, priding yourself on getting things done with a high level of quality
- Strong desire to work in a fast-moving startup environment with high ownership and attention to detail
- You have been a top performer your entire career and hold yourself to very high standards
- B Tech or above degree in CS or equivalent
Preferred Qualifications
- Experience integrating with multiple execution backends or building multi-cloud data connectors
- You have built platforms or services that other engineers and products build on top of
- Familiarity with AI/agent execution paths , running and validating LLM- or agent-generated code
- Experience building an execution engine, orchestrator, or query engine (e.g. Airflow, Spark internals, query planners/optimizers such as Apache Calcite, or dbt)
- You have shipped software over multiple releases, including systems where you drove the delivery of the first version
- Ability to assess new technologies and make pragmatic choices that guide us toward a long-term vision
What You'll Have At Prophecy
- Great company culture.
- Competitive compensation.
- Fair and Open Equity awards for everyone.
- Amazing work.
- Private medical insurance.
- Manage your own growth and career.
- Experience building a transformative product.
- High learning opportunity.
- End-to-end ownership of your project.
