We are looking for Multiple Spark Developers who will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and product.
- Experienced Spark developer who understands how to fully utilize the Spark cluster.
- Will clean, transform and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.
- Optimize Spark code, Spark SQL queries, and partitioning strategy for better scalability, reliability, and performance. This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.
- Good understanding of the complex transformation logic and translate them to Spark-SQL
- Experience with key open source big-data projects as a contributor or committer including Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka, Apache Storm and Apache Hive.
- Excellent programming skills in Java, Scala and/or Python.
- In-depth knowledge of distributed systems and scheduler (e.g., partitioning, replication, consistency, consensus and Kubernetes, Hadoop YARN, Spark standalone, and/or Mesos).
- Experience using Apache Spark internals with Spark query tuning and performance optimization.
- Expertise with Spark and Spark SQL.
- Strong client facing, communication and presentation skills for Technical and Business audience.
- Experience of contributing to the development of spark connections.
- Committer-ship at major open source big-data projects (such as Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka) would be a huge plus.
About The Modern Data Company
Founded in 2018, The Modern Data Company began with the realization that enterprise-wide data access has been siloed. Data engineers and database administrators have been the longstanding data gatekeepers who funneled data to analysts and data scientists. We aim to change that by freeing enterprises to make better data-driven decisions by democratizing access to data. When all employees, irrespective of their technical skills or background, can easily explore and analyze enterprise data, then both productivity and market expansion are realized at a faster pace.
Modern Data is proud to be an equal opportunity workplace.