We are looking for Data Engineers at multiple levels with experience in building high- performing, scalable, enterprise-grade data processing pipelines.
You will be part of a talented software team that works on mission-critical applications. Data Engineer responsibilities include managing application development while providing expertise during the full software development lifecycle, from concept and
design to testing.
- Should understand how various components in the Big Data ecosystem are connected and how data flows between them.
- Identify and use appropriate stack like benthos, flare, alpha etc. to create solutions for ingesting and transforming customers’ data.
- Develop low-level design specifications for creating complex workflows in Flare.
- Create a structural approach by breaking down complex data transformation jobs
into smaller jobs and create optimised workflows.
- Review the solution provided by the Customer Solution Team and propose an
optimised alternate/new solution when required.
- Help analytical engineers with Spark jobs fine tuning and debugging.
- Allocate sufficient resources for workflow execution.
- Document and demonstrate solutions by developing flowcharts, layouts, diagrams,
charts, code comments, and clear code.
- Exposure in all areas of software development, including designing, coding, testing, debugging, configuring, and documenting software.
- Strong working knowledge of Spark, Java/Scala/Pyspark, Kafka, Git, Unix / Linux, ETL pipelines designing.
- Experience with Spark optimisation/ tuning / resource allocations.
- Excellent understanding of IN memory distributed computing frameworks like Spark
and its parameter tuning, writing optimized workflow sequences.
- Experience of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases
(e.g., Redshift, Bigquery, Cassandra, etc).
- Familiarity with Docker, Kubernetes, Azure Data Lake/Blob storage, AWS S3, Google
cloud storage, etc.
- Have a deep understanding of the various stacks and components of Big Data ecosystem.
- Good functional and Object oriented programming skills.
About The Modern Data Company
Founded in 2018, The Modern Data Company began with the realization that enterprise-wide data access has been siloed. Data engineers and database administrators have been the longstanding data gatekeepers who funneled data to analysts and data scientists. We aim to change that by freeing enterprises to make better data-driven decisions by democratizing access to data. When all employees, irrespective of their technical skills or background, can easily explore and analyze enterprise data, then both productivity and market expansion are realized at a faster pace.
Modern Data is proud to be an equal opportunity workplace.