Data Engineer

Mach-X is a fintech start up providing advisory sevices. We are looking for a Senior Big Data Developer who will participate in all aspects of the software development lifecycle which includes estimating, technical design, implementation, documentation, testing, deployment and support of application.

Responsibilities
- Create data integration pipelines to extract, cleanse, and integrate data from a variety of sources and formats for analysis and use across use cases.
- Perform data profiling, discovery, and analysis to identify/determine location, suitability and coverage of data, and identify the various data types, formats, and data quality which exist within a given data source.
- Create re-usable data extraction/ingestion pipelines and templates to demonstrate the logical flow and manipulation of data required to move data from source systems into the target data lake, warehouse, and/or sandbox.
- Perform hands on data development to build the data extraction, movement and integration, leveraging state of the art tools and practices, including both streaming and batched data ingestion techniques.
- Assist in creation of data requirements and data model design as necessary and appropriate.

Qualifications
Minimum of 3 years of experience working with the Apache Hadoop Ecosystem of tools and technologies to extract, integrate, cleanse and organize data, including experience with either the Hortonworks or Cloudera distributions.
Key Tools and Technologies
Spark
Scala
Python
Java
Experience working with the following types of workloads and data pipelines:
Enterprise-scale ETL and ELT batched workloads
Near real-time micro-batches
Streaming data
Experience working with Data Governance frameworks
Some experience performing conceptual and logical data model design
Strong NoSQL, SparkSQL, and ANSI SQL query language skills
Strong verbal and written communication and English language skills
Experience with AWS or Azure