Montash have been engaged by a leading consultancy to source a HDInsight Technical Architect for an initial 6 month contract based in Westminster, London. You will be required to get BPSS clearance for this role.
You will be working on a large project, delivering a new data platform to support reporting and data analysis within the organisation. The platform requires the ingestion of data from numerous sources, both old and new, and the provision of tooling to allow authorised users to perform their work on the data.
The data platform will be based on a data lake architecture, hosted on the Azure cloud. They need to engage an HDInsight Technical Architect to help lead the development and deployment of schema-on-read data modelling tools within the platform. The candidate should have hands-on experience of developing with Apache Hive and Apache Spark on Azure HDInsight and a good awareness of best practice and anti-patterns in this field.
The main activities for the role are:
- Design and develop data models in Apache Hive on HDInsight to make semi-structured data available to existing ETL processes.
- Work with the Cloud Infrastructure team to build HDInsight deployment processes based on secure and performant configurations.
- Work with data scientist community to build simple data models in Apache Spark on HDInsight to provide access to semi-structured data for R developers.
- Provide input to the design of the Data Platform architecture in order to get most benefit from the use of HDInsight.
- Work collaboratively with the in-house data management teams to educate and evangelise on the use of schema-on-read data modelling patterns.
- Azure HDInsight
- Apache Hive on HDInsight
- Apache Spark on HDInsight
- Experience of working with semi-structured data, in particular JSON.
Useful, but not essential:
- Familiarity/experience of R statistical programming.
- Experience of working with Azure CI pipelines, such as Jenkins, Terraform
The role requires someone who is knowledgeable and enthusiastic about the technology and who is both capable and happy to share his knowledge with others. Good communication skills are a must.