Senior Python Engineer for Data Spaces BigData
Summary
- We are looking for a Python Developer to work on building a full-cycle data management platform, which will include data ingestion, ETL, data quality, data enrichment, data processing pipelines orchestrated into an "elastic data fabric" and, most importantly, utilizing federated learning.
- Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
- Experience with Go, React/Angular will be a big plus
- Start: ASAP
- Duration: Long-term
Project Description
The platform aims to automate data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.
The platform will incorporate scalable and dynamic tools for creating and managing data spaces, handling complex data workflows, ensuring modularity and privacy compliance.
- The project is about building a data spaces platform for industrial manufacturing.
- Data spaces is a shared distributed data management system combining multiple data sources applying ML models and managed data exchange.
- Another source of Data Spaces platforms and definitions https://internationaldataspaces.org/why/data-spaces/
About Role
We are looking for a Senior Go/Python Engineer who will work on a platform that automates data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.
Preliminary Stack:
- Backend: Go, Python, Flask/FastAPI
- Frontend: ReactJS, Angular.
- AI/ML: Azure Machine Learning, Azure Databricks, TensorFlow Federated, PyTorch, and privacy-enhancing techniques.
- Cloud and DevOps: Kubernetes, Docker, Azure DevOps, CI/CD Data pipelines on Azure
- Data Engineering: Apache NiFi. Kafka Connect, Databricks - on Azure.
- Database: Cosmos DB, Postgres/Hyperscale or MySQL/Healwave
*stack may change during the hiring process of qualified specialists in their areas
Responsibilities:
- Oversee system architecture, scalability, and alignment with business goals.
- Mentor the team, address risks, and ensure code quality.
- Define metadata-driven architecture to support scalable and dynamic workflows.
Requirements:
- Strong skills in Python
- Previous experience in development and architecture of data platforms (if not data spaces specifically)
- Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
- Solid understanding of security best practices.
- Problem-solving and debugging abilities.
- Hands-on experience with CI/CD pipelines and testing frameworks.
Would be a plus:
- Familiarity with Data pipelines and the Azure ecosystem
- Experience with Go + React or Angular