Senior Python Engineer for Data Spaces BigData

Back-End Web, Data Engineer, Data Extraction and ETL, Data Science.

Senior

Upper-Intermediate English

Required Skills

Python

Nice to Have

React

Angular

CI/CD

Azure

Flask

Django

ETL

ETL Pipelines

Azure Pipelines

Summary

- We are looking for a Python Developer to work on building a full-cycle data management platform, which will include data ingestion, ETL, data quality, data enrichment, data processing pipelines orchestrated into an "elastic data fabric" and, most importantly, utilizing federated learning.
- Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
- Experience with Go, React/Angular will be a big plus
- Start: ASAP
- Duration: Long-term

Are you a talented developer looking for a remote job that lets you show your skills and get decent compensation? Join Upstaff.com, a platform that connects you with hand-picked startups and scale-ups in the US and Europe.

Project Description

The platform aims to automate data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.

The platform will incorporate scalable and dynamic tools for creating and managing data spaces, handling complex data workflows, ensuring modularity and privacy compliance.

The project is about building a data spaces platform for industrial manufacturing.
Data spaces is a shared distributed data management system combining multiple data sources applying ML models and managed data exchange.
Another source of Data Spaces platforms and definitions https://internationaldataspaces.org/why/data-spaces/

About Role

We are looking for a Senior Go/Python Engineer who will work on a platform that automates data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.

Preliminary Stack:

Backend: Go, Python, Flask/FastAPI
Frontend: ReactJS, Angular.
AI/ML: Azure Machine Learning, Azure Databricks, TensorFlow Federated, PyTorch, and privacy-enhancing techniques.
Cloud and DevOps: Kubernetes, Docker, Azure DevOps, CI/CD Data pipelines on Azure
Data Engineering: Apache NiFi. Kafka Connect, Databricks - on Azure.
Database: Cosmos DB, Postgres/Hyperscale or MySQL/Healwave

*stack may change during the hiring process of qualified specialists in their areas

Responsibilities:

Oversee system architecture, scalability, and alignment with business goals.
Mentor the team, address risks, and ensure code quality.
Define metadata-driven architecture to support scalable and dynamic workflows.

Requirements:

Strong skills in Python
Previous experience in development and architecture of data platforms (if not data spaces specifically)
Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
Solid understanding of security best practices.
Problem-solving and debugging abilities.
Hands-on experience with CI/CD pipelines and testing frameworks.

Would be a plus:

Familiarity with Data pipelines and the Azure ecosystem
Experience with Go + React or Angular

Not your tech stack?

Join the Upstaff community and we are looking for the best project for you. Be ready for the next steps:

Create your profile on our website (import from LinkedIn)
20-30-minute screening call
Technical interview
Feedback
Project Selection (we are looking for the best project for you).

We work with developers from 50+ countries in different regions: Europe, LATAM, the U.S. (W-9 form owners), Canada, Asia (Philippines, Indonesia), Oceania (Australia, New Zealand, Papua New Guinea), and the the UK.

We don’t have a legal and ethical basis to accept applicants from the following countries: Russia, Belarus, Iran, North Korea
We do not provide visa assistance, and our cooperation model does not include the benefits typically offered with direct hire.