Senior Python Engineer for Data Spaces BigData

Back-End Web, Data Engineer, Data Extraction and ETL, Data Science.

Senior

B2 (Upper-Intermediate)

Required Skills

Python

Nice to Have

React

Angular

CI/CD

Azure

Flask

Django

ETL

ETL Pipelines

Azure Pipelines

Summary

- We are looking for a Python Developer to work on building a full-cycle data management platform, which will include data ingestion, ETL, data quality, data enrichment, data processing pipelines orchestrated into an "elastic data fabric" and, most importantly, utilizing federated learning.
- Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
- Experience with Go, React/Angular will be a big plus
- Start: ASAP
- Duration: Long-term

Are you a talented developer looking for a remote job that lets you show your skills and get decent compensation? Join Upstaff.com, a platform that connects you with hand-picked startups and scale-ups in the US and Europe.

Project Description

The platform aims to automate data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.

The platform will incorporate scalable and dynamic tools for creating and managing data spaces, handling complex data workflows, ensuring modularity and privacy compliance.

The project is about building a data spaces platform for industrial manufacturing.
Data spaces is a shared distributed data management system combining multiple data sources applying ML models and managed data exchange.
Another source of Data Spaces platforms and definitions https://internationaldataspaces.org/why/data-spaces/

About Role

We are looking for a Senior Go/Python Engineer who will work on a platform that automates data ingestion, processing, and sharing with user-friendly, privacy-preserving, and scalable solutions for industrial manufacturing.

Preliminary Stack:

Backend: Go, Python, Flask/FastAPI
Frontend: ReactJS, Angular.
AI/ML: Azure Machine Learning, Azure Databricks, TensorFlow Federated, PyTorch, and privacy-enhancing techniques.
Cloud and DevOps: Kubernetes, Docker, Azure DevOps, CI/CD Data pipelines on Azure
Data Engineering: Apache NiFi. Kafka Connect, Databricks - on Azure.
Database: Cosmos DB, Postgres/Hyperscale or MySQL/Healwave

*stack may change during the hiring process of qualified specialists in their areas

Responsibilities:

Oversee system architecture, scalability, and alignment with business goals.
Mentor the team, address risks, and ensure code quality.
Define metadata-driven architecture to support scalable and dynamic workflows.

Requirements:

Strong skills in Python
Previous experience in development and architecture of data platforms (if not data spaces specifically)
Experience with distributed systems, cloud-native design, hybrid platforms, CI/CD, and DevOps.
Solid understanding of security best practices.
Problem-solving and debugging abilities.
Hands-on experience with CI/CD pipelines and testing frameworks.

Would be a plus:

Familiarity with Data pipelines and the Azure ecosystem
Experience with Go + React or Angular

How to Apply for Senior Python Engineer for Data Spaces BigData

Be ready for the next steps:

Create your profile on our website (import from LinkedIn)
20-30-minute screening call
Technical interview feedback
Project Selection (we are looking for the best project for you).

Who can apply:

We work with developers across Europe, LATAM, the U.S. (W-9 form owners), Canada, Asia, Oceania, and the UK.
We do not provide visa assistance, and this cooperation model does not include direct-hire benefits.
We cannot accept applicants from Russia, Belarus, Iran, or North Korea.

See Open Jobs

Not your stack?

Join the Upstaff community and we are looking for the best project for you.