Senior/Expert Data Engineer (consulting and auditing firm)

Role Overview:

The Senior/Expert Data Engineer position is a contract, long-term engagement based in Ukraine, focused on delivering advanced data engineering solutions using Databricks, Power BI, and SQL. The role is embedded within a global consulting and auditing firm supported by a custom product engineering company. The candidate will join a cross-functional development team responsible for engineering innovative data experiences and solving complex business challenges for multinational organizations and scaling startups.

Key Responsibilities:

• Design, develop, and deploy Power BI solutions, including dashboards and reports that provide critical business insights.
• Create and implement robust data models optimized for performance and efficient query execution.
• Optimize Power BI reports and dashboards by improving query speed and data model efficiency.
• Contribute to the design, coding, configuration, and management of data ingestion processes, supporting both near real-time and batch ETL pipelines across multiple data storage systems.
• Perform performance tuning and optimization of complex SQL and DAX queries to enhance data processing efficiency.
• Develop, execute, and maintain data processing workflows using Azure Synapse notebooks.
• Manage and optimize data storage solutions within Azure Data Lake Storage (ADLS), focusing on Parquet file optimization techniques such as partitioning and compression.
• Identify platform gaps and implement improvements to enhance quality, robustness, maintainability, and speed.
• Cross-train team members on emerging technologies and continuously learn from peers to foster knowledge sharing.
• Contribute actively to the unit’s community-building activities, participate in conferences, and uphold best practices and excellence in engineering.

Projects:

The job description does not specify individual project names or dates. However, the candidate’s contributions span multiple projects involving the following contexts and responsibilities:

• Power BI Solution Development: Serving as the lead data engineer responsible for designing and deploying Power BI dashboards and reports, performing advanced data analytics, and implementing complex DAX calculations and Power Query transformations to support business decision-making.
• ETL Pipeline Engineering: Managing and optimizing complex ETL pipelines involving data ingestion, batch processing, and near real-time data flows across diverse data storages, utilizing Azure Data Factory and Azure Synapse Analytics.
• Data Storage Optimization: Overseeing data storage management in Azure Data Lake Storage, applying advanced Parquet file optimization techniques such as partitioning and compression to improve data retrieval and storage efficiency.
• SQL and DAX Performance Tuning: Leading efforts to optimize complicated SQL queries and DAX formulas to enhance report and dashboard performance.
• Azure Synapse Notebook Development: Developing and maintaining data processing workflows within Azure Synapse notebooks to support large-scale backend data processing and analysis systems.

Technologies, Tools, and Methodologies Used:

• Databricks
• Power BI Pro (Desktop and Service)
• DAX (Data Analysis Expressions)
• Power Query
• SQL (Azure SQL Database, Microsoft SQL Server)
• Azure Data Lake Storage (ADLS)
• Parquet file format with optimization techniques (partitioning, compression)
• Azure Synapse Analytics and Notebooks
• Azure Data Factory
• ETL and data integration processes
• Agile software development methodologies, including SCRUM
• Software engineering best practices: coding standards, code reviews, source control, build processes, testing, and operations

Additional Contributions and Skills:

• Strong problem-solving and analytical skills with meticulous attention to detail.
• Excellent communication skills, capable of presenting complex data clearly and concisely to stakeholders at various organizational levels.
• Experience collaborating within cross-functional teams and contributing to community-building and knowledge sharing.
• Demonstrated ability to learn new technologies quickly and independently.
• Proven track record of delivering high-quality software on large, complex, cross-functional projects.