How statistics are calculated
We count how many offers each candidate received and for what salary. For example, if a Data QA developer with Postman with a salary of $4,500 received 10 offers, then we would count him 10 times. If there were no offers, then he would not get into the statistics either.
The graph column is the total number of offers. This is not the number of vacancies, but an indicator of the level of demand. The more offers there are, the more companies try to hire such a specialist. 5k+ includes candidates with salaries >= $5,000 and < $5,500.
Median Salary Expectation – the weighted average of the market offer in the selected specialization, that is, the most frequent job offers for the selected specialization received by candidates. We do not count accepted or rejected offers.
Trending Data QA tech & tools in 2024
Data QA
What is Data Quality
A data quality analyst maintains an organisation’s data so that they can have confidence in the accuracy, completeness, consistency, trustworthiness, and availability of their data. DQA teams are in charge of conducting audits, defining the data quality standards, spotting outliers, and fixing the flaws, and play a key role at all stages in the data lifecycle. Without DQA work, strategic plans will fail, operations will go awry, customers will leave, and organisations will face substantial financial losses, as well as a lack of customer trust and potential legal repercussions due to poor-quality data.
This is a job that has changed as much as the hidden infrastructure that transforms data into insight and then powers the apps that we all use. I mean, it’s changed a lot.
Data Correctness/Validation
This is the largest stream of all the tasks. When we talk about data correctness, we should be asking: what does correctness mean to you, for this dataset? Because it would be different for every dataset and every organisation. The commonsense interpretation is that it must be what your end user (or business) wants from the dataset. Or what would be an expected result of the dataset.
We can obtain this just by asking questions, or else reading through the list of requirements. Here are some of the tests we might run, in this stream:
Finding Duplicates — nobody wants this in their data.
– Your data contains unique/distinct values in that column/field. Will the returned value be a unique/distinct value in that column/field?
– Any value that can be found in your data is returned.
Data with KPIs – If data has any columns we can sum, min or max on it’s called a key performance indicator. So basically any models which are mostly numeric/int column. eg: Budget, Revenue, Sales etc. If there is data comparison between two datasets then below tests applies:
– Comparing counts between two datasets — get the difference in count
– Compare the unique/distinct values and counts for columns – find out which values are not present in either of the datasets.
– Compare the KPIs between two datasets and get the percentage difference between them.
– Replace missing values – missing in any one of the datasets with primary or composite primary key. This can be done in a data source that does not have primary key too.
– Perform the metrics by segment for the individual column value — that can help you determine what might be going wrong if the count of values in the Zoopla-side doesn’t match the count on the Rightmove-side or if some of the values are missing.
Data Freshness
This is an easy set. How do we know if the data is fresh?
An obvious indication here is to check if your dataset has a date column, in which case, you just check the max date. Another one is, when the data was pulled into a particular table, all of this can be converted into a very simple automated checks, which we might talk about in a later blog entry.
Data Completeness
This could be an intermediate step in addition to data correctness, but how do we know to get there if the space of answers is complete?
To do this test, check if any column has all values null in it perhaps that’s okay, but most of the time it’s bad news.
Another test would be one-valuedness: whether everywhere on the column all values are the same, probably in some cases that would be a fine result, but probably in other cases that would be something we’d rather look into.
What are Data Quality Tools and How are They Used?
Data quality tools are used to improve, or sometimes automate, many processes required to ensure that data stays fit for analytics, data science, and machine learning. For example, such tools enable teams to evaluate their existing data pipelines, identify bottlenecks in quality, and even automate many remediation steps. Examples of activities relating to guaranteeing data quality include data profiling, data lineage, and data cleansing. Data cleansing, data profiling, measurement, and visualization tools can be used by teams to ‘understand the shape and values of the data assets that have been acquired – and how they are being collected’. These tools will call outliers and mixed formats. In the data analytics pipeline, data profiling acts as a quality control gate. And each of these are data management chores.
Where is Postman used?
API Craftmanship
- Developers use Postman to sculpt APIs like Michelangelo, carving through stone-cold JSON.
Debugging Dance-offs
- In the disco of development, Postman's the DJ, spinning beats of bug-free rhythms through automated testing.
Documentation Disco
- It waltzes around complexities, partnering with APIs to create docs that even your grandma could follow.
Performance Polka
- Postman prances into the limelight, analyzing API performances, ensuring they're not doing the two-step during rush hour.
Postman Alternatives
Insomnia
Open-source alternative for HTTP and GraphQL requests. Features workspace organization, data chaining, and environment variables.
- Free and open-source
- Plugin support for extended functionality
- Less intuitive UI compared to Postman
- No collaboration in the free version
- Excellent for individual developers
- More focused on API testing than full API lifecycle management
// Example Insomnia GET request
GET /api/puppies
Host: puppy-api.example.com
Authorization: Bearer YourTokenHere
SoapUI
A tool mainly for SOAP APIs but also supports REST and GraphQL. Tailored for testing with extensive automation scripting capabilities.
- Purpose-built for SOAP with REST support
- Test automation through Groovy scripting
- Steep learning curve for beginners
- Outdated interface
- Powerful data-driven testing features
- Can be overkill for simple API exploration
// Example SoapUI request to a SOAP service
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://example.com/webservices/service">
<soapenv:Header/>
<soapenv:Body>
<web:GetPuppyDetailsRequest>
<web:puppyId>1</web:puppyId>
</web:GetPuppyDetailsRequest>
</soapenv:Body>
</soapenv:Envelope>
Curl
Command-line tool for transferring data with URL syntax. Favoured for its ubiquity on UNIX/Linux systems.
- Pre-installed on most Unix-based systems
- Useful for simple API tests from the command line
- Not a GUI, less accessible for non-technicals
- No built-in feature for organizing requests
- Versatile and can handle numerous protocols
- Lacks the collaborative features of Postman
// Curl command to fetch data from an API
curl -X GET "http://puppy-api.example.com/api/puppies" -H "accept: application/json" -H "Authorization: Bearer YourTokenHere"
Quick Facts about Postman
The Birth of Postman from a Side Project
Way back in 2012, a clever chap named Abhinav Asthana was toying around with the idea of API testing for the mere fun of it. Little did he know, his side hustle would morph into the colossal giant we fondly dub as Postman. Imagine fiddling with code over a weekend and accidentally giving birth to a software unicorn!
API Testing Turned Up to Eleven
Breaking away from the boring old regimen of purely code-based API interaction, Postman strutted in with a graphical interface so slick it made developers worldwide drop their command lines like hot potatoes. Not just that, with a shared workspace, now teams can chit-chat and collaborate over APIs without leaving their desks—leaving more time for coffee breaks and less for email tag.
Versions Galore - The Postman Chronicles
Remember when you had to update software manually, and it was about as much fun as watching paint dry? Postman laughed in the face of such outdated shenanigans. Starting from a Chrome extension, this tool transformed so often, undergoing several iterations, and now sits pretty as a standalone entity, version numbers blooming with every update, like flowers in an eternal spring of software development.
What is the difference between Junior, Middle, Senior and Expert Postman developer?
Seniority Name | Years of Experience | Average Salary (USD/year) | Responsibilities & Activities |
---|---|---|---|
Junior Developer | 0-2 years | 40,000 - 60,000 |
|
Middle Developer | 2-5 years | 60,000 - 85,000 |
|
Senior Developer | 5-8 years | 85,000 - 120,000 |
|
Expert/Team Lead | 8+ years | 120,000+ |
|
Top 10 Postman Related Tech
JavaScript and Node.js
Now, whisper this softly; wherever APIs are involved, JavaScript lurks nearby, stewing in its cauldron of callbacks and promises. Node.js strides in, swinging its event-driven architecture like a club, conquering scalability like it's level one of a video game. With these two in your arsenal, custom scripts and Postman test automation turn into a cakewalk — except it's more like a code-walk. Here's a snippet of glory:
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
JSON/XML
JSON and XML are the bread and butter of API responses, akin to the 'swipe right' or 'swipe left' in dating apps — they let you know if you've got a match with the data structure you desire. Mastering their syntax is a no-brainer; it's like learning the ABCs before diving into War and Peace. Dive into the object notation like this:
{
"id": "chuck-norris",
"joke": "When Chuck Norris throws exceptions, it's across the room."
}
REST and GraphQL
REST, the granddaddy of network-based architectures, makes resources play nice with HTTP methods. Then, there's GraphQL—the cool, younger sibling that asks for specific data and gets exactly that, nothing more, like a finicky eater picking out onions from their salad. Postman toys with both, ensuring APIs behave and play fetch with the data you ask for.
HTTP/HTTPS Protocols
HTTP and HTTPS are the foundation to API's castles in the cloud – as important as knowing which end of the sword to hold in a duel. HTTPS throws in some armor with SSL/TLS, making everything as secure as a dragon's treasure horde.
Git
Ah, Git, the time machine for developers, allowing them to leap between versions of their code faster than a caffeinated kangaroo. A lifesaver when you accidentally start an apocalypse in your codebase. Keep your API tests and collections version-controlled, and collaborate with your teammates without the drama of overwriting each other's work.
CICD Tools (Jenkins, GitLab CI, etc.)
Have you ever dreamed of an assembly line for your code that magically spits out ready-to-deploy software? Behold, CICD tools make that dream a reality – you push code changes, and they handle the drudgery of testing and deployment. It's like having a robot butler for your software. Integrate Postman with these, and voilà, your APIs are tested in the blink of an eye.
API Specification Formats (OpenAPI/Swagger, RAML)
API specifications are the hot sauce to your API burrito, giving it that extra kick. They outline everything one needs to know to communicate with your API, like a guidebook for your digital Everest. Formats like OpenAPI/Swagger and RAML lay down the law for your API structure, making everything as predictable as a Marvel movie plot.
Postman API Testing Features
Do you fancy setting up tests for APIs faster than a dog chasing a squirrel? Postman's testing features are your Harry Potter-style spells. Write tests, set up monitors, and perform test runs directly within Postman's magical environment. Once you've tasted this power, manually testing APIs feels like churning butter by hand in the 21st century.
pm.test("Body matches string", function () {
pm.expect(pm.response.text()).to.include("string_you_want_to_test");
});
OAuth and Authentication Mechanisms
Just as a bouncer checks your ID before letting you into a club, OAuth and other authentication mechanisms protect APIs from uninvited guests. Get cozy with OAuth 2.0, bearer tokens, and the gang to ensure your clients are who they say they are, shaking hands like civilized folk before proceeding to the data party.
Environment and Variable Management
Consider environments and variables in Postman as your recipe book for cooking up API requests with different flavors — no mess, no fuss. Seamlessly switch between development, testing, and production environments like a DJ swapping beats. Here's how you might retrieve an environment variable:
let apiUrl = pm.environment.get("base_url");