From Data to Discovery: Crafting an Effective Analytics Workflow
Developing a best practice approach to common Data Analytics tasks
In the world of Software Development, project methodologies such as Waterfall and Agile have become deeply ingrained in Software Engineering projects. These approaches offer structured guidelines and best practices to optimize the software development process, leading to increased efficiency and higher quality output.
As a result, software applications like Jira, a project management and issue tracking software, have had great success in helping organizations organize and manage their software development projects.
It has proven so successful, that these methodologies, and the associated software, has been extended to other IT related areas including Data Analytics. Yet the objective and characteristics of a Data Analytics project is very different to that of Software Development, so why do we still insist on using it and why have so many data teams adopted Jira as a means of managing their projects?
What is an Analytics Workflow?
An Analytics Workflow refers to the different tasks involved in extracting insights and value from data. It involves a series of interconnected stages designed to transform raw data from a variety of sources, into meaningful information and actionable insights.
Examples of standard data tasks include:
- Ad-hoc Data Requests
- Data Collection
- Data Engineering
- Data Modelling
- Machine Learning Modelling
- Data Analysis
- Report or Dashboard Development
- Insights Communication
Key characteristics of Data Analytics projects
- Objective: Data Analytics projects are usually tasked with gaining insights and knowledge from data to support strategic decision-making. On the other hand, Software Development projects focus on design, development, and the delivery of a software product that addresses a specific need or problem.
- Changing Requirements: With Data Analytics projects, the requirements are likely to change depending on what is found in the data, whereas with Software Development projects, the requirements are more stable and any updates are managed by strict change management processes.
- Exploratory Nature: Profiling the data is a common first step in any project. Without that level of understanding, it is difficult to determine what is feasible, how much time is required and the level of complexity of a project.
- Repetitive Tasks: Data is constantly being updated, the insights generated a month ago or even a day ago may no longer be relevant today. It is common to undertake the exact same task again, just with an updated dataset.
Common problems with Data Analytics projects
- Scope Creep and Unclear Objectives: Undefined or changing project objectives can lead to scope creep, making it difficult to deliver meaningful results within the allocated time and resources.
- Lack of Domain Expertise: Data analysts may lack in-depth knowledge of the domain or industry being analyzed, leading them to misinterpret results or overlook critical aspects of the data.
- Insufficient Communication: Inadequate communication between data analysts and business users can lead to misunderstandings, misaligned expectations, and ineffective utilization of any insights.
A new tool for Analytics Workflows
When we set out to develop dablr, a collaborative data analytics platform, it was imperative that we create a solution that was designed to help bridge the gap between data analysts and the end business user, and improve the collaborative nature of Data Analytics projects.
We wanted to develop a platform that would be the Slack/Jira for Analytics.
To achieve the Slack/Jira for Analytics, we designed features that take into account the unique characteristics of a Data Analytics project:
- Specific Data Analytics workflows: This ranges from quick tasks like ad-hoc requests to longer-term projects, like a machine learning project. Each workflow provides a best practice approach to how a particular workflow should be conducted, including what tasks should be undertaken as a part of that workflow.
- Clear Requirements: The platform guides the end business user on what information is helpful to provide as a part of undertaking a particular workflow. For example, if the workflow was to develop a dashboard, key information may include: the preferred data source, date ranges, any exclusions, type of output etc. Any updates to the requirements will also be tracked by the platform.
- Workflows are visually represented as process flow diagrams: Inspired by ETL applications, the visual nature of these workflows makes it easier for business users and data analysts to determine what steps come next and the status of a project, moving away from the bureaucratic nature of forms.
- Discussion Hub: Each workflow has an area where teams can discuss the task at hand. If any assistance is required, users can tag other users or teams on the platform for help.
- Accessible Workflow list: Moving ad-hoc requests and other data tasks to an in-app list from Jira, Slack and email makes it easier to see what data analysts are working on. It also highlights similar or duplicate tasks across the team, and provides end users an idea of where their work lies in a queue.
Our goal with dablr is to make it easier for data teams to work with the business, and better utilize data across different teams. These are just some ways we are trying to achieve this, if you have any ideas or feedback and would like to contribute, we’d love to hear from you.
Visit our website and join the dablr waitlist.
Subscribe to DDIntel Here.
DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.
Register on AItoolverse (alpha) to get 50 DDINs
Support DDI AI Art Series: https://heartq.net/collections/ddi-ai-art-series
Join our network here: https://datadriveninvestor.com/collaborate
