Goal Setting: For You and Your AI Team
It doesn't matter how fast you go if you follow the wrong direction.
In this article, I will cover the following:
How to properly define goals;
Goal-setting for your personal and professional life;
Goal-setting for Data Science & AI teams;
What to do with the final results.
Let’s go!
Defining Goals
SMART is a framework that defines a series of properties a goal should have. I will use a personal example as I go through them. Let’s say you are considering “improve my health” as a goal this year. We should transform this goal into something more:
Specific: Instead of “improve my health,” narrow it down to one component of your health. For instance, you might want to be “less sedentary.”, which is more specific.
Measurable: “Less sedentary” is hard to measure. We should add something that would let us check our progress. We could change the goal to “exercise 300 days this year”.
Achievable: Exercising 300 days a year might not be feasible for your current schedule. It is okay to be ambitious, but it is not okay to be ridiculous. We could tune it to something more achievable like “Exercise 3 times per week”.
Relevant: Is this goal aligned with your values and long-term objectives? If you want to become a more fit person long-term, this goal makes sense!
Timely: With the “Exercise 3 times per week”, it is unclear how long we should sustain it. Set an end date, such as “Exercise at least 3 times per week throughout 2023” or “Exercise 156 times throughout 2023.”
Comment: I prefer “156 times throughout 2023” over “3 times per week throughout 2023” because i) unexpected events might happen, and if you struggle in a week you could still compensate in others; ii) intermediate progress is more accurately tracked if you count by single days rather than "weeks that you did at least 3 times” — helping with the measurable property as well.
Finally, after defining the goal, develop a strategy for achieving it. In the case of “Exercise 156 times throughout 2023”, you already know that you will need to exercise, on average, 3 times per week. So, you could time-block the hours in your agenda (or schedule sessions with a personal trainer). If you have any upcoming trips, decide whether you will compensate in other weeks or how/where you will exercise during your trip.
Goal Setting for You
I classify each personal goal as either a “habit-defining goal” or a “project goal.” In the example earlier, “Exercise 156 times throughout 2023”, I would call it a “Habit-defining goal” because it focuses more on trying to keep a practice going throughout the year than on completing something.
Opposed to that, a “project goal” could be “Publishing my Book by December 2023”. It still respects all SMART goals (some might say it is not measurable, but it is, with “published” or “not published” criteria), but it focuses more on the Output (results) rather than the Input (effort).
For project goals, it is crucial to set intermediate checkpoints to create a roadmap:
By the end of January - Create a draft for each chapter;
Write one chapter per month, finishing the whole book by September
By the end of October - Finish the book review and layout;
By the end of November - Finalize details for publication and promotion;
By the end of December - Have it published on Amazon KDP;
Of course, you could keep breaking down (from year to months to weeks to days to hours), but try to stop at a level that gives you enough clarity to take action. In this example, I would do a quick planning session each month to schedule the activities I need to complete that month.
Organizing your personal goals
First, I separate my goals into 5 areas:
Health: Exercises, Diet, Mental Health, and so on;
Intellectual: Books, Learning Projects, Languages, etc.;
Professional: Career Growth, Side Projects, and more;
Finances: Savings, Expenses, and Investments;
Other: Leisure, social, or anything I cannot fit into the above.
Start with the longer-term goals for each category because you can check whether everything is cohesive when defining the shorter-term goals.
For instance, you might have a long-term goal such as “Retire with 1 million dollars by 50 years old”. Suppose you are 40 years old and currently have $500k. In that case, you can then calculate how much money you need to save per year and then either create financial goals like “Save X this year” or professional goals related to creating more revenue streams (“publish a paid mobile app”) or increasing current ones (“get promoted in my job”).
Also, it’s okay to disregard this connection between short-term and long-term goals because sometimes we want to do something just because it is fun. It is more important to check whether a new goal will impede others. For example, if you have two goals, “visit 10 countries this year” and “save X money this year,” you need to understand if all the trips might get in the way of your saving money goal.
(Example) My personal goals
Here are my goals for 2023 as an example:
Goal-setting for Data Science & AI teams
How to Organize Goals - OKRs
Instead of setting a bunch of goals in a list, the most common approach in the industry is to use a framework named OKR (Objectives and Key Results). It is pretty easy to apply:
Start with an Objective (e.g., I want to improve our fraud detection solution)
Define Key Results, typically 3 to 5, following SMART principles. Examples:
Improve by 50% the solution financial return (from 100k to 150k/month)
Run a modeling experiment data from a new vendor
Implement an automatic retraining pipeline
Reduce scoring time from 1s to 0.5s
Track your progress. This could be done via periodic checkpoints.
How much did your team achieve already?
Do we need to change something? (priority, strategy)
(Example Goals) When first creating a model…
My suggestion is to start with a well-defined business goal. Say a team is developing a machine learning model to detect fraudulent credit card transactions to block the suspicious ones automatically. In this case, we should start with “what is the minimum acceptable performance for a solution that still brings value to the company?”.
Let’s dive a little deeper into this example. We need to understand the value of each thing.
True Positives (TP) - Correctly Identified Frauds: This would stop some frauds from happening, which gives mainly two benefits: i) negates losses the company would have to pay for; ii) prevents attrition with a customer that would have an awful experience.
False Positives (FP) - Incorrectly Appointed Frauds: Blocking a credit card incorrectly could generate customer attrition, impacting satisfaction and reflecting how long they would stay as customers. We could go further and, through experiments/studies, understand how much the churn rate of customers increases when they have incorrectly blocked transactions.
With that, one recommended approach is to find a minimum “True Positive to False Positive” ratio (also known as “precision”) by estimating the actual value of a true positive and a false positive.
One simplistic way to do this is to get the “average savings” in identifying fraud and the “average losses” by incorrectly blocking a transaction.
After some studies, we end up with TPs and FPs worth $4000 and -$100, respectively. With this information, we can easily estimate the expected financial return for a given model and threshold.
Finally, it would not make sense for most businesses to implement a fraud detection system that would bring only $10 in revenue. We should define how much money the solution should bring to the table to be considered an option. For our hypothetical business, it could be “at least” $100,000 positive return per month.
In this example, we have a specific way of measuring the effectiveness of a given model, and we also have a minimum performance that we need to achieve ($100k), so we could have a general goal as:
Fraud Goal Example: Design and Deploy an AI-enabled solution to automatically identify fraud transactions, bringing at least $100k positive return considering a $4000 value for each identified fraud, and -$100 value for each incorrectly identified fraud, by the end of this quarter.
This would be a worthy goal for the team as it clarifies what they should work towards, the requirements, how to measure it, and the deadline.
(Example Goals) For Mature Solutions
All right! Now we have successfully designed and deployed the fraud detection solution. It might be an excellent moment to create goals for the next quarter so we can improve them and bring more value.
We still should look at business requirements to orient “how much improvement is enough to make an impact.” Sometimes, the improvement expectations are way higher than what is possible to do with currently available data. In those cases, we can revisit this project in other cycles or focus on further actions to make more/better data available.
If we already have an excellent model, it might be more complicated to improve than a newer model created using just a few features. For more mature solutions, instead of using “improve the model by X%,” it would be more interesting to focus on “process goals” instead, such as:
Create and validate 3 hypotheses for improving model accuracy;
Reduce the time we take to validate a new feature by X%;
Experiment with automatic retraining strategies and publish results;
This is my experience; I know not all managers agree, but I feel it’s better to push improving the process that will ultimately help with the end goal than trying to set an arbitrary number of “push the recall rate from 98% to 99%”, considering all the uncertainty that can be behind machine learning projects, which could bring unnecessary anxiety to the team.
Other Practical Tips for Team Goal-Setting:
Single-metrics: Avoid using easy-to-fool metrics, such as precision, by themselves. Instead of “Achieve 90% precision”, prefer “Achieve 90% maintaining our current recall of 85%”.
Value the process: Being result-oriented is excellent, but data science is a field that requires testing and validation. Naturally, the hypothesis might come out false in the end. Refrain from only valuing results and consider the process as a whole. (e.g., creating key results for completing experiments)
Involve your team: Engaging everyone in creating a goal is crucial to improve the sense of belonging but also helps make goals more achievable.
Frequency: Most companies create their OKRs quarterly, biannually, or yearly. Generally, high-uncertainty companies (such as startups) might benefit from shorter OKR cycles, as things can change rapidly, whereas more mature companies may prefer extended periods.
What to do with the final results?
Are the team goals a reflection of individual performance?
One mistake many companies make is to assign the team goals results as the performance of the team members. I’m not too fond of this approach, especially for data science & AI teams, because not achieving a given goal might result from the problem being too hard (and we might not have known it beforehand). This creates a problem of people trying to move to teams with “easier OKRs.”
At the same time, we should not completely disregard it. If a team had 5 goals and had poor results in all of them, this could be a strong indicator that the team is underperforming or overestimating tremendously. Either way, you got a problem.
My recommendation is to handle performance reviews separately from OKRs. However, considering their current level, we should try to see whether the person was providing their best effort towards the team OKRs. Therefore, the OKRs would impact the performance indirectly, making this a less complicated process.
Is Goal-Setting a Skill?
Yes. It is also very context-specific. In other words, what can be achieved in a specified period will vary drastically by company and factors such as data availability, infrastructure, team seniority, team size, etc. But experience will be valuable towards better goal-setting after all.
The important thing is that you finish the OKR cycle by reviewing them with your team, listing what you have learned trying to make it better in the next cycle.
Conclusion
Even though data-science projects are filled with uncertainty, defining goals is possible and brings many benefits:
Determines a clear direction for the team (people can understand whether what they are doing contributes to the team goals and adjust)
Promotes a shared sense of responsibility when done correctly.
Enforces a data-oriented culture (we don’t manage what we don’t measure)
Improves project syncing, as we can see all goals and main deadlines.
“If one does not know to which port one is sailing, no wind is favorable.” — Seneca, stoic philosopher
That should be enough to justify implementing them.
Good luck with your goals! Consider subscribing if you found it helpful!