The drift observation data can be captured in tabular format, such as comma-separated values or Parquet, on Amazon Simple Storage Service (S3) and analyzed with Amazon Athena and Amazon QuickSight. Basic architecture on how data drift is detected using Amazon SageMaker You can integrate those tasks into your ML workflow with Amazon SageMaker Pipelines.įigure 2. The white, rectangular boxes in the architecture diagram represent the tasks for detecting data and model drift. You can store the features that you defined for your models in the Amazon SageMaker Feature Store, a fully managed, purpose-built repository to store, update, retrieve, and share ML features. You can use Amazon SageMaker Data Wrangler, a visual data preparation tool, to clean and normalize your input data for your ML task. With Amazon CloudWatch, you can define rules and thresholds that prompt drift notifications.įigure 2 illustrates a basic architecture with the data sources for training and production (on the left) and the observed data concerning drift (on the right). Drift evaluation constitutes the monitoring data and mechanisms to detect changes and triggering consequent actions. Changes of these attributes between re-trained models also signal drift. SageMaker Clarify provides insights into your trained models, including importance of model features and any biases towards certain segments of the input data. For example, product recommendations may require you to ask a selected group of consumers for their feedback to the recommendation. Some use cases can require extra steps to collect actual values. For example, using weekly demand forecasting, you can compare the forecast quantities one week later with the actual demand. You can also detect drift through model quality monitoring, which requires capturing actual values that can be compared with the predictions. Deviations in the data profile signal a drift in the input data. There are three stages to detecting data drift: data quality monitoring, model quality monitoring, and drift evaluation (see Figure 1).ĭata quality monitoring establishes a profile of the input data during model training, and then continuously compares incoming data with the profile. Then, we explain how Amazon SageMaker Clarify can help detect data drift. We also discuss the steps of building a feedback loop to capture the request data in the production environment and create a data pipeline to store the data for profiling and baselining.
#AWS SAGEMAKER CLARIFY HOW TO#
This blog post explains how to approach changing data patterns in the age of disruption and how to mitigate its effects on ML models. To mitigate the effects of the disruption, data drift needs to be detected and the ML models quickly trained and adjusted accordingly.
If there is data drift, the model performance will degrade and no longer provide an accurate guidance.
Data drift is unexpected and undocumented changes to data structure, semantics, and/or infrastructure. With any disruptions, data drift can occur. ML models are dependent on data insights to help plan and support production-ready applications. Any disruption-a pandemic, hurricane, or even blocked sailing routes-has a major impact on the patterns of data and can create anomalous behavior. This is a time where major disruptions are not only lasting longer, but also happening more frequently, as discussed in a McKinsey article on risk and resilience. This data is foundational to power tools, such as data analytics and machine learning (ML), in order to achieve high quality results. As companies continue to embrace the cloud and digital transformation, they use historical data in order to identify trends and insights.
0 kommentar(er)
