It’s difficult to manage what you can’t see, so visualization is key at the definition stage and even more so when the pipeline is running. Minimize the number of tools required, and provide a single tool to visualize and interact with the pipeline and its dependencies. It’s important that meaningful descriptions are provided for each step so when there is a failure, it’s easy to identify what the step performs. We wouldn’t want multiple jobs with the same name. For example, having naming conventions for jobs in a workflow is essential. It’s vital that teams follow standards when building such workflows. They all must have a way of defining their specific aspect of the workflow from a standard interface, and then have the ability to merge their respective workflows that will make up the pipeline. Multiple teams may be involved in creating the flow. For example, if data ingestion succeeds, then proceed down path A otherwise, proceed with path B. The first challenge is understanding the intended workflow through the pipeline, including any dependencies and required decision tree branching. Five Orchestration Challengesįive challenges stand out in simplifying the orchestration of a machine learning data pipeline. In this post, we will walk through the architecture of a predictive maintenance system that we developed to simplify the complex orchestration steps in a ML pipeline used to reduce downtime and costs for a trucking company. BMC is an AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in DevOps and Migration. You need a way to orchestrate the steps in the pipeline and manage the dependencies between them.Ĭontrol-M, a workflow orchestration solution by BMC Software, Inc., simplifies complex application, data, and file transfer workflows, whether on-premises, on the AWS Cloud, or across a hybrid cloud model. However, coordinating and monitoring the actions across the data pipeline in a way that consistently delivers results in the expected timeframe remains a complex task. If, for instance, you need a Hadoop cluster or data warehouse, you can deploy it in a few hours using AWS services. If you’re going to run those ML models in production at scale, you need data engineering expertise to build a pipeline for data ingestion, storage, processing, and analytics.Īmazon Web Service (AWS) offers a diverse collection of services for data scientists and data engineers. To build and train ML models, you need data science expertise. Predictive maintenance, which analyzes sensor data to predict equipment failures, has emerged as one of the most common business use cases of machine learning (ML) and the Internet of Things (IoT). By Scott Kellish, Partner Solutions Architects at AWSīy Basil Faruqui, Principal Solutions Marketing Manager at BMC Digital Business Automationīy Joe Goldberg, Innovation Evangelist at BMC
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |