Delivery Hero’s GitHub-Vertex AI Voucher Fraud Detection

Delivery Hero’s GitHub and Vertex AI Story

Effective model maintenance is essential in machine learning’s ever-changing world. Fraud detection models must be redeployed often to combat human opponents that may reverse-engineer fraud engine logic and change their methods.

At Delivery Hero, a prominent local delivery platform, the Incentive Fraud team builds ML-powered, rule-based services to identify and prevent incentive voucher fraud. These coupons can be given to those who have just registered to encourage users to use the delivery food platform, therefore it should be able to distinguish new clients from those making new profiles for each order. Delivery Hero works in 70+ countries with varied data privacy laws and local limits, making this task difficult.

Tech set up

Model Serving Overview

The team uses a REST API services with rule-based logic and integrated ML models to make choices. Since the API is utilized on each meal order, latency is tight. The service and models operate on neighborhood Kubernetes clusters with horizontally autoscaler and other high reliability strategies to meet latency requirements.

Vertex AI was chosen for its ML model creation environment due to its scalability and close connection with BigQuery, Delivery Hero’s primary data warehouse, as well as additional Google Cloud resources. Vertex AI Pipelines train models and store metadata in Vertex AI Models Registry. Cloud Build builds FastAPI docker images from trained and evaluated models.

For ML CI/CD

All workflows were tightly integrated with GitHub Actions CI/CD to enable quick model development iterations, allowing users to train models and create pictures using software development and MLOps best practices:

• Version Control (VC)- tracks model and data changes (Vertex Pipelines get a snapshot of a BigQuery table saved on GCS as parquet files with `DATASET_VERSION` suffix).
• Continuous Integration (CI)-reliable trunk-based development within a GitHub repository uploading processes to shared GCP Vertex AI settings without local pipeline triggers. Users (data scientist) can execute experiments in Pull Requests (PR) with precise experiment tracking.
• Continuous Deployment (CD)- lets users securely push new models to production without technical help.
• Continuous Testing (CT) – CI/CD integration of model quality metrics and artifacts lineage improves communication between data scientists, ML developers, stakeholders, and decisionmakers.

CT was implemented to facilitate the building and maintenance of numerous models trained on different data subsets using the same code base. A common Delivery Hero setup is to keep tens of models, one per nation, and deploy them in regional clusters. Thus, each model must be chosen independently, but development, assessment, and deployment iterations are shared.

Implementation details

The team’s Python library ml-utils enabled ML Operations processes for the Incentive Fraud use case utilizing the GCP Vertex AI Model Registry at the backend. This package links Kubeflow pipelines (used internally by GCP Vertex AI Pipelines): Pipeline (or Pipeline Runs), Experimentation (grouping of pipelines), and Models and Datasets from a single CLI or Python API interface. Ml-utils loads the massive Kubeflow Pipeline Runs json definitions, identifies the needed artifacts, and distributes them in a pre-defined manner. More crucially, it offers a layer of abstraction to Models that enforces name rules and queries the Vertex ML Metadata to find models by wildcards.

Training pipeline steps for Vertex AI:

1. load GCS dataset snapshot,
2. divide data for training/testing/validation
3. calculate dataset attributes (statistics, visualization, data drift),
4. new model training,
5. use ml-utils to load the production model,
6. Using ml-utils, load the “champion model” (best-ever model by various measures),
7. evaluated all three models against the same test dataset split and saved the evaluation metrics in Vertex AI Model Registry as Experiment information for ml-utils to access.
8. Update the aliases and submit the new model to Vertex AI Model Registry:

• An alias with a git commit hash, such as `pr123-a1b2c3d` for PR commits or `main-a1b2c3d` for main branch commits
• Transfer the champ alias if the model outperforms the champion model.

Data slices, models, and other pipeline artifacts are automatically stored to GCS. After all Vertex AI pipelines succeed, the GitHub Actions task that initiated them queries the Vertex AI Model Registry to get evaluation metrics and prints them as markdown to the job Summary page . Each git change is linked to a set of Vertex pipelines and the model quality report, which data scientists and management use to make choices based on model quality.

When ready to redeploy certain models, the team creates a PR to edit the serving image config, which defines the slice to be pushed to production. Blue dashed line in Vertex Model Registry. This PR initiates another GitHub Actions workflow that submits a Cloud build workflow that imports model pickles, creates the FastAPI server image with the models baked in, runs integration tests, and changes model aliases.

Infrastructure overview

Team employed two settings for Incentive Fraud projects:

• Dev a permissive security environment for model testing and e2e model training pipeline tests.
• Model release candidates are produced in Prod, a secure environment.

Five high-level rules helped the team manage the two projects:

1. GitHub Actions running GCP processes in Dev and Prod behave identically.
2. Each PR commit triggers Dev GCP workflows.
3. Each main branch commit activates Prod GCP workflows.
4. Dev and Prod pipelines share code and dataset snapshots.
5. Each PR is squished into one commit before merging into the main branch.

These criteria allow the team to have a clean linear history of the main branch, where each commit that changes model code, dataset version, or configuration creates per-country release candidate models with expected quality metrics: