Infly (无限光年)

Solution

Infly used Alibaba Cloud MaxCompute as the compute substrate for large-model data preparation so text quality work could scale with training demand. Raw corpora are staged in MaxCompute, which provides fully managed serverless CPU capacity for distributed preprocessing runs without self-managed clusters. On top of that base, MaxFrame supplies a distributed Python execution environment with built-in MinHash operators so very large text collections can be compared and cleaned in parallel. The same MaxFrame layer invokes FastText classification models so datasets can be categorized and scored before later training and fine-tuning stages. The design focuses on CPU-heavy data engineering rather than model inference: MaxCompute supplies elastic resources, MaxFrame runs deduplication and classification logic, and the cleaned outputs are handed to Infly's trustworthy-model training pipeline.

Data flow

Large text datasets are loaded into MaxCompute, where MaxFrame launches distributed MinHash deduplication and FastText classification jobs across elastic CPU capacity. The cleaned, deduplicated, and scored outputs are then passed to downstream large-model training and fine-tuning workflows so Infly can improve data quality before model development.