Source: https://seerepo.com/r/OpenDCAI/DataFlow/built
Markdown source: https://seerepo.com/r/OpenDCAI/DataFlow/built.md
Description: DataFlow is a Python repo built with FastAPI and uvicorn. It reaches its data with sqlglot. It has 78,684 written lines, with 83 test files.

# What is DataFlow built with?

DataFlow is a Python repo built with FastAPI and uvicorn. It reaches its data with sqlglot, uses AI through accelerate, evaluate, sentencepiece, torchaudio, torchvision, and 13 more, connects to Google, makes its content with lxml-html-clean, pypdf, and pymupdf, validates its data with Pydantic, is tested with Pytest, runs in production with psutil and colorlog, and uses db-dtypes, gcsfs, pyarrow, and pymysql. It has 78,684 written lines, with 83 test files.

Read on November 16, 2026, at commit 13d5de8.

Repository: https://github.com/OpenDCAI/DataFlow

## Frontend

### Interface

- **[Gradio](https://seerepo.com/uses/gradio)**: 2 files
- **[Rich](https://seerepo.com/uses/rich)**: 1 file

## Backend

### Data

- **db-dtypes**
- **gcsfs**
- **[pyarrow](https://seerepo.com/uses/pyarrow)**: 1 file
- **[pymysql](https://seerepo.com/uses/pymysql)**: 1 file

### Data access

- **[sqlglot](https://seerepo.com/uses/sqlglot)**

### AI

- **[accelerate](https://seerepo.com/uses/accelerate)**
- **[evaluate](https://seerepo.com/uses/evaluate)**
- **[sentencepiece](https://seerepo.com/uses/sentencepiece)**
- **[torchaudio](https://seerepo.com/uses/torchaudio)**
- **[torchvision](https://seerepo.com/uses/torchvision)**
- **[torch](https://seerepo.com/uses/torch)**: 10 files
- **[transformers](https://seerepo.com/uses/transformers)**: 10 files
- **[nltk](https://seerepo.com/uses/nltk)**: 4 files
- **chonkie**: 2 files
- **[datasets](https://seerepo.com/uses/datasets)**: 1 file
- **fasttext**: 1 file
- **[librosa](https://seerepo.com/uses/librosa)**: 1 file
- **math-verify**: 1 file
- **[ModelScope](https://seerepo.com/uses/modelscope)**: 1 file
- **presidio-analyzer**: 1 file
- **presidio-anonymizer**: 1 file
- **[tiktoken](https://seerepo.com/uses/tiktoken)**: 1 file

### Services

- **[Google](https://seerepo.com/uses/google)**: 2 files · [dataflow/serving/api_google_vertexai_serving.py, line 36](https://github.com/OpenDCAI/DataFlow/blob/13d5de84ad151c0bfc434167d9e3b9e36c3809e4/dataflow/serving/api_google_vertexai_serving.py#L36)
- **[OpenAI](https://seerepo.com/uses/openai)**: 2 files · [dataflow/serving/api_vlm_serving_openai.py, line 7](https://github.com/OpenDCAI/DataFlow/blob/13d5de84ad151c0bfc434167d9e3b9e36c3809e4/dataflow/serving/api_vlm_serving_openai.py#L7)

## Support

### The base

- **[FastAPI](https://seerepo.com/uses/fastapi)**
- **[uvicorn](https://seerepo.com/uses/uvicorn)**

### Content

- **lxml-html-clean**
- **[pypdf](https://seerepo.com/uses/pypdf)**: 2 files
- **[pymupdf](https://seerepo.com/uses/pymupdf)**: 1 file

### Validation

- **[Pydantic](https://seerepo.com/uses/pydantic)**: 1 file

### Testing

- **Pytest**: 52 files

### Deploy

- **[psutil](https://seerepo.com/uses/psutil)**
- **[colorlog](https://seerepo.com/uses/colorlog)**: 1 file

### Tools

- **docstring-parser**
- **Cookiecutter**: 1 file

### Other

- **addict**
- **cloudpickle**
- **func-timeout**
- **httpx**
- **json5**
- **pyfiglet**
- **soundfile**
- **sseclient-py**
- **termcolor**
- **pandas**: 83 files
- **tqdm**: 56 files
- **NumPy**: 37 files
- **requests**: 10 files
- **colorama**: 7 files
- **pyyaml**: 5 files
- **word2number**: 2 files
- **appdirs**: 1 file
- **datasketch**: 1 file
- **google-api-core**: 1 file
- **rapidfuzz**: 1 file
- **SciPy**: 1 file
- **simhash**: 1 file
- **trafilatura**: 1 file

## Other repos with a similar stack

- [run-llama/llama_index](https://seerepo.com/r/run-llama/llama_index/built): gcsfs, pyarrow, pymysql, Google, OpenAI, Rich, sentencepiece, torch, transformers, nltk, chonkie, datasets, ModelScope, presidio-analyzer, presidio-anonymizer, tiktoken, lxml-html-clean, pypdf, pymupdf, Pydantic, psutil · [Compare](https://seerepo.com/compare/OpenDCAI/DataFlow/vs/run-llama/llama_index)
- [bojieli/ai-agent-book](https://seerepo.com/r/bojieli/ai-agent-book/built): pyarrow, Google, OpenAI, FastAPI, uvicorn, Rich, accelerate, sentencepiece, torchaudio, torch, transformers, nltk, datasets, librosa, tiktoken, pypdf, pymupdf, Pydantic, psutil, colorlog · [Compare](https://seerepo.com/compare/bojieli/ai-agent-book/vs/OpenDCAI/DataFlow)
- [ray-project/ray](https://seerepo.com/r/ray-project/ray/built): gcsfs, pyarrow, Google, FastAPI, uvicorn, Gradio, Rich, sqlglot, sentencepiece, torchvision, torch, transformers, nltk, datasets, tiktoken, Pydantic, psutil, colorlog · [Compare](https://seerepo.com/compare/OpenDCAI/DataFlow/vs/ray-project/ray)
- [hpcaitech/ColossalAI](https://seerepo.com/r/hpcaitech/ColossalAI/built): OpenAI, FastAPI, uvicorn, Gradio, Rich, accelerate, sentencepiece, torchaudio, torchvision, torch, transformers, datasets, math-verify, ModelScope, tiktoken, pypdf, Pydantic, psutil · [Compare](https://seerepo.com/compare/hpcaitech/ColossalAI/vs/OpenDCAI/DataFlow)
- [debpalash/VoiceStudio](https://seerepo.com/r/debpalash/VoiceStudio/built): pyarrow, Google, OpenAI, FastAPI, uvicorn, Rich, accelerate, torchaudio, torchvision, torch, transformers, librosa, ModelScope, pypdf, Pydantic, psutil · [Compare](https://seerepo.com/compare/debpalash/VoiceStudio/vs/OpenDCAI/DataFlow)
- [mudler/LocalAI](https://seerepo.com/r/mudler/LocalAI/built): pyarrow, OpenAI, FastAPI, uvicorn, Gradio, accelerate, sentencepiece, torchaudio, torchvision, torch, transformers, datasets, librosa, ModelScope, tiktoken · [Compare](https://seerepo.com/compare/mudler/LocalAI/vs/OpenDCAI/DataFlow)

## Frequently asked questions

### What language is DataFlow written in?

DataFlow is written in Python, with 78,684 written lines.

### What framework does DataFlow use?

DataFlow is built with FastAPI and uvicorn.

### Does DataFlow use AI?

Yes. DataFlow uses AI through accelerate, evaluate, sentencepiece, torchaudio, torchvision, torch, transformers, nltk, chonkie, datasets, fasttext, librosa, math-verify, ModelScope, presidio-analyzer, presidio-anonymizer, tiktoken, and OpenAI.

### What services does DataFlow use?

DataFlow connects to Google and uses db-dtypes, gcsfs, pyarrow, and pymysql.

The proof: [dataflow/serving/api_google_vertexai_serving.py, line 36](https://github.com/OpenDCAI/DataFlow/blob/13d5de84ad151c0bfc434167d9e3b9e36c3809e4/dataflow/serving/api_google_vertexai_serving.py#L36)

### Does DataFlow have tests?

Yes. DataFlow has 83 test files and is tested with Pytest.

### Which repos have a stack like DataFlow's?

The ones that share most of its stack: run-llama/llama_index, bojieli/ai-agent-book, and ray-project/ray.

## What is your repo built with?

Any public repo, with no account: https://seerepo.com/r/owner/repo/built, with the owner and name it has on GitHub.
