Abstract

The modern artificial-intelligence industry is fundamentally a data-and-compute industry as much as it is a model-development industry. The capabilities of systems such as GPT, Gemini, Claude, Llama, Grok, DeepSeek, Qwen, Mistral and Phi emerge from the interaction of enormous training datasets, increasingly sophisticated neural-network architectures, large-scale computing infrastructure, human and machine-generated feedback, reinforcement learning, synthetic data, retrieval systems and continuous post-training.

Yet the exact contents of frontier-model training datasets are rarely disclosed in full. Public documentation generally identifies broad categories—such as publicly available Internet information, licensed material, human-generated examples, contractor data and synthetic data—without publishing a complete list of every website, book, image collection, database or document used. OpenAI, for example, identifies three primary information categories: public Internet information, third-party-accessed information, and information supplied or generated by users, human trainers and researchers. It also states that synthetic data is used in model development.

This thesis examines the information ecosystem behind major AI-model companies, explains the distinction between training data, post-training data, retrieval data and user-context data, and compares the principal architectures currently used across the industry. It argues that the competitive frontier is moving away from the simplistic idea that “the company with the most data wins.” Instead, advantage increasingly depends on data quality, data rights, synthetic-data generation, reasoning-oriented post-training, multimodality, mixture-of-experts architectures, reinforcement learning, inference-time computation and agentic systems.

1. Introduction

Artificial intelligence models are often described as if they were primarily algorithms. In reality, a modern foundation model is the product of several interconnected layers:

Data
Data acquisition
Data cleaning and filtering
Tokenization or multimodal representation
Pre-training
Post-training
Human or AI feedback
Reinforcement learning
Evaluation
Inference infrastructure
Retrieval and external tools
Continuous model improvement

The central question is therefore not simply:

“What model architecture does an AI company use?”

It is also:

“What information does the company have legal or practical access to, how does it transform that information into training material, and how does the resulting model learn from it?”

This distinction is particularly important because the largest AI companies increasingly treat their data recipes as proprietary intellectual property.

Mistral AI, for example, explicitly states that it does not disclose the datasets used to train its models and considers training datasets, training logic and the resources required to produce its models proprietary.

Consequently, any serious comparison must distinguish between confirmed disclosures, partial disclosures and reasonable technical inference.

2. The AI Training Data Supply Chain

A useful way to understand modern AI is to imagine a data supply chain.

Raw information

↓

Data acquisition

↓

Rights/licensing assessment

↓

Cleaning and deduplication

↓

Quality classification

↓

Safety filtering

↓

Tokenization / multimodal processing

↓

Pre-training dataset

↓

Foundation model

↓

Supervised fine-tuning

↓

Preference / reinforcement data

↓

Synthetic data

↓

Evaluation

↓

Production model

↓

User interaction and feedback

This pipeline demonstrates an important fact: the training dataset is not the same thing as the information the AI can access after deployment.

A model may be trained on historical Internet information but subsequently use search engines, databases, APIs, enterprise documents or other tools during inference.

3. The Major Categories of AI Training Information

3.1 Publicly Available Internet Information

The public Internet remains one of the most important sources of large-scale training material.

It can include:

Web pages
Documentation
Encyclopedic material
Educational resources
Public discussions
Public-domain material
Public code repositories
News and articles
Scientific information
Technical documentation
Publicly available images
Other multimedia

OpenAI identifies publicly available Internet information as one of the three primary categories used to develop its foundation models.

Anthropic similarly states that Claude models have been trained using a proprietary mixture that includes publicly available Internet information.

xAI says Grok has primarily been pretrained on publicly available information, including webpage data, metadata extracts and Internet text.

DeepSeek’s disclosure likewise identifies publicly available Internet information as a major pre-training source.

4. Licensed and Third-Party Data

Public Internet information is only one component of the modern training ecosystem.

Companies increasingly acquire additional material through:

Commercial licensing
Data providers
Institutional partnerships
Publishers
Specialist databases
Enterprise partnerships
Human-data contractors
Research organizations

OpenAI states that its models also use information obtained through partnerships with third parties.

Anthropic reports using non-public data from third parties as well as data generated by labeling services and paid contractors.

DeepSeek explicitly describes a two-part pre-training data strategy involving public information and licensed proprietary datasets obtained through third-party agreements.

This illustrates an important industry transition:

The future training-data market is increasingly becoming a commercial data market.

5. Human-Generated Training Data

Large models require more than raw Internet information.

A webpage can tell a model what words and concepts exist, but it does not necessarily teach the model:

How to follow instructions
How to reason through a problem
How to refuse unsafe requests
How to communicate clearly
How to prioritize competing instructions
How to produce useful code
How to evaluate an answer

Consequently, AI laboratories employ human researchers, contractors, domain specialists and evaluators to create or assess training examples.

Human-generated information may include:

Question-and-answer pairs
Demonstrations
Corrections
Preference rankings
Safety examples
Coding examples
Reasoning examples
Evaluation datasets
Domain-specific annotations

Anthropic reports using data provided by labeling services and paid contractors.

OpenAI similarly identifies human trainers and researchers as sources of information used in model development.

6. Synthetic Data: The New Training-Data Factory

One of the most important developments in AI is the rise of synthetic data.

Synthetic data is information generated by AI systems or other computational processes rather than directly collected from humans or the physical world.

It can include:

Artificial question-answer pairs
Generated mathematics problems
Synthetic programming problems
Generated reasoning chains
Artificial conversations
Simulated environments
Automatically generated explanations
Critiques of AI-generated answers
Machine-generated preference data

Microsoft’s Phi-4 research is a particularly clear example. Microsoft describes Phi-4 as a 14-billion-parameter model whose training recipe places unusual emphasis on data quality and strategically incorporates synthetic data throughout training.

Meta’s Llama 3.1 training methodology also describes iterative post-training involving supervised fine-tuning, direct preference optimization and synthetic data generation.

Qwen3 provides another example: Alibaba reports using models such as Qwen2.5-Math and Qwen2.5-Coder to generate synthetic mathematics, textbooks, question-answer pairs and code-related material.

This changes the economics of AI.

The industry is gradually moving from:

“Collect as much human-generated information as possible”

toward:

“Collect high-quality information and use models to generate additional high-quality training environments.”

7. User Data and Feedback

User interactions can occupy a complicated position in the AI-development lifecycle.

A crucial distinction must be made between:

A. Information used to train the foundation model

and

B. Information temporarily supplied to a model during inference.

These are not automatically the same.

OpenAI says information supplied by users can form part of model development, subject to its policies and controls.

Anthropic says commercial-product inputs and outputs are not used for training by default, while specific feedback or explicit opt-in can permit use for training.

xAI states that consumer interactions may be used for training and provides controls allowing users to opt out; it also says business and enterprise customer data is not used to improve its models.

Mistral’s current policies similarly distinguish between consumer and enterprise use and provide controls concerning model training.

Therefore, the phrase “AI trains on everything users type” is an oversimplification.

The actual situation depends on:

Product
Subscription type
Privacy settings
Enterprise agreement
Feedback mechanism
Applicable law
Company’s current policy

8. OpenAI

8.1 Data Sources

OpenAI publicly identifies three major information categories:

Publicly available Internet information
Information accessed through third-party partnerships
Information supplied or generated by users, human trainers and researchers

OpenAI additionally reports the use of synthetic data. Its current training-data summary says its systems are developed using datasets containing trillions of tokens spanning textual, image, audio and audiovisual content.

OpenAI also notes that its datasets can contain material protected by copyright as well as public-domain material.

The company applies processing and filtering techniques intended to improve data quality, accuracy, privacy and safety.

8.2 Architecture

OpenAI does not publicly disclose every architectural detail of its current frontier systems.

Historically, GPT systems have been based on Transformer-derived architectures, but modern frontier models should not be thought of simply as “a Transformer.”

The contemporary architecture stack may include:

Transformer components
Large-scale attention mechanisms
Specialized routing
Multimodal encoders
Reasoning-oriented computation
Tool-use systems
Retrieval
Reinforcement learning
Distillation
Synthetic-data pipelines
Agentic orchestration

The exact internal architecture of current proprietary frontier models remains substantially undisclosed.

9. Google DeepMind and Gemini

Google has a unique position because it combines:

Search
YouTube
Android
Cloud
Books
Maps
Scientific research
DeepMind
Large-scale computing
Extensive multimodal infrastructure

This does not mean that all Google user information is automatically used to train Gemini. Data access and training use remain governed by product policies, technical controls and legal requirements.

Gemini represents a particularly important development in AI architecture because Google has pursued models designed from the beginning for multimodality.

Rather than treating text as the only fundamental information format, modern Gemini systems are designed to work across combinations of:

Text
Images
Audio
Video
Code
Documents

The strategic significance is enormous.

Traditional LLM:

Text → tokens → Transformer → text

Multimodal foundation model:

Text + image + audio + video + code + other representations → shared model system → multimodal output

Google’s AI ecosystem therefore illustrates the transition from language models toward general-purpose multimodal models.

10. Anthropic and Claude

Anthropic’s Claude models represent another major branch of frontier AI.

Anthropic reports that Claude Opus 4 and Sonnet 4 were pretrained on large, diverse datasets and subsequently subjected to post-training involving human feedback and Constitutional AI.

Its disclosed training-data categories include:

Public Internet information
Non-public third-party data
Data from labeling services
Paid-contractor data
Opted-in user data
Internally generated data

Constitutional AI

One of Anthropic’s distinguishing contributions is Constitutional AI.

Instead of depending exclusively on human preference labels, the methodology uses explicit principles to help guide model behavior.

Conceptually:

Base model

↓

Principles

↓

Self-critique

↓

Improved responses

↓

Preference / reinforcement training

↓

Aligned model

Anthropic therefore demonstrates that training-data strategy is not merely about accumulating more documents. It is also about generating behavioral supervision.

11. Meta and Llama

Meta’s Llama family has become one of the most important open-weight model ecosystems.

Llama 3 provides unusually informative public documentation.

Meta states that Llama 3 was pretrained on more than 15 trillion tokens collected from publicly available sources, with the dataset seven times larger than Llama 2’s and containing four times more code.

Meta also describes:

A 128K-token vocabulary
Decoder-only Transformer architecture
Grouped-query attention
Extensive data filtering
Quality classifiers
Multilingual data

Llama 3.1 continued this philosophy.

Meta describes its 405-billion-parameter model as a standard decoder-only Transformer with adaptations rather than a mixture-of-experts architecture, emphasizing training stability.

This is important because it demonstrates that larger parameter count does not automatically require MoE.

12. xAI and Grok

xAI’s Grok family represents another important frontier.

The original Grok-1 was an autoregressive Transformer-based language model trained using next-token prediction and subsequently fine-tuned using human and model-generated feedback.

xAI later disclosed Grok-1 as a 314-billion-parameter Mixture-of-Experts model, with approximately 25% of its parameters active for an individual token.

Grok’s disclosed data ecosystem includes:

Public Internet information
Third-party generated data
User data
Contractor data
Internally generated data

Its Grok 4 model card also describes reinforcement learning involving human feedback, verifiable rewards and model grading.

xAI’s later framework states that it uses synthetic data in reinforcement learning, fine-tuning and post-training.

Grok therefore illustrates the evolution toward:

Internet data + synthetic data + reinforcement learning + verification + agentic capability.

13. DeepSeek

DeepSeek has become particularly influential because it demonstrated that efficient architectures and training methodologies can dramatically reduce the cost of achieving high capability.

DeepSeek-V3 was reported as having:

671 billion total parameters
Approximately 37 billion activated parameters
14.8 trillion training tokens

Its architecture is based on a Mixture-of-Experts approach.

DeepSeek’s own model disclosure identifies public Internet data and licensed third-party data as major pre-training sources. It also describes filtering, human and algorithmic review, synthetic or automatically constructed question-answer data, and safety-oriented training.

DeepSeek’s importance therefore extends beyond its model itself.

It demonstrates the strategic value of:

Sparse computation
Efficient expert routing
High-quality datasets
Synthetic data
Reinforcement learning
Open technical publication

14. Alibaba Qwen

Qwen illustrates another major development: massive multilingual and multimodal training combined with synthetic data generation.

Qwen3 reportedly expanded pre-training to approximately 36 trillion tokens, covering 119 languages and dialects. Alibaba says the data included Web material and PDF-like documents and that models were used to extract and improve information from documents.

The Qwen training ecosystem also generated synthetic mathematics and coding data using specialized models.

This represents an important architectural principle:

A model can become part of the machinery used to create the training data for the next model.

That creates a feedback loop:

Model A

→ generates data

→ Model B training

→ improves reasoning

→ generates better data

→ Model C

This is increasingly important in frontier AI.

15. Mistral AI

Mistral has contributed significantly to the development of open and efficient model architectures.

Its Mixtral family is especially important because it demonstrates the power of Sparse Mixture-of-Experts architectures.

Mixtral 8×7B consists of eight expert groups with routing that selects two experts for each token. Mistral explains that this increases total model capacity while activating only a fraction of the parameters for each token.

Mixtral 8×22B continued the strategy, with 141 billion total parameters but approximately 39 billion active parameters.

However, Mistral is considerably less transparent about the exact datasets underlying its proprietary models.

The company explicitly states that its training datasets remain undisclosed proprietary assets.

This demonstrates a fundamental industry tension:

Open model weights do not necessarily mean open training data.

16. Microsoft Phi

Microsoft’s Phi family demonstrates a different philosophy.

Instead of relying primarily on gigantic model size, Microsoft has emphasized high-quality training data, synthetic data and efficient small models.

Phi-4 contains approximately 14 billion parameters and was designed with a strong emphasis on data quality. Microsoft reports that synthetic data plays a significant role throughout the training process.

The significance is profound.

A smaller model trained on exceptionally good data can outperform a larger model trained on lower-quality data for particular tasks.

This leads to a broader principle:

Parameter count is not a complete measure of intelligence.

17. Comparing Major Companies

Company / Family	Major disclosed data categories	Architecture direction	Openness
OpenAI / GPT	Public Internet, third-party data, user/human/researcher data, synthetic data	Proprietary frontier architecture; Transformer-derived ecosystem	Mostly closed
Google / Gemini	Large-scale multimodal and Google/third-party information ecosystem	Multimodal frontier architecture	Mostly closed
Anthropic / Claude	Public Internet, third-party, contractors, opt-in users, internal data	Proprietary Transformer-derived frontier systems	Closed
Meta / Llama	Publicly available sources, extensive code and multilingual data	Decoder-only Transformer; GQA; large dense models in disclosed Llama generations	Open-weight
xAI / Grok	Public Internet, third-party, user/contractor and internal data, synthetic data	Transformer + MoE in disclosed Grok generations	Mixed
DeepSeek	Public Internet + licensed data + generated optimization data	MoE and efficiency-oriented architectures	Relatively open
Alibaba / Qwen	Web, documents, synthetic math/code data	Dense and MoE variants; multilingual/multimodal	Open-weight for many models
Mistral	Not publicly disclosed in full	Dense and sparse MoE architectures	Mixed/open-weight
Microsoft / Phi	Organic data plus extensive synthetic data	Compact Transformer-derived models	Open-weight for several releases

The table should be interpreted carefully: companies frequently have multiple model families, and the architecture of one generation cannot automatically be assigned to every current model.

18. The Transformer Remains the Foundation

Despite enormous innovation, the Transformer remains the dominant foundation for modern generative AI.

The basic Transformer contains mechanisms that allow the model to process relationships among tokens.

At a simplified level:

Input

↓

Tokenization

↓

Embeddings

↓

Attention

↓

Feed-forward computation

↓

Repeated layers

↓

Output probabilities

↓

Next token

The original Transformer architecture has evolved substantially.

Modern systems may incorporate:

Multi-head attention
Grouped-query attention
Multi-query attention
Rotary positional representations
Mixture-of-experts
Long-context mechanisms
Multimodal encoders
Retrieval systems
Tool calling
Reasoning-time computation
Reinforcement learning

Thus, saying that a model is “a Transformer” is technically correct but increasingly incomplete.

19. Mixture-of-Experts Architecture

Mixture-of-Experts, or MoE, is one of the most important architectural developments.

A dense model activates essentially the same major parameter pathway for each token.

An MoE model contains multiple expert networks.

Conceptually:

Token

↓

Router

↙ ↓ ↘

Expert A | Expert B | Expert C

↓

Selected expert outputs

↓

Combined representation

↓

Next layer

The key advantage is that a model can contain enormous total parameter capacity while activating only a fraction of that capacity for each token.

DeepSeek-V3, Grok-1 and Mixtral are prominent examples of the MoE direction.

20. Dense Models

Dense models activate their primary parameters for each token.

Advantages can include:

Simpler architecture
Predictable computation
Straightforward scaling
Stable training characteristics

Meta’s disclosed Llama 3.1 405B architecture is a notable example of a large decoder-only Transformer that did not adopt MoE.

Dense architecture therefore remains highly relevant.

21. Multimodal Models

The next generation of AI is increasingly multimodal.

Instead of:

Text → Text

the system becomes:

Text + Image + Audio + Video + Code + Documents

↓

Unified or coordinated representation

↓

Reasoning

↓

Text / Image / Audio / Action

This is particularly important for systems intended to operate as general-purpose assistants.

A multimodal model can potentially:

Read documents
Interpret diagrams
Analyze images
Understand speech
Process video
Write software
Generate images
Reason over multiple information formats

The architecture becomes less like a traditional language model and more like a general information-processing system.

22. Retrieval-Augmented Generation

Another important distinction is between training and retrieval.

A model does not necessarily need to memorize every piece of information it uses.

A retrieval system can work as follows:

User question

↓

Search / retrieval

↓

Relevant documents

↓

Context window

↓

AI model

↓

Answer

This is called Retrieval-Augmented Generation (RAG).

RAG is particularly important for:

Enterprise databases
Legal information
Scientific literature
Corporate documents
Product manuals
Internal knowledge
Frequently changing information

It allows organizations to connect a foundation model to current information without retraining the entire model every time the underlying information changes.

23. AI Agents

The frontier is increasingly moving beyond standalone chatbots.

An AI agent can combine:

Foundation model
Memory
Retrieval
Planning
Tools
APIs
Code execution
Browser/search capability
External databases
Feedback loops

The conceptual architecture becomes:

User

↓

Agent

↓

Reasoning model

↙ ↓ ↘

Search | Code | Database

↘ ↓ ↙

Observation

↓

Reasoning

↓

Action

↓

Result

This changes what “AI training data” means.

The model itself may have been trained on historical data, while the agent obtains current information through external systems.

24. Reinforcement Learning

Modern frontier systems increasingly use reinforcement learning or related preference-optimization techniques.

The basic concept is:

Model generates answer

↓

Answer evaluated

↓

Reward / preference signal

↓

Model parameters updated

↓

Better future behavior

The reward can come from:

Humans
AI evaluators
Verifiable mathematical answers
Coding tests
Simulated environments
Safety classifiers
Rule-based systems

xAI explicitly reports human feedback, verifiable rewards and model grading in its Grok 4 training methodology.

Anthropic describes human feedback and Constitutional AI in Claude’s post-training process.

25. The Rise of Verifiable Training

One of the most significant developments is the shift toward tasks where correctness can be automatically checked.

Examples include:

Mathematics
Programming
Formal logic
Some scientific problems
Games
Simulated environments

For these tasks, the system can receive a reward based on whether the answer actually works.

For example:

AI writes code

↓

Compiler executes code

↓

Tests run

↓

Tests pass

↓

Positive reward

This creates a much stronger learning signal than simply asking a human whether the response “looks good.”

26. Data Quality Versus Data Quantity

The early AI race emphasized enormous datasets.

Modern research increasingly emphasizes data quality.

A high-quality dataset can contain:

Correct information
Diverse examples
High-value reasoning
Strong code
Verified mathematics
Accurate scientific material
Good instructional examples
Carefully selected multilingual information

Low-quality data can contain:

Duplicates
Spam
Incorrect information
SEO-generated pages
Toxic material
Machine-generated repetition
Poor translations
Contradictions

Therefore:

More tokens ≠ automatically better intelligence.

The modern equation is closer to:

Capability = data quality × data diversity × architecture × compute × optimization × post-training

27. The Problem of Data Contamination

As AI-generated content spreads across the Internet, a major problem emerges.

Suppose:

AI Model A

generates millions of webpages.

Those webpages are later scraped by:

AI Model B

Model B is therefore learning from information originally generated by Model A.

If this continues repeatedly, training datasets can become increasingly synthetic and repetitive.

This creates concerns about:

Information degradation
Error propagation
Model collapse
Reduced diversity
Circular knowledge
Hidden bias amplification

Consequently, future AI laboratories will need increasingly sophisticated systems for identifying:

Human-generated data

versus

Machine-generated data.

28. Copyright and Data Rights

One of the largest unresolved issues surrounding AI concerns the legal status of training data.

Training datasets may contain:

Copyrighted works
Public-domain works
Licensed material
User-generated content
Public personal information
Commercial databases

Different jurisdictions have different rules concerning:

Copyright
Text and data mining
Fair use
Licensing
Privacy
Database rights
Opt-out mechanisms

OpenAI explicitly acknowledges that its training datasets may contain copyright-protected material.

The future AI economy will therefore involve an expanding market for legally licensed training information.

29. Privacy

Privacy is another major dimension.

A large training dataset may inadvertently contain:

Names
Addresses
Telephone numbers
Public profiles
Personal documents
Sensitive information

Companies therefore employ:

Filtering
Deduplication
Redaction
Classification
Data minimization
Privacy controls

DeepSeek states that it attempts to remove sensitive and identifying information from training datasets and uses filtering and testing procedures.

Anthropic similarly describes processes addressing personal data within training information.

30. Why AI Companies Do Not Publish Everything

There are several reasons.

30.1 Competitive advantage

The dataset itself can be valuable intellectual property.

30.2 Legal uncertainty

Publishing a complete dataset inventory could expose companies to additional disputes.

30.3 Security

Detailed disclosure can reveal weaknesses in filtering or safety systems.

30.4 Cost

Large datasets may involve expensive licensing agreements.

30.5 Privacy

Publishing data sources at excessive granularity could expose sensitive information.

30.6 Strategic secrecy

Knowing the exact data recipe can help competitors reproduce a model.

Thus, the modern AI industry often publishes categories and methodologies rather than complete datasets.

31. Open-Weight Does Not Mean Open-Source Everything

This distinction is critical.

An AI company can release:

Model weights

without releasing:

Original training datasets
Data licenses
Filtering systems
Training infrastructure
Proprietary evaluation datasets
Complete post-training data
Internal research methods

Meta, DeepSeek, Mistral and other companies demonstrate different points along this openness spectrum.

Consequently, the AI industry contains several levels of openness:

Closed model

↓

Open API

↓

Open technical paper

↓

Open weights

↓

Open architecture

↓

Open training methodology

↓

Open datasets

↓

Fully reproducible system

Very few frontier systems are completely open across all layers.

32. The New Strategic Resource: Data Generation

An important development is that leading AI companies increasingly possess systems capable of creating their own training material.

This produces a powerful cycle:

Human data

↓

Foundation model

↓

Synthetic-data generator

↓

Higher-quality training data

↓

Improved model

↓

Better synthetic-data generator

↓

Even better model

This could eventually reduce dependence on raw Internet data.

The critical challenge becomes determining:

How do we verify that synthetic information is actually better than the information from which it was generated?

33. AI Models Are Becoming Training Ecosystems

A modern AI company is therefore not simply building one neural network.

It is building an ecosystem:

Layer 1 — Data

Public, licensed, proprietary and synthetic information.

Layer 2 — Data engineering

Cleaning, filtering, deduplication and classification.

Layer 3 — Foundation model

Transformer, MoE, multimodal or other neural architecture.

Layer 4 — Post-training

Supervised fine-tuning and preference optimization.

Layer 5 — Reinforcement learning

Human, AI or verifiable rewards.

Layer 6 — Evaluation

Benchmarks, adversarial testing and real-world evaluation.

Layer 7 — Retrieval

Search and databases.

Layer 8 — Tools

Code execution, APIs and external systems.

Layer 9 — Agent

Planning and autonomous task execution.

Layer 10 — Product

Chatbots, coding assistants, enterprise systems, robotics and other applications.

The model is therefore only one component.

34. The Emerging Architecture of Frontier AI

A useful conceptual model of the modern frontier system is:

                    HUMAN KNOWLEDGE
                           │
        ┌──────────────────┼──────────────────┐
        │                  │                  │
   PUBLIC WEB          LICENSED DATA     HUMAN DATA
        │                  │                  │
        └──────────────────┼──────────────────┘
                           ↓
                    DATA PROCESSING
                           ↓
              FILTERING + DEDUPLICATION
                           ↓
                    PRE-TRAINING
                           ↓
                 FOUNDATION MODEL
                           ↓
        ┌──────────────────┼──────────────────┐
        │                  │                  │
   FINE-TUNING        RL / PREFERENCE    SYNTHETIC DATA
        │                  │                  │
        └──────────────────┼──────────────────┘
                           ↓
                       REASONING
                           ↓
                 RETRIEVAL + TOOLS
                           ↓
                        AGENT
                           ↓
                    USER / WORLD
                           │
                           ↓
                    FEEDBACK LOOP
                           │
                           └──────→ FUTURE TRAINING

This architecture captures the fundamental transformation taking place in AI.

35. What Each Major Company Appears to Optimize

A simplified strategic interpretation is useful.

OpenAI

Focus:

Frontier capability
Multimodal intelligence
Reasoning
Agents
Large-scale deployment

Google DeepMind

Focus:

Multimodality
Scientific AI
Search integration
Large-scale infrastructure
General intelligence research

Anthropic

Focus:

Reasoning
Coding
Reliability
Safety
Constitutional alignment
Enterprise use

xAI

Focus:

Frontier models
Real-time information
Reasoning
Coding
Agentic systems

DeepSeek

Focus:

Efficient training
MoE
Reasoning
Open models
Cost efficiency

Alibaba/Qwen

Focus:

Multilingual AI
Reasoning
Coding
Multimodal systems
Open model ecosystem

Mistral

Focus:

Efficient models
Open-weight systems
Enterprise AI
European AI sovereignty
Specialized models

Microsoft/Phi

Focus:

Small efficient models
Synthetic data
On-device AI
Efficient reasoning

36. The AI Data Race Is Becoming a Data-Quality Race

The industry’s competitive structure is changing.

The first phase was:

More Internet data

The second phase:

More compute

The third phase:

Bigger models

The fourth phase:

Better post-training

The emerging phase is:

Better information + synthetic data + reasoning + efficient architectures + inference-time computation.

This means the next generation of AI companies may not win by merely possessing the largest raw dataset.

They may win through the ability to transform mediocre information into high-value learning experiences.

37. The Importance of Specialized Data

General Internet information is useful for general intelligence.

But specialized applications require specialized data.

Examples:

Medicine

Clinical literature
Medical textbooks
Structured medical data
Carefully validated expert examples

Engineering

Technical specifications
CAD information
Engineering standards
Simulation results

Law

Legislation
Regulations
Court decisions
Legal commentary

Agriculture

Crop data
Weather
Soil information
Satellite imagery
Agricultural research

Finance

Market information
Economic statistics
Company filings
Financial reports

Science

Scientific publications
Experimental results
Databases
Simulations

This explains why enterprise-specific AI training is becoming increasingly important.

Mistral’s Forge initiative is explicitly aimed at allowing enterprises to build models grounded in proprietary organizational knowledge, with partnerships involving organizations such as ASML, Ericsson, the European Space Agency and others.

38. The Future of Training Data

The next decade is likely to produce several major trends.

38.1 More licensed data

Companies will increasingly negotiate explicit rights to high-value datasets.

38.2 More synthetic data

AI will increasingly manufacture specialized training examples.

38.3 More verified data

Mathematics, code and scientific reasoning will benefit from automatically verifiable training.

38.4 More multimodal data

Text alone will no longer dominate the foundation-model ecosystem.

38.5 More private enterprise data

Companies will build AI around internal knowledge.

38.6 More regional datasets

African, Asian, Latin American and other underrepresented languages and cultural contexts will become strategically important.

38.7 More continual data pipelines

AI systems will increasingly interact with continuously updated information rather than relying exclusively on static training datasets.

39. Africa and the Training-Data Opportunity

Africa represents an important underdeveloped AI-data opportunity.

Many African languages remain significantly underrepresented in major datasets.

Potential training resources include:

African language corpora
Local literature
Educational material
Agricultural knowledge
Government documents
Historical archives
Indigenous knowledge where appropriate and legally permissible
African scientific research
Local business information
Geospatial data

The challenge is not simply collecting the information.

It is ensuring:

Consent
Ownership
Cultural respect
Linguistic quality
Compensation
Privacy
Data sovereignty

African AI development therefore presents an opportunity to move from being primarily a consumer of imported AI to becoming a producer of globally valuable AI data and models.

40. The Fundamental Difference Between Knowledge and Intelligence

An important conceptual conclusion emerges from studying these companies.

A model can have enormous quantities of information without possessing reliable reasoning.

Therefore:

Knowledge ≠ intelligence.

A modern AI system needs:

Knowledge

Representation

Reasoning

Planning

Verification

Memory

Tools

Feedback

The evolution of frontier AI is therefore moving from simply teaching models more facts toward creating systems capable of using knowledge effectively.

41. Major Findings

This thesis produces several central findings.

Finding 1

There is no single universal source of AI training data.

Finding 2

Public Internet information remains fundamental, but it is increasingly supplemented by licensed, proprietary, human-generated and synthetic information.

Finding 3

Exact datasets are often proprietary.

Finding 4

Open model weights do not necessarily mean open datasets.

Finding 5

Synthetic data has become a major component of modern AI development.

Finding 6

The Transformer remains the dominant foundational architecture.

Finding 7

Mixture-of-Experts is increasingly important because it allows large parameter capacity with sparse computation.

Finding 8

Multimodal AI is replacing text-only thinking about foundation models.

Finding 9

Post-training and reinforcement learning are increasingly as important as pre-training.

Finding 10

Retrieval, tools and agents extend the capabilities of foundation models beyond their static training data.

Finding 11

Data quality is increasingly more important than raw data quantity.

Finding 12

The competitive frontier is moving toward reasoning, verification, agents and inference-time computation.

42. Conclusion

The question “Where do AI companies get their information?” has no simple one-line answer.

The modern AI training ecosystem is a combination of:

public information + licensed information + human-generated information + proprietary information + synthetic data + feedback + reinforcement signals.

The second question—“What architectures do they use?”—is equally complex.

The dominant foundation remains the Transformer, but the modern AI architecture has evolved into a much broader system incorporating:

Transformers + Mixture-of-Experts + multimodality + reinforcement learning + synthetic data + retrieval + tools + agents + inference-time reasoning.

OpenAI, Google DeepMind, Anthropic, Meta, xAI, DeepSeek, Alibaba and Mistral are pursuing different combinations of these technologies. Meta provides unusually detailed public information about Llama’s training data and architecture; DeepSeek has emphasized efficient MoE systems and open technical disclosure; Anthropic emphasizes human feedback and Constitutional AI; Microsoft demonstrates the power of synthetic-data-centered small models; Mistral has pioneered efficient sparse MoE models; Qwen demonstrates enormous multilingual and synthetic-data pipelines; and xAI combines large-scale Internet data with reinforcement-learning and synthetic-data strategies.

The deeper lesson is that the foundation model itself is no longer the entire AI system.

The emerging AI machine is better understood as:

Data → Model → Post-training → Reasoning → Retrieval → Tools → Agent → Feedback → New Data.

That creates a self-improving technological ecosystem in which data produces models, models produce synthetic data, synthetic data improves models, and increasingly capable models interact with the external world through tools and agents.

The great strategic question of the coming AI era may therefore not simply be who has the largest model, but rather:

Who can build the most effective closed-loop system for acquiring, governing, generating, learning from, verifying and deploying information?

That is likely to be one of the defining competitive advantages of the next generation of artificial intelligence.

Selected Sources and Further Reading

Research note: Exact training-data inventories for leading proprietary models are generally unavailable publicly. Where companies disclose only broad categories, this thesis deliberately avoids presenting speculation as fact. Architecture descriptions likewise distinguish publicly documented architectures from proprietary systems whose complete internal designs have not been released.

Where Major AI Model Companies Source Their Information and Training Data, and What Types of AI Models or Architectures They Are Currently Using?

Abstract

1. Introduction

2. The AI Training Data Supply Chain

Raw information

Data acquisition

Rights/licensing assessment

Cleaning and deduplication

Quality classification

Safety filtering

Tokenization / multimodal processing

Pre-training dataset

Foundation model

Supervised fine-tuning

Preference / reinforcement data

Synthetic data

Evaluation

Production model

User interaction and feedback

3. The Major Categories of AI Training Information

3.1 Publicly Available Internet Information

4. Licensed and Third-Party Data

5. Human-Generated Training Data

6. Synthetic Data: The New Training-Data Factory

7. User Data and Feedback

A. Information used to train the foundation model

B. Information temporarily supplied to a model during inference.

8. OpenAI

8.1 Data Sources

8.2 Architecture

9. Google DeepMind and Gemini

10. Anthropic and Claude

Constitutional AI

11. Meta and Llama

12. xAI and Grok

13. DeepSeek

14. Alibaba Qwen

15. Mistral AI

16. Microsoft Phi

17. Comparing Major Companies

18. The Transformer Remains the Foundation

19. Mixture-of-Experts Architecture

20. Dense Models

21. Multimodal Models

22. Retrieval-Augmented Generation

23. AI Agents

24. Reinforcement Learning

25. The Rise of Verifiable Training

26. Data Quality Versus Data Quantity

27. The Problem of Data Contamination

28. Copyright and Data Rights

29. Privacy

30. Why AI Companies Do Not Publish Everything

30.1 Competitive advantage

30.2 Legal uncertainty

30.3 Security

30.4 Cost

30.5 Privacy

30.6 Strategic secrecy

31. Open-Weight Does Not Mean Open-Source Everything

Model weights

32. The New Strategic Resource: Data Generation

33. AI Models Are Becoming Training Ecosystems

Layer 1 — Data

Layer 2 — Data engineering

Layer 3 — Foundation model

Layer 4 — Post-training

Layer 5 — Reinforcement learning

Layer 6 — Evaluation

Layer 7 — Retrieval

Layer 8 — Tools

Layer 9 — Agent

Layer 10 — Product

34. The Emerging Architecture of Frontier AI

35. What Each Major Company Appears to Optimize

OpenAI

Google DeepMind

Anthropic

Meta

xAI