CustomGPT Experiments

A shared from-scratch GPT architecture used to study arithmetic loss masking, corpus quality, model capacity, context length, and consumer-GPU constraints.

Engineering projectPythonPyTorchTransformers
Contents

CustomGPT is the shared decoder-only architecture behind two focused experiments:

  • SumGPT studies three-digit addition, reversed output digits, and loss masking that excludes inherently unpredictable operands.
  • BookGPT studies native-Italian corpus quality, model capacity, context length, and training on consumer AMD hardware.

The experiments are presented together because the useful result is not another small transformer implementation, but what changed when the objective, data, model size, and hardware constraints changed.

Technologies

  • Python
  • PyTorch
  • Transformers
  • ROCm