How Grammar-Based Synthetic Data Shapes Behavior Learning:
An Empirical Study of Input Generation Strategies

Zusammenfassung

Grammar-based fuzzing can synthesize valid inputs for programs that process structured data. For behavior learning, the same inputs can be paired with the outputs a program produces, creating datasets that train neural program mocks. The fuzzer then serves as a data generator, and its selection policy determines which behaviors the model observes during training. To study how grammar-based input generation affects neural behavior modeling, we compare four generator configurations, train encoder-decoder Transformer models on 10,000 and 100,000 generated input-output pairs, and evaluate the resulting models on synthetic hold-out data and available real-world samples. On synthetic hold-out data, the larger training corpus improves the BLEU score in all 48 paired comparisons; random generation and k-path guidance are strongest on average, though the best generator remains subject-dependent. On real-world data, the larger corpus improves the BLEU score in 35 of 40 paired comparisons, but the gains are smaller and less consistent. JSON-to-YAML models retain high real-world accuracy, whereas msticpy and py-asciimath models expose large synthetic-to-real quality gaps. Overall, generator selection affects learned behavior models, but practical performance depends strongly on how well grammar-generated inputs match the deployment distribution.