The universal expressibility assumption of Deep Neural Networks (DNN) is the key motivation behind recent work in the system biology community to employ DNNs to solve important problems in functional genomics and molecular genetics. Because of the black-box nature of DNN, such assumptions, while useful in practice, are unsatisfactory for scientific analysis. In this paper, we given an example of a DNN in which every layer is interpretable. Moreover, this DNN is biologically validated and predictive. We derive our DNN from a System Biology model that was not previously recognized as having a DNN structure. The DNN is concerned with a key unsolved problem in Biology: To understand the DNA regulatory code which controls how genes in multicellular organisms are turned on and off. Although we apply our DNN to data from the early embryo of the fruit fly Drosophila, this system serves as a testbed for the analysis of much larger data sets obtained by System Biology studies on Genomic scale.