Introduction

While the idea of applying a stack of layers composed of computational nodes to estimate complex functions origins in the 1960s rosenblatt1958perceptron, it was not until the 1990s, when the first convolutional neural networks were introduced LeCun1990Handwritten, that artificial neural networks were successfully applied on real world classification tasks. With the beginning of this decade and the massive increase in broadly available computing power the advent of Deep Learning begun. Groundbreaking work by Krizhevsky in image classification Krizhevsky2012ImageNet paved the way for many applications in image, video, sound and natural language processing. There has also been successful work on biological and medical data alipanahi2015predicting, kadurin2017cornucopia.

Background

Artificial neural networks are powerful function approximators, able to untangle complex relations in the input space cybenko1989approximation. However it were the convolutional neural networks proposed in the early 1990s LeCun1990Handwritten that made deep learning possible. Convolutional neural networks rely on trainable filters or kernels to extract valuable information (features) from the input space. The application of trainable kernels for feature extraction has been demonstrated to be extremely powerful in representation learning oquab2014learning, detection lee2009unsupervised and classification Krizhevsky2012ImageNet tasks. Similar to the visual cortex of mammals, convolutional neural networks comprise different layers of abstraction. While the lower layers detect simple properties like edges and corners, higher layers assemble the features from the lower layers and detect more complex shapes lecun2015deep. With increasing depth the layers have a larger receptive field and are thus able to combine more signals from the layers below lecun2015deep. A convolutional neural network can thus be trained to extract the relevant information present in the input space and encode the input in a compressed representation. Handwritten, human-designed feature extraction thus becomes obsolete. Often a convolutional neural network is complemented by a small fully connected (dense) neural network part processing the extracted features to perform the real classification or detection task Krizhevsky2012ImageNet.

Protein representation learning

The protein space is extremely complex. The amino acid alphabet knows 20 basic letters and an average protein has a length of 500 residues, making the combinatory complexity of the space tremendous. Comparable to images however, functional protein sequences reside on a thin manyfold within the total sequence space. Learning the properties of the protein distribution of a certain functionality would enable not only a decent classification of sequences into functions but also unlimited sampling from this distribution resulting in de novo protein sequence generation. Attempts for protein sequence classification have been made with CNNs szalkai2017near as well as with recurrent neural networks liu2017deep with good success, however without the possibility for generative modelling.
Also handwritten feature extractors exist for protein sequences bandyopadhyay2005efficientsaeys2007review. Along with support vector machines they were applied in protein-protein interaction prediciton hamp2015evolutionary as well as in protein family classification cai2003svmleslie2002spectrum. However they are outperformed by trainable approaches applying CNNs szalkai2017near or word2vec models leslie2002spectrum.

Network Architecture

To harness the strenghts of convolutional networks in representation learning and feature extraction we implemented a fully convolutional architecture to classify protein sequences to functions based on the ResNet architecture he2016deep. The motivation behind applying a residual neural network is an optimized training stability and facilitated learining process. As the identity is added to the output of a residual block through a shortcut, the fear of fading gradients and accuracy saturation is reduced.
We define a residual block as a set of two convolutional layers, a convolutional layer with kernel size 1 to squeeze the channels Iandola2016SqueezeNet and a pooling layer. Further a batch normalisation layer is attached to every convolutional layer. The input of a residual block is added to its output after being resized by a pooling layer and padded on the channels. A schematic view on a residual block is depicted in figure 1, together with an overview of the overall architecture. The model consists of an input processing layer, 30 residual blocks and two independent convolutional outlayers.

Label selection

To harness the strenghts of convolutional networks in representation learning and feature extraction we implemented a fully convolutional architecture to classify protein sequences to functions. Function labels were thereby defined by the gene ontology (GO) annotation gene2004gene. The gene ontology annotation is hierarchical and best described as a directed acyclic graph (DAG). It contains labels providing information on the cellular location, pathway and molecular function of a particular protein. As we were interested solely in protein function classification, we considered on GO-labels in the molecular function sub-DAG. The molecular function sub-DAG has up to 12 levels and 11135 GO-terms. As the population between terms varies greatly and strongly depends on the terms' level in the DAG, with terms towards the roots being stronger populated than leaf terms.
Thus, we thresholded the considered labels based on their minimum population, ending with a set of 1509 GO terms with a minimum population of 50 samples when considering the manually annotated SwissProt database apweiler2004uniprot. As the hierarchy in the DAG is fully inferable towards the root term, a classification by leaf terms is fully sufficient for a comprehensive classification, we further reduced the set of considered labels to all leaf nodes in the 1509-terms sub-DAG. Our final set of classes had 886 GO-terms of all levels.

Data preprocessing

In order to convert the protein sequences into a machine readable format we preprocessed the whole UniProt database (release 08/17) as well as the SwissProt database (release 08/17) apweiler2004uniprot. For the classification task of 886 GO-labels we genererated a dataset containing 180774 sequences for the SwissProt and ~7 million sequences for Uniprot database respectively.
During datapreprocessing the full GO-annotation was infered through the DAG from the annotated Go-terms to facilitate the subsequent preprocessing steps. Sequences were then filtered for a minimum length of 175 amino acids and sequences containing non canonical amino acids were excluded.
To ensure a random distribution of sequences over in the validation and train sets for each label and at the same time account for the extreme class in-balance among the GO-terms, the validation set was created on the fly during generation of the training set. This was done by randomly sampling sequences from the preprocessing streams, ensuring the validation set to contain at least 5 sequences per GO-term. Training and validation sets were mutually exclusive.
Prior to training the generated datasets were processed to a binary file format, to speed up the feed streams for the GPUs. Thereby sequences were one-hot encoded and clipped or zero padded to a window of 1000 residues. The labels were also one-hot encoded. In the uniprot dataset the average sequence had 1.3 labels assigned.

Training

The model was trained using the Adagrad optimizer duchi2011adaptive on a Nvidia Tesla K80 GPU for overall 4.8 epochs (600.000 steps) with a mini-batchsize of 64 and an initial learing rate of 0.01. We minimized the cross entropy between sigmoid logits and labels, while for metrics calculations a label was considered predicted with an output value higher than 0.5. During the training process the independent outlayers where updated alternatingly depending on their argmax. The signal for both outlayers was averaged for inference and the variance between the outlayers was considered as a metric for model uncertainty. Upon loss convergence the training was stopped and the model was reinitialized with a factor 10 reduced learining rate with an ultimate learining rate of 0.001.

Results

The performance of the network was asserted on an exclusive validation set of 4425 sequences. For each GO-label the validation set contained at least 5 distinct samples. Our model achieved an area under the curve (AUC) for the reciever operating characteristic (ROC) of 99.8% with an average F1 score of 78% (Figure 3).

Wet Lab Validation

To assert the value of DeeProtein in sequence activtiy evaluation context, we validated the correlation between the DeeProtein classification score and enzyme activity in the wetlab. First we predicted a set of 25 single and double mutant beta-Lactamase variants with both higher and lower scores as the wildtype and subsequently asserted the activity in the wetlab.
In order to derive a measure for enzyme activity we investigated the minimum inhibitory concentration (MIC) of Carbenicillin for all predicted mutants. The MIC was asserted by OD600-measurement in Carbenicillin containing media. As the OD was measuren in a 96-well plate the values are not absolute. From the measurements the MIC-score was calculated as the first Carbenicillin concentration where the OD fell below a threshold of 0.08. Next the classification scores were averaged for each MIC-score and then plotted against the Carbenicilline concentration (Figure 2).

Protein Sequence Embedding

A protein representation first described by Asgari et al is prot2vec asgari2015continuous. The technique originates in the natural language processing and is based on the word2vec model mikolov2013efficient originally deriving vectorized word representations. Applied to proteins a word is defined as a k-mer of 3 amino acid residues. A protein sequence can thus be respresented as the sum over all internal k-mers. Interesting properties have been described in the resulting vectorspace, for example clustering of hydrophobic and hydrophilic k-mers and sequences asgari2015continuous. However there are limitations to the prot2vec model, the most important being the information loss on the sequence order. This has been addressed by application of the continuous bag of words model, with a paragraph embedding kimothi2016distributed. However training is here extremely slow as a proteinsequence itself is embedded in the paragraph context, where a paragraph is a greater set of protein sequences (e.g. SwissProt-DB). Further new protein sequences can not be added to the embedding as the paragraph context may not change.
Thus we intended to find a optimized word2vec approach for fast, reproducible and simple protein sequence embedding. Therefore we applied a word2vec model mikolov2013efficient on kmers of length 3 with a total dimension size of 100. As the quality of the representation estimate scales with the number of training samples we trained our model on the whole UniProt database (Release 8/2017, apweiler2004uniprot), composed of over 87 million sequences, exceeding the training base of 324,018 sequences derived by asgari2015continuous.

Results

We visualized our 100 dimensional embedding through PCA dimensionality reduction as shown in Fig 5. Highlighted in sequence are all kmers containing a certain amino acid. Clear clusters can be observed for the aminoacids Cysteine (top right corner), Lysine (top left corner), Tryptophane (center right), Glutamate (center left), Proline (bottom) and Arginine (center) even after dimensionality reduction. In contrast, for aminoacids like Glycine, Serine and Valine are distributed over the whole kmer space.

Application

Our kmer embedding provides a great base to explore the protein space for future research. The embedding may be applied in classification as demonstrated by asgari2015continuous but also in alternation of existing sequences. By exploiting the intrinsic properties of the vector space, a sequence defined as the path through that vector space (namely by its start and ending point) may be altered by exchanging vector components along that path to similar ones. For instance if the kmer 'AAG' and 'GAG' cluster closely in figure 3 and the distance between their embedding vectors is close, out hypothesis is they're exchangeable, without huge perturbations to the sequence because the two kmers are likely in the same sequence context.

Neural Networks 101

1. Neural Network Basics

The idea of neural networks origins in the late 1960 where Rosenblatt et al. first descriped the perceptron as a the functional unit of neural networks rosenblatt1958perceptron and subsequently their application in mulitlayered perceptrons (MLPs) rosenblatt1961principles. Most neural networks are feedforward neural networks where the information flows from the input layer through the networks until the last layer generates an output.
In general a neural network can be seen as a function approximator mapping an input to an output function, in terms of a classifier for instance:
$$y = f(x)$$ In contrast to a "handdesigned" classification algorithm, a neural network learns the parameters \(\theta\) required for a successfull classification: $$y = f(x, \theta)$$ And when we consider the chain like structure of neural networks: $$y = f_3(f_2(f_1(x, \theta_1), \theta_2), \theta_3)$$ A single neuron thereby consists of a linear mapping and an non-linear activation function applied after the linear activation: $$z_j = w_{ij} i + b_j$$ $$y_j = a(z_j)$$ In the context of a layer with multiple neurons: $$ z_j = \sum w_{ij} i + b_j$$ $$ y_j = a(z_j)$$ The weights \(w\) and biases \(b\) thereby describe the trainable parameters of the layer.

2. Training - Backpropagation

Like other machine learning models neural networks are trained with gradient based methods Goodfellow-et-al-2016. The most common technique that is applied on artificial neural networks is error backpropagation with stochastic gradient descent (SGD). Here the network is shown small batches of the training data and the trainable parameters are updated after each step with the goal to minimize the error of a cost or loss function. A training cycle consists of a forward and backward pass. In the forward pass the information flows from input through the network, an output is generated and the error is estimated by computing the loss function. Subsequently, during the backward pass, the error is backpropagated through the network and the trainable parameters of the network are updated to minimize the value of the loss function. Technically the gradient of the loss function with respect to the wights and biases is computed and distributed throught the network (see Fig. 6).

The most common cost or loss functions for classification tasks include the mean squared error (MSE) and the cross entropy (CE): $$ MSE = \frac{1}{2}(y - \hat{y})^2 $$ $$ CE = \sum_y y log(\hat{y})$$

3. Fully Connected vs. Convolutional Nets

A convolutional neural network (CNN) is a special neural network architecture first described by Yann LeCun LeCun1990Handwritten. CNNs are specialized on analyzing spatial or time-series data, with a grid like topology. Instead of matrix multiplication, CNNs apply a mathematical operation called convolution in at least one of their layers Goodfellow-et-al-2016. Inspired by the structure of neurons on the visual cortex of mammals CNNs, have achieved tremendous results on various tasks like image, video, sound and language processing.
A convolution is defined as: $$ s(t)=\int x(a)w(t-a) da $$ Where the input function \(x()\) is smoothed by the weighting function (or Kernel) \(w\), leading to the output \(s\). Typically a convolution is denoted with an asterix: $$ s(t)=(x \ast w)(t) $$ Applied on a two dimenstional discrete input, a convolution is described as: $$ S(i,j) = \sum_m \sum_n I(i-m,j-n)K(m,n) $$
A ConvNet in abstract representation could look like this:

$$Input \rightarrow Conv \rightarrow Conv2 \rightarrow Conv3 \rightarrow Fully Connected \rightarrow Fully Connected \rightarrow Output$$

The information is propagated from the input throught the convolutional layers and fully connected layers to generate an output. By repeated application of convolutions and different kernel sizes the size of the image is reduced as the information proceedes through the network. The kernels are thereby modular, and optimized during the training process. The result is a trainable feature extractor (Conv1-Conv3) that can be examined by a small fully connected neural network. Another advantage is that CNNs rely on parameter sharing. As a kernel is often much smaller as the image it is applied on, and a kernel is applied on the whole image, the size of parameters that needs to be restored is greatly reduced compared to conventional neural networks that rely on general matrix multiplication Goodfellow-et-al-2016.

More Ressources

We provide a brief overview on what a neural network does in our neuralnetworks101. For a comprehensice explanation please rely on these ressources:
- colah's Blog provides decent explanations of virtually all neural network architectures
- Adit Deshpande's Blog is a very good resource for introductory explanations and overviews on applications
- The Deep Learning Book is a comprehensive work by three of the fields most prominent figures

Practical tips are given in these resources:
- Andrew Ng's tips on practical implementation
- Comprehensive TensoFlow tutorials by the Hvass labs