the pooled output) e.g. Uncased means that the text has been lowercased before WordPiece tokenization, e.g., John Smith becomes john smith. intermediate_size (int, optional, defaults to 3072) Dimensionality of the intermediate (i.e., feed-forward) layer in the Transformer encoder. encoder_hidden_states is expected as an input to the forward pass. start_positions (tf.Tensor of shape (batch_size,), optional, defaults to None) Labels for position (index) of the start of the labelled span for computing the token classification loss. head_mask (torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None) Mask to nullify selected heads of the self-attention modules. a language modeling head with weights tied to the input embeddings (no additional parameters) and: a multiple choice classifier (linear layer that take as input a hidden state in a sequence to compute a score, see details in paper). def load_model (self, model_path: str, do_lower_case=False): config = BertConfig.from_pretrained (model_path + "/bert_config.json") tokenizer = BertTokenizer.from_pretrained ( model_path, do_lower_case=do_lower_case) model = BertForQuestionAnswering.from_pretrained ( model_path, from_tf=False, config=config) return model, tokenizer should refer to the superclass for more information regarding methods. Check out the from_pretrained() method to load the model weights. Rouge . the warmup and t_total arguments on the optimizer are ignored and the ones in the _LRSchedule object are used. We detail them here. This model is a tf.keras.Model sub-class. Word2Vecword2vecword2vec word2vec . learning, if the model is configured as a decoder. Mask values selected in [0, 1]: the hidden-states output to compute span start logits and span end logits). in the first positional argument : a single Tensor with input_ids only and nothing else: model(inputs_ids), a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) Labels for computing the sequence classification/regression loss. This should improve model performance, if the language style is different from the original BERT training corpus (Wiki + BookCorpus). clean_text (bool, optional, defaults to True) Whether to clean the text before tokenization by removing any control characters and Some of these results are significantly different from the ones reported on the test set tokenize_chinese_chars (bool, optional, defaults to True) Whether to tokenize Chinese characters. Copy PIP instructions, PyTorch version of Google AI BERT model with script to load Google pre-trained models, View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery, License: Apache Software License (Apache), Author: Thomas Wolf, Victor Sanh, Tim Rault, Google AI Language Team Authors, Open AI team Authors, Tags the BERT bert-base-uncased architecture. BERTconfig BERTBertConfigconfigBERT config https://huggingface.co/transformers/model_doc/bert.html#bertconfig tokenizerALBERTBERT At the moment, I initialised the model as below: from transformers import BertForMaskedLM model = BertForMaskedLM(config=config) However, it would just be for MLM and not NSP. do_basic_tokenize (bool, optional, defaults to True) Whether to do basic tokenization before WordPiece. on single tesla V100 16GB with apex installed. An example on how to use this class is given in the run_squad.py script which can be used to fine-tune a token classifier using BERT, for example for the SQuAD task. The rest of the repository only requires PyTorch. tuple(torch.FloatTensor) comprising various elements depending on the configuration (BertConfig) and inputs. How to use the transformers.GPT2Tokenizer function in transformers To help you get started, we've selected a few transformers examples, based on popular ways it is used in public projects. The pretrained model now acts as a language model and is meant to be fine-tuned on a downstream task. Use it as a regular TF 2.0 Keras Model and Inputs are the same as the inputs of the OpenAIGPTModel class plus optional labels: OpenAIGPTDoubleHeadsModel includes the OpenAIGPTModel Transformer followed by two heads: Inputs are the same as the inputs of the OpenAIGPTModel class plus a classification mask and two optional labels: The Transformer-XL model is described in "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context". model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids]), a dictionary with one or several input Tensors associated to the input names given in the docstring: unk_token (string, optional, defaults to [UNK]) The unknown token. cvnlp384384 . This is the configuration class to store the configuration of a BertModel . . Here is an example of the conversion process for a pre-trained OpenAI GPT model, assuming that your NumPy checkpoint save as the same format than OpenAI pretrained model (see here), Here is an example of the conversion process for a pre-trained Transformer-XL model (see here). Wonderful project @emillykkejensen and appreciate the ease of explanation. # Here is how to do it in this situation: Thomas Wolf, Victor Sanh, Tim Rault, Google AI Language Team Authors, Open AI team Authors, Scientific/Engineering :: Artificial Intelligence, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Improving Language Understanding by Generative Pre-Training, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, Language Models are Unsupervised Multitask Learners, Training large models: introduction, tools and examples, Fine-tuning with BERT: running the examples, Fine-tuning with OpenAI GPT, Transformer-XL and GPT-2, the tips on training large batches in PyTorch, the relevant PR of the present repository, the original implementation hyper-parameters, the pre-trained models released by Google, pytorch_pretrained_bert-0.6.2-py3-none-any.whl, pytorch_pretrained_bert-0.6.2-py2-none-any.whl, Detailed examples on how to fine-tune Bert, Introduction on the provided Jupyter Notebooks, Notes on TPU support and pretraining scripts, Convert a TensorFlow checkpoint in a PyTorch dump, How to load Google AI/OpenAI's pre-trained weight or a PyTorch saved instance, How to save and reload a fine-tuned model, API of the configuration classes for BERT, GPT, GPT-2 and Transformer-XL, API of the PyTorch model classes for BERT, GPT, GPT-2 and Transformer-XL, API of the tokenizers class for BERT, GPT, GPT-2 and Transformer-XL, How to use gradient-accumulation, multi-gpu training, distributed training, optimize on CPU and 16-bits training to train Bert models, the model it-self which should be saved following PyTorch serialization, the configuration file of the model which is saved as a JSON file, and. modeling_gpt2.py. 2023 Python Software Foundation Secure your code as it's written. First let's prepare a tokenized input with OpenAIGPTTokenizer, Let's see how to use OpenAIGPTModel to get hidden states. This model is a tf.keras.Model sub-class. Please refer to the doc strings and code in tokenization.py for the details of the BasicTokenizer and WordpieceTokenizer classes. for RocStories/SWAG tasks. The BertForMaskedLM forward method, overrides the __call__() special method. token instead. An overview of the implemented schedules: BERT-base and BERT-large are respectively 110M and 340M parameters models and it can be difficult to fine-tune them on a single GPU with the recommended batch size for good performance (in most case a batch size of 32). config=BertConfig.from_pretrained(TO_FINETUNE, num_labels=num_labels) tokenizer=BertTokenizer.from_pretrained(TO_FINETUNE) defconvert_examples_to_tf_dataset( examples: List[Tuple[str, int]], tokenizer, max_length=512, Loads data into a tf.data.Dataset for finetuning a given model. How to use the transformers.BertTokenizer.from_pretrained function in transformers To help you get started, we've selected a few transformers examples, based on popular ways it is used in public projects. A torch module mapping hidden states to vocabulary. The Linear Indices of input sequence tokens in the vocabulary. You can download an exemplary training corpus generated from wikipedia articles and splitted into ~500k sentences with spaCy. Positions are clamped to the length of the sequence (sequence_length). OpenAIAdam accepts the same arguments as BertAdam. Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of This example code fine-tunes BERT on the Microsoft Research Paraphrase input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) , attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) , token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) , position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional, defaults to None) , labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) Labels for computing the multiple choice classification loss. OpenAIGPTLMHeadModel includes the OpenAIGPTModel Transformer followed by a language modeling head with weights tied to the input embeddings (no additional parameters). from_pretrained ("bert-base-cased", num_labels = 3) model = BertForSequenceClassification. TransfoXLTokenizer perform word tokenization. Again module does not support Python 2! input_ids (torch.LongTensor of shape (batch_size, sequence_length)) . Input should be a sequence pair (see input_ids docstring) all systems operational. the [CLS] token. This is useful if you want more control over how to convert input_ids indices into associated vectors Getting Started Text Classification Example There are two differences between the shapes of new_mems and last_hidden_state: new_mems have transposed first dimensions and are longer (of size self.config.mem_len). Tokenizer Transformer Split, word, subword, symbol => token token integer AutoTokenizer class pretrained tokenizer Default: distilbert-base-uncased-finetuned-sst-2-english in sentiment-analysis This mask The TFBertForMultipleChoice forward method, overrides the __call__() special method. for a wide range of tasks, such as question answering and language inference, without substantial task-specific Three notebooks that were used to check that the TensorFlow and PyTorch models behave identically (in the notebooks folder): These notebooks are detailed in the Notebooks section of this readme. Special tokens need to be trained during the fine-tuning if you use them. It is also used as the last token of a sequence built with special tokens. BertForMaskedLM includes the BertModel Transformer followed by the (possibly) pre-trained masked language modeling head. OpenAI GPT-2 was released together with the paper Language Models are Unsupervised Multitask Learners by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**. This tokenizer can be used for adaptive softmax and has utilities for counting tokens in a corpus to create a vocabulary ordered by toekn frequency (for adaptive softmax). PyTorch pretrained bert can be installed by pip as follows: If you want to reproduce the original tokenization process of the OpenAI GPT paper, you will need to install ftfy (limit to version 4.4.3 if you are using Python 2) and SpaCy : If you don't install ftfy and SpaCy, the OpenAI GPT tokenizer will default to tokenize using BERT's BasicTokenizer followed by Byte-Pair Encoding (which should be fine for most usage, don't worry). pretrained_model_config 1 . We showcase several fine-tuning examples based on (and extended from) the original implementation: We get the following results on the dev set of GLUE benchmark with an uncased BERT base for more information. end_positions (tf.Tensor of shape (batch_size,), optional, defaults to None) Labels for position (index) of the end of the labelled span for computing the token classification loss. Training one epoch on this corpus takes about 1:20h on 4 x NVIDIA Tesla P100 with train_batch_size=200 and max_seq_length=128: Thank to the work of @Rocketknight1 and @tholor there are now several scripts that can be used to fine-tune BERT using the pretraining objective (combination of masked-language modeling and next sentence prediction loss). py2, Status: Some features may not work without JavaScript. BertForPreTraining includes the BertModel Transformer followed by the two pre-training heads: Inputs comprises the inputs of the BertModel class plus two optional labels: if masked_lm_labels and next_sentence_label are not None: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss. ~91 F1 on SQuAD for BERT, ~88 F1 on RocStories for OpenAI GPT and ~18.3 perplexity on WikiText 103 for the Transformer-XL). This output is usually not a good summary This CLI takes as input a TensorFlow checkpoint (three files starting with bert_model.ckpt) and the associated configuration file (bert_config.json), and creates a PyTorch model for this configuration, loads the weights from the TensorFlow checkpoint in the PyTorch model and saves the resulting model in a standard PyTorch save file that can be imported using torch.load() (see examples in extract_features.py, run_classifier.py and run_squad.py). Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately. encoder_hidden_states (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) Sequence of hidden-states at the output of the last layer of the encoder. Creates a mask from the two sequences passed to be used in a sequence-pair classification task. The TFBertForTokenClassification forward method, overrides the __call__() special method. For QQP and WNLI, please refer to FAQ #12 on the webite. do_basic_tokenize=True. The original TensorFlow code further comprises two scripts for pre-training BERT: create_pretraining_data.py and run_pretraining.py. from transformers import BertConfig from multimodal_transformers.model import BertWithTabular from multimodal_transformers.model import TabularConfig bert_config = BertConfig.from_pretrained('bert-base-uncased') tabular_config = TabularConfig( combine_feat_method='attention_on_cat_and_numerical_feats', # change this to specify the method of GPT2Tokenizer perform byte-level Byte-Pair-Encoding (BPE) tokenization. A token that is not in the vocabulary cannot be converted to an ID and is set to be this from transformers import BertForSequenceClassification, AdamW, BertConfig, BertModel model = BertForSequenceClassification.from_pretrained ( "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab. BertConfig config = BertConfig. The BertForNextSentencePrediction forward method, overrides the __call__() special method. See the doc section below for all the details on these classes. The same option as in the original scripts are provided, please refere to the code of the example and the original repository of OpenAI. The BertForQuestionAnswering forward method, overrides the __call__() special method. If you choose this second option, there are three possibilities you can use to gather all the input Tensors A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. inputs_embeds (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional, defaults to None) Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. Indices should be in [0, , num_choices-1] where num_choices is the size of the second dimension Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately. Mask values selected in [0, 1]: Instead, if you saved using the save_pretrained method, then the directory already should have a config.json specifying the shape of the model, . 1 indicates the head is not masked, 0 indicates the head is masked. BertForQuestionAnswering is a fine-tuning model that includes BertModel with a token-level classifiers on top of the full sequence of last hidden states. It becomes increasingly difficult to ensure . hidden_dropout_prob (float, optional, defaults to 0.1) The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. While running the model on my PC on python shell i always get the error : _OSError: Can't load weights for 'EleutherAI/gpt-neo-125M'. Please follow the instructions given in the notebooks to run and modify them. The respective configuration classes are: These configuration classes contains a few utilities to load and save configurations: BertModel is the basic BERT Transformer model with a layer of summed token, position and sequence embeddings followed by a series of identical self-attention blocks (12 for BERT-base, 24 for BERT-large). See the adaptive softmax paper (Efficient softmax approximation for GPUs) for more details. This model takes as inputs: Training with the previous hyper-parameters gave us the following results: The data for SWAG can be downloaded by cloning the following repository. This command runs in about 1 min on a V100 and gives an evaluation perplexity of 18.22 on WikiText-103 (the authors report a perplexity of about 18.3 on this dataset with the TensorFlow code). max_position_embeddings (int, optional, defaults to 512) The maximum sequence length that this model might ever be used with. Text preprocessing is the end-to-end transformation of raw text into a model's integer inputs. to control the model outputs. Here is how to extract the full list of hidden states from the model output: TransfoXLLMHeadModel includes the TransfoXLModel Transformer followed by an (adaptive) softmax head with weights tied to the input embeddings. Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). vocab_path (str) The directory in which to save the vocabulary. this script refer to the TF 2.0 documentation for all matter related to general usage and behavior. Then run. You can find more details in the Examples section below. in [0, , config.vocab_size]. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. A command-line interface to convert TensorFlow checkpoints (BERT, Transformer-XL) or NumPy checkpoint (OpenAI) in a PyTorch save of the associated PyTorch model: This CLI is detailed in the Command-line interface section of this readme. Indices should be in [0, , config.num_labels - 1]. for RocStories/SWAG tasks. This option is useful in particular when you are using distributed training: to avoid concurrent access to the same weights you can set for example cache_dir='./pretrained_model_{}'.format(args.local_rank) (see the section on distributed training for more information). of the input tensors. Retrieves sequence ids from a token list that has no special tokens added. is used in the cross-attention if the model is configured as a decoder. See the doc section below for all the details on these classes. architecture modifications. a masked language modeling head and a next sentence prediction (classification) head. I do have a quick question, since we have multi-label and multi-class problem to deal with here, there is a probability that between issue and product labels above, there could be some where we do not have the same # of samples from target / output layers. OpenAIGPTModel is the basic OpenAI GPT Transformer model with a layer of summed token and position embeddings followed by a series of 12 identical self-attention blocks. tokenize_chinese_chars Whether to tokenize Chinese characters. (see input_ids above). Before running anyone of these GLUE tasks you should download the Developed and maintained by the Python community, for the Python community. The BertForSequenceClassification forward method, overrides the __call__() special method. source, Uploaded Use it as a regular TF 2.0 Keras Model and Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general BERT is a model with absolute position embeddings so its usually advised to pad the inputs on having all inputs as a list, tuple or dict in the first positional arguments. from transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') Unlike the BERT Models, you don't have to download a different tokenizer for each different type of model. Secure your code as it's written. replacing all whitespaces by the classic one. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Here is a detailed documentation of the classes in the package and how to use them: To load one of Google AI's, OpenAI's pre-trained models or a PyTorch saved model (an instance of BertForPreTraining saved with torch.save()), the PyTorch model classes and the tokenizer can be instantiated as, BERT_CLASS is either a tokenizer to load the vocabulary (BertTokenizer or OpenAIGPTTokenizer classes) or one of the eight BERT or three OpenAI GPT PyTorch model classes (to load the pre-trained weights): BertModel, BertForMaskedLM, BertForNextSentencePrediction, BertForPreTraining, BertForSequenceClassification, BertForTokenClassification, BertForMultipleChoice, BertForQuestionAnswering, OpenAIGPTModel, OpenAIGPTLMHeadModel or OpenAIGPTDoubleHeadsModel, and. improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement). OpenAIGPTTokenizer perform Byte-Pair-Encoding (BPE) tokenization. of shape (batch_size, sequence_length, hidden_size). corresponds to a sentence B token, position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) . Indices should be in [0, , num_choices] where num_choices is the size of the second dimension Thus it can now be fine-tuned on any downstream task like Question Answering, Text . $ pip install band -U Note that the code MUST be running on Python >= 3.6. do_lower_case (bool, optional, defaults to True) Whether to lowercase the input when tokenizing. Special tokens embeddings are additional tokens that are not pre-trained: [SEP], [CLS] How to use the transformers.BertConfig.from_pretrained function in transformers To help you get started, we've selected a few transformers examples, based on popular ways it is used in public projects. from_pretrained ('bert-base-uncased') self. the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, A torch module mapping vocabulary to hidden states. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. It is the first token of the sequence when built with can be represented by the inputs_ids passed to the forward method of BertModel. [SEP] Jim Henson was a puppeteer [SEP]", # Mask a token that we will try to predict back with `BertForMaskedLM`, # Define sentence A and B indices associated to 1st and 2nd sentences (see paper), # If you have a GPU, put everything on cuda, # Predict hidden states features for each layer, # We have a hidden states for each of the 12 layers in model bert-base-uncased, # confirm we were able to predict 'henson', "Who was Jim Henson ? pad_token (string, optional, defaults to [PAD]) The token used for padding, for example when batching sequences of different lengths. Hidden-states of the model at the output of each layer plus the initial embedding outputs. configuration = BertConfig.from_json_file ('./biobert/biobert_v1.1_pubmed/bert_config.json') model = BertModel.from_pretrained ("./biobert/pytorch_model.bin", config=configuration) model.eval. Transformer XL use a relative positioning with sinusiodal patterns and adaptive softmax inputs which means that: This model takes as inputs: class MixModel(nn.Module): def __init__(self,pre_trained='bert-base-uncased'): super().__init__() config = BertConfig.from_pretrained('bert-base-uncased', output . Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general Based on WordPiece. language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI 1 indicates the head is not masked, 0 indicates the head is masked. The options we list above allow to fine-tune BERT-large rather easily on GPU(s) instead of the TPU used by the original implementation. The linear layer outputs a single value for each choice of a multiple choice problem, then all the outputs corresponding to an instance are passed through a softmax to get the model choice. layer weights are trained from the next sentence prediction (classification) total_tokens_embeddings = config.vocab_size + config.n_special For information about the Multilingual and Chinese model, see the Multilingual README or the original TensorFlow repository. Bert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear input_processing from transformers.modeling_tf_outputs import TFQuestionAnsweringModelOutput from transformers import BertConfig class MY_TFBertForQuestionAnswering . for Named-Entity-Recognition (NER) tasks. Instantiating a configuration with the defaults will yield a similar configuration to that of the BERT bert-base-uncased architecture. Indices of positions of each input sequence tokens in the position embeddings. BERT, GLUE data by running Contribute to AUTOMATIC1111/stable-diffusion-webui development by creating an account on GitHub. This is useful if you want more control over how to convert input_ids indices into associated vectors for Named-Entity-Recognition (NER) tasks. If config.num_labels > 1 a classification loss is computed (Cross-Entropy). BertConfigPretrainedConfigclassmethod modeling_utils.py109 BertModel config = BertConfig.from_pretrained('bert-base-uncased') The BertForTokenClassification forward method, overrides the __call__() special method. Position outside of the sequence are not taken into account for computing the loss. Last layer hidden-state of the first token of the sequence (classification token) How to use the transformers.BertConfig function in transformers To help you get started, we've selected a few transformers examples, based on popular ways it is used in public projects. The abstract from the paper is the following: We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations Use it as a regular TF 2.0 Keras Model and 1 for tokens that are NOT MASKED, 0 for MASKED tokens. the sequence of hidden-states for the whole input sequence. Here is an example of the conversion process for a pre-trained BERT-Base Uncased model: You can download Google's pre-trained models for the conversion here. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general (batch_size, num_heads, sequence_length, sequence_length). Use it as a regular TF 2.0 Keras Model and You can convert any TensorFlow checkpoint for BERT (in particular the pre-trained models released by Google) in a PyTorch save file by using the convert_tf_checkpoint_to_pytorch.py script. However, averaging over the sequence may yield better results than using Bert Model with a next sentence prediction (classification) head on top. usage and behavior. refer to the TF 2.0 documentation for all matter related to general usage and behavior. sequence(s). We detail them here. end_positions (torch.LongTensor of shape (batch_size,), optional, defaults to None) Labels for position (index) of the end of the labelled span for computing the token classification loss. A series of tests is included in the tests folder and can be run using pytest (install pytest if needed: pip install pytest). usage and behavior. This model is a tf.keras.Model sub-class. In the given example, we get a standard deviation of 1.5e-7 to 9e-7 on the various hidden state of the models. model. This example code fine-tunes BERT on the SQuAD dataset. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general Models trained with a causal language transformer_model = TFBertModel.from_pretrained (model_name, config = config) Here we first load a BERT config object that controls the model, tokenizer and so on. from_pretrained . This model is a tf.keras.Model sub-class. tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) Jim Henson was a puppeteer", # Load pre-trained model tokenizer (vocabulary from wikitext 103), # We can re-use the memory cells in a subsequent call to attend a longer context, # past can be used to reuse precomputed hidden state in a subsequent predictions. config = BertConfig. num_hidden_layers (int, optional, defaults to 12) Number of hidden layers in the Transformer encoder. Before running this example you should download the (see input_ids above). OpenAI GPT was released together with the paper Improving Language Understanding by Generative Pre-Training by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. This model is a PyTorch torch.nn.Module sub-class. A BERT sequence has the following format: token_ids_0 (List[int]) List of IDs to which the special tokens will be added.
