Update README.md

README.md

@@ -1,77 +1,114 @@
 ---
 library_name: transformers
-tags: []
+license: mit
+datasets:
+- teknium/OpenHermes-2.5
+- Open-Orca/OpenOrca
+- cognitivecomputations/dolphin
+- LDJnr/Capybara
+- abacusai/SystemChat
 ---
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+Walsh_Instruct-1.7b
 
+## Model Details
 
+- Model Dimension: 2048
+- Hidden Layers: 32
+- Attention Heads: 32
+- Feedforward Dimension: 8192
+- Feedforward Network Type: Conventional MLP with GeLU activation
+- Vocabulary Size: 32000
+- Max Sequence Length: 16K (14-bit absolute positional encoding via Walsh matrix)
+- Weight Initialization: DeepNet, https://arxiv.org/abs/2203.00555
+- Pretraining Datasets: RedPajama-Data-1T, mostly "books" and some Wikipedia.
 
-## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+This is an instruction tuned fork of my "dinalt/walsh-1-7b" model... mostly for fun.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+Hadamard-Walsh 1.7B is an experimental model using a new positional encoder. The encoder represents absolute positions by using a combination of rows from the Hadamard-Walsh matrix (https://en.wikipedia.org/wiki/Hadamard_code). Each row corresponds to a binary digit is the positional code, where the presence of a row codes for a 1 and the absence, a zero. While training, the base offset in the sequence is randomly chosen for each batch. The result is that the model is very proficient at sequences much longer than those seen in training.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+Aside from the unsual positional encoder, the most interesting aspect of this model is the application of DITTO training:
 
-### Model Sources [optional]
+Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation
+https://arxiv.org/abs/2206.02369
 
-<!-- Provide the basic links for the model. -->
+As described in the paper, the procedure is very effective at eliminating sentence level repition. As described in the paper, it also reduces perplexity slightly.
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+I will see about posting the code for running the training and generating a DITTO dataset later, althogh the "ditto-loss" function is already in the model implementation.
 
-## Uses
+- **Developed by:** Jason dinAlt
+- **Model type:** Causal language model. Instruction following. Text generation.
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Model Sources [optional]
 
-### Direct Use
+- **Repository:** https://huggingface.co/dinalt/walsh-1-7b
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+## Uses
 
-[More Information Needed]
+This is a toy instruciton following model. It's occasionally reliable at following directions.
 
-### Downstream Use [optional]
+### Direct Use
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
 [More Information Needed]
 
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+This is an uncensored instruction following model. No attempt has been made to make the model "safe." It may offend your sensibilities.
+It will likely provide inaccurate information. Use at your own risk. Whatever you do, don't put it in charge of the global defense grid!
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
+The easiest way to get started with the model is to use text-generation-webui, which needs to be started with the "--trust-remote-code" flag.
+
+https://github.com/oobabooga/text-generation-webui
+
+It appears to work best with the "Big O" and "Simple-1" generation presets.
+
+### Prompt Format
+As an instruction model, the model has been trained to use the ChatML instruction format:
+```
+<|im_start|>system 
+Provide some context and/or instructions to the model.
+<|im_end|> 
+<|im_start|>user 
+The user’s message goes here
+<|im_end|> 
+<|im_start|>assistant 
+```
+For details, see: https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/ai-services/openai/includes/chat-markup-language.md#chatml
+
+### Loading:
+The model implementation is all my own, so you will need to use "trust_remote_code" to load the model.
+
+```
+from transformers import (
+AutoTokenizer,
+AutoModelForCausalLM,
+)
+
+model_id = "dinalt/walsh-1-7b"
+model = AutoModelForCausalLM.from_pretrained(
+model_id,
+trust_remote_code=True,
+# flash_attention_2 requires bfloat16 or float16
+torch_dtype=torch.bfloat16,
+# One of ["flash_attention_2", "sdpa", "eager"]
+attn_implementation="flash_attention_2",
+)
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+```
+
+For batch instruction generation, see my example code here:
+https://discuss.huggingface.co/t/implimentation-of-stopping-criteria-list/20040/16?u=dinalt
 
 ## Training Details
 
@@ -140,15 +177,7 @@ Use the code below to get started with the model.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+It keeps my house warm in the winter...
 
 ## Technical Specifications [optional]
 
@@ -162,23 +191,12 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 #### Hardware
 
-[More Information Needed]
+6 x RTX4090
 
 #### Software
 
 [More Information Needed]
 
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
 
 ## Glossary [optional]
 
@@ -196,6 +214,4 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 ## Model Card Contact
 
-[More Information Needed]
-
-
+[More Information Needed]