Qwen7b微调保姆级教程

前方干货预警：这可能是你能够找到的， 最容易理解，最容易跑通的 ， 适用于各种开源LLM模型的 ， 同时支持多轮和单轮对话数据集 的大模型高效微调范例。

我们构造了一个修改大模型自我认知的3轮对话的玩具数据集，使用QLoRA算法，只需要 5分钟的训练时间 ，就可以完成微调，并成功修改了LLM模型的自我认知(以 Qwen7b-Chat 为例)。

公众号 算法美食屋 后台回复关键词： torchkeras ，可获取本文notebook源码~

通过借鉴FastChat对各种开源LLM模型进行数据预处理方法统一管理的方法，因此本范例适用于非常多不同的开源LLM模型，包括 Qwen-7b-Chat，Llama-13b-chat, BaiChuan2-13b-chat, Intern-7b-chat, ChatGLM2-6b-chat 以及其它许许多多FastChat支持的模型。

在多轮对话模式下,我们按照如下格式构造包括多轮对话中所有机器人回复内容的标签。

(注：llm.build_inputs_labels(messages,multi_rounds=True) 时采用)

        
          
  
inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>  
labels = <-100> <assistant1> <-100> <assistant2> <-100> <assistant3>  
  

      

在单轮对话模式下，我们仅将最后一轮机器人的回复作为要学习的标签。

(注：llm.build_inputs_labels(messages,multi_rounds=False)时采用)

        
          
inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>  
labels = <-100> <-100> <-100> <-100> <-100> <assistant3>  
  

      

〇，预训练模型

        
          
import warnings  
warnings.filterwarnings('ignore')  

      

        
          
import torch  
from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig  
from transformers.generation.utils import GenerationConfig  
import torch.nn as nn  
  
  
#使用QLoRA引入的 NF4量化数据类型以节约显存  
model_name_or_path ='qwen\_7b'  #远程：'Qwen/Qwen-7b-Chat'  
  
bnb_config=BitsAndBytesConfig(  
            load_in_4bit=True,  
            bnb_4bit_compute_dtype=torch.float16,  
            bnb_4bit_use_double_quant=True,  
            bnb_4bit_quant_type="nf4",  
            llm_int8_threshold=6.0,  
            llm_int8_has_fp16_weight=False,  
        )  
  
tokenizer = AutoTokenizer.from_pretrained(  
   model_name_or_path, trust_remote_code=True)  
  
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,  
                quantization_config=bnb_config,  
                trust_remote_code=True)   
  
model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)  
  

      

微调前输出如下：

一，准备数据

下面我设计了一个改变LLM自我认知的玩具数据集，这个数据集有三轮对话。

第一轮问题是 who are you?

第二轮问题是 where are you from?

第三轮问题是 what can you do?

差不多是哲学三问吧：你是谁？你从哪里来？你要到哪里去?

通过这三个问题，我们希望初步地改变 大模型的自我认知。

在提问的方式上，我们稍微作了一些数据增强。

所以，总共是有 27个样本。

1，导入样本

        
          
who_are_you = ['请介绍一下你自己。','你是谁呀？','你是？',]  
i_am = ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']  
where_you_from = ['你多大了？','你是谁开发的呀？','你从哪里来呀']  
i_from = ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']  
what_you_can = ['你能干什么','你有什么作用呀？','你能帮助我干什么']  
i_can = ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。']  
  
conversation = [(who_are_you,i_am),(where_you_from,i_from),(what_you_can,i_can)]  
print(conversation)  

      

        
          
import random  
def get\_messages(conversation):  
    select = random.choice  
    messages,history = [],[]  
    for t in conversation:  
        history.append((select(t[0]),select(t[-1])))  
          
    for prompt,response in history:  
        pair = [{"role": "user", "content": prompt},  
            {"role": "assistant", "content": response}]  
        messages.extend(pair)  
    return messages   
  

      

2，做数据集

        
          
from torch.utils.data import Dataset,DataLoader   
from copy import deepcopy  
class MyDataset(Dataset):  
    def \_\_init\_\_(self,conv,size=8  
                ):  
        self.conv = conv  
        self.index_list = list(range(size))  
        self.size = size   
          
    def \_\_len\_\_(self):  
        return self.size  
          
    def get(self,index):  
        idx = self.index_list[index]  
        messages = get_messages(self.conv)  
        return messages  
  
      
    def \_\_getitem\_\_(self,index):  
        messages = self.get(index)  
        input_ids, labels = llm.build_inputs_labels(messages,multi_rounds=True) #支持多轮  
        return {'input\_ids':input_ids,'labels':labels}  
      

      

        
          
ds_train = ds_val = MyDataset(conversation)  

      

3，创建管道

        
          
#如果pad\_token\_id为None，需要使用unk\_token\_id或eos\_token\_id代替  
if tokenizer.pad_token_id is None:  
    tokenizer.pad_token_id = tokenizer.unk_token_id if tokenizer.unk_token_id is not None else tokenizer.eos_token_id  
      
  
def data\_collator(examples: list):  
      
    len_ids = [len(example["input\_ids"]) for example in examples]  
    longest = max(len_ids) #之后按照batch中最长的input\_ids进行padding  
      
    input_ids = []  
    labels_list = []  
      
    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):  
        ids = example["input\_ids"]  
        labs = example["labels"]  
          
        ids = ids + [tokenizer.pad_token_id] * (longest - length)  
        labs = labs + [-100] * (longest - length)  
          
        input_ids.append(torch.LongTensor(ids))  
        labels_list.append(torch.LongTensor(labs))  
            
    input_ids = torch.stack(input_ids)  
    labels = torch.stack(labels_list)  
    return {  
        "input\_ids": input_ids,  
        "labels": labels,  
    }  
  

      

        
          
import torch   
dl_train = torch.utils.data.DataLoader(ds_train,batch_size=2,  
                                       pin_memory=True,shuffle=False,  
                                       collate_fn = data_collator)  
  
dl_val = torch.utils.data.DataLoader(ds_val,batch_size=2,  
                                    pin_memory=True,shuffle=False,  
                                     collate_fn = data_collator)  
  

      

二，定义模型

下面我们将使用QLoRA(实际上用的是量化的AdaLoRA）算法来微调Baichuan-13b模型。

        
          
from peft import get_peft_config, get_peft_model, TaskType  
model.supports_gradient_checkpointing = True  #  
model.gradient_checkpointing_enable()  
model.enable_input_require_grads()  
  
model.config.use_cache = False  # silence the warnings. Please re-enable for inference!  
  

      

        
          
import bitsandbytes as bnb   
def find\_all\_linear\_names(model):  
    """  
    找出所有全连接层，为所有全连接添加adapter  
    """  
    cls = bnb.nn.Linear4bit  
    lora_module_names = set()  
    for name, module in model.named_modules():  
        if isinstance(module, cls):  
            names = name.split('.')  
            lora_module_names.add(names[0] if len(names) == 1 else names[-1])  
  
    if 'lm\_head' in lora_module_names:  # needed for 16-bit  
        lora_module_names.remove('lm\_head')  
    return list(lora_module_names)  
  

      

        
          
from peft import prepare_model_for_kbit_training   
model = prepare_model_for_kbit_training(model)  
  

      

        
          
lora_modules = find_all_linear_names(model)  
print(lora_modules)   
  

      

        
          
from peft import AdaLoraConfig  
peft_config = AdaLoraConfig(  
    task_type=TaskType.CAUSAL_LM, inference_mode=False,  
    r=16,  
    lora_alpha=16, lora_dropout=0.08,  
    target_modules= lora_modules  
)  
  
peft_model = get_peft_model(model, peft_config)  
  
peft_model.is_parallelizable = True  
peft_model.model_parallel = True  
peft_model.print_trainable_parameters()  
  

      

trainable params: 26,838,912 || all params: 7,748,163,616 || trainable%: 0.34639062015388394

三，训练模型

        
          
from torchkeras import KerasModel   
from accelerate import Accelerator   
  
class StepRunner:  
    def \_\_init\_\_(self, net, loss\_fn, accelerator=None, stage = "train", metrics\_dict = None,   
                 optimizer = None, lr\_scheduler = None  
                 ):  
        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage  
        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler  
        self.accelerator = accelerator if accelerator is not None else Accelerator()   
        if self.stage=='train':  
            self.net.train()   
        else:  
            self.net.eval()  
      
    def \_\_call\_\_(self, batch):  
          
        #loss  
        with self.accelerator.autocast():  
            loss = self.net.forward(**batch)[0]  
  
        #backward()  
        if self.optimizer is not None and self.stage=="train":  
            self.accelerator.backward(loss)  
            if self.accelerator.sync_gradients:  
                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)  
            self.optimizer.step()  
            if self.lr_scheduler is not None:  
                self.lr_scheduler.step()  
            self.optimizer.zero_grad()  
              
        all_loss = self.accelerator.gather(loss).sum()  
          
        #losses (or plain metrics that can be averaged)  
        step_losses = {self.stage+"\_loss":all_loss.item()}  
          
        #metrics (stateful metrics)  
        step_metrics = {}  
          
        if self.stage=="train":  
            if self.optimizer is not None:  
                step_metrics['lr'] = self.optimizer.state_dict()['param\_groups'][0]['lr']  
            else:  
                step_metrics['lr'] = 0.0  
        return step_losses,step_metrics  
      
KerasModel.StepRunner = StepRunner   
  
#仅仅保存QLora可训练参数  
def save\_ckpt(self, ckpt\_path='checkpoint', accelerator = None):  
    unwrap_net = accelerator.unwrap_model(self.net)  
    unwrap_net.save_pretrained(ckpt_path)  
      
def load\_ckpt(self, ckpt\_path='checkpoint'):  
    import os  
    self.net.load_state_dict(  
        torch.load(os.path.join(ckpt_path,'adapter\_model.bin')),strict =False)  
    self.from_scratch = False  
      
KerasModel.save_ckpt = save_ckpt   
KerasModel.load_ckpt = load_ckpt   
  

      

        
          
optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),  
                                  lr=6e-03,is_paged=True)  #'paged\_adamw'  
keras_model = KerasModel(peft_model,loss_fn =None,  
        optimizer=optimizer)   
  
ckpt_path = 'qwen7b\_multirounds'  
  
  

      

        
          
keras_model.fit(train_data = dl_train,  
                val_data = dl_val,  
                epochs=100,patience=15,  
                monitor='val\_loss',mode='min',  
                ckpt_path = ckpt_path  
               )  

      

四，保存模型

为减少GPU压力，此处可重启kernel释放显存

        
          
import warnings   
warnings.filterwarnings('ignore')  

      

        
          
import torch  
from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig  
from transformers.generation.utils import GenerationConfig  
import torch.nn as nn  
#使用QLoRA引入的 NF4量化数据类型以节约显存  
model_name_or_path ='qwen\_7b'  
ckpt_path = 'qwen7b\_multirounds'  
  
  
  
tokenizer = AutoTokenizer.from_pretrained(  
   model_name_or_path, trust_remote_code=True)  
  
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,  
                trust_remote_code=True)   
  
model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)  
  

      

        
          
from peft import PeftModel  
  
#可能需要5分钟左右  
peft_model = PeftModel.from_pretrained(model, ckpt_path)  
model_new = peft_model.merge_and_unload()  
  

      

        
          
from transformers.generation.utils import GenerationConfig  
model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)  
  

      

        
          
save_path = 'qwen\_torchkeras'  

      

        
          
tokenizer.save_pretrained(save_path)  
model_new.save_pretrained(save_path)  

      

        
          
!cp qwen_7b/*.py  qwen_torchkeras/  

      

五，使用模型

为减少GPU压力，此处可再次重启kernel释放显存。

        
          
  
import warnings  
warnings.filterwarnings('ignore')  
  

      

        
          
import torch  
from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig  
from transformers.generation.utils import GenerationConfig  
import torch.nn as nn  
  
model_name_or_path =  'qwen\_torchkeras'  
  
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=False, trust_remote_code=True)  
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto",   
                                             torch_dtype=torch.float16, trust_remote_code=True)  
model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)  
  

      

我们测试一下微调后的效果。

非常棒，粗浅的测试表明，我们的多轮对话训练是成功的。已经在Qwen的自我认知中，种下了一颗梦中情炉的种子。😋😋

公众号 算法美食屋 后台回复关键词 ： torchkeras ， 可 获取本文notebook 源码以及更多有趣范例~