Notebooks
G
Google Gemini
[Gemma 2]Custom Vocabulary

[Gemma 2]Custom Vocabulary

recurrentgemmagemmapaligemmagemma-cookbookcodegemma
Copyright 2024 Google LLC.
[ ]

Gemma - How to use a custom vocabulary

This notebook demonstrates how to use a custom vocabulary in Gemma.

Consider a document containing a length word, such as "MyCustomWordInMyDocument" with a high frequency of occurrence. It will be fragmented into several tokens, resulting in inefficiencies. Gemma Tokenizer offers a potential solution in the form of "<unused[0-98]>" tokens. However, this approach requires additional training, as the base model lacks knowledge of these tokens.

In this demo, you will fine-tune the model with a simple prompt "日本の珍しい名字" which means "Rare Japanese surnames".

<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<unused[0-98]>さん<end_of_turn>

Based on your application's requirements, you must replace your custom vocabulary with unused tokens. Then, feed the converted training datasets to the model. During the tuning process, the base model will learn the patterns in your document and start generating unused tokens accordingly. Finally, before printing the output text, convert it back to your custom vocabulary.

Due to the fixed size of the vocabulary, adding a significant number of new tokens is not feasible. To do so, you would need to resize the tokenizer and retrain the model, which is inefficient and not recommended unless absolutely necessary.

Setup

Select the Colab runtime

To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:

  1. In the upper-right of the Colab window, select ▾ (Additional connection options).
  2. Select Change runtime type.
  3. Under Hardware accelerator, select T4 GPU.

Gemma setup on Kaggle

To complete this tutorial, you'll first need to complete the setup instructions at Gemma setup. The Gemma setup instructions show you how to do the following:

  • Get access to Gemma on kaggle.com.
  • Select a Colab runtime with sufficient resources to run the Gemma 2B model.
  • Generate and configure a Kaggle username and API key.

After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment.

Set environment variables

Set environment variables for KAGGLE_USERNAME and KAGGLE_KEY.

[ ]

Install dependencies

Install Keras and KerasNLP

[ ]

Identify the presence of Surnames in the Tokenizer's vocabulary

Regarding the base Gemma Tokenizer, see below that it includes frequently used names, such as "佐藤" and "鈴木", within its vocabulary. However, for less common names, the tokenizer begins to split them into multiple tokens, like below:

235585 -> 小
240763 -> 嶋
  4758 -> さん

It's getting worse if you go very rare surnames like below:

235771 -> 加
   450 -> <0xE9>
   389 -> <0xAC>
   391 -> <0xAE>
  4758 -> さん

In the case of "加鬮", tokenizer typically splits this character into its UTF-8 representation. This is because the tokenizer hasn't encountered this character frequently during the training process. If you fine-tune the model, it will significantly improve efficiency. For example "加鬮", 4 tokens can be reduced to 1 token.

[ ]
[   108  56985  32012 235465  78638   4758 235394  79900   4758 235394
 122224   4758 235394  98647   4758 235394 119177   4758 235394 169944
   4758 235394 113698   4758 235394 119392   4758 235394 117774   4758
 235394 140762   4758    108 235308 235276 235274 235893 235465 235585
 240763   4758 235394 239713 235493   4758 235394 235469 236872   4758
 235394 241122 235722   4758 235394 238303 235841   4758 235394 157967
   4758 235394 238803 238313   4758 235394 235861 235875   4758 235394
 236447 236063   4758 235394 240052 236872   4758    108 235304 235324
 235276 235276 235274 235893 235465 235661 236316   4758 235394 238881
 235842   4758 235394 235469 239319   4758 235394 237338 235502   4758
 235394 237550 236228   4758 235394 235648 235858   4758 235394 243141
 238022   4758 235394 235771    450    389    391   4758 235394 236497
 241335   4758 235394 235990 242446   4758    108]
   108 -> 

 56985 -> トップ
 32012 -> テン
235465 -> :
 78638 -> 佐藤
  4758 -> さん
235394 -> 、
 79900 -> 鈴木
  4758 -> さん
235394 -> 、
122224 -> 高橋
  4758 -> さん
235394 -> 、
 98647 -> 田中
  4758 -> さん
235394 -> 、
119177 -> 伊藤
  4758 -> さん
235394 -> 、
169944 -> 渡辺
  4758 -> さん
235394 -> 、
113698 -> 山本
  4758 -> さん
235394 -> 、
119392 -> 中村
  4758 -> さん
235394 -> 、
117774 -> 小林
  4758 -> さん
235394 -> 、
140762 -> 加藤
  4758 -> さん
   108 -> 

235308 -> 5
235276 -> 0
235274 -> 1
235893 -> 位
235465 -> :
235585 -> 小
240763 -> 嶋
  4758 -> さん
235394 -> 、
239713 -> 畑
235493 -> 中
  4758 -> さん
235394 -> 、
235469 -> 大
236872 -> 井
  4758 -> さん
235394 -> 、
241122 -> 磯
235722 -> 部
  4758 -> さん
235394 -> 、
238303 -> 浅
235841 -> 見
  4758 -> さん
235394 -> 、
157967 -> 秋田
  4758 -> さん
235394 -> 、
238803 -> 芳
238313 -> 賀
  4758 -> さん
235394 -> 、
235861 -> 相
235875 -> 原
  4758 -> さん
235394 -> 、
236447 -> 細
236063 -> 田
  4758 -> さん
235394 -> 、
240052 -> 坪
236872 -> 井
  4758 -> さん
   108 -> 

235304 -> 3
235324 -> 7
235276 -> 0
235276 -> 0
235274 -> 1
235893 -> 位
235465 -> :
235661 -> 法
236316 -> 川
  4758 -> さん
235394 -> 、
238881 -> 乙
235842 -> 間
  4758 -> さん
235394 -> 、
235469 -> 大
239319 -> 舌
  4758 -> さん
235394 -> 、
237338 -> 巻
235502 -> 上
  4758 -> さん
235394 -> 、
237550 -> 戸
236228 -> 住
  4758 -> さん
235394 -> 、
235648 -> 前
235858 -> 更
  4758 -> さん
235394 -> 、
243141 -> 梶
238022 -> 浜
  4758 -> さん
235394 -> 、
235771 -> 加
   450 -> �
   389 -> �
   391 -> �
  4758 -> さん
235394 -> 、
236497 -> 千
241335 -> 艘
  4758 -> さん
235394 -> 、
235990 -> 西
242446 -> 胤
  4758 -> さん
   108 -> 

[ ]
<0xE9>
<0xAC>
<0xAE>
[ ]
['<pad>',
, '<eos>',
, '<bos>',
, '<unk>',
, '<mask>',
, '<2mass>',
, '[@BOS@]',
, '<unused0>',
, '<unused1>',
, '<unused2>',
, '<unused3>',
, '<unused4>',
, '<unused5>',
, '<unused6>',
, '<unused7>',
, '<unused8>',
, '<unused9>',
, '<unused10>',
, '<unused11>',
, '<unused12>',
, '<unused13>',
, '<unused14>',
, '<unused15>',
, '<unused16>',
, '<unused17>',
, '<unused18>',
, '<unused19>',
, '<unused20>',
, '<unused21>',
, '<unused22>',
, '<unused23>',
, '<unused24>',
, '<unused25>',
, '<unused26>',
, '<unused27>',
, '<unused28>',
, '<unused29>',
, '<unused30>',
, '<unused31>',
, '<unused32>',
, '<unused33>',
, '<unused34>',
, '<unused35>',
, '<unused36>',
, '<unused37>',
, '<unused38>',
, '<unused39>',
, '<unused40>',
, '<unused41>',
, '<unused42>',
, '<unused43>',
, '<unused44>',
, '<unused45>',
, '<unused46>',
, '<unused47>',
, '<unused48>',
, '<unused49>',
, '<unused50>',
, '<unused51>',
, '<unused52>',
, '<unused53>',
, '<unused54>',
, '<unused55>',
, '<unused56>',
, '<unused57>',
, '<unused58>',
, '<unused59>',
, '<unused60>',
, '<unused61>',
, '<unused62>',
, '<unused63>',
, '<unused64>',
, '<unused65>',
, '<unused66>',
, '<unused67>',
, '<unused68>',
, '<unused69>',
, '<unused70>',
, '<unused71>',
, '<unused72>',
, '<unused73>',
, '<unused74>',
, '<unused75>',
, '<unused76>',
, '<unused77>',
, '<unused78>',
, '<unused79>',
, '<unused80>',
, '<unused81>',
, '<unused82>',
, '<unused83>',
, '<unused84>',
, '<unused85>',
, '<unused86>',
, '<unused87>',
, '<unused88>',
, '<unused89>',
, '<unused90>',
, '<unused91>',
, '<unused92>',
, '<unused93>',
, '<unused94>',
, '<unused95>',
, '<unused96>',
, '<unused97>',
, '<unused98>',
, '<start_of_turn>',
, '<end_of_turn>',
, '\n',
, '\n\n',
, '\n\n\n',
, '\n\n\n\n',
, '\n\n\n\n\n',
, '\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n',
, '▁▁',
, '▁▁▁',
, '▁▁▁▁',
, '▁▁▁▁▁',
, '▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁',
, '<table>',
, '<caption>',
, '<thead>',
, '<tbody>',
, '<tfoot>',
, '<tr>',
, '<th>',
, '<td>',
, '</table>',
, '</caption>',
, '</thead>',
, '</tbody>',
, '</tfoot>',
, '</tr>',
, '</th>',
, '</td>',
, '<h1>',
, '<h2>',
, '<h3>',
, '<h4>',
, '<h5>',
, '<h6>',
, '<blockquote>',
, '</h1>',
, '</h2>',
, '</h3>',
, '</h4>',
, '</h5>',
, '</h6>',
, '</blockquote>',
, '<strong>',
, '<em>',
, '<b>',
, '<i>',
, '<u>',
, '<s>',
, '<sub>',
, '<sup>',
, '<code>',
, '</strong>',
, '</em>',
, '</b>',
, '</i>',
, '</u>',
, '</s>',
, '</sub>',
, '</sup>',
, '</code>',
, '<0x00>',
, '<0x01>',
, '<0x02>',
, '<0x03>',
, '<0x04>',
, '<0x05>',
, '<0x06>',
, '<0x07>',
, '<0x08>',
, '<0x09>',
, '<0x0A>',
, '<0x0B>',
, '<0x0C>',
, '<0x0D>',
, '<0x0E>',
, '<0x0F>',
, '<0x10>',
, '<0x11>',
, '<0x12>',
, '<0x13>',
, '<0x14>',
, '<0x15>',
, '<0x16>',
, '<0x17>',
, '<0x18>',
, '<0x19>',
, '<0x1A>',
, '<0x1B>',
, '<0x1C>',
, '<0x1D>',
, '<0x1E>',
, '<0x1F>',
, '<0x20>',
, '<0x21>',
, '<0x22>',
, '<0x23>',
, '<0x24>',
, '<0x25>',
, '<0x26>',
, '<0x27>',
, '<0x28>',
, '<0x29>',
, '<0x2A>',
, '<0x2B>',
, '<0x2C>',
, '<0x2D>',
, '<0x2E>',
, '<0x2F>',
, '<0x30>',
, '<0x31>',
, '<0x32>',
, '<0x33>',
, '<0x34>',
, '<0x35>',
, '<0x36>',
, '<0x37>',
, '<0x38>',
, '<0x39>',
, '<0x3A>',
, '<0x3B>',
, '<0x3C>',
, '<0x3D>',
, '<0x3E>',
, '<0x3F>',
, '<0x40>',
, '<0x41>',
, '<0x42>',
, '<0x43>',
, '<0x44>',
, '<0x45>',
, '<0x46>',
, '<0x47>',
, '<0x48>',
, '<0x49>',
, '<0x4A>',
, '<0x4B>',
, '<0x4C>',
, '<0x4D>',
, '<0x4E>',
, '<0x4F>',
, '<0x50>',
, '<0x51>',
, '<0x52>',
, '<0x53>',
, '<0x54>',
, '<0x55>',
, '<0x56>',
, '<0x57>',
, '<0x58>',
, '<0x59>',
, '<0x5A>',
, '<0x5B>',
, '<0x5C>',
, '<0x5D>',
, '<0x5E>',
, '<0x5F>',
, '<0x60>',
, '<0x61>',
, '<0x62>',
, '<0x63>',
, '<0x64>',
, '<0x65>',
, '<0x66>',
, '<0x67>',
, '<0x68>',
, '<0x69>',
, '<0x6A>',
, '<0x6B>',
, '<0x6C>',
, '<0x6D>',
, '<0x6E>',
, '<0x6F>',
, '<0x70>',
, '<0x71>',
, '<0x72>',
, '<0x73>',
, '<0x74>',
, '<0x75>',
, '<0x76>',
, '<0x77>',
, '<0x78>',
, '<0x79>',
, '<0x7A>',
, '<0x7B>',
, '<0x7C>',
, '<0x7D>',
, '<0x7E>',
, '<0x7F>',
, '<0x80>',
, '<0x81>',
, '<0x82>',
, '<0x83>',
, '<0x84>',
, '<0x85>',
, '<0x86>',
, '<0x87>',
, '<0x88>',
, '<0x89>',
, '<0x8A>',
, '<0x8B>',
, '<0x8C>',
, '<0x8D>',
, '<0x8E>',
, '<0x8F>',
, '<0x90>',
, '<0x91>',
, '<0x92>',
, '<0x93>',
, '<0x94>',
, '<0x95>',
, '<0x96>',
, '<0x97>',
, '<0x98>',
, '<0x99>',
, '<0x9A>',
, '<0x9B>',
, '<0x9C>',
, '<0x9D>',
, '<0x9E>',
, '<0x9F>',
, '<0xA0>',
, '<0xA1>',
, '<0xA2>',
, '<0xA3>',
, '<0xA4>',
, '<0xA5>',
, '<0xA6>',
, '<0xA7>',
, '<0xA8>',
, '<0xA9>',
, '<0xAA>',
, '<0xAB>',
, '<0xAC>',
, '<0xAD>',
, '<0xAE>',
, '<0xAF>',
, '<0xB0>',
, '<0xB1>',
, '<0xB2>',
, '<0xB3>',
, '<0xB4>',
, '<0xB5>',
, '<0xB6>',
, '<0xB7>',
, '<0xB8>',
, '<0xB9>',
, '<0xBA>',
, '<0xBB>',
, '<0xBC>',
, '<0xBD>',
, '<0xBE>',
, '<0xBF>',
, '<0xC0>',
, '<0xC1>',
, '<0xC2>',
, '<0xC3>',
, '<0xC4>',
, '<0xC5>',
, '<0xC6>',
, '<0xC7>',
, '<0xC8>',
, '<0xC9>',
, '<0xCA>',
, '<0xCB>',
, '<0xCC>',
, '<0xCD>',
, '<0xCE>',
, '<0xCF>',
, '<0xD0>',
, '<0xD1>',
, '<0xD2>',
, '<0xD3>',
, '<0xD4>',
, '<0xD5>',
, '<0xD6>',
, '<0xD7>',
, '<0xD8>',
, '<0xD9>',
, '<0xDA>',
, '<0xDB>',
, '<0xDC>',
, '<0xDD>',
, '<0xDE>',
, '<0xDF>',
, '<0xE0>',
, '<0xE1>',
, '<0xE2>',
, '<0xE3>',
, '<0xE4>',
, '<0xE5>',
, '<0xE6>',
, '<0xE7>',
, '<0xE8>',
, '<0xE9>',
, '<0xEA>',
, '<0xEB>',
, '<0xEC>',
, '<0xED>',
, '<0xEE>',
, '<0xEF>',
, '<0xF0>',
, '<0xF1>',
, '<0xF2>',
, '<0xF3>',
, '<0xF4>',
, '<0xF5>',
, '<0xF6>',
, '<0xF7>',
, '<0xF8>',
, '<0xF9>',
, '<0xFA>',
, '<0xFB>',
, '<0xFC>',
, '<0xFD>',
, '<0xFE>',
, '<0xFF>',
, 'in',
, '▁t',
, 'er',
, '▁a',
, 'on',
, 're',
, 'en',
, 'he',
, 'an',
, 'at',
, 'or',
, 'es',
, '▁s',
, 'ar',
, 'ti',
, 'te',
, 'th',
, 'st',
, 'nd',
, 'al',
, '▁o',
, 'le',
, 'de',
, '▁i',
, 'se',
, '▁c',
, '▁d',
, 'it',
, 'nt',
, 'is',
, '▁p',
, 'me',
, 'ri',
, 'ra',
, 'ou',
, 'as',
, 'ed',
, 'ne',
, 'to',
, 'ng',
, '▁w',
, 'ro',
, 'li',
, 'ta',
, '▁f',
, '▁b',
, '▁m',
, 'ic',
, 'el',
, 'la',
, 'et',
, 've',
, 'ur',
, '▁e',
, 'ha',
, 'co',
, 'll',
, 'ch',
, '▁h',
, 'ce',
, '▁l',
, 'ma',
, 'ea',
, 'io',
, 'om',
, 'tr',
, 'id',
, 'il',
, 'ge',
, 'si',
, 'di',
, 'hi',
, 'lo',
, 'ec',
, 'ie',
, '▁r',
, 'un',
, 'ac',
, 'ct',
, '▁n',
, 'us',
, 'pe',
, 'be',
, 'na',
, 'ca',
, 'ns',
, 'of',
, 'ut',
, 'ol',
, 'ot',
, 'am',
, 'ss',
, 'em',
, 'ad',
, 'po',
, 'os',
, 'pa',
, '▁S',
, 'im',
, 'ho',
, '▁the',
, 'ing',
, '▁in',
, '▁of',
, '▁to',
, '▁and',
, 'ent',
, '▁th',
, '▁de',
, '▁re',
, '▁g',
, '▁T',
, '▁C',
, '▁A',
, 'ion',
, 'tion',
, '▁=',
, '▁I',
, '▁(',
, 'qu',
, '▁v',
, ');',
, '▁M',
, '▁P',
, '▁y',
, 'ver',
, '▁B',
, 'um',
, 'ul',
, 'ig',
, '▁is',
, '▁for',
, 'ly',
, '▁u',
, 'ate',
, '▁D',
, 'ow',
, 'ter',
, '▁on',
, '▁{',
, '▁wi',
, '▁be',
, 'end',
, 'ir',
, 'ts',
, 'ers',
, 'ue',
, 'hat',
, 'ati',
, 'ine',
, 'ck',
, 'res',
, '▁R',
, 'ay',
, '="',
, 'gh',
, '▁L',
, 'ons',
, 'men',
, '▁con',
, '▁F',
, '//',
, 'int',
, 'cti',
, '▁E',
, '▁k',
, 'and',
, '▁H',
, 'ort',
, 'our',
, 'od',
, 'est',
, '()',
, '▁N',
, '▁W',
, 'if',
, '▁*',
, 'ht',
, 'The',
, 'ith',
, '▁G',
, 'ere',
, 'ke',
, '▁pro',
, 'ub',
, 'pp',
, '▁en',
, 'ble',
, '▁ha',
, '\\[',
, '\\]',
, '▁"',
, '▁it',
, 'oo',
, 'ci',
, 'all',
, 'ess',
, 'ab',
, '▁an',
, 'ret',
, 'ation',
, '▁that',
, '▁with',
, 'ction',
, 'ment',
, '▁j',
, '▁ne',
, '▁st',
, '▁com',
, '▁me',
, '▁la',
, 'ity',
, '▁as',
, 'tem',
, '▁O',
, '**',
, '▁or',
, 'urn',
, 'ted',
, '▁you',
, '▁he',
, 'ist',
, 'rom',
, '▁at',
, '▁$',
, '▁ex',
, '▁se',
, '▁qu',
, 'mp',
, '▁li',
, 'line',
, 'per',
, '▁mo',
, 'ld',
, 'port',
, '▁are',
, '▁le',
, '}\r',
, 'ure',
, '▁so',
, '▁J',
, '▁The',
, 'op',
, 'iz',
, '▁al',
, '▁ma',
, 'ass',
, 'lic',
, 'xt',
, 'ore',
, '..',
, '▁&',
, 'ite',
, '/*',
, '</',
, '▁-',
, '▁was',
, '▁\\',
, '▁by',
, 'age',
, '▁wh',
, 'ame',
, 'ard',
, '▁this',
, 'ag',
, 'ak',
, 'ust',
, 'ia',
, 'ant',
, '▁su',
, 'hl',
, '▁V',
, 'out',
, 'for',
, '▁K',
, '▁un',
, '▁do',
, 'ap',
, '▁di',
, '▁U',
, 'ath',
, 'ang',
, '▁par',
, 'ty',
, 'ord',
, 'ber',
, 'con',
, 'art',
, 'ff',
, 'ys',
, 'du',
, 'ome',
, 'tu',
, 'ze',
, 'ind',
, '">',
, '▁po',
, 'ex',
, 'mat',
, 'las',
, 'return',
, '▁from',
, '/**',
, 'hline',
, "▁'",
, 'gin',
, 'ans',
, '▁not',
, 'rou',
, 'av',
, '▁we',
, 'ata',
, 'one',
, 'cl',
, 'ack',
, '▁ch',
, '},',
, 'ther',
, '▁have',
, 'au',
, '▁no',
, '--',
, 'ase',
, 'ould',
, 'ich',
, '▁can',
, '▁tr',
, 'sed',
, 'sp',
, 'tic',
, 'ud',
, 'ote',
, 'ublic',
, 'ell',
, '▁id',
, 'ong',
, 'import',
, 'ep',
, 'ect',
, 'ks',
, '▁ar',
, '};',
, 'ght',
, 'oc',
, 'sion',
, 'get',
, 'ous',
, 'ru',
, '();',
, '▁el',
, '),',
, '",',
, 'mo',
, '▁lo',
, 'ran',
, 'ary',
, '▁*/',
, 'cc',
, 'ry',
, '▁all',
, 'ps',
, 'ount',
, '▁ad',
, 'ph',
, 'ork',
, 'lass',
, 'ound',
, 'ated',
, '▁ab',
, '▁per',
, 'lock',
, 'ire',
, 'ear',
, ').',
, '▁sp',
, '▁des',
, 'ER',
, 'ven',
, 'ance',
, 'va',
, 'lu',
, 'red',
, 'ult',
, 'iew',
, 'erv',
, 'ally',
, '});',
, '*/',
, '▁your',
, 'ile',
, 'ice',
, 'che',
, '▁:',
, 'der',
, '▁sa',
, '▁z',
, 'og',
, 'fer',
, '▁go',
, 'com',
, 'ON',
, 'row',
, '▁es',
, 'ence',
, '▁will',
, '▁In',
, 'public',
, 'ign',
, 'set',
, 'ime',
, 'this',
, 'item',
, '("',
, 'In',
, 'able',
, '▁new',
, 'ize',
, '▁Y',
, 'ull',
, '▁[',
, 'vi',
, 'vel',
, 'are',
, 'ont',
, 'ast',
, '▁п',
, '▁ro',
, 'ple',
, '▁but',
, 'begin',
, 'ations',
, '▁Th',
, '__',
, '▁uid',
, '▁que',
, '▁up',
, 'tring',
, '▁co',
, 'ight',
, 'ст',
, 'div',
, 'ide',
, 'ces',
, '▁man',
, '▁us',
, 'ial',
, '▁has',
, "',",
, '▁out',
, '▁ا',
, 'block',
, 'ies',
, 'lay',
, '▁his',
, 'section',
, 'IN',
, 'ory',
, '▁und',
, 'so',
, 'sel',
, 'ook',
, 'val',
, 'ft',
, 'ill',
, 'log',
, 'ption',
, 'ded',
, 'ni',
, '▁с',
, '▁comp',
, '▁sh',
, 'ree',
, 'quote',
, 'Th',
, '▁get',
, '▁which',
, '->',
, 'ater',
, '▁pid',
, 'ens',
, '▁pre',
, '▁.',
, '...',
, 'no',
, 'iv',
, 'Con',
, '▁ti',
, 'pri',
, 'form',
, '{\r',
, '▁+',
, 'par',
, 'ten',
, 'we',
, 'ach',
, '▁<',
, 'wer',
, '▁my',
, 'ond',
, '▁ver',
, 'blockquote',
, '▁one',
, '▁в',
, 'ors',
, 'mer',
, '▁more',
, 'ink',
, 'EN',
, 'на',
, 'les',
, ';\r',
, '▁they',
, 'ain',
, 'ystem',
, 'Re',
, 'ES',
, '▁dis',
, 'ents',
, 'ild',
, '▁au',
, 'RE',
, '_{',
, 'mb',
, 'ни',
, '▁St',
, 'ра',
, 'ms',
, ...]

Load Model

[ ]

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
Here are some Japanese names that are considered uncommon or unique:

**Nature-Inspired:**

* **Aoi:**  (青) Blue, a beautiful color often associated with the sky and water.
* **Kiku:** (菊) Chrysanthemum, a symbol of longevity and beauty.
* **Sakura:** (桜) Cherry blossom, a symbol of spring and renewal.
* **Tsuki:** (月) Moon, a celestial body that inspires wonder and mystery.
* **Yume:** (夢) Dream, a symbol of hope and imagination.
TOTAL TIME ELAPSED: 25.95s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
Here are some examples of common Japanese surnames, often considered "ordinary" or "unremarkable":

**Common and Neutral:**

* **Yamada:**  (山田) -  Meaning "mountain field"
* **Suzuki:** (鈴木) -  Meaning "small, humble"
* **Nakamura:** (中村) -  Meaning "village of the mountain"
* **Kato:** (加藤) -  Meaning "a person who lives in a village"
* **Tanaka:** (田中) -  Meaning "a person who lives
TOTAL TIME ELAPSED: 5.38s
[ ]
{'<unused0>': '内大久保', '<unused1>': '諏訪戸', '<unused2>': '宮艸', '<unused3>': '宝坂', '<unused4>': '埜上', '<unused5>': '篠垣', '<unused6>': '池呂', '<unused7>': '奥須賀', '<unused8>': '勝居', '<unused9>': '真喜', '<unused10>': '高河原', '<unused11>': '小盛', '<unused12>': '溜口', '<unused13>': '伊勢井', '<unused14>': '落久保', '<unused15>': '志渡澤', '<unused16>': '嘉多', '<unused17>': '板藤', '<unused18>': '南波留', '<unused19>': '風ん', '<unused20>': '采尾', '<unused21>': '一上', '<unused22>': '本邑', '<unused23>': '周本', '<unused24>': '畠尾', '<unused25>': '鳥水', '<unused26>': '下小薗', '<unused27>': '阪辺', '<unused28>': '稲味', '<unused29>': '武浪', '<unused30>': '安楽城', '<unused31>': '江古', '<unused32>': '賎機', '<unused33>': '鶴原谷', '<unused34>': '西明寺', '<unused35>': '布塚', '<unused36>': '寺端', '<unused37>': '炭吉', '<unused38>': '管生', '<unused39>': '村片', '<unused40>': '昌司', '<unused41>': '伊秩', '<unused42>': '後道', '<unused43>': '佐保井', '<unused44>': '神尊', '<unused45>': '為積', '<unused46>': '聖川', '<unused47>': '登喜', '<unused48>': '弦田', '<unused49>': '犬居'}
70
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
山本さん<end_of_turn>
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<unused0>さん<end_of_turn>
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
松本さん<end_of_turn>

LoRA Fine-tuning

[ ]

Note that enabling LoRA reduces the number of trainable parameters significantly. In practice, we recommend beginning with a relatively small rank (such as 4, 8, 16). This is computationally efficient for experimentation.

To monitor the learning progress, we will evaluate the model at the end of each epoch and save the all lora weights.

[ ]
Epoch 1/5
35/35 ━━━━━━━━━━━━━━━━━━━━ 0s 3s/step - loss: 1.3578 - sparse_categorical_accuracy: 0.2155
Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<end_of_turn>さん<end_of_turn>
TOTAL TIME ELAPSED: 22.53s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
ⓧさん<end_of_turn>
TOTAL TIME ELAPSED: 0.40s
35/35 ━━━━━━━━━━━━━━━━━━━━ 167s 4s/step - loss: 1.3449 - sparse_categorical_accuracy: 0.2209
Epoch 2/5
35/35 ━━━━━━━━━━━━━━━━━━━━ 0s 735ms/step - loss: 0.2559 - sparse_categorical_accuracy: 0.7883
Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<end_of_turn>さん<end_of_turn>
TOTAL TIME ELAPSED: 0.43s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
井上さん<end_of_turn>
TOTAL TIME ELAPSED: 0.40s
35/35 ━━━━━━━━━━━━━━━━━━━━ 56s 783ms/step - loss: 0.2550 - sparse_categorical_accuracy: 0.7891
Epoch 3/5
34/35 ━━━━━━━━━━━━━━━━━━━━ 0s 779ms/step - loss: 0.1715 - sparse_categorical_accuracy: 0.8500
Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<start_of_turn>さん<end_of_turn>
TOTAL TIME ELAPSED: 0.41s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
井上さん<end_of_turn>
TOTAL TIME ELAPSED: 0.41s
35/35 ━━━━━━━━━━━━━━━━━━━━ 28s 811ms/step - loss: 0.1708 - sparse_categorical_accuracy: 0.8501
Epoch 4/5
35/35 ━━━━━━━━━━━━━━━━━━━━ 0s 792ms/step - loss: 0.1328 - sparse_categorical_accuracy: 0.8413
Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<start_of_turn>さん<end_of_turn>
TOTAL TIME ELAPSED: 0.42s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
清水さん<end_of_turn>
TOTAL TIME ELAPSED: 0.41s
35/35 ━━━━━━━━━━━━━━━━━━━━ 29s 837ms/step - loss: 0.1325 - sparse_categorical_accuracy: 0.8414
Epoch 5/5
35/35 ━━━━━━━━━━━━━━━━━━━━ 0s 806ms/step - loss: 0.1024 - sparse_categorical_accuracy: 0.8451
Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<unused26>さん<end_of_turn>
TOTAL TIME ELAPSED: 0.42s

Gemma output:
<start_of_turn>user
日本の平凡な名字<end_of_turn>
<start_of_turn>model
井上さん<end_of_turn>
TOTAL TIME ELAPSED: 0.41s
35/35 ━━━━━━━━━━━━━━━━━━━━ 30s 851ms/step - loss: 0.1022 - sparse_categorical_accuracy: 0.8452
Output

Try a different sampler

The top-K algorithm randomly picks the next token from the tokens of top K probability.

NOTE: Due to randomness of the sampler, you may encounter <unusedXX> tokens that surpass the number you trained with.

[ ]
--------------------------------------------------------------------------------
[   106   1645    108  47172 142742  39742    107    108    106   2516
    108     23   4758    107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
142742 -> 珍しい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    23 -> <unused16>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
嘉多さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 22.22s
--------------------------------------------------------------------------------
[   106   1645    108  47172 142742  39742    107    108    106   2516
    108     59   4758    107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
142742 -> 珍しい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    59 -> <unused52>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<unused52>さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.54s
--------------------------------------------------------------------------------
[   106   1645    108  47172 142742  39742    107    108    106   2516
    108     75   4758    107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
142742 -> 珍しい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    75 -> <unused68>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
<unused68>さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.51s
--------------------------------------------------------------------------------
[   106   1645    108  47172 142742  39742    107    108    106   2516
    108     30   4758    107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
142742 -> 珍しい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    30 -> <unused23>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
周本さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.53s
--------------------------------------------------------------------------------
[   106   1645    108  47172 142742  39742    107    108    106   2516
    108     33   4758    107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
142742 -> 珍しい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    33 -> <unused26>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の珍しい名字<end_of_turn>
<start_of_turn>model
下小薗さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.51s

Try a slight different prompts

[ ]
--------------------------------------------------------------------------------
[  106  1645   108 47172 39742   107   108   106  2516   108 98647  4758
   107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

 98647 -> 田中
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の名字<end_of_turn>
<start_of_turn>model
田中さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.69s
--------------------------------------------------------------------------------
[  106  1645   108 47172 77427 39742   107   108   106  2516   108   106
  4758   107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
 77427 -> 面白い
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

   106 -> <start_of_turn>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の面白い名字<end_of_turn>
<start_of_turn>model
<start_of_turn>さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.50s
--------------------------------------------------------------------------------
[  106  1645   108 47172 46116  4302  3581 39742   107   108   106  2516
   108    51  4758   107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
 46116 -> 見た
  4302 -> こと
  3581 -> ない
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    51 -> <unused44>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本の見たことない名字<end_of_turn>
<start_of_turn>model
神尊さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.58s
--------------------------------------------------------------------------------
[  106  1645   108 47172 59956 39742   107   108   106  2516   108    16
  4758   107]
   106 -> <start_of_turn>
  1645 -> user
   108 -> 

 47172 -> 日本の
 59956 -> すごい
 39742 -> 名字
   107 -> <end_of_turn>
   108 -> 

   106 -> <start_of_turn>
  2516 -> model
   108 -> 

    16 -> <unused9>
  4758 -> さん
   107 -> <end_of_turn>

Gemma output:
<start_of_turn>user
日本のすごい名字<end_of_turn>
<start_of_turn>model
真喜さん<end_of_turn>
--------------------------------------------------------------------------------
TOTAL TIME ELAPSED: 0.49s