Milvus Export Hf To Onnx

Export Hf To Onnx

onnx_exampleimage-searchvector-databasesemantic-searchtutorialsmilvusembeddingsunstructured-dataquestion-answeringLLMmilvus-bootcampdeep-learningimage-recognitionimage-classificationaudio-searchPythonquickstartragNLP

alph-notebooks/milvus-bootcamp / export_hf_to_onnx.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

HuggingFace-to-Onnx

Example of exporting a text embedding model and tokenizer from HuggingFace to ONNX

[2]

Attempt 1: Open model and export to ONNX

[ ]

2025-05-01 08:52:23.154699 [W:onnxruntime:, helper.cc:83 IsInputSupported] CoreML does not support input dim > 16384. Input:embeddings.word_embeddings.weight, shape: {30522,384}
2025-05-01 08:52:23.155121 [W:onnxruntime:, coreml_execution_provider.cc:112 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 55 number of nodes in the graph: 418 number of nodes supported by CoreML: 278
2025-05-01 08:52:23.641367 [W:onnxruntime:, session_state.cc:1263 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2025-05-01 08:52:23.641376 [W:onnxruntime:, session_state.cc:1265 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

[ ]

/Users/stefanwebb/opt/anaconda3/envs/huggingface/lib/python3.11/site-packages/optimum/onnxruntime/configuration.py:784: FutureWarning: disable_embed_layer_norm will be deprecated soon, use disable_embed_layer_norm_fusion instead, disable_embed_layer_norm_fusion is set to True.
  warnings.warn(

Examine saved model in ONNX

[8]

None

[ ]

inputs ['input_ids', 'attention_mask', 'token_type_ids']
outputs ['last_hidden_state']

Gotcha

The output being last_hidden_state suggests the ONNX model from sentence-transformers doesn't include pooling and normalization modules. See these:

[ ]

Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: ORTModelForFeatureExtraction  

Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) 

Normalize()

This particular sentence-transformers model has three modules: the encoder-type transformer, a mean pooling layer on the output, and a final normalization to the text embedding.

Attempt 2: Open model and export to ONNX

Instead of implementing the code to convert and join all three modules into ONNX, we can use existing libraries.

One option is the Optimum library from HuggingFace: optimum-cli export onnx --model sentence-transformers/all-MiniLM-L6-v2 model.onnx.

We will use a library from .txtai (you may know them from the Python package outlines for constrained LLM sampling).

[ ]

WARNING:root:Please consider to run pre-processing before quantization. Refer to example: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md

[7]

None

[ ]

inputs ['input_ids', 'attention_mask', 'token_type_ids']
outputs ['embeddings']

Now the output node is labelled embeddings, indicating it has applied all three modules.

Tokenizer

Let's export the tokenizer using ONNXRuntimeExtensions.

[1]

[4]

[ ]

I haven't been able to get model and tokenizer in same IR version in order to combine graphs.

[ ]

(7, 8)

Testing Use

[14]

[ ]

Inference is performed via the ONNX Runtime.

[ ]

Let's tokenize a string of text.

[ ]

[21]

[array([  101,  1996,  4248,  2829,  4419, 14523,  2058,  1996, 13971,
,         3899,  1012,   102], dtype=int64),
, array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64),
, array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int64),
, array([[ 0,  0],
,        [ 0,  3],
,        [ 4,  9],
,        [10, 15],
,        [16, 19],
,        [20, 25],
,        [26, 30],
,        [31, 34],
,        [35, 39],
,        [40, 43],
,        [44, 45],
,        [ 0,  0]], dtype=int64)]

[ ]

[57]

transformers.models.bert.tokenization_bert_fast.BertTokenizerFast

Encoding the tokens with our ONNX model and decoding with the HuggingFace model gives back the input (plus additional special tokens).

[ ]

'[CLS] the quick brown fox jumps over the lazy dog. [SEP]'

Now let's perform inference with the embedding model.

[ ]

array([[ 1.46325007e-01,  3.28532130e-01,  2.66175002e-01,
,         5.18237472e-01,  2.02143028e-01, -1.79584488e-01,
,         1.52321756e-01, -3.98070544e-01, -3.71623226e-02,
,        -5.72629236e-02,  1.29877284e-01,  1.32518455e-01,
,        -1.79732755e-01, -1.65457316e-02, -6.52377785e-04,
,        -1.31084666e-01, -2.06912145e-01, -1.84920907e-01,
,         3.07158738e-01, -2.62583256e-01, -2.98586309e-01,
,        -3.02038938e-01,  1.37112692e-01,  1.63912699e-01,
,        -4.19944048e-01, -1.17152214e-01, -3.97956759e-01,
,        -3.00221205e-01,  4.11090940e-01, -5.13568342e-01,
,        -8.72481465e-02,  1.85722232e-01, -2.18171403e-01,
,        -2.64097247e-02, -1.63030490e-01, -3.72051567e-01,
,         3.35423738e-01,  4.56916727e-02,  1.79710209e-01,
,         1.38806954e-01,  1.49378225e-01, -8.92430544e-02,
,        -2.62890935e-01,  1.50573924e-01, -5.16011655e-01,
,         2.48664081e-01, -4.35304970e-01, -2.22753058e-03,
,         1.70447230e-02,  7.45331869e-03, -1.46014124e-01,
,        -8.62182751e-02, -7.91217908e-02,  2.18953371e-01,
,         3.39172959e-01,  3.44479419e-02, -5.18480949e-02,
,        -1.10342443e-01,  2.76021272e-01, -1.12608306e-01,
,         2.10698023e-02,  2.01664343e-01,  6.60073757e-02,
,         1.04513913e-01,  1.17197432e-01,  2.06166461e-01,
,        -1.37191221e-01, -3.48047376e-01, -5.76119602e-01,
,        -1.08443588e-01,  1.12067312e-02, -2.37340689e-01,
,        -6.14458136e-02,  7.51624256e-02,  2.37712208e-02,
,        -1.32521585e-01,  1.76032782e-01, -2.44395554e-01,
,         3.55587602e-01,  4.43052351e-02, -1.96507916e-01,
,        -3.38534117e-01, -2.27761611e-01,  2.40894511e-01,
,         1.44513309e-01, -1.42351985e-02,  1.04805082e-01,
,        -1.56732649e-02,  4.09836089e-03, -1.51045784e-01,
,        -1.11955576e-01, -3.63453507e-01, -1.65750131e-01,
,         1.50050357e-01,  6.13481961e-02,  1.19610719e-01,
,         8.84542763e-02, -1.50378391e-01, -2.19579145e-01,
,         2.01723680e-01,  9.95668024e-03,  1.88402653e-01,
,         5.16420066e-01,  4.52098846e-02, -7.67205581e-02,
,        -3.01945925e-01,  5.09440787e-02,  7.86306337e-02,
,         5.74932456e-01,  1.45094290e-01, -6.92986771e-02,
,        -8.87666568e-02,  1.24641120e-01,  4.92059857e-01,
,         1.17545694e-01,  1.14440821e-01, -4.18735355e-01,
,        -1.51425242e-01, -1.66200414e-01, -5.23970008e-01,
,         4.58409876e-01,  5.81893623e-01, -4.24026817e-01,
,         4.09295820e-02, -6.36414215e-02, -3.07693064e-01,
,        -1.04793571e-01, -1.75027156e-32,  3.56341749e-01,
,        -2.12330595e-01, -2.92983502e-01, -2.12103054e-01,
,        -2.53180683e-01, -3.95887256e-01, -3.13732550e-02,
,        -9.28076282e-02, -4.89635654e-02,  2.85974532e-01,
,        -4.67964441e-01, -1.11648105e-02,  1.11177489e-02,
,        -8.95037353e-02, -1.16933107e-01, -5.45736372e-01,
,         4.04342897e-02,  3.75743099e-02,  1.06830813e-01,
,         4.15194005e-01,  2.11188182e-01,  5.39109290e-01,
,        -1.45160258e-01, -1.01240963e-01,  2.00557768e-01,
,        -5.98138981e-02, -4.35719937e-01, -1.57140091e-01,
,        -3.84878814e-02,  2.87303954e-01, -2.08185717e-01,
,        -2.38358498e-01, -1.90930799e-01,  3.89428496e-01,
,        -1.49624467e-01, -6.01740420e-01,  2.69364476e-01,
,        -3.24530840e-01, -2.34665081e-01,  2.68453032e-01,
,         8.67292881e-02,  4.94738556e-02,  1.31433710e-01,
,         1.81901753e-01, -2.81626821e-01,  3.68114449e-02,
,         1.42404586e-01,  5.62638827e-02, -1.00267582e-01,
,         1.79056808e-01, -2.83027172e-01,  1.22709595e-01,
,         1.81330502e-01, -2.99587250e-01, -2.72828817e-01,
,         4.83315468e-01,  3.96138549e-01, -1.58888772e-01,
,        -1.10800594e-01,  5.38129628e-01,  1.35027781e-01,
,        -2.59890407e-02, -3.66259068e-02, -1.36844873e-01,
,         6.73519433e-01, -2.57645577e-01, -3.58156651e-01,
,        -3.11016291e-02, -1.01776607e-01,  3.11954111e-01,
,         1.31663784e-01,  2.56282479e-01,  9.76647213e-02,
,        -6.26498520e-01,  2.41229713e-01, -1.31417632e-01,
,         7.46836066e-02, -1.74299583e-01,  2.50860751e-01,
,        -7.69787505e-02,  6.86030865e-01, -3.88147742e-01,
,        -2.19014227e-01,  1.94530666e-01, -8.02200288e-03,
,         2.50101060e-01,  1.68782342e-02,  1.21686339e-01,
,        -1.37575686e-01, -1.79390803e-01, -1.98672712e-01,
,        -5.26131801e-02,  3.92878443e-01, -2.85057366e-01,
,         2.16094866e-01,  1.32624430e-32,  5.44442713e-01,
,        -2.71417975e-01,  3.13715249e-01,  2.58446604e-01,
,        -1.75100997e-01,  3.46659094e-01, -2.04824403e-01,
,         3.73328142e-02, -4.10647869e-01,  4.31406826e-01,
,        -1.04935966e-01, -1.09297028e-02,  8.07947814e-02,
,         2.65660044e-02,  5.28253198e-01, -6.85167611e-02,
,         4.18209076e-01, -5.96915781e-02,  2.79007554e-02,
,         1.20604277e-01, -2.68359244e-01,  8.88191089e-02,
,        -1.49447560e-01,  2.87605405e-01,  9.49743390e-02,
,         3.15899789e-01,  3.47899109e-01, -1.02624476e-01,
,        -9.21735466e-02, -3.75352710e-01, -3.32795233e-01,
,        -3.38638902e-01, -4.06765044e-02, -2.14362755e-01,
,         1.57808051e-01,  2.25305691e-01, -2.47538805e-01,
,        -7.18037188e-02, -1.16640396e-01, -4.27364230e-01,
,         1.48355708e-01, -7.62912408e-02, -3.29614095e-02,
,         3.61372143e-01,  2.07773611e-01,  5.31713329e-02,
,        -3.35338622e-01,  3.46878201e-01, -1.34133786e-01,
,         1.89251855e-01, -1.14382684e-01,  9.03625786e-02,
,         2.31320247e-01,  1.61370143e-01, -2.07244858e-01,
,        -2.24654302e-01, -9.58347097e-02, -1.54911563e-01,
,         2.20494702e-01, -3.91075760e-02,  1.33875282e-02,
,         1.39622509e-01,  3.13203894e-02,  1.45047948e-01,
,        -5.83880544e-02, -3.80364448e-01, -2.66321570e-01,
,        -2.07957789e-01,  3.50972027e-01, -3.84614110e-01,
,         1.22801336e-02,  8.69480893e-02,  1.16526447e-01,
,        -3.53367805e-01,  3.95663194e-02,  4.23076600e-01,
,         5.65669000e-01,  6.49911910e-03,  4.14586693e-01,
,        -2.24571243e-01,  6.85512349e-02, -7.96958283e-02,
,        -4.68175672e-02,  7.71496585e-03, -3.33937734e-01,
,         2.56769598e-01, -5.27394831e-01,  3.01520050e-01,
,         3.27662021e-01, -2.91941971e-01,  1.20466866e-01,
,         3.20452690e-01,  3.24546158e-01,  3.36270601e-01,
,        -3.42106260e-02, -9.61046069e-08, -2.56941348e-01,
,         2.96434369e-02, -4.93199140e-01,  2.23714873e-01,
,         3.49879354e-01,  9.27421227e-02, -1.68685894e-02,
,         6.42052591e-02,  2.99592167e-01, -1.27725139e-01,
,         1.84475496e-01,  9.58684981e-02,  3.41700077e-01,
,         2.70333618e-01, -1.39427297e-02,  1.43404886e-01,
,        -8.02055672e-02,  3.56187187e-02,  1.25088707e-01,
,         8.66966724e-01, -1.10178441e-02, -3.94232154e-01,
,        -2.17166692e-02, -1.17384158e-01, -8.86405110e-02,
,        -1.37166992e-01, -4.28660393e-01,  8.69664550e-03,
,        -1.08561195e-01, -3.27548794e-02, -2.67236739e-01,
,         3.73793632e-01, -5.23531735e-01,  4.73097414e-02,
,         2.37779871e-01,  9.53797176e-02,  4.09780592e-01,
,         3.93795259e-02,  1.75428391e-01, -1.76602349e-04,
,         1.61984116e-02,  2.48420998e-01,  7.18881888e-03,
,         1.27824517e-02, -1.80594325e-01, -4.81582470e-02,
,        -1.35565490e-01, -4.38419223e-01,  2.96767384e-01,
,        -2.98838168e-01, -3.22318554e-01, -2.64659286e-01,
,         3.03042233e-01,  2.19225392e-01, -3.27207357e-01,
,        -5.72266392e-02, -3.24612297e-02, -3.40771407e-01,
,        -2.66784966e-01,  9.14191380e-02,  3.43744278e-01,
,         3.26167315e-01,  2.76887029e-01,  3.44392538e-01]], dtype=float32)

The embeddings between the ONNX model and the saved HuggingFace weights will differ as they're from different trainings.

In the corresponding Java app, you can compare the embedding below to the one produced via Java.

[37]

array([[ 4.39401269e-02,  5.89273572e-02,  4.81781922e-02,
,         7.75616020e-02,  2.67397407e-02, -3.76246534e-02,
,        -2.59578507e-03, -5.99470101e-02, -2.48484872e-03,
,         2.20740736e-02,  4.80036773e-02,  5.57535887e-02,
,        -3.89535986e-02, -2.66309399e-02,  7.69358641e-03,
,        -2.62365304e-02, -3.64078879e-02, -3.78273763e-02,
,         7.40729570e-02, -4.95132506e-02, -5.85304871e-02,
,        -6.36074990e-02,  3.24228741e-02,  2.20151860e-02,
,        -7.10863322e-02, -3.31508964e-02, -6.93992078e-02,
,        -5.00420891e-02,  7.46240765e-02, -1.11135170e-01,
,        -1.23101575e-02,  3.77289020e-02, -2.80298274e-02,
,         1.45433918e-02, -3.15793417e-02, -8.05702582e-02,
,         5.83476461e-02,  2.58636032e-03,  3.92938629e-02,
,         2.57627461e-02,  4.98468950e-02, -1.74043898e-03,
,        -4.55198474e-02,  2.92620845e-02, -1.02021821e-01,
,         5.22407517e-02, -7.91030079e-02, -1.02924807e-02,
,         9.20308568e-03,  1.30610717e-02, -4.04580906e-02,
,        -2.77928729e-02,  1.24553507e-02,  6.72849938e-02,
,         6.81194738e-02, -7.57987006e-03, -6.11215830e-03,
,        -4.23792452e-02,  5.17623462e-02, -1.56695489e-02,
,         9.55946930e-03,  4.12514731e-02,  2.14898121e-02,
,         1.04386294e-02,  2.73313932e-02,  1.86888538e-02,
,        -2.69680247e-02, -7.00684041e-02, -1.04705125e-01,
,        -1.88911334e-03,  1.77195203e-02, -5.74545749e-02,
,        -1.44010559e-02,  4.37871087e-04,  2.33907066e-03,
,        -2.51983870e-02,  4.93028425e-02, -5.09553850e-02,
,         6.32023290e-02,  1.49240633e-02, -2.70748269e-02,
,        -4.52742428e-02, -4.90396172e-02,  3.75088751e-02,
,         3.84690724e-02,  1.57746568e-03,  3.09814643e-02,
,         2.01616567e-02, -1.24542136e-02, -3.06513142e-02,
,        -2.78925058e-02, -6.89273179e-02, -5.13948761e-02,
,         2.14811154e-02,  1.15653016e-02,  1.26375072e-03,
,         1.88614838e-02, -4.42327373e-02, -4.49605435e-02,
,        -3.41741089e-03,  1.31102754e-02,  2.00236458e-02,
,         1.21111892e-01,  2.30980087e-02, -2.20298115e-02,
,        -3.28951739e-02, -3.17868311e-03,  1.26769912e-04,
,         9.91549939e-02,  1.65064428e-02, -4.69857594e-03,
,        -1.45556359e-02, -3.71646765e-03,  9.64998454e-02,
,         2.86016371e-02,  2.13641711e-02, -7.17547163e-02,
,        -2.41155028e-02, -4.40934822e-02, -1.07332557e-01,
,         6.79869130e-02,  1.30458891e-01, -7.96964020e-02,
,         6.80233585e-03, -2.37495080e-02, -4.61554676e-02,
,        -2.99669486e-02, -3.69382533e-33,  7.30794892e-02,
,        -2.20270064e-02, -8.61630291e-02, -7.14560002e-02,
,        -6.36703521e-02, -7.21969232e-02, -5.94721921e-03,
,        -2.33739484e-02, -2.83885989e-02,  4.77444157e-02,
,        -8.05938616e-02, -1.56685722e-03,  1.38301672e-02,
,        -2.86189746e-02, -3.35150510e-02, -1.13776058e-01,
,        -9.18037072e-03, -1.07963523e-02,  3.23242322e-02,
,         5.88202700e-02,  3.34382616e-02,  1.07973419e-01,
,        -3.72813791e-02, -2.96640303e-02,  5.17315492e-02,
,        -2.25261170e-02, -6.96105510e-02, -2.14469377e-02,
,        -2.33503617e-02,  4.82224151e-02, -3.58696878e-02,
,        -4.68909293e-02, -3.97889279e-02,  1.10801756e-01,
,        -1.43186236e-02, -1.18443616e-01,  5.82839511e-02,
,        -6.25948012e-02, -2.94011869e-02,  6.03312738e-02,
,        -2.44144560e-03,  1.60117093e-02,  2.67059617e-02,
,         2.49468610e-02, -6.49329200e-02, -1.06925191e-02,
,         2.81261113e-02,  1.03477063e-02, -6.63481245e-04,
,         1.98137816e-02, -3.04280929e-02,  6.28596032e-03,
,         5.15406691e-02, -4.75625619e-02, -6.44389465e-02,
,         9.55016539e-02,  7.55907595e-02, -2.81462483e-02,
,        -3.49858776e-02,  1.01820581e-01,  1.99138932e-02,
,        -3.68067063e-02,  2.93822680e-03, -5.00899516e-02,
,         1.50941163e-01, -6.16309680e-02, -8.58869255e-02,
,         7.14878691e-03, -1.33048026e-02,  7.80549571e-02,
,         1.75089724e-02,  4.21180353e-02,  3.57964635e-02,
,        -1.32943705e-01,  3.57068107e-02, -2.03077011e-02,
,         1.25009790e-02, -3.80389169e-02,  4.91672866e-02,
,        -1.56585220e-02,  1.21424727e-01, -8.08397382e-02,
,        -4.68668118e-02,  4.10758965e-02, -1.84301455e-02,
,         6.69724271e-02,  4.32531629e-03,  2.27229483e-02,
,        -1.36410547e-02, -4.53305878e-02, -3.92800644e-02,
,        -6.30411552e-03,  5.29699847e-02, -3.69061418e-02,
,         7.11443201e-02,  2.33313496e-33,  1.05220012e-01,
,        -4.81849834e-02,  6.96004704e-02,  6.57009482e-02,
,        -4.65167277e-02,  5.14504798e-02, -1.24384668e-02,
,         3.21013927e-02, -9.23414752e-02,  5.01018241e-02,
,        -3.28826420e-02,  1.39206154e-02, -8.61003587e-04,
,        -4.89492156e-03,  1.03941709e-01,  3.22781882e-04,
,         5.28177321e-02, -1.17869880e-02,  2.31530294e-02,
,         1.31383296e-02, -5.26194572e-02,  3.26648988e-02,
,         3.16695048e-04,  6.41062036e-02,  3.88260931e-02,
,         5.88018186e-02,  8.29776898e-02, -1.88321713e-02,
,        -2.26444788e-02, -1.00478262e-01, -3.83906588e-02,
,        -5.88144399e-02,  1.83725182e-03, -4.27011475e-02,
,         2.50020102e-02,  6.40217662e-02, -3.77539247e-02,
,        -6.85244752e-03, -2.56094942e-03, -9.75910723e-02,
,         1.88639015e-02, -8.91217496e-04,  1.73602179e-02,
,         7.10833147e-02,  3.30262259e-02,  6.93262694e-03,
,        -5.60433976e-02,  5.14697060e-02, -4.29480150e-02,
,         4.60065417e-02, -8.79782904e-03,  3.17413360e-02,
,         4.94121015e-02,  2.95144580e-02, -5.05192168e-02,
,        -5.42992204e-02,  1.39859767e-04, -2.76574250e-02,
,         3.46951261e-02, -2.10896023e-02,  1.37992427e-02,
,         3.00023723e-02,  1.39710652e-02, -4.27215593e-03,
,        -1.50279254e-02, -8.76220539e-02, -6.85213506e-02,
,        -4.28084843e-02,  7.76905119e-02, -7.10421503e-02,
,        -7.37382937e-03,  2.13691555e-02,  1.35557186e-02,
,        -7.90344328e-02,  5.49237523e-03,  8.30597505e-02,
,         1.14154063e-01,  1.80078775e-03,  8.75332281e-02,
,        -4.15831767e-02,  1.55292638e-02, -1.01214005e-02,
,        -7.32931029e-03,  1.07971337e-02, -6.62568957e-02,
,         3.98299992e-02, -1.16720736e-01,  6.43076599e-02,
,         4.02892269e-02, -6.54920787e-02,  1.94968916e-02,
,         8.10138956e-02,  5.36315180e-02,  7.67823160e-02,
,        -1.35062365e-02, -1.76907768e-08, -4.44088206e-02,
,         9.20132268e-03, -8.79707709e-02,  4.26976942e-02,
,         7.31645375e-02,  1.68446675e-02, -4.03056219e-02,
,         1.85057744e-02,  8.44005570e-02, -3.74557786e-02,
,         3.03045101e-02,  2.90730894e-02,  6.36792630e-02,
,         2.89727822e-02, -1.47402417e-02,  1.77620705e-02,
,        -3.37012634e-02,  1.73157491e-02,  3.37989070e-02,
,         1.76823482e-01, -1.75572690e-02, -6.02978468e-02,
,        -1.43410871e-02, -2.38625929e-02, -4.45390865e-02,
,        -2.89787799e-02, -8.97043422e-02, -1.74434332e-03,
,        -2.61579156e-02,  5.95270703e-03, -5.18231578e-02,
,         8.57392848e-02, -8.18344206e-02,  8.36842507e-03,
,         4.00998592e-02,  4.17756364e-02,  1.04569227e-01,
,        -2.85369623e-03,  1.96612384e-02,  5.82248531e-03,
,         1.33426916e-02,  4.50896770e-02, -2.17425618e-02,
,        -1.39383934e-02, -6.87028542e-02, -2.91238283e-03,
,        -3.10636591e-02, -1.05849534e-01,  6.91420212e-02,
,        -4.24087532e-02, -4.67772000e-02, -3.64743397e-02,
,         4.50434908e-02,  6.09990060e-02, -6.56444505e-02,
,        -5.47300931e-03, -1.86086707e-02, -6.31589964e-02,
,        -3.87502834e-02,  3.46755087e-02,  5.55316061e-02,
,         5.21691255e-02,  5.61158992e-02,  1.02072954e-01]], dtype=float32)

[ ]

(1, 384)

The similarity scores between sentences, however, should be similar across different trainings of the same model.

[ ]

tensor([[1.0000, 0.6290, 0.2128],
,        [0.6290, 1.0000, 0.1674],
,        [0.2128, 0.1674, 1.0000]])

[ ]

The embeddings from the ONNX version of the model exhibit similar similiarities between sentences as for the HuggingFace weights.

[98]

tensor([[1.0000, 0.6378, 0.2185],
,        [0.6378, 1.0000, 0.1812],
,        [0.2185, 0.1812, 1.0000]])

[ ]

None None

So we can debug the Java version, we print the tokenized strings here. It will be useful for comparing the ONNX embeddings between Python and Java.

[99]

array([  101,  1996,  4248,  2829,  4419, 14523,  2058,  1996, 13971,
,        3899,  1012,   102], dtype=int64)

[100]

array([  101,  1037,  3435,  4702, 29195,  2058,  1996,  7367, 16454,
,        5649, 19598,  1012,   102], dtype=int64)

[101]

array([  101,  1996,  4937,  2938,  2006,  1996, 13523,  1012,   102],
,      dtype=int64)