Qdrant Hnsw Performance Tuning

Hnsw Performance Tuning

day_2qdrant-examplescourse

alph-notebooks/qdrant-examples / hnsw_performance_tuning.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Day 2: HNSW Tuning and Filtering Optimization

In this tutorial, we'll explore how Qdrant's HNSW algorithm affects search performance and how payload indexes can dramatically improve filtering speed. You'll learn to:

Tune our HNSW configurations
Compare filtering performance with and without payload indexes

1. Installing Required Dependencies

We'll need several libraries to work with Qdrant and perform our performance tests:

[39]

Now let's install the necessary libraries. These packages will enable us to:

datasets: Load and work with the DBpedia dataset
qdrant-client: Interact with our Qdrant vector database
tqdm: Show progress bars during data processing
openai: Generate embeddings for our queries
time - for performance timing

[2]

2. Connecting to Qdrant Cloud

Now we'll establish a connection to our Qdrant cloud instance. Unlike the Day 1 tutorial which used an in-memory database, we're using Qdrant Cloud for this performance testing. This allows us to test with larger datasets and measure real-world performance with cloud infrastructure.

[3]

3. Loading the DBpedia Dataset

We'll use the DBpedia entities dataset, which contains 100K Wikipedia articles with pre-computed embeddings with OpenAI's text-embedding-3-large model with 1536-dimensional vectors (first 1536 dimensions of a text-embedding-3-large embedding), which is ideal for testing HNSW performance on high-dimensional data.

The 100K vectors dataset allow us to see real performance differences and includes titles, text, and categories for filtering tests.

[38]

4. Creating Our Collection

We're starting with m=0 for a specific reason: bulk upload speed. When m=0, Qdrant doesn't build any HNSW connections during indexing, which makes uploading 100K vectors much faster. This is perfect for our testing workflow, where we'll:

Upload data quickly with m=0
Test full scan performance
Update to m=16 and test HNSW performance
Compare the difference

Warning: Don't use this technique on subsequent bulk uploads. Setting m=0 will delete the existing HNSW index. Rebuilding from scratch is slow and resource-intensive.

The strict_mode_config with enabled=False and unindexed_filtering_retrieve=True allows us to test filtering without payload indexes, so we can measure the performance impact of adding indexes later.

Qdrant (Managed) Cloud runs in strict mode by default.

[5]

Created collection: dbpedia_100K

5. Exploring Our Dataset

Before we start uploading data, let's examine the dataset structure to understand what we're working with. This helps us verify the data format and plan our upload strategy.

[6]

Dataset info:
DatasetDict({
    train: Dataset({
        features: ['_id', 'title', 'text', 'text-embedding-3-large-1536-embedding'],
        num_rows: 100000
    })
})

First example (proper access):
{'_id': '<dbpedia:Parabolic_reflector>', 'title': 'Parabolic reflector', 'text': 'A parabolic (or paraboloid or paraboloidal) reflector (or dish or mirror) is a reflective surface used to collect or project energy such as light, sound, or radio waves. Its shape is part of a circular paraboloid, that is, the surface generated by a parabola revolving around its axis. The parabolic reflector transforms an incoming plane wave traveling along the axis into a spherical wave converging toward the focus.', 'text-embedding-3-large-1536-embedding': [-0.02738949842751026, 0.03491175174713135, -0.021398993209004402, -0.018760139122605324, -0.042221687734127045, 0.019897576421499252, -0.024750644341111183, -0.015590478666126728, -0.03924918174743652, 0.0045838737860322, -0.004727949388325214, 0.005251170601695776, 0.008606611751019955, -0.040553443133831024, 0.0018720327643677592, 0.03312218561768532, 0.024614151567220688, -0.004254017025232315, 0.026919357478618622, -0.009835043922066689, -0.00032867208938114345, -0.046255797147750854, -0.08153153210878372, -0.021019848063588142, -0.019017957150936127, 0.018744973465800285, 0.032121241092681885, -0.039340175688266754, -0.05620458722114563, -0.025508934631943703, 0.06970217823982239, 0.011510869488120079, 0.010593336075544357, -0.03545772284269333, -0.040735434740781784, -0.01607578620314598, -0.03160560131072998, -0.06051168218255043, -0.03206057474017143, 0.0011326983803883195, -0.015029342845082283, 0.013573423027992249, -0.02282458171248436, 0.004489087499678135, -0.02534211054444313, 0.037732597440481186, -0.0015601853374391794, 0.029133569449186325, -0.02189946547150612, -0.038005582988262177, 0.005296668037772179, -0.07534386962652206, -0.011723190546035767, 0.05875244736671448, -0.018183836713433266, 0.02467481419444084, 0.029770534485578537, 0.0629078820347786, 0.0020265348721295595, -0.0036852979101240635, 0.04458755627274513, -0.028026463463902473, -0.04301030933856964, 0.0448908731341362, 0.004272974096238613, -0.005967756267637014, 0.0010644521098583937, 0.009569642134010792, -0.023218894377350807, -0.007052113302052021, 0.033516496419906616, -0.01188243180513382, -0.0016511803260073066, 0.026995187625288963, 0.013080532662570477, 0.04200936481356621, 0.03010418266057968, 0.037368617951869965, 0.05380838364362717, -0.03779326379299164, -0.039522167295217514, 0.011093808338046074, 0.021747808903455734, 0.02851177006959915, 0.010676748119294643, 0.03075631521642208, 0.011738356202840805, -0.041281405836343765, 0.04713541641831398, 0.011821769177913666, -0.031120294705033302, 0.0269496887922287, 0.03585203364491463, 0.06230125203728676, 0.08232015371322632, -0.006377233657985926, 0.0031241620890796185, 0.035882364958524704, -0.04813636094331741, 0.034638769924640656, -0.017061565071344376, -0.002250230871140957, -0.006180077791213989, 0.016864409670233727, -0.005505198147147894, -0.050926875323057175, -0.013816076330840588, -0.008667275309562683, 0.016712751239538193, 0.004216102417558432, 0.016667252406477928, -0.011283381842076778, 0.0552036426961422, -0.01841132342815399, -0.02684352919459343, -0.029012242332100868, 0.013436930254101753, -0.020079566165804863, -0.04067477211356163, 0.03136294707655907, -0.032606545835733414, -0.006119414698332548, -0.013262523338198662, 0.018350660800933838, -0.014847353100776672, -0.040068138390779495, 0.01916961558163166, -0.017835022881627083, 0.023052070289850235, -0.013186694122850895, -0.003920368384569883, -0.004951645154505968, -0.012519396841526031, -0.0012786694569513202, 0.009599973447620869, 0.03151460736989975, -0.014938347972929478, 0.018851133063435555, -0.006597138475626707, 0.021762974560260773, 0.008849265053868294, -0.04759038984775543, 0.07146141678094864, -0.006532683502882719, 0.005910884588956833, 0.004530793521553278, -0.022324109449982643, 0.03976482152938843, -0.016909906640648842, -0.02984636463224888, 0.004242642316967249, 0.0026976228691637516, -0.04637712612748146, -0.018183836713433266, 0.016591424122452736, -0.04804536700248718, -0.006126997526735067, 0.028572434559464455, 0.02834494598209858, -0.07382728904485703, -0.008204717189073563, 0.01777435839176178, -0.029057741165161133, 0.048106029629707336, 0.012549729086458683, 0.023598039522767067, 0.0030976219568401575, -0.02403784915804863, -0.040917422622442245, 0.0055734445340931416, 0.0067942943423986435, 0.010343099944293499, 0.023689035326242447, -0.01023693848401308, -0.07231070101261139, -0.018001846969127655, -0.007802822161465883, 0.01768336445093155, 0.0060739172622561455, -0.011556366458535194, -0.031059630215168, -0.05647757276892662, 0.036246348172426224, 0.0033573368564248085, 0.02450799010694027, 0.03788425773382187, -0.00373079557903111, 0.004481504205614328, -0.05656856670975685, -0.02470514550805092, -0.026054905727505684, 0.004856858868151903, 0.06069367378950119, -0.03336483612656593, 0.047469064593315125, -0.0872945487499237, -0.022035958245396614, -0.010790491476655006, -0.02748049423098564, -0.007143108639866114, 0.0025023629423230886, 0.025296613574028015, -0.032879531383514404, 0.0034672890324145555, -0.04304064065217972, 0.05032024160027504, 0.054384686052799225, -0.018547816202044487, 0.003628426231443882, -0.03524539992213249, 0.025463437661528587, -0.050835881382226944, -8.761824574321508e-05, 0.0007848319946788251, 0.023158229887485504, 0.004128898493945599, 0.0009094761917367578, -0.02488713525235653, 0.008735520765185356, -0.02629755809903145, 0.01953359693288803, -0.03115062601864338, 0.022020792588591576, 0.03345583379268646, -0.009440732188522816, 0.00981987826526165, -0.041584718972444534, -0.004617996979504824, -0.053838714957237244, -0.018274832516908646, -0.0674576386809349, 0.041281405836343765, 0.015355408191680908, -0.03104446455836296, -0.02403784915804863, -0.00958480779081583, 0.0037497528828680515, -0.019852079451084137, 0.0021611314732581377, 0.01842649094760418, 0.006437897216528654, -0.02001890353858471, -0.03136294707655907, -0.003821790451183915, 0.028663428500294685, -0.027374332770705223, 0.024917468428611755, -0.06909554451704025, -0.05235246568918228, -0.026585709303617477, -0.03776292875409126, 0.03133261576294899, -0.05665956065058708, 0.05399037525057793, -0.004382926505059004, 0.006305195856839418, 0.00523600447922945, 0.01982174627482891, 0.01094973273575306, 0.04237334430217743, 0.03266720846295357, -0.04492120444774628, 0.02796580083668232, -0.021641647443175316, -0.02619139850139618, 0.02112600952386856, -0.014824604615569115, -0.026798030361533165, -0.011290964670479298, 0.023719366639852524, -0.008803767152130604, 0.03560938313603401, 0.02347671240568161, -0.041008420288562775, 0.035033080726861954, 0.0023791403509676456, -0.010403763502836227, 0.014089060947299004, 0.042525000870227814, -0.019017957150936127, 0.027616987004876137, -0.056447237730026245, 0.010835989378392696, -0.013634085655212402, 0.02282458171248436, 0.013406598940491676, 0.010001868940889835, 0.014195222407579422, 0.020428379997611046, 0.00539903761819005, -0.022506099194288254, -0.031575269997119904, -0.044648218899965286, 0.07182539999485016, 0.04564916342496872, -0.022035958245396614, -0.010032200254499912, -0.028071962296962738, -0.004060652572661638, 0.044648218899965286, -0.001304261852055788, 0.023082401603460312, -0.04021979495882988, 0.03442644700407982, -0.01750137470662594, -0.028269117698073387, -0.007939314469695091, 0.03733828663825989, -0.0172890517860651, -0.034092798829078674, 0.0561135895550251, -0.015014177188277245, -0.011146889068186283, -0.016621755436062813, 0.002627481007948518, 0.0026312724221497774, -0.0009071065578609705, -0.04728707671165466, -0.010555421933531761, -0.010373431257903576, -0.032970525324344635, 0.01785018853843212, -0.0794689804315567, 0.00869002379477024, 0.039704158902168274, 0.0065933470614254475, 0.059996046125888824, -0.01795634999871254, 0.008341209031641483, 0.04674110561609268, -0.009016089141368866, -0.004140273202210665, 0.04589181765913963, -0.020549707114696503, -0.006745005492120981, 0.007855902425944805, -0.007567751687020063, 0.007215146441012621, -0.030741147696971893, -0.0025440689641982317, 0.016758248209953308, 0.02208145707845688, -0.01374024711549282, -0.017744027078151703, -0.011996176093816757, -0.06454579532146454, -0.0107980752363801, -0.01944260112941265, 0.015370573848485947, 0.000463979784399271, 0.0006545005599036813, -0.020792359486222267, -0.07588984072208405, 0.04555816948413849, 0.039886146783828735, -0.017653033137321472, 5.05034186062403e-05, -0.05526430532336235, 0.01552223227918148, 0.012178165838122368, -0.026449216529726982, -0.059723060578107834, -0.028951579704880714, -0.014104226604104042, -0.03172692656517029, -0.01374024711549282, 0.003110891906544566, -0.0062104095704853535, 0.01619711145758629, -0.012951623648405075, 0.011632195673882961, 0.008811349980533123, -0.029012242332100868, 0.01916961558163166, 0.03767193481326103, 0.03077148087322712, 0.041099414229393005, -0.031666263937950134, -0.007745950482785702, 0.002784826559945941, -0.004708991851657629, 0.01757720299065113, -0.010540255345404148, -0.004568708129227161, 0.02384069375693798, 0.037823595106601715, -0.012375322170555592, -0.04516385868191719, -0.03924918174743652, -0.026403719559311867, 0.01098764780908823, -0.009084335528314114, 0.004197144880890846, -0.01514308713376522, 0.0060928743332624435, -0.013436930254101753, 0.016667252406477928, 0.017471041530370712, -0.01309569925069809, 0.015431237407028675, 0.029421720653772354, 0.047924041748046875, -0.002581983571872115, -0.0024435953237116337, 0.005319416988641024, 0.009660637006163597, -0.03542739152908325, 0.036792315542697906, -0.016667252406477928, 0.01121513545513153, 0.04434490203857422, -0.028875751420855522, 0.03500274941325188, 0.02617623284459114, 0.016758248209953308, 0.03852122277021408, -0.010631250217556953, 0.04871266335248947, -0.0187904704362154, -0.01299712061882019, -0.028178121894598007, 0.014544036239385605, 0.02843594178557396, -0.019700421020388603, -0.019017957150936127, 0.009675802662968636, 0.04856100678443909, -0.0015355407958850265, 0.017258720472455025, -0.030437830835580826, -0.041857704520225525, 0.007962063886225224, 0.013884322717785835, -0.004458755720406771, -0.018562981858849525, 0.002204733435064554, 0.0010919400956481695, 0.024341166019439697, 0.011753522790968418, -0.0027753477916121483, -0.022005626931786537, 0.02449282445013523, 0.017319384962320328, -0.011298547498881817, -0.022187616676092148, -0.006142163183540106, 0.0036132603418082, -0.03982548415660858, 0.07218937575817108, 0.005922258831560612, 0.0105705875903368, 0.012739301659166813, -0.05617425590753555, -0.021732641384005547, -0.010964899323880672, -0.004083401057869196, -0.007700453046709299, -0.024553487077355385, -0.014885267242789268, 0.015529815107584, 0.0038691838271915913, 0.02523595094680786, 0.01464261393994093, -0.056447237730026245, 0.051199860870838165, 0.013641669414937496, -0.011988593265414238, -0.0021706102415919304, -0.0033402752596884966, -0.04379893094301224, -0.003890036838129163, -0.06105765327811241, -0.011245466768741608, 0.020155394449830055, -0.008758270181715488, -0.01853265054523945, -0.048682332038879395, -0.0014853039756417274, 0.005239796359091997, -0.03715629503130913, -0.03482075780630112, 0.031575269997119904, 0.007764907553792, -0.0029971483163535595, 0.03040749952197075, -0.0032094698399305344, -0.00921324547380209, 3.450820076977834e-05, 0.006945952773094177, -0.009895707480609417, 0.052413128316402435, -0.00015059200813993812, -0.014953513629734516, -0.017880519852042198, -0.022566763684153557, -0.019518429413437843, -0.010858737863600254, 0.03573070839047432, 0.031180957332253456, 0.07807371765375137, 0.013224608264863491, -0.004015155136585236, -0.01627294160425663, 0.03251555189490318, -0.023704200983047485, 0.06202826648950577, 0.029694706201553345, -0.02497813105583191, -0.007165857125073671, 0.016758248209953308, 0.023491879925131798, -0.01599995605647564, 0.02778381109237671, -0.02907290682196617, 0.025326944887638092, -0.008144053630530834, -0.02738949842751026, 0.006415148265659809, -0.04552783817052841, 0.009152581915259361, -0.013262523338198662, -0.014415126293897629, 0.005838846787810326, -0.022794250398874283, 0.03645866736769676, -0.01104072853922844, 0.05638657510280609, -0.0021933589596301317, 0.027844473719596863, -0.008530782535672188, 0.00023862493981141597, -0.03039233386516571, 0.043738268315792084, 0.012504231184720993, -0.013072949834167957, 0.000957817304879427, -0.010790491476655006, 0.048227354884147644, 0.025417940691113472, -0.02385585941374302, -0.0370653010904789, 0.004375343676656485, 0.006551641039550304, 0.006005670875310898, 0.01145778875797987, -0.0029061532113701105, 0.010244522243738174, -0.020443545654416084, -0.0062748645432293415, -0.016803745180368423, 0.07728509604930878, 0.006782920099794865, 0.023264391347765923, 0.07546519488096237, 0.017653033137321472, 0.008667275309562683, -0.007787656504660845, -0.008075807243585587, 0.010441677644848824, -0.0011507077142596245, -0.01145778875797987, 0.0011867266148328781, -0.02318856306374073, 0.02963404171168804, -0.024811306968331337, 0.021747808903455734, -0.0019772457890212536, 0.03721696138381958, -0.0041099414229393005, -0.0034255830105394125, -0.005907092709094286, 0.01663692109286785, -0.022293778136372566, 0.027525991201400757, 0.005744060035794973, -0.013626502826809883, -0.006312779150903225, 0.003927951212972403, -0.015969624742865562, 0.023734532296657562, 0.008068224415183067, -0.008591446094214916, -0.009789546951651573, 0.01374024711549282, 0.009948788210749626, -0.029785700142383575, 0.027616987004876137, 0.040280457586050034, 0.004159230273216963, -0.006312779150903225, -0.027905136346817017, 0.038763877004384995, 0.008712772279977798, 0.028466273099184036, 0.0196397565305233, 0.014779106713831425, 0.014187638647854328, -0.04112974554300308, 0.021853968501091003, 0.02086818963289261, 0.011806602589786053, 0.03739894926548004, -0.0008033153717406094, -0.04216102138161659, -0.014104226604104042, 0.04640745744109154, 0.015696639195084572, -0.044102247804403305, -0.0302103441208601, 0.01380091067403555, -0.04983493685722351, -0.0396738238632679, -0.010926984250545502, 0.02440182864665985, -0.014308965764939785, 0.0023260600864887238, -0.012595226056873798, -0.025645427405834198, -0.008561113849282265, -0.02030705288052559, -0.00892509426921606, -0.023886190727353096, 0.01505209133028984, -0.0491979718208313, -0.019958239048719406, -0.013831241987645626, 0.033698488026857376, -0.015082423575222492, -0.03049849532544613, 0.029421720653772354, -0.010502341203391552, -0.06527375429868698, -0.028132624924182892, -0.013891905546188354, 0.03843022882938385, 0.023446381092071533, 0.0018805635627359152, -0.01926061138510704, 0.00963030569255352, 0.005854012444615364, -0.03263687714934349, 0.012185748666524887, 0.036337342113256454, -0.05171550065279007, 0.010487175546586514, 0.021489989012479782, -0.023067235946655273, -0.025417940691113472, 0.01486251875758171, 0.021019848063588142, -0.009175330400466919, -0.016151614487171173, -0.0041099414229393005, 0.014726025983691216, 0.024902300909161568, -0.008856847882270813, -0.017986681312322617, 0.03882453963160515, 0.007753533311188221, -0.02347671240568161, -0.00163127516862005, 0.00240757642313838, -0.044011253863573074, 0.012678638100624084, -0.020352551713585854, 0.007385761942714453, -0.02000373788177967, 0.04385959729552269, 0.0029914609622210264, 0.0020208475179970264, -0.028269117698073387, 0.006782920099794865, -0.03245488926768303, -0.021671978756785393, 0.04461788758635521, -0.035670045763254166, -0.009372486732900143, 0.01739521324634552, 0.01841132342815399, -0.017804691568017006, -0.013080532662570477, 0.024750644341111183, 0.0014435979537665844, 0.01674308255314827, 0.01637910306453705, -0.005641690921038389, 0.027450162917375565, -0.055051982402801514, -0.033152516931295395, -0.019518429413437843, -0.06497044116258621, -0.01647009700536728, -0.02534211054444313, -0.00640756543725729, -0.00279240938834846, -0.03524539992213249, -0.027177177369594574, -0.014346879906952381, -0.02720750868320465, -0.04170604795217514, 0.010911818593740463, -0.009903290309011936, -0.0028739257249981165, -0.009516561403870583, -0.02385585941374302, 0.010267270728945732, -0.014604699797928333, -0.028390444815158844, -0.019518429413437843, -0.017046399414539337, 0.014066312462091446, -0.036246348172426224, 0.0009497604332864285, 0.01234498992562294, 0.026858694851398468, -0.013270106166601181, -0.001313740503974259, 0.010873904451727867, -0.04367760568857193, 0.02141415886580944, 0.020807527005672455, 0.01906345598399639, 0.0058691781014204025, 0.013725081458687782, 0.04771171882748604, 0.02675253339111805, 0.005122261121869087, -0.03545772284269333, -0.03597336262464523, 0.015742138028144836, -0.03994680941104889, 0.01029001921415329, -0.008614194579422474, -0.01127579901367426, -0.012087170965969563, 0.03394113853573799, -0.04740840196609497, 0.033061522990465164, -0.02280941605567932, 0.00252321595326066, -0.00011812763841589913, -0.01352034229785204, 0.02963404171168804, -0.0159847903996706, -0.010676748119294643, -0.014574367552995682, 0.006779128219932318, 0.02393168769776821, 0.013762995600700378, -0.007711827289313078, -0.004674868658185005, 0.011723190546035767, -0.030907971784472466, 0.011078642681241035, -0.026858694851398468, 0.02553926780819893, -0.024644482880830765, 0.009410400874912739, -0.0015819862019270658, 0.036155350506305695, -0.027541156858205795, 0.04173637926578522, 0.0035848242696374655, -0.0006293821497820318, -0.04734773933887482, -0.049622613936662674, 0.018638812005519867, 0.01533265970647335, 0.011116557754576206, -0.02132316492497921, 0.025872915983200073, 0.023052070289850235, -0.0026142108254134655, -0.017273886129260063, -0.01572697050869465, 0.015029342845082283, 0.00617628637701273, -0.05444534868001938, 0.01589379645884037, 0.008485284633934498, -0.00481515284627676, -0.019958239048719406, -0.05638657510280609, -0.034365784376859665, -0.02431083470582962, -0.025463437661528587, 0.014142141677439213, -0.00017452558677177876, 0.00981987826526165, 0.022035958245396614, 0.03767193481326103, 0.024068180471658707, -0.013194276951253414, -0.0055734445340931416, -0.004735532216727734, 0.021929798647761345, 0.00847011897712946, 0.0007753533427603543, -0.027738312259316444, -0.024144010618329048, 0.05435435473918915, -0.0019450184190645814, -0.004443589597940445, 0.001195257413201034, 0.003863496473059058, -0.04143306240439415, 0.04768138751387596, -0.03479042649269104, -0.025311779230833054, -0.003361128270626068, -0.02552410028874874, 0.036337342113256454, 0.018911797553300858, 0.02534211054444313, -0.0191847812384367, -0.015355408191680908, -0.009531727991998196, -0.034274786710739136, -0.009137416258454323, -0.005414203274995089, 0.01210991945117712, -0.049319297075271606, -0.024917468428611755, 0.03654966503381729, -0.03330417349934578, 0.012322241440415382, 0.002917527686804533, -0.004197144880890846, 0.018441656604409218, -0.010631250217556953, 0.019093787297606468, -0.03148427605628967, -0.009471064433455467, 0.010123195126652718, -0.03439611569046974, -0.004128898493945599, 0.030452998355031013, 0.02365870401263237, 0.013254940509796143, 0.01257247757166624, 0.03721696138381958, -0.02907290682196617, -0.01935160532593727, -0.025493768975138664, -0.041857704520225525, 0.027328835800290108, 0.046346794813871384, 0.009084335528314114, 0.011086225509643555, 0.0340624675154686, 0.016758248209953308, -0.013186694122850895, 0.004284348338842392, 0.032121241092681885, 0.006202826742082834, -0.017926016822457314, -0.02226344682276249, 0.011981010437011719, 0.0005374393076635897, 0.0018445447785779834, 0.015878628939390182, 0.022399939596652985, 0.0017099479446187615, -0.036792315542697906, 0.0042464337311685085, -0.004602830857038498, 0.03178759291768074, -0.010873904451727867, 0.012708970345556736, -0.0065705981105566025, -0.009706134907901287, 0.01927577704191208, 0.016151614487171173, -0.05620458722114563, 0.03676198422908783, 0.012314658612012863, -0.009986702352762222, -0.03542739152908325, 0.026782864704728127, 0.05656856670975685, 0.03621601685881615, 0.016106117516756058, -0.024280503392219543, 0.024841638281941414, 0.026858694851398468, -0.026600874960422516, -0.07552585750818253, -0.008060641586780548, 0.03284920006990433, 0.015544981695711613, -0.021641647443175316, -0.0021516529377549887, -0.0005488136666826904, -0.0039507001638412476, -0.027268171310424805, 0.038581885397434235, -0.014324131421744823, -0.020989516749978065, -0.011010396294295788, -0.006892872042953968, 0.01692507229745388, -0.0005199037841521204, -0.02086818963289261, -0.012481482699513435, 0.028587600216269493, -0.005835055373609066, 0.041099414229393005, 0.033152516931295395, -0.0004222737334202975, -0.012921291403472424, 0.01627294160425663, -0.03397146984934807, 0.04759038984775543, -0.009129832498729229, 0.05013825371861458, 0.007704244460910559, 0.019154449924826622, 0.030073851346969604, 0.007374387700110674, -0.039340175688266754, 0.0401894636452198, 0.02010989747941494, 0.04704442247748375, -0.004530793521553278, 0.01005494873970747, -0.028921248391270638, -0.012701387517154217, 0.006081500090658665, 0.026070071384310722, 0.024432161822915077, 0.024265335872769356, -0.00841703824698925, 0.0474993959069252, 0.03733828663825989, 0.02142932638525963, 0.017653033137321472, -0.030255841091275215, -0.01010802946984768, 0.014999011531472206, -0.010744994506239891, 0.01897246018052101, 0.012671055272221565, 0.017137393355369568, -0.013118447735905647, 0.02356770820915699, -0.025842582806944847, -0.057569511234760284, 0.011336461640894413, -0.010365848429501057, 0.025008462369441986, 0.016106117516756058, 0.0002462078700773418, -0.03348616510629654, -0.044466231018304825, 0.009471064433455467, -0.006066333968192339, 0.007886234670877457, -0.009046420454978943, 0.041190408170223236, 0.003129849210381508, -0.02320372872054577, 0.013831241987645626, 0.03048332966864109, -0.0003400464775040746, 0.0047469064593315125, -0.010858737863600254, 0.003562075551599264, -0.040341123938560486, 0.005907092709094286, -0.030923139303922653, 0.0015772469341754913, -0.02020089328289032, -0.018669143319129944, 0.010919401422142982, 0.05496098846197128, 0.012648306787014008, 0.06685100495815277, -0.003071081591770053, 0.019761083647608757, 0.027935469523072243, 0.03943117335438728, 0.03573070839047432, -0.005262544844299555, -0.01417247299104929, -0.008917511440813541, -0.012663472443819046, 0.0034066257067024708, 0.0006971544935368001, -0.02599424123764038, 0.001672981190495193, -0.013368683867156506, -0.02758665382862091, 0.007143108639866114, 0.01628810726106167, 0.004356386139988899, 0.009266325272619724, -0.011389542371034622, -0.013072949834167957, -0.0053497483022511005, -0.026161065325140953, -0.002953546354547143, -0.03218190371990204, -0.00897817499935627, 0.019033122807741165, -0.038581885397434235, 0.03506341204047203, -0.009152581915259361, 0.007726992946118116, -0.004163021687418222, 0.013914654031395912, 0.010934567078948021, 0.018214168027043343, -0.052140142768621445, -0.020746862515807152, 0.039704158902168274, 0.029998023062944412, 0.0021383827552199364, 0.014498538337647915, 0.06478844583034515, 0.043252963572740555, -0.016970569267868996, 0.02030705288052559, 0.0015544980997219682, 0.028572434559464455, -0.027814142405986786, 0.02385585941374302, -0.02264259196817875, -0.0036416961811482906, 0.0629078820347786, -0.011010396294295788, 0.006062542553991079, -0.027616987004876137, 0.0048303185030817986, -0.020701365545392036, 0.0004232215869706124, 0.024053014814853668, -0.02729850448668003, -0.05835813656449318, -0.0035412225406616926, -0.03086247481405735, -0.02514495514333248, 0.0017630283255130053, -0.014263467863202095, 0.005550695583224297, -0.001238859142176807, -0.019412269815802574, -0.012898542918264866, -0.00511846924200654, -0.03448710963129997, -0.007954481057822704, -0.01628810726106167, -0.05414203181862831, -0.030634988099336624, 0.029512716457247734, -0.02394685335457325, -0.029967691749334335, -0.03533639758825302, 0.013391432352364063, 0.004663494415581226, 0.021656813099980354, -0.02224828116595745, -0.009675802662968636, 0.04052311182022095, 0.009554476477205753, 0.03321317955851555, -0.0017781942151486874, 0.04610414057970047, -0.043434951454401016, 0.01757720299065113, 0.003370607038959861, -0.021368661895394325, 0.010426511988043785, 0.015271996147930622, 0.022566763684153557, 0.037368617951869965, 0.04728707671165466, 0.015453985892236233, -0.001555445953272283, -0.015302328392863274, -0.022672923281788826, -0.010418929159641266, -0.010418929159641266, -0.017425544559955597, -0.00032345883664675057, -0.009266325272619724, 0.0030369586311280727, 0.067336305975914, -0.018365826457738876, -0.010744994506239891, 0.011670110747218132, -0.006684341933578253, 0.04525485262274742, 0.020428379997611046, 0.021353496238589287, -0.0033099434804171324, -0.0053307912312448025, 0.00016990475705824792, 0.020079566165804863, -0.004894773475825787, 0.0003170607378706336, -0.006020836532115936, 0.05744818598031998, -0.02535727620124817, 0.024053014814853668, 0.022415105253458023, 0.05432402342557907, -0.002115634037181735, 0.00040521216578781605, -0.022142119705677032, -0.036337342113256454, -0.03433545306324959, -0.01028243638575077, -0.02309756726026535, -0.006540266331285238, 0.01730421744287014, -0.021914632990956306, -0.01663692109286785, -0.009880541823804379, 0.018653977662324905, -0.013907071202993393, 0.032121241092681885, -0.008667275309562683, 0.009008506312966347, 0.022308943793177605, -0.03421412408351898, 0.015939293429255486, 0.0006625574314966798, 0.007764907553792, -0.02271842211484909, 0.015226499177515507, -0.01757720299065113, 0.005922258831560612, 0.05365672707557678, 0.012708970345556736, 0.008326043374836445, 0.02047387696802616, 0.030240675434470177, -0.010835989378392696, -0.007654955610632896, 0.0018369618337601423, 0.010638833977282047, -0.021262500435113907, -0.037641603499650955, 0.022005626931786537, -0.004762072116136551, -0.004538376349955797, 0.003402834292501211, 0.02544827200472355, -0.01869947463274002, 0.03500274941325188, 0.008401872590184212, -0.023294722661376, 0.010244522243738174, 0.0024379079695791006, 0.001664450392127037, 0.008591446094214916, -0.00959239061921835, -0.020140228793025017, 0.007518462836742401, -0.01906345598399639, -0.0010056844912469387, 0.01992790773510933, 0.03852122277021408, 0.009835043922066689, 0.02224828116595745, -0.00523600447922945, -0.03154493868350983, -0.004447381477802992, -0.00446633854880929, -0.019017957150936127, 0.014005648903548717, 0.004917521961033344, -0.007040739059448242, -0.011776271276175976, -0.011670110747218132, -0.024007517844438553, -0.009918455965816975, -0.0044284239411354065, -0.015590478666126728, -0.007662538439035416, 0.024811306968331337, 0.016576258465647697, 0.008909928612411022, -0.03227289766073227, 0.01323977392166853, -0.011859683319926262, -0.00012653993326239288, 0.011708024889230728, 0.01692507229745388, -0.027616987004876137, 0.059237752109766006, 0.0224454365670681, -0.03843022882938385, 0.03906719386577606, -0.009910873137414455, -0.005880552809685469, 0.011078642681241035, 0.01258764322847128, -0.004542167764157057, 0.004034112207591534, -0.01486251875758171, 0.023628370836377144, 0.019002791494131088, -0.01080565806478262, 0.016712751239538193, 0.0159847903996706, -0.02751082554459572, -0.016515593975782394, -0.011783854104578495, 0.013110864907503128, 0.018305163830518723, 0.001071087084710598, -0.021459657698869705, -0.008674858137965202, 0.015939293429255486, -0.0034995165187865496, 0.01581796631217003, 0.014362046495079994, 0.006566806696355343, 0.0006113727577030659, -0.029118403792381287, 0.023355387151241302, 0.0030331669840961695, 0.0063696508295834064, 0.05292876437306404, -0.011905181221663952, 0.05814581364393234, -0.021444492042064667, 0.014892850071191788, 0.03788425773382187, -0.005395245738327503, -0.01692507229745388, 0.007848319597542286, 0.03095347061753273, -0.015544981695711613, 0.0278596393764019, 0.0039507001638412476, 0.011981010437011719, -0.013452095910906792, 0.010600918903946877, 0.00012535510177258402, -0.02647954784333706, 0.04862166941165924, 0.038854870945215225, -0.015438820235431194, -0.019488098099827766, -0.011245466768741608, 0.013254940509796143, 0.007848319597542286, -0.02666153945028782, -0.006346902344375849, -0.004921313375234604, 0.020079566165804863, 0.016515593975782394, 0.023127898573875427, 0.01304261852055788, 0.02095918543636799, 0.03903685882687569, -0.0146805290132761, -0.052898433059453964, -0.01589379645884037, -0.03900652751326561, -0.0072492691688239574, -0.003639800474047661, -0.029876695945858955, 0.023598039522767067, -0.02124733477830887, 0.0015611331909894943, 0.011366793885827065, 0.009463481605052948, 0.020322220399975777, 0.03630701079964638, 0.014308965764939785, 0.007825571112334728, -0.00066066172439605, -0.006688133347779512, -0.013308020308613777, -0.018259664997458458, 0.003135536564514041, -0.009569642134010792, 0.026115568354725838, -0.007901400327682495, 0.019093787297606468, -0.006062542553991079, -0.0003696672501973808, 0.020170561969280243, 0.000715163943823427, -0.012822713702917099, -0.004288139753043652, -0.0057137287221848965, 0.024068180471658707, 0.013368683867156506, 0.028648262843489647, -0.0013516551116481423, 0.0020227432250976562, 0.00605875113978982, -0.011799019761383533, 0.019215114414691925, 0.016712751239538193, -0.008113722316920757, 0.03218190371990204, -0.03239422291517258, 0.02496296539902687, -0.02001890353858471, -0.015370573848485947, -0.03958282992243767, -0.0105705875903368, -0.00940281804651022, 0.030346836894750595, 0.05062355846166611, 0.04434490203857422, -0.010980064980685711, 0.02133833058178425, 0.01804734393954277, 0.025948744267225266, -0.015393323265016079, 0.04291931539773941, -0.0056682308204472065, -0.029800865799188614, -0.01860848069190979, 0.005262544844299555, -0.016970569267868996, -0.0027071016374975443, -0.03233356028795242, -0.019381938502192497, 0.01860848069190979, 0.0034066257067024708, -0.012557311914861202, 0.007893817499279976, -0.04813636094331741, 0.03160560131072998, -0.005717520136386156, -0.025190452113747597, -0.003347858088091016, 0.00264075119048357, -0.007817988283932209, 0.00940281804651022, -0.0018625541124492884, -0.03973449021577835, -0.011890014633536339, 0.011905181221663952, -0.019321274012327194, -0.0001368479715893045, 0.0422823503613472, 0.0012786694569513202, 0.00011475087376311421, -0.0064530628733336926, -0.03946150466799736, -0.02039804868400097, 0.03882453963160515, 0.013831241987645626, -0.0005454961210489273, 0.018487153574824333, 0.018775304779410362, 0.026691870763897896, -0.0007421780610457063, 0.01333076972514391, 0.018380992114543915, 0.03309185430407524, 0.03579137101769447, 0.006263489834964275, -0.002976295305415988, -0.037368617951869965, -0.03518473729491234, -0.01619711145758629, 0.03991647809743881, 0.012928875163197517, -0.01417247299104929, 0.003010418266057968, 0.011730773374438286, -0.0037345869932323694, 0.016970569267868996, 0.028572434559464455, -1.4491963156615384e-05, 0.013376266695559025, 0.003347858088091016, 0.002030326286330819, -0.001514687784947455, -0.01103314571082592, -0.025599930435419083, -0.007347847335040569, -0.013163944706320763, 0.03776292875409126, -0.0031999913044273853, -0.0008090025512501597, -0.0013848303351551294, 0.011586698703467846, -0.018729805946350098, 0.007287183776497841, 0.019002791494131088, 0.011518452316522598, 0.009054004214704037, 0.021580982953310013, 0.021171506494283676, 0.02635822258889675, -0.035882364958524704, -0.0003594776790123433, -0.011541200801730156, -0.020595204085111618, -0.0013781952438876033, -0.00217440165579319, -0.03949183598160744, 0.009622722864151001, 0.020898520946502686, 0.023294722661376, 0.009038837626576424, -0.039795152842998505, 0.0452851839363575, -0.026813197880983353, 0.0051298439502716064, 0.009038837626576424, -0.013998066075146198, -0.004223685245960951, 0.008576279506087303, -0.014043563976883888, -0.022308943793177605, -0.029315559193491936, 0.023537376895546913, 0.017076730728149414, 0.01853265054523945, 0.02161131612956524, -0.020792359486222267, -0.00039644440403208137, -0.017152559012174606, 0.0004677712277043611, -0.041493725031614304, -0.002483405638486147, 0.015924127772450447, 0.029512716457247734, -0.014475789852440357, 0.04243400692939758, -0.006502351723611355, -0.012557311914861202, -0.016242610290646553, 0.0025535474997013807, -0.00869760662317276, -0.050168585032224655, 0.022945908829569817, 0.023734532296657562, -0.02470514550805092, -0.002204733435064554, -0.003939325921237469, -0.024720311164855957, -0.019017957150936127, -0.010623667389154434, 0.0063696508295834064, -0.014938347972929478, -0.01121513545513153, 0.023340221494436264, 0.028678594157099724, -0.04947095364332199, -0.004515627399086952, -0.0409780889749527, 0.011715607717633247, 0.01285304594784975, -0.019154449924826622, 0.017471041530370712, 0.029512716457247734, -0.015651142224669456, -0.020807527005672455, 0.003992406185716391, 0.018183836713433266, 0.006305195856839418, 0.005876761395484209, -0.001638858113437891, -0.005103303585201502, -0.014058729633688927, -0.015484318137168884, 0.007730784825980663, -0.004280556924641132, -0.003827477805316448, 0.0022881454788148403, 0.014627448283135891, 0.019230280071496964, 0.05210981145501137, 0.005292876623570919, -0.03748994693160057, 0.005842638202011585, 0.009099501185119152, -0.008545948192477226, -0.029482383280992508, 0.04701409116387367, -0.048500340431928635, 0.0375809408724308, 0.007321306969970465, -0.010608501732349396, -0.009478647261857986, 0.013778161257505417, -0.00804547592997551, 0.01721322350203991, 0.029254896566271782, -0.02907290682196617, 0.01374782994389534, -0.0038691838271915913, -0.006926995236426592, -0.019472932443022728, 0.007268226705491543, 0.015067257918417454, -0.0006763014825992286, 0.019503263756632805, 0.018092840909957886, -0.032788537442684174, 0.03657999634742737, -0.010009451769292355, 0.009895707480609417, -0.009175330400466919, -0.043252963572740555, 0.0017279573949053884, 0.005804723594337702, 0.001999994507059455, -0.007620832417160273, -0.008955425582826138, -0.027798976749181747, 0.022051123902201653, 0.015757303684949875, 0.018760139122605324, 0.007734576240181923, -0.028375277295708656, -0.02021605893969536, 0.02458381839096546, -0.010176275856792927, -0.009023671969771385, -0.028936414048075676, 0.011010396294295788, -0.018911797553300858, -0.020792359486222267, -0.06272589415311813, -0.038945864886045456, 0.025023628026247025, 0.007359221577644348, 0.010403763502836227, -0.023825528100132942, 0.0035146824084222317, -0.004894773475825787, 0.0020530750043690205, 0.005562070291489363]}

Dataset features:
{'_id': Value('string'), 'title': Value('string'), 'text': Value('string'), 'text-embedding-3-large-1536-embedding': List(Value('float64'))}

Available columns:
['_id', 'title', 'text', 'text-embedding-3-large-1536-embedding']

6. Bulk Uploading Points

We're uploading 100K vectors in large batches (10K each) to speed up the process. With m=0, the upload is much faster since Qdrant isn't building HNSW connections during indexing.

Warning: Don't use this technique on subsequent bulk uploads. Setting m=0 will delete the existing HNSW index. Rebuilding from scratch is slow and resource-intensive.

The payload includes fields we'll use for filtering tests: length (text length) and has_numbers (a boolean flag). These will let us test different filter types later.

[7]

Uploading 100000 points in batches of 10000

Uploading points: 100%|██████████| 10/10 [04:58<00:00, 29.87s/it]

Upload completed! Total points uploaded: 100000

7. Updating to HNSW Configuration

Now we'll update the collection to use m=16, which builds HNSW connections between vectors. This should dramatically improve search speed by creating a navigable graph structure for vector search. Search becomes near‑logarithmic instead of linear scanning.

The m=16 parameter means each node connects to 16 nearest neighbors, creating a balance between search speed and index size.

[8]

HNSW indexing enabled with m=16

Now it will take some time to build the HNSW index. You can check whether it's fully built and optimized by, for example, looking at the collection status — once it changes from YELLOW to GREEN, the process is complete.

[19]

<CollectionStatus.GREEN: 'green'>

However, you can start querying the collection right away!

8. Creating a Test Query

We're using OpenAI to generate an embedding for our query. We must use the same model (text-embedding-3-large) and dimensions (1536, first 1536 dimensions out of 3072) that were used to create our dataset embeddings.

We've provided the embedding of the test query for the experiments, so you don't have to use the OpenAI API.

[17]

[18]

Embedding dimensions: 1536
First 5 values: [-0.012367082759737968, -0.01300355140119791, -0.003559330478310585, -0.0013035375159233809, 0.024401243776082993]

9. Performing Similarity Search

After all vectors are indexed, we'll test the baseline performance with m=16.

[20]

Running baseline performance test...
Warming up caches...
Average search time: 62.10ms
Search times: ['62.15ms', '61.49ms', '62.65ms']
Found 10 results
Top result: 'A.I. Artificial Intelligence' (score: 0.5392)

The first time you run a query, Qdrant may need to load parts of the index from disk into memory, which can make it slower. After that, those parts stay cached in memory, so repeated queries are much faster. But if your machine is low on memory or you wait too long, the system might remove that cached data, and the process would repeat.

10. Testing Filtering Without Indexes

Now we'll test filtering performance without any payload indexes. This forces Qdrant to scan through all 100K vectors and check each one against the filter condition.

We're comparing the search time with and without a filter to see the overhead of full scan filtering.

[35]

Testing filtering without payload indexes
Filtered search (WITHOUT index): 492.02ms
Individual times: ['473.81ms', '504.70ms', '497.55ms']
Overhead vs baseline: 429.92ms
Found 10 matching results
Top result: 'Cyc (/ˈsaɪk/) is an artificial intelligence project that attempts to assemble a comprehensive ontology and knowledge base of everyday common sense knowledge, with the goal of enabling AI applications to perform human-like reasoning.The project was started in 1984 by Douglas Lenat at MCC and is developed by the Cycorp company.Parts of the project are released as OpenCyc, which provides an API, RDF endpoint, and data dump under an open source license.'
Score: 0.3030

11. Creating Payload Indexes

Now we'll create a full text index on our text payload field. This should dramatically improve filtering performance by allowing Qdrant to quickly locate matching records instead of scanning all vectors.

Warning: The Filterable HNSW index (Qdrant’s native vector index structure designed for vector search with filtering) is built only if the payload indices are created before the HNSW index is built.

[36]

Payload index created for 'text' field

12. Testing Filtering With Indexes

Now we'll test the same filter query, but this time with the payload index in place. The performance difference should be dramatic - indexed filtering should be much faster than the full scan we just tested.

This comparison shows the real-world impact of adding indexes to your vector search engine.

[37]

Testing filtering WITH payload indexes...
Filtered search (WITH index): 73.49ms
Individual times: ['107.42ms', '56.56ms', '56.49ms']
Overhead vs baseline: 11.39ms
Found 10 matching results
Top result: 'Cyc (/ˈsaɪk/) is an artificial intelligence project that attempts to assemble a comprehensive ontology and knowledge base of everyday common sense knowledge, with the goal of enabling AI applications to perform human-like reasoning.The project was started in 1984 by Douglas Lenat at MCC and is developed by the Cycorp company.Parts of the project are released as OpenCyc, which provides an API, RDF endpoint, and data dump under an open source license.'
Score: 0.3030

Next Steps

In this tutorial, you've learned how to:

Optimize initial upload speed by starting with m=0 and building HNSW later
Measure filtering overhead with and without payload indexes
Tune our HNSW index and parameters

These techniques help you understand the performance trade-offs in vector search engines and optimize your applications for production use.