The inconsistency in the number of returned nodes from Pinecone index B compared to index A could be due to several factors specific to the configuration and behavior of your vector databases. Here are some potential reasons and areas to investigate:

1. Sparse Vector Handling and Chunking Differences: The way sparse vectors are generated and utilized, especially if index B uses them more extensively or differently due to its unique chunking strategy, could impact search results.

2. Filtering Mechanism: Differences in metadata due to chunking could affect filtering. If filters are applied based on metadata that vary significantly between indexes, this might result in varying numbers of results.

3. Query Mode and Parameters: The query mode (VectorStoreQueryMode) and parameters like the alpha parameter for blending sparse and dense vector queries could lead to inconsistencies if they're not applied uniformly across both indexes.

4. Dimensionality and Vector Handling: Ensure that the dimensionality of vectors and the handling of default query vectors are consistent across both indexes. Mismatches here could affect search outcomes.

5. Namespace and Additional Filtering: Differences in namespace configuration or additional filters applied during queries could also lead to discrepancies in the number of returned nodes.

6. Batch Size and Insertion Parameters: The efficiency of data insertion, influenced by batch_size and insert_kwargs, might differ between the two indexes due to the chunking strategy, potentially affecting the completeness of the index and, subsequently, query results.

To address these issues, review the configuration and behavior of your indexes with a focus on the areas mentioned above. Ensure that both indexes are configured similarly in terms of vector handling, filtering, querying parameters, and data insertion practices.

For more detailed investigation, consider the specific implementation of the VectorIndexRetriever class and how it interacts with Pinecone's query functionality, particularly regarding the similarity_top_k parameter and how it might be influenced by the underlying data and index configuration.

If the problem persists, examining the Pinecone documentation or reaching out to their support with specifics about your index configurations and the observed behavior might provide additional insights.

To continue the conversation, mention @dosu.

@dosu
The only difference is that index A chunks on a 1024 basis whereas index B chunks on a 2048 basis. Otherwise, the indexes are identical.

There is no filtering. The parameters are identical between the indexes. Both indexes use 3072 dimensions.

The indexes are both capable of returning the number of documents I request, since index B if asked for 100 docs can return 70, so it must be capable of returning 17 if only asked for 17.

Given the details you've shared, the difference in chunking between index A and index B (1024 vs. 2048) is a significant factor to consider, especially since both indexes are otherwise identical in configuration and parameters. This chunking difference can affect the internal representation and handling of vectors within Pinecone, potentially leading to the observed discrepancies in the number of returned nodes.

Chunking affects how data is stored, indexed, and retrieved. Larger chunks might lead to a situation where fewer, more densely packed vectors are considered closely similar to the query vector, especially in a high-dimensional space like 3072 dimensions. This could result in fewer nodes being returned as the system may deem fewer vectors as being within the top K most similar, based on the internal similarity calculations and thresholds.

To mitigate this issue, consider the following steps:

1. Review Vector Representation: Ensure that the vector representations are optimized for the chunking strategy used. This might involve adjusting how vectors are generated or pre-processed before being indexed.

2. Adjust Query Parameters: Experiment with adjusting query parameters, such as increasing the similarity_top_k value or modifying other query-related settings that might influence the retrieval process.

3. Debug and Logging: Utilize debug logging to gain insights into the retrieval process. This can help identify if there are any unexpected behaviors or discrepancies in how queries are processed and results are returned.

4. Consult Pinecone Documentation or Support: Since the behavior is specific to how Pinecone handles vector indexing and retrieval, consulting the Pinecone documentation for any nuances related to chunking and vector representation or reaching out to Pinecone support for guidance might provide additional insights.

5. Experiment with Chunk Sizes: If feasible, consider experimenting with adjusting the chunk size for index B to align more closely with index A's chunking strategy and observe if this brings the number of returned nodes closer to expectations.

The goal of these steps is to identify and adjust factors that might be influencing the retrieval process due to the difference in chunking strategies. By closely monitoring how these changes affect the number of nodes returned, you can better understand the underlying cause and find a more consistent and predictable retrieval performance across both indexes.

To continue the conversation, mention @dosu.

@Jez77 there is a step that removes duplicates in the base retriever class -- perhaps you have duplicate data in your pinecone index?

Could you manually set the similarity thresholds to see if it does anything? I've never used pinecone but wondering if they have some kind of dynamic threshold