Feature/select improve

AGENTS.md

@@ -156,7 +156,8 @@ unavoidable external C library dependency.
 ## Technology Stack
 
 - **Language**: C++20 (`std::span`, `std::popcount`, and `<bit>` are required).
-- **Build**: CMake 3.18 or newer.
+- **Build**: CMake 3.18 or newer. The optional Pasta comparison backend
+  requires CMake 3.25 or newer.
 - **Testing**: Google Test 1.17.0, registered with CTest.
 - **Benchmarking**: Google Benchmark 1.9.4.
 - **SIMD**: AVX-512 when available, AVX2 fallback, and scalar fallback.
@@ -172,16 +173,20 @@ dependencies only as needed. Direct CMake defaults and their effects are:
 | `PIXIE_TESTS` | `ON` standalone, `OFF` downstream | Builds tests and fetches Google Test. |
 | `PIXIE_BENCHMARKS` | `OFF` | Builds native Google Benchmark targets. |
 | `PIXIE_THIRD_PARTY_BACKENDS` | `OFF` | Enables optional SDSL adapters and their comparison targets. |
+| `PIXIE_3STAR_SOURCE_DIR` | empty | Permitted local 3star-rankselect checkout at Pixie's pinned revision for its benchmark-only adapter; Pixie never fetches or redistributes it. |
 | `PIXIE_DIAGNOSTICS` | `OFF` | Enables diagnostic logging for profiling experiments. |
 | `PIXIE_DOCS` | `OFF` | Enables the Doxygen `docs` target. |
 | `PIXIE_COVERAGE` | `OFF` | Adds GCC coverage instrumentation. |
 
 `MappedFile` uses native POSIX memory mapping on Linux/Unix. A default
 FetchContent consumer therefore receives no third-party dependency, while a
 standalone default build fetches Google Test. Enabling third-party backends also
-fetches SDSL and pasta-toolbox dependencies, but only SDSL currently has a
-registered Pixie adapter/comparison benchmark. Do not describe pasta-toolbox as
-an available backend until Pixie adds and registers one.
+fetches SDSL and pasta-toolbox dependencies. SDSL has registered RmM comparison
+targets; Pasta provides an owning rank/select comparison backend and is not a
+default library dependency. Pasta is GPLv3-or-later; keep its adapter strictly
+behind the optional comparison-backend build option. The 3-star research
+artifact has no published license; its benchmark-only adapter requires a
+user-provided, permitted local checkout and must not fetch or redistribute it.
 
 ## Build and Test Presets
 
@@ -275,6 +280,9 @@ when debugging a focused Google Test filter.
   cases over more undirected random input. Cover border correction, same-leaf
   paths, prefix/suffix selectors, sparse-overlay hit/miss paths, partial final
   blocks, and first-minimum ties.
+- **Rank/select boundaries**: test a target at the final position of a full
+  2^16-bit superblock; its local one or zero rank is 65,536 and cannot be
+  passed through a 16-bit basic-block search key.
 - **Storage tests**: test owners and views through the same specification,
   including nested byte subranges, alignment constraints, serialization, and
   owner/view lifetime rules.

CMakeLists.txt

@@ -50,9 +50,40 @@ option(PIXIE_BENCHMARKS "Build benchmarks" OFF)
 option(PIXIE_THIRD_PARTY_BACKENDS "Build optional third-party backend integrations" OFF)
 option(PIXIE_DIAGNOSTICS "Include diagnostic logs" OFF)
 option(PIXIE_DOCS "Build Doxygen documentation" OFF)
+set(PIXIE_3STAR_SOURCE_DIR "" CACHE PATH
+        "Path to a permitted local 3star-rankselect checkout for benchmark-only use")
+
+if (PIXIE_3STAR_SOURCE_DIR)
+    if (NOT PIXIE_THIRD_PARTY_BACKENDS)
+        message(FATAL_ERROR
+                "PIXIE_3STAR_SOURCE_DIR requires PIXIE_THIRD_PARTY_BACKENDS")
+    endif ()
+    foreach (three_star_file m3.hpp Tree3.hpp bv.hpp alignedallocator.hpp)
+        if (NOT EXISTS "${PIXIE_3STAR_SOURCE_DIR}/${three_star_file}")
+            message(FATAL_ERROR
+                    "PIXIE_3STAR_SOURCE_DIR is missing ${three_star_file}")
+        endif ()
+    endforeach ()
+    execute_process(
+            COMMAND git -C "${PIXIE_3STAR_SOURCE_DIR}" rev-parse HEAD
+            OUTPUT_VARIABLE PIXIE_3STAR_REVISION
+            OUTPUT_STRIP_TRAILING_WHITESPACE
+            RESULT_VARIABLE PIXIE_3STAR_REVISION_RESULT)
+    if (NOT PIXIE_3STAR_REVISION_RESULT EQUAL 0 OR
+            NOT PIXIE_3STAR_REVISION STREQUAL "8b66e5b24944b483f8a886e215e147a11457cb30")
+        message(FATAL_ERROR
+                "PIXIE_3STAR_SOURCE_DIR must be 3star-rankselect revision "
+                "8b66e5b24944b483f8a886e215e147a11457cb30")
+    endif ()
+    set(PIXIE_3STAR_SUPPORT ON)
+endif ()
 
 if (PIXIE_THIRD_PARTY_BACKENDS)
-    add_compile_definitions(SDSL_SUPPORT)
+    if (CMAKE_VERSION VERSION_LESS 3.25)
+        message(FATAL_ERROR
+                "PIXIE_THIRD_PARTY_BACKENDS requires CMake 3.25 for pasta::bit_vector")
+    endif ()
+    add_compile_definitions(SDSL_SUPPORT PIXIE_PASTA_SUPPORT)
 endif ()
 
 if (PIXIE_DIAGNOSTICS)
@@ -97,7 +128,7 @@ if (PIXIE_THIRD_PARTY_BACKENDS)
     FetchContent_Declare(
             pasta_bit_vector
             GIT_REPOSITORY https://github.com/pasta-toolbox/bit_vector.git
-            GIT_TAG origin/main
+            GIT_TAG 3ffb6e5a2e58c76425de8197bfe554eb1bf9fd94
     )
     FetchContent_MakeAvailable(pasta_bit_vector)
 
@@ -148,15 +179,38 @@ function (pixie_enable_project_warnings target)
     endif ()
 endfunction ()
 
+if (PIXIE_3STAR_SUPPORT)
+    function (pixie_enable_three_star target)
+        target_include_directories(${target}
+                PRIVATE src/benchmarks)
+        target_include_directories(${target}
+                SYSTEM PRIVATE ${PIXIE_3STAR_SOURCE_DIR})
+        target_compile_definitions(${target} PRIVATE
+                PIXIE_3STAR_SUPPORT
+                ALIGNMENT=2097152
+                CHECK_ALL_DENSE_SPARSE=0
+                COUNT_HIT_TYPES=0
+                CUSTOM_L0=2048
+                POP=2
+                PREFETCH_NT=1
+                PREFETCH_T0=1
+                REVERSE=2
+                TREE_DIVISION_EXP=3
+                TREE_EXP_LVL0=19
+                TREE_TYPE=3)
+    endfunction ()
+endif ()
+
 if (PIXIE_DIAGNOSTICS)
     target_compile_definitions(pixie INTERFACE PIXIE_DIAGNOSTICS)
     target_link_libraries(pixie INTERFACE spdlog::spdlog_header_only)
 endif ()
 
 if (PIXIE_THIRD_PARTY_BACKENDS)
-    target_compile_definitions(pixie INTERFACE SDSL_SUPPORT)
+    target_compile_definitions(pixie INTERFACE SDSL_SUPPORT PIXIE_PASTA_SUPPORT)
     target_include_directories(pixie INTERFACE
             $<BUILD_INTERFACE:${sdsl_lite_SOURCE_DIR}/include>)
+    target_link_libraries(pixie INTERFACE pasta_bit_vector)
 endif ()
 
 # ---------------------------------------------------------------------------
@@ -191,6 +245,27 @@ if (PIXIE_TESTS)
             gtest
             gtest_main
             ${PIXIE_DIAGNOSTICS_LIBS})
+    if (PIXIE_THIRD_PARTY_BACKENDS)
+        target_link_libraries(rank_select_tests pasta_bit_vector)
+        if (CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU")
+            # Pasta's debug-assertion helper is a static function that becomes
+            # unused in optimized builds.
+            target_compile_options(rank_select_tests PRIVATE -Wno-unused-function)
+        endif ()
+    endif ()
+
+    if (PIXIE_3STAR_SUPPORT)
+        add_executable(three_star_benchmark_tests
+                src/tests/three_star_benchmark_tests.cpp
+                src/benchmarks/three_star_adapter.cpp)
+        target_include_directories(three_star_benchmark_tests
+                PUBLIC include)
+        target_link_libraries(three_star_benchmark_tests
+                gtest
+                gtest_main
+                ${PIXIE_DIAGNOSTICS_LIBS})
+        pixie_enable_three_star(three_star_benchmark_tests)
+    endif ()
 
     add_executable(benchmark_tests
             src/tests/benchmark_tests.cpp)
@@ -325,6 +400,9 @@ if (PIXIE_TESTS)
             select512_experimental_tests
             excess_record_lows_tests
             rmq_tests)
+    if (PIXIE_3STAR_SUPPORT)
+        list(APPEND PIXIE_TEST_TARGETS three_star_benchmark_tests)
+    endif ()
     foreach (test_target IN LISTS PIXIE_TEST_TARGETS)
         pixie_enable_project_warnings(${test_target})
         gtest_discover_tests(${test_target}
@@ -345,6 +423,18 @@ if (PIXIE_BENCHMARKS)
     target_link_libraries(rank_select_benchmarks
             benchmark
             ${PIXIE_DIAGNOSTICS_LIBS})
+    if (PIXIE_THIRD_PARTY_BACKENDS)
+        target_link_libraries(rank_select_benchmarks pasta_bit_vector)
+        if (CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU")
+            # See the matching rank_select_tests exception above.
+            target_compile_options(rank_select_benchmarks PRIVATE -Wno-unused-function)
+        endif ()
+    endif ()
+    if (PIXIE_3STAR_SUPPORT)
+        target_sources(rank_select_benchmarks PRIVATE
+                src/benchmarks/three_star_adapter.cpp)
+        pixie_enable_three_star(rank_select_benchmarks)
+    endif ()
 
     add_executable(rmm_benchmarks
             src/benchmarks/rmm_benchmarks.cpp)
@@ -431,6 +521,15 @@ if (PIXIE_BENCHMARKS)
             benchmark
             benchmark_main
             ${PIXIE_DIAGNOSTICS_LIBS})
+    if (PIXIE_THIRD_PARTY_BACKENDS)
+        target_include_directories(serialization_benchmarks
+                PRIVATE ${sdsl_lite_SOURCE_DIR}/include)
+        target_link_libraries(serialization_benchmarks pasta_bit_vector)
+        if (CMAKE_CXX_COMPILER_ID MATCHES "Clang|GNU")
+            target_compile_options(serialization_benchmarks
+                    PRIVATE -Wno-unused-function)
+        endif ()
+    endif ()
 
     add_executable(integer_vector_benchmarks
             src/benchmarks/integer_vector_benchmarks.cpp)

include/pixie/rank_select/implementations.h

@@ -7,6 +7,8 @@
  * - `RankSelectSupport<MetadataStorage>`: non-owning source bits with
  *   storage-backed rank/select metadata.
  * - `RankSelectSupportView`: non-owning source bits with read-only metadata.
+ * - `PastaRankSelectSupport`: optional Pasta backend owning a converted source
+ *   copy and its rank/select metadata.
  */
 
 // clang-format off
@@ -76,8 +78,43 @@
  * | FNBP       | 2^22 |   0.139 |   0.020 |          0.041 |
  * | FNBP       | 2^26 |   2.289 |   0.291 |          0.036 |
  * | FNBP       | 2^30 |  36.093 |   4.627 |          0.036 |
+ *
+ * Current native 2^30 snapshot, 2026-09-13.
+ *
+ * One CPU-pinned Release pass per row with the standard 0.2 s warmup and 1.0 s
+ * minimum time. Query CPU times are ns; build CPU time is ms. The index uses
+ * SelectSupport::kBoth and is non-owning; its source remains external.
+ *
+ * | source       | build ms |   rank1 |   rank0 | select1 | select0 | aux MiB |
+ * | :----------- | -------: | ------: | ------: | ------: | ------: | ------: |
+ * | random 12.5% |   39.700 |  40.400 |  66.500 | 180.000 | 152.000 |   4.625 |
+ * | random 50%   |   39.800 |  38.100 |  37.200 | 140.000 | 153.000 |   4.625 |
+ * | random 87.5% |   45.400 |  37.800 |  51.200 | 134.000 | 169.000 |   4.625 |
+ *
+ * Pasta FlatRankSelect<> snapshot, 2026-09-13.
+ *
+ * One CPU-pinned Release pass per row with a deterministic random 50% source
+ * and the standard 0.2 s warmup and 1.0 s minimum time. Both query pools and
+ * index construction are outside timed query regions. Pasta uses its default
+ * `FlatRankSelect<>` configuration (`DONT_CARE` and linear L2 search). Build
+ * time is CPU ms; query times are CPU ns. `aux MiB` is logical index storage;
+ * `source MiB` is the copied source that native RankSelectSupport does not own.
+ *
+ * |    N | build ms |   rank1 |   rank0 | select1 | select0 | aux MiB | source MiB |
+ * | ---: | -------: | ------: | ------: | ------: | ------: | ------: | ---------: |
+ * | 2^10 |    0.000 |  10.900 |  11.400 |  17.200 |  16.600 |   0.000 |      0.000 |
+ * | 2^14 |    0.000 |  10.300 |  11.100 |  30.600 |  33.400 |   0.000 |      0.002 |
+ * | 2^18 |    0.003 |  11.300 |  10.100 |  30.100 |  36.300 |   0.001 |      0.031 |
+ * | 2^22 |    0.057 |  12.600 |  11.500 |  35.300 |  47.000 |   0.018 |      0.500 |
+ * | 2^26 |    1.540 |  38.300 |  29.400 |  71.600 | 146.000 |   0.281 |      8.000 |
+ * | 2^30 |  143.000 |  47.100 |  49.100 | 140.000 | 325.000 |   4.500 |    128.000 |
+ * | 2^34 | 2889.000 | 109.000 |  88.700 | 291.000 | 552.000 |  72.000 |   2048.000 |
  */
 // clang-format on
 
 #include <pixie/rank_select.h>
 #include <pixie/rank_select/support.h>
+
+#ifdef PIXIE_PASTA_SUPPORT
+#include <pixie/rank_select/pasta.h>
+#endif

include/pixie/rank_select/pasta.h

@@ -0,0 +1,132 @@
+#pragma once
+
+#ifndef PIXIE_PASTA_SUPPORT
+#error "PastaRankSelectSupport requires PIXIE_THIRD_PARTY_BACKENDS"
+#endif
+
+#include <pixie/rank_select.h>
+
+#include <algorithm>
+#include <bit>
+#include <cstddef>
+#include <cstdint>
+#include <pasta/bit_vector/bit_vector.hpp>
+#include <pasta/bit_vector/support/flat_rank.hpp>
+#include <pasta/bit_vector/support/flat_rank_select.hpp>
+#include <pasta/bit_vector/support/l12_type.hpp>
+#include <span>
+
+namespace pixie {
+
+/**
+ * @brief Pasta flat rank/select adapter over a copied packed bit sequence.
+ *
+ * @details The constructor owns a normalized copy of the logical source words;
+ * source mutations do not affect this index. Both select directions are always
+ * available. Public positions are zero-based, prefix ranks use `[0, end)`, and
+ * selects use one-based ranks with the Pixie rank-zero and out-of-range
+ * sentinels.
+ */
+class PastaRankSelectSupport : public RankSelectBase<PastaRankSelectSupport> {
+ private:
+  using PastaSupport = pasta::FlatRankSelect<>;
+
+  static uint64_t logical_word(std::span<const uint64_t> source,
+                               std::size_t word_index,
+                               std::size_t num_bits) {
+    uint64_t word = source[word_index];
+    const std::size_t remaining = num_bits - word_index * 64;
+    if (remaining < 64) {
+      word &= (uint64_t{1} << remaining) - 1;
+    }
+    return word;
+  }
+
+  pasta::BitVector bits_;
+  PastaSupport support_;
+  std::size_t num_bits_ = 0;
+  std::size_t max_rank_ = 0;
+
+ public:
+  /**
+   * @brief Construct an owning Pasta index from caller-provided packed words.
+   * @param source_words Packed least-significant-bit-first source words.
+   * @param num_bits Number of valid input bits, clamped to @p source_words.
+   */
+  explicit PastaRankSelectSupport(std::span<const uint64_t> source_words,
+                                  std::size_t num_bits)
+      : bits_(std::min(num_bits, source_words.size() * 64)),
+        num_bits_(std::min(num_bits, source_words.size() * 64)) {
+    std::fill(bits_.data().begin(), bits_.data().end(), 0);
+    const std::size_t word_count = (num_bits_ + 63) / 64;
+    for (std::size_t word_index = 0; word_index < word_count; ++word_index) {
+      bits_.data()[word_index] =
+          logical_word(source_words, word_index, num_bits_);
+    }
+    support_ = PastaSupport(bits_);
+    max_rank_ = num_bits_ == 0 ? 0 : support_.rank1(num_bits_);
+  }
+
+  /** @brief Return the number of valid source bits. */
+  std::size_t size_impl() const { return num_bits_; }
+
+  /** @brief Return the bit at a zero-based valid position. */
+  int bit_impl(std::size_t position) const {
+    return static_cast<bool>(bits_[position]);
+  }
+
+  /** @brief Return the one rank in `[0, end_position)`. */
+  uint64_t rank_impl(std::size_t end_position) const {
+    return end_position >= num_bits_ ? max_rank_ : support_.rank1(end_position);
+  }
+
+  /** @brief Return the zero-based position of a one-based one rank. */
+  uint64_t select_impl(std::size_t rank) const {
+    if (rank == 0) {
+      return 0;
+    }
+    return rank > max_rank_ ? num_bits_ : support_.select1(rank);
+  }
+
+  /** @brief Return the zero-based position of a one-based zero rank. */
+  uint64_t select0_impl(std::size_t rank) const {
+    if (rank == 0) {
+      return 0;
+    }
+    return rank > num_bits_ - max_rank_ ? num_bits_ : support_.select0(rank);
+  }
+
+  /** @brief Pasta constructs support for one selects unconditionally. */
+  bool supports_select1_impl() const { return true; }
+
+  /** @brief Pasta constructs support for zero selects unconditionally. */
+  bool supports_select0_impl() const { return true; }
+
+  /**
+   * @brief Return bytes owned by the converted source and Pasta metadata.
+   * @details Pasta's `FlatRankSelect::space_usage()` omits its inherited L1/L2
+   * table, so account for that table explicitly from its documented layout.
+   */
+  std::size_t memory_usage_bytes_impl() const {
+    return source_copy_bytes() + index_logical_bytes() +
+           (sizeof(*this) - sizeof(bits_) - sizeof(support_));
+  }
+
+  /** @brief Return bytes in Pasta's owned normalized source copy. */
+  std::size_t source_copy_bytes() const {
+    return bits_.space_usage() - sizeof(pasta::BitVector);
+  }
+
+  /**
+   * @brief Return Pasta's logical index bytes, excluding its owned source.
+   * @details Pasta reports logical vector sizes rather than allocator capacity.
+   */
+  std::size_t index_logical_bytes() const {
+    const std::size_t l12_entries =
+        bits_.data().size() / pasta::FlatRankSelectConfig::L1_WORD_SIZE + 1;
+    return (support_.space_usage() - sizeof(PastaSupport)) +
+           l12_entries * sizeof(pasta::BigL12Type);
+  }
+};
+
+}  // namespace pixie