/*
Copyright (c) 2018, 2019 MariaDB
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */
#ifndef ROWID_FILTER_INCLUDED
#define ROWID_FILTER_INCLUDED
#include "mariadb.h"
#include "sql_array.h"
/*
What rowid / primary filters are
--------------------------------
Consider a join query Q of the form
SELECT * FROM T1, ... , Tk WHERE P.
For any of the table reference Ti(Q) from the from clause of Q different
rowid / primary key filters (pk-filters for short) can be built.
A pk-filter F built for Ti(Q) is a set of rowids / primary keys of Ti
F= {pk1,...,pkN} such that for any row r=r1||...||rk from the result set of Q
ri's rowid / primary key pk(ri) is contained in F.
When pk-filters are useful
--------------------------
If building a pk-filter F for Ti(Q )is not too costly and its cardinality #F
is much less than the cardinality of T - #T then using the pk-filter when
executing Q might be quite beneficial.
Let r be a random row from Ti. Let s(F) be the probability that pk(r)
belongs to F. Let BC(F) be the cost of building F.
Suppose that the optimizer has chosen for Q a plan with this join order
T1 => ... Tk and that the table Ti is accessed by a ref access using index I.
Let K = {k1,...,kM} be the set of all rowid/primary keys values used to access
rows of Ti when looking for matches in this table.to join Ti by index I.
Let's assume that two set sets K and F are uncorrelated. With this assumption
if before accessing data from Ti by the rowid / primary key k we first
check whether k is in F then we can expect saving on M*(1-s(S)) accesses of
data rows from Ti. If we can guarantee that test whether k is in F is
relatively cheap then we can gain a lot assuming that BC(F) is much less
then the cost of fetching M*(1-s(S)) records from Ti and following
evaluation of conditions pushed into Ti.
Making pk-filter test cheap
---------------------------
If the search structure to test whether an element is in F can be fully
placed in RAM then this test is expected to be much cheaper than a random
access of a record from Ti. We'll consider two search structures for
pk-filters: ordered array and bloom filter. Ordered array is easy to
implement, but it's space consuming. If a filter contains primary keys
then at least space for each primary key from the filter must be allocated
in the search structure. On a the opposite a bloom filter requires a
fixed number of bits and this number does not depend on the cardinality
of the pk-filter (10 bits per element will serve pk-filter of any size).
*/
/*
How and when the optimizer builds and uses range rowid filters
--------------------------------------------------------------
1. In make_join_statistics()
for each join table s
after the call of get_quick_record_count()
the TABLE::method init_cost_info_for_usable_range_rowid_filters()
is called
The method build an array of Range_rowid_filter_cost_info elements
containing the cost info on possible range filters for s->table.
The array is optimized for further usage.
2. For each partial join order when the optimizer considers joining
table s to this partial join
In the function best_access_path()
a. When evaluating a ref access r by index idx to join s
the optimizer estimates the effect of usage of each possible
range filter f and chooses one with the best gain. The gain
is taken into account when the cost of thr ref access r is
calculated. If it turns out that this is the best ref access
to join s then the info about the chosen filter together
with the info on r is remembered in the corresponding element
of the array of POSITION structures.
[We evaluate every pair (ref access, range_filter) rather then
every pair (best ref access, range filter) because if the index
ref_idx used for ref access r correlates with the index rf_idx
used by the filter f then the pair (r,f) is not evaluated
at all as we don't know how to estimate the effect of correlation
between ref_idx and rf_idx.]
b. When evaluating the best range access to join table s the
optimizer estimates the effect of usage of each possible
range filter f and chooses one with the best gain.
[Here we should have evaluated every pair (range access,
range filter) as well, but it's not done yet.]
3. When the cheapest execution plan has been chosen and after the
call of JOIN::get_best_combination()
The method JOIN::make_range_rowid_filters() is called
For each range rowid filter used in the chosen execution plan
the method creates a quick select object to be able to perform
index range scan to fill the filter at the execution stage.
The method also creates Range_rowid_filter objects that are
used at the execution stage.
4. Just before the execution stage
The method JOIN::init_range_rowid_filters() is called.
For each join table s that is to be accessed with usage of a range
filter the method allocates containers for the range filter and
it lets the engine know that the filter will be used when
accessing s.
5. At the execution stage
In the function sub_select() just before the first access of a join
table s employing a range filter
The method JOIN_TAB::build_range_rowid_filter_if_needed() is called
The method fills the filter using the quick select created by
JOIN::make_range_rowid_filters().
6. The accessed key tuples are checked against the filter within the engine
using the info pushed into it.
*/
struct TABLE;
class SQL_SELECT;
class Rowid_filter_container;
class Range_rowid_filter_cost_info;
/* Cost to write rowid into array */
#define ARRAY_WRITE_COST 0.005
/* Factor used to calculate cost of sorting rowids in array */
#define ARRAY_SORT_C 0.01
/* Cost to evaluate condition */
#define COST_COND_EVAL 0.2
typedef enum
{
SORTED_ARRAY_CONTAINER,
BLOOM_FILTER_CONTAINER // Not used yet
} Rowid_filter_container_type;
/**
@class Rowid_filter_container
The interface for different types of containers to store info on the set
of rowids / primary keys that defines a pk-filter.
There will be two implementations of this abstract class.
- sorted array
- bloom filter
*/
class Rowid_filter_container : public Sql_alloc
{
public:
virtual Rowid_filter_container_type get_type() = 0;
/* Allocate memory for the container */
virtual bool alloc() = 0;
/*
@brief Add info on a rowid / primary to the container
@param ctxt The context info (opaque)
@param elem The rowid / primary key to be added to the container
@retval true if elem is successfully added
*/
virtual bool add(void *ctxt, char *elem) = 0;
/*
@brief Check whether a rowid / primary key is in container
@param ctxt The context info (opaque)
@param elem The rowid / primary key to be checked against the container
@retval False if elem is definitely not in the container
*/
virtual bool check(void *ctxt, char *elem) = 0;
/* True if the container does not contain any element */
virtual bool is_empty() = 0;
virtual ~Rowid_filter_container() = default;
};
/**
@class Rowid_filter
The interface for different types of pk-filters
Currently we support only range pk filters.
*/
class Rowid_filter : public Sql_alloc
{
protected:
/* The container to store info the set of elements in the filter */
Rowid_filter_container *container;
Rowid_filter_tracker *tracker;
public:
enum build_return_code {
SUCCESS,
NON_FATAL_ERROR,
FATAL_ERROR,
};
Rowid_filter(Rowid_filter_container *container_arg)
: container(container_arg) {}
/*
Build the filter :
fill it with info on the set of elements placed there
*/
virtual build_return_code build() = 0;
/*
Check whether an element is in the filter.
Returns false is the elements is definitely not in the filter.
*/
virtual bool check(char *elem) = 0;
virtual ~Rowid_filter() = default;
bool is_empty() { return container->is_empty(); }
Rowid_filter_container *get_container() { return container; }
void set_tracker(Rowid_filter_tracker *track_arg) { tracker= track_arg; }
Rowid_filter_tracker *get_tracker() { return tracker; }
};
/**
@class Rowid_filter_container
The implementation of the Rowid_interface used for pk-filters
that are filled when performing range index scans.
*/
class Range_rowid_filter: public Rowid_filter
{
/* The table for which the rowid filter is built */
TABLE *table;
/* The select to perform the range scan to fill the filter */
SQL_SELECT *select;
/* The cost info on the filter (used for EXPLAIN/ANALYZE) */
Range_rowid_filter_cost_info *cost_info;
public:
Range_rowid_filter(TABLE *tab,
Range_rowid_filter_cost_info *cost_arg,
Rowid_filter_container *container_arg,
SQL_SELECT *sel)
: Rowid_filter(container_arg), table(tab), select(sel), cost_info(cost_arg)
{}
~Range_rowid_filter();
build_return_code build() override;
bool check(char *elem) override
{
if (container->is_empty())
return false;
bool was_checked= container->check(table, elem);
tracker->increment_checked_elements_count(was_checked);
return was_checked;
}
SQL_SELECT *get_select() { return select; }
};
/**
@class Refpos_container_sorted_array
The wrapper class over Dynamic_array<char> to facilitate operations over
array of elements of the type char[N] where N is the same for all elements
*/
class Refpos_container_sorted_array : public Sql_alloc
{
/*
Maximum number of elements in the array
(Now is used only at the initialization of the dynamic array)
*/
uint max_elements;
/* Number of bytes allocated for an element */
uint elem_size;
/* The dynamic array over which the wrapper is built */
Dynamic_array<char> *array;
public:
Refpos_container_sorted_array(uint max_elems, uint elem_sz)
: max_elements(max_elems), elem_size(elem_sz), array(0) {}
~Refpos_container_sorted_array()
{
delete array;
array= 0;
}
bool alloc()
{
array= new Dynamic_array<char> (PSI_INSTRUMENT_MEM,
elem_size * max_elements,
elem_size * max_elements/sizeof(char) + 1);
return array == NULL;
}
bool add(char *elem)
{
for (uint i= 0; i < elem_size; i++)
{
if (array->append(elem[i]))
return true;
}
return false;
}
char *get_pos(uint n)
{
return array->get_pos(n * elem_size);
}
uint elements() { return (uint) (array->elements() / elem_size); }
void sort(qsort_cmp2 cmp, void *cmp_arg)
{
my_qsort2(array->front(), array->elements() / elem_size, elem_size, cmp,
cmp_arg);
}
bool is_empty() { return elements() == 0; }
};
/**
@class Rowid_filter_sorted_array
The implementation of the Rowid_filter_container interface as
a sorted array container of rowids / primary keys
*/
class Rowid_filter_sorted_array: public Rowid_filter_container
{
/* The dynamic array to store rowids / primary keys */
Refpos_container_sorted_array refpos_container;
/* Initially false, becomes true after the first call of (check() */
bool is_checked;
public:
Rowid_filter_sorted_array(uint elems, uint elem_size)
: refpos_container(elems, elem_size), is_checked(false) {}
Rowid_filter_container_type get_type() override
{ return SORTED_ARRAY_CONTAINER; }
bool alloc() override { return refpos_container.alloc(); }
bool add(void *ctxt, char *elem) override { return refpos_container.add(elem); }
bool check(void *ctxt, char *elem) override;
bool is_empty() override { return refpos_container.is_empty(); }
};
/**
@class Range_rowid_filter_cost_info
An objects of this class is created for each potentially usable
range filter. It contains the info that allows to figure out
whether usage of the range filter promises some gain.
*/
class Range_rowid_filter_cost_info : public Sql_alloc
{
/* The table for which the range filter is to be built (if needed) */
TABLE *table;
/* Estimated number of elements in the filter */
ulonglong est_elements;
/* The cost of building the range filter */
double b;
/*
a*N-b yields the gain of the filter
for N key tuples of the index key_no
*/
double a;
/* The value of N where the gain is 0 */
double cross_x;
/* Used for pruning of the potential range filters */
key_map abs_independent;
/*
These two parameters are used to choose the best range filter
in the function TABLE::best_range_rowid_filter_for_partial_join
*/
double a_adj;
double cross_x_adj;
public:
/* The type of the container of the range filter */
Rowid_filter_container_type container_type;
/* The index whose range scan would be used to build the range filter */
uint key_no;
/* The selectivity of the range filter */
double selectivity;
Range_rowid_filter_cost_info() : table(0), key_no(0) {}
void init(Rowid_filter_container_type cont_type,
TABLE *tab, uint key_no);
double build_cost(Rowid_filter_container_type container_type);
inline double lookup_cost(Rowid_filter_container_type cont_type);
inline double
avg_access_and_eval_gain_per_row(Rowid_filter_container_type cont_type);
inline double avg_adjusted_gain_per_row(double access_cost_factor);
inline void set_adjusted_gain_param(double access_cost_factor);
/* Get the gain that usage of filter promises for r key tuples */
inline double get_gain(double r)
{
return r * a - b;
}
/* Get the adjusted gain that usage of filter promises for r key tuples */
inline double get_adjusted_gain(double r)
{
return r * a_adj - b;
}
/*
The gain promised by usage of the filter for r key tuples
due to less condition evaluations
*/
inline double get_cmp_gain(double r)
{
return r * (1 - selectivity) / TIME_FOR_COMPARE;
}
Rowid_filter_container *create_container();
double get_a() const { return a; }
void trace_info(THD *thd);
friend
void TABLE::prune_range_rowid_filters();
friend
void TABLE::init_cost_info_for_usable_range_rowid_filters(THD *thd);
friend
Range_rowid_filter_cost_info *
TABLE::best_range_rowid_filter_for_partial_join(uint access_key_no,
double records,
double access_cost_factor);
};
#endif /* ROWID_FILTER_INCLUDED */
