Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Preview calcite upgrade 1.38.0 RC0 #5490

Draft
wants to merge 89 commits into
base: master
Choose a base branch
from
Draft

Preview calcite upgrade 1.38.0 RC0 #5490

wants to merge 89 commits into from

Conversation

zabetak
Copy link
Contributor

@zabetak zabetak commented Oct 6, 2024

No description provided.

zabetak and others added 20 commits October 11, 2024 11:08
@zabetak
Bump Calcite version to 1.32.0
89c98ce
@zabetak
Resolve compilation failure in hive-exec due to missing Value annotation
a0fca15 
The problem as well as root cause analysis can be found under:
* https://issues.apache.org/jira/browse/CALCITE-5300
* https://issues.apache.org/jira/browse/LOG4J2-3609

The problem appears only during compilation thus we don't need the
value-annotations at runtime. Additionally, we don't want to propagate
the dependency to other projects thus we declare it at provided scope.
@zabetak
Compilation fails in JdbcHiveTableScan due to breaking changes in sup…
a0ecc66 
…erclass constructor

Breaking change caused by CALCITE-4640.
@zabetak
Compilation error in HiveFilterJoinRule due to RelRule.Config.EMPTY
59d0c07 
RelRule.Config.EMPTY has been removed in CALCITE-4839 which is a known
breaking change.

Refactor HiveFilterJoinRule using Config interface instead of
inheritance to parameterize Hive variants.

The refactored version does not use RelRule.Config.EMPTY so it solves
the compilation problem.
@zabetak
Revert "Compilation error in HiveFilterJoinRule due to RelRule.Config…
b67f658 
….EMPTY"

This reverts commit 844a656b95f34607a61eaaa6c0900458c89c8d6c.
@zabetak
Compilation error in HiveFilterJoinRule due to RelRule.Config.EMPTY
1a75b2c 
RelRule.Config.EMPTY has been removed in CALCITE-4839 which is a known
breaking change.

Add new parameter to the constructor of HiveFilterJoinRule and let
the problem of initializing the configuration to subclasses. There are
other ways to deal with the problem (see two previous commits) but this
is the one with minimal changes to the code and easier to review.
@zabetak
Compilation fails in JDBCExtractJoinFilterRule due to removed supercl…
a3fc01e 
…ass constructor

The constructor was accidentally removed from CALCITE-4787 along with a
bunch of other changes around Config.EMPTY.

Adapt the rule by passing a config starting of from
JDBCExtractJoinFilterRule.
@zabetak
Compilation fails in HiveDruidRules due to RelRule.Config.EMPTY
8ea58b2 
Documented breaking change CALCITE-4830
@zabetak
Simplify HiveDruidRules by using the config from respective rules in …
b0efc68 
…Calcite

Simplification becomes possible due to the Calcite upgrade.
@zabetak
Compilation fails in HiveRelFactories due to ProjectFactory signature…
09b34c2 
… change

The signature of ProjectFactory#createProject method was changed adding
wildcards with extends for fieldNames in CALCITE-4199.
@zabetak
NoSuchFieldError: BACK_TICK_BACKSLASH when running TestMiniLlapLocalC…
3a92af2 
…liDriver

The error occurs because we are using an old version of Avatica which is
incompatible with Calcite 1.32.0.

java.lang.NoSuchFieldError: BACK_TICK_BACKSLASH
	at org.apache.calcite.config.Lex.<clinit>(Lex.java:38)
	at org.apache.calcite.config.CalciteConnectionProperty.<clinit>(CalciteConnectionProperty.java:80)
	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:181)
	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:135)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1340)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:567)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12739)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
	at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
	at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:522)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:474)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:439)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:433)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:200)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:126)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
	at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:727)
	at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:697)
	at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:114)
	at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
	at org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver(TestMiniLlapLocalCliDriver.java:62)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:95)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
@zabetak
Compilation fails in HiveRemoveEmptySingleRules due to RelRule.Config…
f220cbf 
….EMPTY

Documented breaking change CALCITE-4830.

Add a base Hive rule config to use as an extension point for creating
the prune empty rule variants in Hive since it is not possible (without
ugly casts) to derive a config from the respective Calcite classes.
@zabetak
Compilation fails in HiveReduceExpressionsRule due to missing XxReduc…
f1cdb22 
…eExpressionsRule.Config

Documented breaking change CALCITE-4839 removing deprecated Config interfaces.
@zabetak
Bump Calcite version to 1.33.0 and Avatica to 1.23.0
f85f31d
@zabetak
Compilation fails in HiveProjectFactoryImpl due to new method added i…
63c47f7 
…n ProjectFactory interface

Undocumented breaking change from CALCITE-5127.

For the moment, just thrown an exception if someone tries to set variables in project since Hive does not know how to handle them.
@zabetak
StackOverflowError while running cbo_query13.q
445fdd4 
The query runs for a very long time and eventually fails with StackOverflowError.

jstack (attached at the end) shows that the query is stuck in a very deep call to RelMdPredicates.getPredicates.

Analysing the CBO logs we can see that the only rule that fires is HivePreFilteringRule.
The rule constantly adds a new HiveFilter till the query plan becomes huge and eventually crashes due to a very deep stack.

Turns out that a simplification that was added in CALCITE-5036 turns OR expressions into SEARCH is triggering the problem.

Example:
OR(=($76, 3), =($76, 1)) => SEARCH($76, Sarg[1, 3])

In HiveCalciteUtil#getPredsNotPushedAlready, we compare simplified with not simplified predicates and thus fail to detect the equivalence between OR and SEARCH.

A quick workaround, not sure if it is going be the final solution, is to also simplify the input to getPredsNotPushedAlready. This is sufficient to overcome the StackOverflowError reported here.

Worth noting that the HivePreFilteringRule seems in general buggy so there might be more and better solutions to this problem.

   java.lang.Thread.State: RUNNABLE
	at java.util.TreeMap.compare(TreeMap.java:1294)
	at java.util.TreeMap.put(TreeMap.java:538)
	at org.apache.hive.com.google.common.collect.TreeRangeSet.replaceRangeWithSameLowerBound(TreeRangeSet.java:272)
	at org.apache.hive.com.google.common.collect.TreeRangeSet.add(TreeRangeSet.java:222)
	at org.apache.hive.com.google.common.collect.RangeSet.addAll(RangeSet.java:225)
	at org.apache.hive.com.google.common.collect.AbstractRangeSet.addAll(AbstractRangeSet.java:64)
	at org.apache.hive.com.google.common.collect.TreeRangeSet.addAll(TreeRangeSet.java:41)
	at org.apache.calcite.rex.RexSimplify$RexSargBuilder.addSarg(RexSimplify.java:3056)
	at org.apache.calcite.rex.RexSimplify$SargCollector.accept2b(RexSimplify.java:2894)
	at org.apache.calcite.rex.RexSimplify$SargCollector.accept2(RexSimplify.java:2812)
	at org.apache.calcite.rex.RexSimplify$SargCollector.accept_(RexSimplify.java:2793)
	at org.apache.calcite.rex.RexSimplify$SargCollector.accept(RexSimplify.java:2778)
	at org.apache.calcite.rex.RexSimplify$SargCollector.access$400(RexSimplify.java:2761)
	at org.apache.calcite.rex.RexSimplify.lambda$simplifyAnd$3(RexSimplify.java:1488)
	at org.apache.calcite.rex.RexSimplify$$Lambda$1099/1247334493.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.apache.calcite.rex.RexSimplify.simplifyAnd(RexSimplify.java:1488)
	at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:279)
	at org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:248)
	at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:223)
	at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:299)
	at org.apache.calcite.rel.metadata.janino.GeneratedMetadata_PredicatesHandler.getPredicates_$(Unknown Source)
	at org.apache.calcite.rel.metadata.janino.GeneratedMetadata_PredicatesHandler.getPredicates(Unknown Source)
	at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:841)
	at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:292)
	...
	at org.apache.calcite.rel.metadata.janino.GeneratedMetadata_PredicatesHandler.getPredicates_$(Unknown Source)
	at org.apache.calcite.rel.metadata.janino.GeneratedMetadata_PredicatesHandler.getPredicates(Unknown Source)
	at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:841)
	at org.apache.calcite.rel.metadata.RelMdPredicates.getPredicates(RelMdPredicates.java:292)
@zabetak
ClassCastException in RelMdSize#typeValueSize when compiling cbo_quer…
9e3a368 
…y13.q due to the presence of Sarg literals

The compilation fails in HiveFilterSortPredicates rule since we are trying to cast a Sarg to a NlsString.

Sarg literals were introduced in CALCITE-4173 (1.26.0) and there are various places in the code (both in Hive and Calcite) that we cannot deal with them.

One way to overcome this along with few other problems related to Sargs is to expand the SEARCH operator back into classic conjunctions and disjunctions. The HiveSearchExpandRule is quick POC (definitely needs revisiting and polishing) covering some frequent cases.

java.lang.ClassCastException: org.apache.calcite.util.Sarg cannot be cast to org.apache.calcite.util.NlsString
        at org.apache.calcite.rel.metadata.RelMdSize.typeValueSize(RelMdSize.java:392) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexFunctionCost.visitCall(HiveFilterSortPredicates.java:210) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexFunctionCost.visitCall(HiveFilterSortPredicates.java:189) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rex.RexCall.accept(RexCall.java:189) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexSortPredicatesShuttle.costPerTuple(HiveFilterSortPredicates.java:179) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexSortPredicatesShuttle.rankingAnd(HiveFilterSortPredicates.java:153) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexSortPredicatesShuttle.lambda$visitCall$0(HiveFilterSortPredicates.java:117) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_261]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_261]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_261]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_261]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_261]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_261]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_261]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexSortPredicatesShuttle.visitCall(HiveFilterSortPredicates.java:120) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates$RexSortPredicatesShuttle.visitCall(HiveFilterSortPredicates.java:101) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rex.RexCall.accept(RexCall.java:189) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.rewriteFilter(HiveFilterSortPredicates.java:76) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:62) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.accept(HiveFilter.java:96) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:136) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin.accept(HiveJoin.java:231) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin.accept(HiveJoin.java:231) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveJoin.accept(HiveJoin.java:231) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate.accept(HiveAggregate.java:137) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:57) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:71) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:141) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveFilterSortPredicates.visit(HiveFilterSortPredicates.java:60) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:136) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1742) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1590) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:140) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:936) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:191) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:135) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1342) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:569) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12816) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:464) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:326) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:326) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:223) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:518) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:470) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:435) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:429) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
@kasakrisz @zabetak
use RexSimplify in point look up rule
fb86b9d
@kasakrisz @zabetak
Correlate related prune empty rules config
92f0609
@kasakrisz @zabetak
Iceberg MV related rules config
3fa48c8
soumyakanti3578 and others added 17 commits October 11, 2024 11:11
@soumyakanti3578 @zabetak
benign changes for split0
849b4d8
@soumyakanti3578 @zabetak
Resolve review comments
7d7398f
@soumyakanti3578 @zabetak
benign changes after rebasing
4c24de4
@soumyakanti3578 @zabetak
Add Calcite version check for 1.35
820c773
@soumyakanti3578 @zabetak
Refactor method that transforms ORs to INs and BETWEENs
aa4a132
@soumyakanti3578 @zabetak
benign changes for correlationoptimizer8.q
e4eda6a
@soumyakanti3578 @zabetak
Refactor method makeNotBetween to use the RexBuilder.makeBetween
13ed489 
`NOT(SEARCH(...))` is simplified to `SEARCH[(..), (...)]` in
RexSimplify# simplifyNot

Removed testComputeRangePredicateSelectivityNotBetweenRightLowerThanLeft
as it is not possible to create a RexNode with right < left
@soumyakanti3578 @zabetak
Retry simplify until a RexNode is simplified completely, upto a max of 5
b4c45ec 
tries.

This can be removed after upgrading to Calcite 1.38
https://issues.apache.org/jira/browse/CALCITE-6453
@soumyakanti3578 @zabetak
benign changes after rebasing
0fd0886
@soumyakanti3578 @zabetak
Revert "Retry simplify until a RexNode is simplified completely, upto…
8c9059a 
… a max of 5"

This reverts commit 9724d26.
@soumyakanti3578 @zabetak
Check if HivePreFilteringRule has been applied on a Filter based on i…
495778a 
…ts condition
@zabetak
Revert "Check if HivePreFilteringRule has been applied on a Filter ba…
bf3e302 
…sed on its condition"

This reverts commit 9ec32be.
@zabetak
Preview Calcite version 1.38.0-rc0
6e553b5
@zabetak
Preview Calcite 1.38.0 RC1
a867c84
@zabetak
Compilation errors due to RelRule.Config.EMPTY
5fa8191
@zabetak
Set nex update policy to always for getting the latest RC
57b59ca
@zabetak
IllegalArgumentException in HepRelVertex.init
ca44365 
java.lang.IllegalArgumentException: null
        at org.apache.hive.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
        at org.apache.calcite.plan.hep.HepRelVertex.<init>(HepRelVertex.java:54) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
        at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:837) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
        at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:813) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
        at org.apache.calcite.plan.hep.HepPlanner.setRoot(HepPlanner.java:163) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveHepExtractRelNodeRule.execute(HiveHepExtractRelNodeRule.java:39) ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]

TODO: Check more carefully the removal of the extraction rules; may it can be done independently of the upgrade
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Oct 15, 2024

Quality Gate Passed Quality Gate passed

Issues
37 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
tests failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zabetak @asf-ci-hive @kasakrisz @soumyakanti3578