You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Alexander Bubenchikov via dev <de...@hive.apache.org> on 2026/05/04 14:43:53 UTC

Developer workflow and the patched-iceberg-api setup in IDEA

Hi everyone.

My name is Alexander Bubenchikov, and I’m a software engineer at JetBrains. I’m responsible for how IntelliJ IDEA imports project structures.

I ’ve been digging into the Hive codebase lately to see if we can improve the "out-of-the-box" experience for Hive contributors. However, I’ve run into a bit of a snag with how the Iceberg dependency is handled.

The way patched-iceberg-api unpacks dependencies directly into target/classes seems to be a real headache for IDEs. Since these classes aren't in the source tree and aren't attached as standard libraries, you end up with "red code" everywhere Iceberg is used. This happens for me in both IntelliJ IDEA and Eclipse. 

I've spent some time looking through the documentation and the wiki for any "IDE setup" guides, but haven't found anything that addresses this specific part of the build. It’s entirely possible I missed a README or a wiki page - if so, please just point me in the right direction.


I’m really curious to hear from the people who work on Hive every day: 
 * What’s your local setup? Do you use any specific Maven profiles or scripts to make the IDE happy? If not,
 * Do you just ignore the red code in these modules, or 
 * Is there a trick to getting the symbols to resolve?

I'd love to understand the current workflow better. Maybe there's something we can improve in IDEA, to make such setups more IDE-friendly.

Thanks, Alex

-- 
Alexander Bubenchikov
Software developer
JetBrains
http://www.jetbrains.com <http://www.jetbrains.com/>
The Drive to Develop

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Denys Kuzmenko <dk...@apache.org>.
Sorry, I forgot to mention that I usually build the project from the CLI and then open it in the IDE. I haven’t had a positive experience building Hive from scratch in IntelliJ (not just the Iceberg modules). After that, everything seems fine: `hive-iceberg-shading` is resolved from the Maven repository, and both iceberg-catalog and iceberg-handler modules compile without issues.

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Ayush Saxena <ay...@gmail.com>.
Hi Alex,

Here are the steps I followed:

* Run mvn clean install -DskipTests
* Remove the hive-shading* projects from IntelliJ (as Maven projects)
* Add the hive-iceberg-shading JAR as a dependency to iceberg-catalog

I’ve attached some screenshots for reference.

Lemme know if it still doesn't works for you.

-Ayush


-Ayush

On Tue, 5 May 2026 at 20:43, Simhadri G <si...@apache.org> wrote:
>
> Hi @Alex,
>
> I think this Jira can give you a bit more info on the reasoning behind patched-iceberg modules: https://issues.apache.org/jira/browse/HIVE-25027
>
> There is one additional step needed in IntelliJ (mentioned in the Jira), but it's fairly straightforward.
>
> Thanks!
> Simhadri G
>
> On Tue, May 5, 2026 at 7:07 PM Alexander Bubenchikov via dev <de...@hive.apache.org> wrote:
>>
>> Thank you for your answer. This is interesting, because it should not :)
>>
>> It indeed works when you compile and run tests using maven, but not if you build the project with Idea.
>>
>> Could you please rebuild say iceberg/iceberg-handler module? Or open IcebergSnapshotRefExec and HiveOperationsBase classes? I do see red code in every import from org.apache.iceberg package.
>>
>>
>> https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergSnapshotRefExec.java
>> https://github.com/apache/hive/blob/master/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java
>>
>> Thanks,
>> Alex
>>
>> > On 5 May 2026, at 1:31 PM, Denys Kuzmenko <dk...@apache.org> wrote:
>> >
>> > Hi,
>> >
>> > I’m not experiencing any issues with iceberg modules in IntelliJ: it works fine on my end, compiles and runs tests without problems.
>> >
>> > Regards,
>> > Denys
>>

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Zsolt Miskolczi <zs...@gmail.com>.
Hi Alexander,

I feel your pain.
Let's take, as an example, patched-iceberg-core: if I open any file inside
of that, it is red.

I found two different ways to solve this problem:
Trivial way: I just added iceberg-api as a dependency. In that way, I can
work with IDE support but I need a proper attention to not commit this
change:

<!--    We don't need this dependency to compile. But when we add this
dependency, we get IDE support -->
<!--    Uncomment it during development-->
<!--    <dependency>-->
<!--      <groupId>org.apache.iceberg</groupId>-->
<!--      <artifactId>iceberg-api</artifactId>-->

*<!--    </dependency>-->*
I can stash this change and pick it up anytime when I work with
Iceberg. (Of course, need to uncomment it)

Not trivial, but a one-time setup is to add iceberg-core as a library
dependency in IntelliJ. That will be stored in the .ide folder so it
won't be committed.
[image: image.png]

Both solutions have their drawbacks: once I want to use worktrees, I
have to do it manually again.

I hope it helps.

Zsolt


Alexander Bubenchikov via dev <de...@hive.apache.org> ezt írta (időpont:
2026. máj. 6., Sze, 10:04):

> Thanks @Simhadri.
>
> I see that the patch was actually reverted in
> https://issues.apache.org/jira/browse/HIVE-28494
> The additional step you have mentioned - "manual classpath setup for the
> IntelliJ", right?
>
>
> @Denis
> Yes, after mvn install it works fine, thanks. Despite the fact we resolve
> to binaries instead of sources, but I see you somehow found a way to live
> with that
>
>
> Thank you folks for your feedback, this gives me a lot of food for thought.
>
>
> Thanks,
> Alex
>

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Alexander Bubenchikov via dev <de...@hive.apache.org>.
Thanks @Simhadri.

I see that the patch was actually reverted in https://issues.apache.org/jira/browse/HIVE-28494
The additional step you have mentioned - "manual classpath setup for the IntelliJ", right?


@Denis
Yes, after mvn install it works fine, thanks. Despite the fact we resolve to binaries instead of sources, but I see you somehow found a way to live with that


Thank you folks for your feedback, this gives me a lot of food for thought.


Thanks, 
Alex

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Simhadri G <si...@apache.org>.
Hi @Alex,

I think this Jira can give you a bit more info on the reasoning behind
patched-iceberg modules: https://issues.apache.org/jira/browse/HIVE-25027

There is one additional step needed in IntelliJ (mentioned in the Jira),
but it's fairly straightforward.

Thanks!
Simhadri G

On Tue, May 5, 2026 at 7:07 PM Alexander Bubenchikov via dev <
dev@hive.apache.org> wrote:

> Thank you for your answer. This is interesting, because it should not :)
>
> It indeed works when you compile and run tests using maven, but not if you
> build the project with Idea.
>
> Could you please rebuild say iceberg/iceberg-handler module? Or open
> IcebergSnapshotRefExec and HiveOperationsBase classes? I do see red code in
> every import from org.apache.iceberg package.
>
>
>
> https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergSnapshotRefExec.java
>
> https://github.com/apache/hive/blob/master/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java
>
> Thanks,
> Alex
>
> > On 5 May 2026, at 1:31 PM, Denys Kuzmenko <dk...@apache.org> wrote:
> >
> > Hi,
> >
> > I’m not experiencing any issues with iceberg modules in IntelliJ: it
> works fine on my end, compiles and runs tests without problems.
> >
> > Regards,
> > Denys
>
>

Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Alexander Bubenchikov via dev <de...@hive.apache.org>.
Thank you for your answer. This is interesting, because it should not :)

It indeed works when you compile and run tests using maven, but not if you build the project with Idea.

Could you please rebuild say iceberg/iceberg-handler module? Or open IcebergSnapshotRefExec and HiveOperationsBase classes? I do see red code in every import from org.apache.iceberg package.


https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergSnapshotRefExec.java
https://github.com/apache/hive/blob/master/iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveOperationsBase.java

Thanks, 
Alex

> On 5 May 2026, at 1:31 PM, Denys Kuzmenko <dk...@apache.org> wrote:
> 
> Hi,
> 
> I’m not experiencing any issues with iceberg modules in IntelliJ: it works fine on my end, compiles and runs tests without problems. 
> 
> Regards,
> Denys


Re: Developer workflow and the patched-iceberg-api setup in IDEA

Posted by Denys Kuzmenko <dk...@apache.org>.
Hi,

I’m not experiencing any issues with iceberg modules in IntelliJ: it works fine on my end, compiles and runs tests without problems. 

Regards,
Denys