Skip to content

[Improvement]: use seperate threadpool with table.expireSnapshots() #4264

Description

@Aireed

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

issue:
some tables keep in committing status for long time

Image

i found async-optimizing-commit-executor is waiting for other future to be completed.

`
"async-optimizing-commit-executor-4" #207 prio=5 os_prio=0 tid=0x00007fec4c001000 nid=0x16cbfd sleeping[0x00007fec96cea000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.iceberg.util.Tasks.waitFor(Tasks.java:522)
at org.apache.iceberg.util.Tasks.access$800(Tasks.java:42)
at org.apache.iceberg.util.Tasks$Builder.runParallel(Tasks.java:358)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:201)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.writeManifests(SnapshotProducer.java:666)
at org.apache.iceberg.SnapshotProducer.writeDataManifests(SnapshotProducer.java:611)
at org.apache.iceberg.MergingSnapshotProducer.lambda$newDataFilesAsManifests$19(MergingSnapshotProducer.java:1058)
at org.apache.iceberg.MergingSnapshotProducer$$Lambda$1080/137728188.accept(Unknown Source)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.iceberg.MergingSnapshotProducer.newDataFilesAsManifests(MergingSnapshotProducer.java:1055)
at org.apache.iceberg.MergingSnapshotProducer.prepareNewDataManifests(MergingSnapshotProducer.java:1037)
at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:940)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:257)
at org.apache.iceberg.SnapshotProducer.lambda$commit$4(SnapshotProducer.java:434)
at org.apache.iceberg.SnapshotProducer$$Lambda$793/149360830.run(Unknown Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:432)
at org.apache.amoro.op.MixedUpdate.commit(MixedUpdate.java:143)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.rewriteFiles(UnKeyedTableCommit.java:283)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.commit(UnKeyedTableCommit.java:230)
at org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit(OptimizingQueue.java:788)
at org.apache.amoro.server.table.executor.OptimizingCommitExecutor.execute(OptimizingCommitExecutor.java:53)
at org.apache.amoro.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:82)
at org.apache.amoro.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$2(BaseTableExecutor.java:93)
at org.apache.amoro.server.table.executor.BaseTableExecutor$$Lambda$665/1073662255.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000005c3a18998> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x00000006173222c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)`
the futures is worked with iceberg-work-pool which is shared with expiresnapshots plan.

Image Image Image

How should we improve?

add planwith to set seperate thread pool, like IcebergThreadPools.getPlanningExecutor()
Image

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions