Search before asking
What would you like to be improved?
issue:
some tables keep in committing status for long time
i found async-optimizing-commit-executor is waiting for other future to be completed.
`
"async-optimizing-commit-executor-4" #207 prio=5 os_prio=0 tid=0x00007fec4c001000 nid=0x16cbfd sleeping[0x00007fec96cea000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.iceberg.util.Tasks.waitFor(Tasks.java:522)
at org.apache.iceberg.util.Tasks.access$800(Tasks.java:42)
at org.apache.iceberg.util.Tasks$Builder.runParallel(Tasks.java:358)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:201)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.writeManifests(SnapshotProducer.java:666)
at org.apache.iceberg.SnapshotProducer.writeDataManifests(SnapshotProducer.java:611)
at org.apache.iceberg.MergingSnapshotProducer.lambda$newDataFilesAsManifests$19(MergingSnapshotProducer.java:1058)
at org.apache.iceberg.MergingSnapshotProducer$$Lambda$1080/137728188.accept(Unknown Source)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.iceberg.MergingSnapshotProducer.newDataFilesAsManifests(MergingSnapshotProducer.java:1055)
at org.apache.iceberg.MergingSnapshotProducer.prepareNewDataManifests(MergingSnapshotProducer.java:1037)
at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:940)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:257)
at org.apache.iceberg.SnapshotProducer.lambda$commit$4(SnapshotProducer.java:434)
at org.apache.iceberg.SnapshotProducer$$Lambda$793/149360830.run(Unknown Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:432)
at org.apache.amoro.op.MixedUpdate.commit(MixedUpdate.java:143)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.rewriteFiles(UnKeyedTableCommit.java:283)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.commit(UnKeyedTableCommit.java:230)
at org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit(OptimizingQueue.java:788)
at org.apache.amoro.server.table.executor.OptimizingCommitExecutor.execute(OptimizingCommitExecutor.java:53)
at org.apache.amoro.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:82)
at org.apache.amoro.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$2(BaseTableExecutor.java:93)
at org.apache.amoro.server.table.executor.BaseTableExecutor$$Lambda$665/1073662255.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000005c3a18998> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x00000006173222c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)`
the futures is worked with iceberg-work-pool which is shared with expiresnapshots plan.
How should we improve?
add planwith to set seperate thread pool, like IcebergThreadPools.getPlanningExecutor()

Are you willing to submit PR?
Subtasks
No response
Code of Conduct
Search before asking
What would you like to be improved?
issue:
some tables keep in committing status for long time
i found async-optimizing-commit-executor is waiting for other future to be completed.
`
"async-optimizing-commit-executor-4" #207 prio=5 os_prio=0 tid=0x00007fec4c001000 nid=0x16cbfd sleeping[0x00007fec96cea000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.iceberg.util.Tasks.waitFor(Tasks.java:522)
at org.apache.iceberg.util.Tasks.access$800(Tasks.java:42)
at org.apache.iceberg.util.Tasks$Builder.runParallel(Tasks.java:358)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:201)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.writeManifests(SnapshotProducer.java:666)
at org.apache.iceberg.SnapshotProducer.writeDataManifests(SnapshotProducer.java:611)
at org.apache.iceberg.MergingSnapshotProducer.lambda$newDataFilesAsManifests$19(MergingSnapshotProducer.java:1058)
at org.apache.iceberg.MergingSnapshotProducer$$Lambda$1080/137728188.accept(Unknown Source)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.iceberg.MergingSnapshotProducer.newDataFilesAsManifests(MergingSnapshotProducer.java:1055)
at org.apache.iceberg.MergingSnapshotProducer.prepareNewDataManifests(MergingSnapshotProducer.java:1037)
at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:940)
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:257)
at org.apache.iceberg.SnapshotProducer.lambda$commit$4(SnapshotProducer.java:434)
at org.apache.iceberg.SnapshotProducer$$Lambda$793/149360830.run(Unknown Source)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:432)
at org.apache.amoro.op.MixedUpdate.commit(MixedUpdate.java:143)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.rewriteFiles(UnKeyedTableCommit.java:283)
at org.apache.amoro.server.optimizing.UnKeyedTableCommit.commit(UnKeyedTableCommit.java:230)
at org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit(OptimizingQueue.java:788)
at org.apache.amoro.server.table.executor.OptimizingCommitExecutor.execute(OptimizingCommitExecutor.java:53)
at org.apache.amoro.server.table.executor.BaseTableExecutor.executeTask(BaseTableExecutor.java:82)
at org.apache.amoro.server.table.executor.BaseTableExecutor.lambda$scheduleIfNecessary$2(BaseTableExecutor.java:93)
at org.apache.amoro.server.table.executor.BaseTableExecutor$$Lambda$665/1073662255.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000005c3a18998> (a java.util.concurrent.ThreadPoolExecutor$Worker)
- <0x00000006173222c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)`
the futures is worked with iceberg-work-pool which is shared with expiresnapshots plan.
How should we improve?
add planwith to set seperate thread pool, like IcebergThreadPools.getPlanningExecutor()

Are you willing to submit PR?
Subtasks
No response
Code of Conduct